Okay, I was able to replicate the exact behaviour you're describing by crushing the model browser down into a really tiny size, then dragging it back to the normal size.
This causes Hammer to freak out and think that the position of the line dividers between the windows is some ridiculously high value.
To fix this, close Hammer, locate this "hammer.vdf" file, delete it and run Hammer again: