There is No define size if you ask me. I recommend you build your layout and run around and judge if it is suitable or not. Maps like upward have some huge sightlines which almost never really factor negatively on the map's gameplay. It's really a matter of how it feels rather how it looks.
I think you should really experiement with the geometry. See what sticks and what doesn't. I am surely going to get some backlash from what I am going to suggest but check 3kliksphilip's making of de_sparity. There are 13 videos on a playlist with voice over ,gameplay etc.
From watching this somebody who is mapping for 4 years now learned a ton of new things. I am certain you will see another angle of level design which you haven't seen yet.
Staying on the Team Fortress subject, Paper also has on youtube a few videos on level design called "My TF2 Mapping Projects" there are as of writing at least 6 parts. Watching someone else work and seeing their past maps , what they learned , the difficulties they faces will surely give you a boost.
Lastly since I am answering your questions one at a time, there is also the smoothing tool while playing with displacements. A reason why your displacements may look weird, is because you have one giant displacement which makes it harder to work on. If you load up a valve map for example you will notice that the ground is consisted of many faces. That allows the map maker to add more detail not only to the texture but to the displacement as far gameplay is concerned. Now for which areas to rework/keep that's up to you. I won't tell you how to design your maps since we are two seperate people, we work and live differently.
Anywayz, I hope I answered all your questions and gave you a better look on how to work. Good luck with your future endeavors!
