I have to say, what you are creating might just be the Holy Grail, and it's a safe bet I'm not the only hair-pulling-head-bashing-frustrated Hammer user who feels that way.
As a user of several very different game engines, 3D apps, etc, I think a good way to go about this is simply looking at who does what best, and, well, steal from them!
I am mainly a UnrealEd 3.0 user, and have to say, LOVE it, it really is a fantastic game engine. Hammer could learn a lot from it, and I would like to outline as much as I can think of below.
I enjoy Hammer for many things...but so much more of it just seems idiotic to me. Please keep in mind as you read this that I do NOT "hate" hammer, I just believe it has a lot it REALLY needs to learn and grow from!
***What Hammer Could Learn From the Unreal 3 Engine***
-get rid of the whole silly "leak/no leak" thing. I can compile WAY more complex maps in U3 in a fraction of the time, with excellent lighting, and never have to worry about "leaks" or such
-Axis constrained movement! U3 stole this from max/maya/every other 3D app, now Hammer needs to get in with the cool crowd and grab it. Having the ability to visually move things around in the 3D view, with the "xyz" arrows, would be WONDERFUL. It would be great in the 2D views, too. Applying this to Scale and Rotation would be great as well (again, U3 already does this so easily).
-real-time view of the 3D skybox. It is HORRIBLY annoying that I am forced to compile my entire map, or even with the cordon tool, just to see if my skybox is correct. Wouldn't it be fairly easy to set up a "virtual camera" that could be manually told to "snapshot" from the skybox, and apply it as a temporary texture to the skybox texture in your main level? This would still be fairly restrictive, but would give a HUGE amount of visual aid and control for us mappers using 3D skyboxes!!
-real-time particles systems. Another thing U3 does great, and is lovely to have.
-LIGHTMAPPING ON ALL SURFACES! O, the horror and crushing sadness my heart felt when I realized that, alas, Hammer would NOT light my custom models with anything other than ugly, ugly vertex lighting. WHAT??! I can't imagine the reasoning behind this, I know BSP can used for a lot, but some things it just can't do! Or, I can simply do it much better, faster, and more efficiently in 3DS Max/Maya/etc. Support for lightmaps would be AMAZING.
-Compile errors with no reason? How about the compiler just WORKS (for large, complex, open-air environments, and other admittedly tough maps). Even with VBCT (love it!), I run smack into a wall if I try to get too complex and detailed, if there arn't enough visplanes to chop everything up. Sure, this isn't the most efficient map idea, but what if I DO want a map that is completely open environment, and cant be broken up into different vis leafs? Shouldn't it just chug away, compile it, let me run it- and I'll deal with making the FPS work for players? Nope, it just gives up mid way through the compile each time... (this is a particularly sore issue at the moment...)
-real "WYSIWYG" editing (What You See Is What You Get)? U3 is a true-to-the-bones WYSIWYG editor, amazingly so, in fact. If you turn on all the right checkyboxes, EXACTLY what you see in-editor is EXACTLY what you will see in game. Even better, you can even HEAR everything! Particles, specular, normal, parallax, audio, animations, complete lighting, it is all there. This is a wonderful time saving feature, obviously not all machines can handle it, but it is available and great!
**a note on WYSIWYG- Unreal 3 accomplishes this by simply saving all those lightmaps that it (and hammer) compiles, however it then applies all those lightmaps to the editor- genius! Hammer should be able to access those lightmaps somehow, as well...??
-Multiple property windows
-A "generic browser" like Unreal3, which allows you to view very easily anything at all, from textures to shaders to models to animations to particles, etc. You can even edit them in this viewer, AND it doesn't lock you out of selecting/changing/manipulating the rest of the world (unlike Hammer).
-"zoom to selected" key! Just like all 3D apps have, and Unreal 3 has, a key that quickly zooms your 3D view into the object you have selected would be great.
-advanced terrain painting (see U3, again). The displacement painting in Hammer is just plain awful, sorry to say, if you go try the U3 system, or others, this becomes immediately obvious. I see you and others have already hit this point several times
-A 3Dapp-to-Hammer Worlflow That DOESN'T Make Me Want To Dig My Brains Out with a Rusty Shovel! There are SO many, so complex, so ridiculous steps required just to get a simple freakn' custom 3D model into Hammer, or even just a material, it is beyond my imagination how such a convoluted and obviously insane system was developed.
Okay, so there were actually very likely GOOD reasons for it back in the day (maybe, maybe), but these days, it is just not cool.
When I make an object for unreal, I just export it as the super-common-used-by-everything "ASCII" file, then import it BAM into U3 and I'm done!
(this is the point where those of you who have only used Hammer say "HOLY MOLY, that was soooooo EASY!, I never imagined a world where things could be so beautiful!")
but wait, there's more! To make a U3 material, I just save it out from Photoshop as the super-common-used-by-all Targa file type, and BAM, import to U3 editor, DONE! That was super easy, and you won't even have to download a SINGLE plugin, U3 does it all for you!!!
(this is the point where you say "thud" as you faint from pure ecstasy imagining the simplistic ease and joy that importing models and materials this way would be).
Really, the import process for Hammer is just absurd, okay? We all know it, or at least now you do. If the U3 method of importing is a graceful 20foot leaping slamdunk by Michael Jordan, then Hammer importing is the drunken fan who leaps out of his seat, tumbles down the aisles and onto the court, breaking every bone in his body and pancaking several newborn infants, and happens to land (inexplicably) in the hoop.
Okay, I'm done ranting. I DO love Hammer for some things, but it is definitely a love/hate relationship. Especially coming from U3 mapping, an editor that Hammer is SO far behind in many ways. I realize these are geared toward completely different game types/styles, in many ways, however so much could still be translated, especially in "usability".
Thanks for listening, and I really hope your project works, I will be it's #1 fan!!