Wow, that's a lot of displacements. Though I shouldn't imagine its a problem. You also shouldn't need to stretch your walls through your displacements and I'm not really sure the effect that would have for your vis leafs, because it's hard to understand the structure even with a few pics, but I imaging the compile would try and creating more vis leafs which are totally not necessary.
It's also not recommended to make a giant box around your map to seal leaks. If you do have leaks in your main structure (and then box them in like that), the compiler doesn't know that those areas outside the play area of the map yet inside the giant box aren't reachable by players and thus it will try and calculate all the vis leafs from those places which will add tens of thousands of useless calculations. Going through and carefully sealing your map will greatly improve both compile times and performance.
Lastly, if your concerned about performance I recommend loading your map in the game and running +showbudget in the console. It will give you a nice readout of engine performance which will help you narrow in on trouble areas.