Faster compilation

DrSquishy

we've all had better times to die
aa
Feb 10, 2017
1,297
974
When making my map, I always try and keep the numportals around 2000, but recently compilation times have spiked up by an incredible amount. I suspect it is due to a power 4 displacement, but am not completely sure. Extra notes: I am fairly new to optimization in hammer, and only really know about simple things like func_detail. Any and all help is greatly appreciated
Here is my compile log:
Starting a 'Full - Both' compile.
Starting compilation of C:\Users\John\Documents\MvM_origins_displaced.vmf
Valve Software - vbsp.exe (Jun 14 2017)
2 threads
materialPath: C:\Program Files\Steam\steamapps\common\team fortress 2\tf\materials
Loading C:\Users\John\Documents\MvM_origins_displaced.vmf
ConVarRef mat_reduceparticles doesn't point to an existing ConVar
Patching WVT material: maps/mvm_origins_displaced/construction/concrete/concreteblendground003_wvt_patch
Patching WVT material: maps/mvm_origins_displaced/nightfall/cliffwall_wvt_patch
Patching WVT material: maps/mvm_origins_displaced/blends/blendconcretesnow003_wvt_patch
Patching WVT material: maps/mvm_origins_displaced/blends/blendnightcliffsnow002_wvt_patch
Patching WVT material: maps/mvm_origins_displaced/dev/dev_blendmeasure2_wvt_patch
fixing up env_cubemap materials on brush sides...
ProcessBlock_Thread: 0...1...2...3...4...5...6...7...8...9...10 (0)
ProcessBlock_Thread: 0...1...2...3...4...5...6...7...8...9...10 (0)
Processing areas...done (0)
Building Faces...done (1)
Chop Details...done (0)
Find Visible Detail Sides...
Merged 2540 detail faces...done (0)
Merging details...done (0)
FixTjuncs...
PruneNodes...
WriteBSP...
NODRAW on terrain surface!
NODRAW on terrain surface!
NODRAW on terrain surface!
NODRAW on terrain surface!
NODRAW on terrain surface!
done (0)
writing C:\Users\John\Documents\MvM_origins_displaced.prt...Building visibility clusters...
done (0)
*** Error: Skybox vtf files for skybox/sky_alpinestorm_01 weren't compiled with the same size texture and/or same flags!
Can't load skybox file skybox/sky_alpinestorm_01 to build the default cubemap!
*** Error: Skybox vtf files for skybox/sky_alpinestorm_01 weren't compiled with the same size texture and/or same flags!
Can't load skybox file skybox/sky_alpinestorm_01 to build the default cubemap!
Finding displacement neighbors...
Found a displacement edge abutting multiple other edges.
Warning: overflowed 3 displacement corner-neighbor lists.Finding lightmap sample positions...
Displacement Alpha : 0...1...2...3...4...5...6...7...8...9...10
Building Physics collision data...
WARNING: Map using power 4 displacements, terrain physics cannot be compressed, map will need additional memory and CPU.
done (0) (2793306 bytes)
Placing detail props : 0...1...2...3...4...5...6...7...8...9...10
Water found with no water_lod_control entity, creating a default one.
Compacting texture/material tables...
Reduced 1343 texinfos to 927
Reduced 52 texdatas to 45 (1601 bytes to 1256)
Writing C:\Users\John\Documents\MvM_origins_displaced.bsp
Wrote ZIP buffer, estimated size 3672, actual size 2788
5 seconds elapsed
Valve Software - vvis.exe (Jun 14 2017)
2 threads
reading c:\users\john\documents\MvM_origins_displaced.bsp
reading c:\users\john\documents\MvM_origins_displaced.prt
658 portalclusters
2017 numportals
BasePortalVis: 0...1...2...3...4...5...6...7...8...9...10 (2)
PortalFlow: 0...1...2...3...4...5...6...7...8...9...10 (6233)
Optimized: 140 visible clusters (0.04%)
Total clusters visible: 397100
Average clusters visible: 603
Building PAS...
Average clusters audible: 658
visdatasize:114325 compressed from 115808
writing c:\users\john\documents\MvM_origins_displaced.bsp
1 hour, 43 minutes, 59 seconds elapsed
C:\Users\John\Documents\MvM_origins_displaced.bsp -> C:\Program Files\Steam\steamapps\common\team fortress 2\tf\maps\MvM_origins_displaced.bsp
1 File(s) copied

CompilePal - Automated Packaging
Finding sources of game content...
Found search paths...
Found wildcard path: C:\Program Files\Steam\steamapps\common\team fortress 2\tf\custom\
Found search path: C:\Program Files\Steam\steamapps\common\team fortress 2\tf
Found search path: C:\Program Files\Steam\steamapps\common\team fortress 2\tf\bin
Found search path: C:\Program Files\Steam\steamapps\common\team fortress 2\tf\download
Reading BSP...
Initializing pak file...
Writing file list...
Running bspzip...
Finished!
---------------------
21 materials found
8 models found
0 particle files found
0 sounds found
additional files:
---------------------
'Full - Both' compile finished in 01:44:15

I have also included the .vmf just in case
 

Attachments

  • MvM_origins_displaced.vmf
    2.9 MB · Views: 75

Tumby

aa
May 12, 2013
1,084
1,192
You have nodraw on displacements. Nodraw does not work on displacements. You have to delete sides of a displacement that you don't want to exist. (Select and click destroy)
Alternativly, if you only want it to be invisible, use the invisibledisplacement tool texture. (It's literally just a fully transparent texture)
Where to learn more about optimization: http://www.optimization.interlopers.net/
 

DrSquishy

we've all had better times to die
aa
Feb 10, 2017
1,297
974
You have nodraw on displacements. Nodraw does not work on displacements. You have to delete sides of a displacement that you don't want to exist. (Select and click destroy)
Alternativly, if you only want it to be invisible, use the invisibledisplacement tool texture. (It's literally just a fully transparent texture)
Where to learn more about optimization: http://www.optimization.interlopers.net/
Yeah. Finished compiling and noticed that. was actually just about to remove it
 

killohurtz

Distinction in Applied Carving
aa
Feb 22, 2014
1,016
1,277
I'm going to toss you this guide as well: http://www.nodraw.net/2009/12/optimization-in-source-a-practical-demonstration/

From your vmf, I can immediately see you're making a fundamental optimization mistake: your map is just a giant box with very few world brushes sprinkled about. Because you've used so many func_details and displacements - which don't block visibility - you've essentially told the compiler that the entire map should render no matter where you are. So it will spend hours calculating visibility from every single visleaf to every other visleaf, and give you a map with terrible performance as a result. You need more world brushes in there to break up the map and lessen the load on both the compiler and player's computers.

Also, try to "shrink-wrap" your skybox to cut out all that wasted space around the map and save the compiler from doing unnecessary calculations (specifically, you don't need a ceiling nearly that high, or any space below ground except where the lava pit is).

This map is also extremely overscaled. In my honest opinion, it would benefit from a complete rework with smaller, more contained areas. Throw in a few prop_static playermodels or walk around the map in-game to see just how massive it is.
 

worMatty

Repacking Evangelist
aa
Jul 22, 2014
1,257
999
As killo says, the number of portals doesn't matter, because it's the amount that can see each other that has the biggest effect on compile time. Your compile log says VVIS is taking an hour and forty-three minutes. VVIS shouldn't last very long at all. It's the most common cause of inflated compile times.

Don't focus on reducing numportals. Focus instead on reducing the possibility of them seeing each other. With visibility optimisation, your goal is to limit what the player can see, finding a balance between complexity and rendering performance. Players can 'see' through func_detail and displacements. Only world brushes will really block the line of sight between one numportal and the other. This is why many maps have areas separated by corridors or massive turns, or shrink-wrapped skybox brushes (which are also world brushes).

IMO visibility optimisation begins with the initial planning of a map. Rather than start with a big rectangle, think about building a map into a giant S or C shape. Doing so will make it much easier to divide outdoor spaces. Also, put some thought into how large buildings can be placed to limit line of sight.

<horizontal rule>

Look at this
tbSRwC4.png
 

DrSquishy

we've all had better times to die
aa
Feb 10, 2017
1,297
974
Wow. As soon as possible, I have a lot of optimisation to do.
 

Crowbar

Spiritual preprocessor
aa
Dec 19, 2015
1,455
1,297
the number of portals doesn't matter, because it's the amount that can see each other that has the biggest effect on compile time.

The amount of portals that can see each other out of the total amount of them matters in game but not while compiling, because each is going to be tested against each anyway, so what matters during compile is the total number. Do correct me if I'm being wrong.
 

worMatty

Repacking Evangelist
aa
Jul 22, 2014
1,257
999
I created a hollow box with dimensions of 10240 x 5184 x 10240 and filled it with 512HU-high wedges to create a 3D grid of visleafs with dimensions of 512x512x512. I performed two VBSP+VVIS-only compiles, one with the full hollow space and one with a floor dividing the chamber in half, cutting the number of portals that can see each other at once by roughly 50%.

wlpZWMi.png


Here are the results:

Starting a 'Normal' compile.
Starting compilation of C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.vmf
Valve Software - vbsp.exe (Jun 14 2017)
8 threads
materialPath: C:\Users\Matty\CloudStation\Steam\steamapps\common\Team Fortress 2\tf\materials
Loading C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.vmf
fixing up env_cubemap materials on brush sides...
ProcessBlock_Thread: 0...1...2...3...4...5...6...7...8...9...10 (0)
ProcessBlock_Thread: 0...1...2...3...4...5...6...7...8...9...10 (0)
Processing areas...done (0)
Building Faces...done (0)
FixTjuncs...
PruneNodes...
WriteBSP...
done (0)
writing C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.prt...Building visibility clusters...
done (0)
*** Error: Skybox vtf files for skybox/sky_tf2_04 weren't compiled with the same size texture and/or same flags!
Can't load skybox file skybox/sky_tf2_04 to build the default cubemap!
*** Error: Skybox vtf files for skybox/sky_tf2_04 weren't compiled with the same size texture and/or same flags!
Can't load skybox file skybox/sky_tf2_04 to build the default cubemap!
Finding displacement neighbors...
Finding lightmap sample positions...
Displacement Alpha : 0...1...2...3...4...5...6...7...8...9...10
Building Physics collision data...
done (0) (2282 bytes)
Placing detail props : 0...1...2...3...4...5...6...7...8...9...10
Compacting texture/material tables...
Reduced 6 texinfos to 4
Reduced 2 texdatas to 2 (36 bytes to 36)
Writing C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.bsp
ZIP Output overshot buffer estimate: Estimated 0, actual 54
0 seconds elapsed
Valve Software - vvis.exe (Jun 14 2017)
8 threads
reading c:\users\matty\cloudstation\tf2 mapping\work\- experiments\portal_vis_calc.bsp
reading c:\users\matty\cloudstation\tf2 mapping\work\- experiments\portal_vis_calc.prt
1000 portalclusters
2650 numportals
BasePortalVis: 0...1...2...3...4...5...6...7...8...9...10 (0)
PortalFlow: 0...1...2...3...4...5...6...7...8...9...10 (71)
Optimized: 0 visible clusters (0.00%)
Total clusters visible: 1000000
Average clusters visible: 1000
Building PAS...
Average clusters audible: 1000
visdatasize:258004 compressed from 256000
writing c:\users\matty\cloudstation\tf2 mapping\work\- experiments\portal_vis_calc.bsp
ZIP Output overshot buffer estimate: Estimated 0, actual 54
1 minute, 11 seconds elapsed
'Normal' compile finished in 00:01:12
2 errors/warnings logged:
(1) - Can't load skybox file [sub:1] to build the default cubemap! (Caution)
(2) - Can't load skybox file [sub:1] to build the default cubemap! (Caution)

Starting a 'Normal' compile.
Starting compilation of C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.vmf
Valve Software - vbsp.exe (Jun 14 2017)
8 threads
materialPath: C:\Users\Matty\CloudStation\Steam\steamapps\common\Team Fortress 2\tf\materials
Loading C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.vmf
fixing up env_cubemap materials on brush sides...
ProcessBlock_Thread: 0...1...2...3...4...5...6...7...8...9...10 (0)
ProcessBlock_Thread: 0...1...2...3...4...5...6...7...8...9...10 (0)
Processing areas...done (0)
Building Faces...done (0)
FixTjuncs...
PruneNodes...
WriteBSP...
done (0)
writing C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.prt...Building visibility clusters...
done (0)
*** Error: Skybox vtf files for skybox/sky_tf2_04 weren't compiled with the same size texture and/or same flags!
Can't load skybox file skybox/sky_tf2_04 to build the default cubemap!
*** Error: Skybox vtf files for skybox/sky_tf2_04 weren't compiled with the same size texture and/or same flags!
Can't load skybox file skybox/sky_tf2_04 to build the default cubemap!
Finding displacement neighbors...
Finding lightmap sample positions...
Displacement Alpha : 0...1...2...3...4...5...6...7...8...9...10
Building Physics collision data...
done (0) (2674 bytes)
Placing detail props : 0...1...2...3...4...5...6...7...8...9...10
Compacting texture/material tables...
Reduced 6 texinfos to 4
Reduced 2 texdatas to 2 (36 bytes to 36)
Writing C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.bsp
ZIP Output overshot buffer estimate: Estimated 0, actual 54
0 seconds elapsed
Valve Software - vvis.exe (Jun 14 2017)
8 threads
reading c:\users\matty\cloudstation\tf2 mapping\work\- experiments\portal_vis_calc.bsp
reading c:\users\matty\cloudstation\tf2 mapping\work\- experiments\portal_vis_calc.prt
1000 portalclusters
2600 numportals
BasePortalVis: 0...1...2...3...4...5...6...7...8...9...10 (0)
PortalFlow: 0...1...2...3...4...5...6...7...8...9...10 (13)
Optimized: 0 visible clusters (0.00%)
Total clusters visible: 500000
Average clusters visible: 500
Building PAS...
Average clusters audible: 500
visdatasize:308004 compressed from 256000
writing c:\users\matty\cloudstation\tf2 mapping\work\- experiments\portal_vis_calc.bsp
ZIP Output overshot buffer estimate: Estimated 0, actual 54
13 seconds elapsed
'Normal' compile finished in 00:00:14
2 errors/warnings logged:
(1) - Can't load skybox file [sub:1] to build the default cubemap! (Caution)
(2) - Can't load skybox file [sub:1] to build the default cubemap! (Caution)

I just noticed that the one hour forty-three minute time in the OP's log is for the whole compile, but VRAD is missing (no lighting). @DrSquishy You have some other errors in your compile log you should take care of. IIRC, displacements with too many edges abutting can increase compile time, but it's been some time since I encountered the error. What it means is you have three or more displacement edges aligned along the same plane, touching each other. Normally you would only have two because you'd choose only the top faces of your ground brushes and convert them into displacements. I suspect you've selected the whole of a brush and turned all of its faces in to displacements, and the extra displacements are underground.

Avoid using power 4 displacements, they use more memory and can rarely cause crashes. If you wish to obtain similar resolution, use multiple power 3 displacements instead.

Are you familiar with the Interlopers compile log checker?
 

henke37

aa
Sep 23, 2011
2,075
515
The amount of portals that can see each other out of the total amount of them matters in game but not while compiling, because each is going to be tested against each anyway, so what matters during compile is the total number. Do correct me if I'm being wrong.
VVIS is smart enough to use the portal data that VBSP compiled. It does a tree traversal from the visleaf and only compares against neighboring leafs. Not only that, it performs portal visibility checks along the way.
 

DrSquishy

we've all had better times to die
aa
Feb 10, 2017
1,297
974
I created a hollow box with dimensions of 10240 x 5184 x 10240 and filled it with 512HU-high wedges to create a 3D grid of visleafs with dimensions of 512x512x512. I performed two VBSP+VVIS-only compiles, one with the full hollow space and one with a floor dividing the chamber in half, cutting the number of portals that can see each other at once by roughly 50%.

wlpZWMi.png


Here are the results:

Starting a 'Normal' compile.
Starting compilation of C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.vmf
Valve Software - vbsp.exe (Jun 14 2017)
8 threads
materialPath: C:\Users\Matty\CloudStation\Steam\steamapps\common\Team Fortress 2\tf\materials
Loading C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.vmf
fixing up env_cubemap materials on brush sides...
ProcessBlock_Thread: 0...1...2...3...4...5...6...7...8...9...10 (0)
ProcessBlock_Thread: 0...1...2...3...4...5...6...7...8...9...10 (0)
Processing areas...done (0)
Building Faces...done (0)
FixTjuncs...
PruneNodes...
WriteBSP...
done (0)
writing C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.prt...Building visibility clusters...
done (0)
*** Error: Skybox vtf files for skybox/sky_tf2_04 weren't compiled with the same size texture and/or same flags!
Can't load skybox file skybox/sky_tf2_04 to build the default cubemap!
*** Error: Skybox vtf files for skybox/sky_tf2_04 weren't compiled with the same size texture and/or same flags!
Can't load skybox file skybox/sky_tf2_04 to build the default cubemap!
Finding displacement neighbors...
Finding lightmap sample positions...
Displacement Alpha : 0...1...2...3...4...5...6...7...8...9...10
Building Physics collision data...
done (0) (2282 bytes)
Placing detail props : 0...1...2...3...4...5...6...7...8...9...10
Compacting texture/material tables...
Reduced 6 texinfos to 4
Reduced 2 texdatas to 2 (36 bytes to 36)
Writing C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.bsp
ZIP Output overshot buffer estimate: Estimated 0, actual 54
0 seconds elapsed
Valve Software - vvis.exe (Jun 14 2017)
8 threads
reading c:\users\matty\cloudstation\tf2 mapping\work\- experiments\portal_vis_calc.bsp
reading c:\users\matty\cloudstation\tf2 mapping\work\- experiments\portal_vis_calc.prt
1000 portalclusters
2650 numportals
BasePortalVis: 0...1...2...3...4...5...6...7...8...9...10 (0)
PortalFlow: 0...1...2...3...4...5...6...7...8...9...10 (71)
Optimized: 0 visible clusters (0.00%)
Total clusters visible: 1000000
Average clusters visible: 1000
Building PAS...
Average clusters audible: 1000
visdatasize:258004 compressed from 256000
writing c:\users\matty\cloudstation\tf2 mapping\work\- experiments\portal_vis_calc.bsp
ZIP Output overshot buffer estimate: Estimated 0, actual 54
1 minute, 11 seconds elapsed
'Normal' compile finished in 00:01:12
2 errors/warnings logged:
(1) - Can't load skybox file [sub:1] to build the default cubemap! (Caution)
(2) - Can't load skybox file [sub:1] to build the default cubemap! (Caution)

Starting a 'Normal' compile.
Starting compilation of C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.vmf
Valve Software - vbsp.exe (Jun 14 2017)
8 threads
materialPath: C:\Users\Matty\CloudStation\Steam\steamapps\common\Team Fortress 2\tf\materials
Loading C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.vmf
fixing up env_cubemap materials on brush sides...
ProcessBlock_Thread: 0...1...2...3...4...5...6...7...8...9...10 (0)
ProcessBlock_Thread: 0...1...2...3...4...5...6...7...8...9...10 (0)
Processing areas...done (0)
Building Faces...done (0)
FixTjuncs...
PruneNodes...
WriteBSP...
done (0)
writing C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.prt...Building visibility clusters...
done (0)
*** Error: Skybox vtf files for skybox/sky_tf2_04 weren't compiled with the same size texture and/or same flags!
Can't load skybox file skybox/sky_tf2_04 to build the default cubemap!
*** Error: Skybox vtf files for skybox/sky_tf2_04 weren't compiled with the same size texture and/or same flags!
Can't load skybox file skybox/sky_tf2_04 to build the default cubemap!
Finding displacement neighbors...
Finding lightmap sample positions...
Displacement Alpha : 0...1...2...3...4...5...6...7...8...9...10
Building Physics collision data...
done (0) (2674 bytes)
Placing detail props : 0...1...2...3...4...5...6...7...8...9...10
Compacting texture/material tables...
Reduced 6 texinfos to 4
Reduced 2 texdatas to 2 (36 bytes to 36)
Writing C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.bsp
ZIP Output overshot buffer estimate: Estimated 0, actual 54
0 seconds elapsed
Valve Software - vvis.exe (Jun 14 2017)
8 threads
reading c:\users\matty\cloudstation\tf2 mapping\work\- experiments\portal_vis_calc.bsp
reading c:\users\matty\cloudstation\tf2 mapping\work\- experiments\portal_vis_calc.prt
1000 portalclusters
2600 numportals
BasePortalVis: 0...1...2...3...4...5...6...7...8...9...10 (0)
PortalFlow: 0...1...2...3...4...5...6...7...8...9...10 (13)
Optimized: 0 visible clusters (0.00%)
Total clusters visible: 500000
Average clusters visible: 500
Building PAS...
Average clusters audible: 500
visdatasize:308004 compressed from 256000
writing c:\users\matty\cloudstation\tf2 mapping\work\- experiments\portal_vis_calc.bsp
ZIP Output overshot buffer estimate: Estimated 0, actual 54
13 seconds elapsed
'Normal' compile finished in 00:00:14
2 errors/warnings logged:
(1) - Can't load skybox file [sub:1] to build the default cubemap! (Caution)
(2) - Can't load skybox file [sub:1] to build the default cubemap! (Caution)

I just noticed that the one hour forty-three minute time in the OP's log is for the whole compile, but VRAD is missing (no lighting). @DrSquishy You have some other errors in your compile log you should take care of. IIRC, displacements with too many edges abutting can increase compile time, but it's been some time since I encountered the error. What it means is you have three or more displacement edges aligned along the same plane, touching each other. Normally you would only have two because you'd choose only the top faces of your ground brushes and convert them into displacements. I suspect you've selected the whole of a brush and turned all of its faces in to displacements, and the extra displacements are underground.

Avoid using power 4 displacements, they use more memory and can rarely cause crashes. If you wish to obtain similar resolution, use multiple power 3 displacements instead.

Are you familiar with the Interlopers compile log checker?
I now use compilepal which highlights the errors and described in detail what they are
 

MoonFox

L10: Glamorous Member
Mar 17, 2015
739
74
besides the technical, using skybox textures only on visible areas helps, and nodrawing any non-visible surfaces helps. outside of that, these guys are better at it.
 

DrSquishy

we've all had better times to die
aa
Feb 10, 2017
1,297
974
besides the technical, using skybox textures only on visible areas helps, and nodrawing any non-visible surfaces helps. outside of that, these guys are better at it.
Thanks