Faster compilation

Discussion in 'Mapping Questions & Discussion' started by DrSquishy, Jun 21, 2017.

  1. DrSquishy

    DrSquishy L10: Glamorous Member

    Messages:
    751
    Positive Ratings:
    260
    When making my map, I always try and keep the numportals around 2000, but recently compilation times have spiked up by an incredible amount. I suspect it is due to a power 4 displacement, but am not completely sure. Extra notes: I am fairly new to optimization in hammer, and only really know about simple things like func_detail. Any and all help is greatly appreciated
    Here is my compile log:
    Starting a 'Full - Both' compile.
    Starting compilation of C:\Users\John\Documents\MvM_origins_displaced.vmf
    Valve Software - vbsp.exe (Jun 14 2017)
    2 threads
    materialPath: C:\Program Files\Steam\steamapps\common\team fortress 2\tf\materials
    Loading C:\Users\John\Documents\MvM_origins_displaced.vmf
    ConVarRef mat_reduceparticles doesn't point to an existing ConVar
    Patching WVT material: maps/mvm_origins_displaced/construction/concrete/concreteblendground003_wvt_patch
    Patching WVT material: maps/mvm_origins_displaced/nightfall/cliffwall_wvt_patch
    Patching WVT material: maps/mvm_origins_displaced/blends/blendconcretesnow003_wvt_patch
    Patching WVT material: maps/mvm_origins_displaced/blends/blendnightcliffsnow002_wvt_patch
    Patching WVT material: maps/mvm_origins_displaced/dev/dev_blendmeasure2_wvt_patch
    fixing up env_cubemap materials on brush sides...
    ProcessBlock_Thread: 0...1...2...3...4...5...6...7...8...9...10 (0)
    ProcessBlock_Thread: 0...1...2...3...4...5...6...7...8...9...10 (0)
    Processing areas...done (0)
    Building Faces...done (1)
    Chop Details...done (0)
    Find Visible Detail Sides...
    Merged 2540 detail faces...done (0)
    Merging details...done (0)
    FixTjuncs...
    PruneNodes...
    WriteBSP...
    NODRAW on terrain surface!
    NODRAW on terrain surface!
    NODRAW on terrain surface!
    NODRAW on terrain surface!
    NODRAW on terrain surface!
    done (0)
    writing C:\Users\John\Documents\MvM_origins_displaced.prt...Building visibility clusters...
    done (0)
    *** Error: Skybox vtf files for skybox/sky_alpinestorm_01 weren't compiled with the same size texture and/or same flags!
    Can't load skybox file skybox/sky_alpinestorm_01 to build the default cubemap!
    *** Error: Skybox vtf files for skybox/sky_alpinestorm_01 weren't compiled with the same size texture and/or same flags!
    Can't load skybox file skybox/sky_alpinestorm_01 to build the default cubemap!
    Finding displacement neighbors...
    Found a displacement edge abutting multiple other edges.
    Warning: overflowed 3 displacement corner-neighbor lists.Finding lightmap sample positions...
    Displacement Alpha : 0...1...2...3...4...5...6...7...8...9...10
    Building Physics collision data...
    WARNING: Map using power 4 displacements, terrain physics cannot be compressed, map will need additional memory and CPU.
    done (0) (2793306 bytes)
    Placing detail props : 0...1...2...3...4...5...6...7...8...9...10
    Water found with no water_lod_control entity, creating a default one.
    Compacting texture/material tables...
    Reduced 1343 texinfos to 927
    Reduced 52 texdatas to 45 (1601 bytes to 1256)
    Writing C:\Users\John\Documents\MvM_origins_displaced.bsp
    Wrote ZIP buffer, estimated size 3672, actual size 2788
    5 seconds elapsed
    Valve Software - vvis.exe (Jun 14 2017)
    2 threads
    reading c:\users\john\documents\MvM_origins_displaced.bsp
    reading c:\users\john\documents\MvM_origins_displaced.prt
    658 portalclusters
    2017 numportals
    BasePortalVis: 0...1...2...3...4...5...6...7...8...9...10 (2)
    PortalFlow: 0...1...2...3...4...5...6...7...8...9...10 (6233)
    Optimized: 140 visible clusters (0.04%)
    Total clusters visible: 397100
    Average clusters visible: 603
    Building PAS...
    Average clusters audible: 658
    visdatasize:114325 compressed from 115808
    writing c:\users\john\documents\MvM_origins_displaced.bsp
    1 hour, 43 minutes, 59 seconds elapsed
    C:\Users\John\Documents\MvM_origins_displaced.bsp -> C:\Program Files\Steam\steamapps\common\team fortress 2\tf\maps\MvM_origins_displaced.bsp
    1 File(s) copied

    CompilePal - Automated Packaging
    Finding sources of game content...
    Found search paths...
    Found wildcard path: C:\Program Files\Steam\steamapps\common\team fortress 2\tf\custom\
    Found search path: C:\Program Files\Steam\steamapps\common\team fortress 2\tf
    Found search path: C:\Program Files\Steam\steamapps\common\team fortress 2\tf\bin
    Found search path: C:\Program Files\Steam\steamapps\common\team fortress 2\tf\download
    Reading BSP...
    Initializing pak file...
    Writing file list...
    Running bspzip...
    Finished!
    ---------------------
    21 materials found
    8 models found
    0 particle files found
    0 sounds found
    additional files:
    ---------------------
    'Full - Both' compile finished in 01:44:15

    I have also included the .vmf just in case
     

    Attached Files:

  2. Tumbolisu

    aa Tumbolisu  I ⌄ I 

    Messages:
    955
    Positive Ratings:
    939
    You have nodraw on displacements. Nodraw does not work on displacements. You have to delete sides of a displacement that you don't want to exist. (Select and click destroy)
    Alternativly, if you only want it to be invisible, use the invisibledisplacement tool texture. (It's literally just a fully transparent texture)
    Where to learn more about optimization: http://www.optimization.interlopers.net/
     
  3. DrSquishy

    DrSquishy L10: Glamorous Member

    Messages:
    751
    Positive Ratings:
    260
    Yeah. Finished compiling and noticed that. was actually just about to remove it
     
  4. killohurtz

    aa killohurtz Distinction in Applied Carving

    Messages:
    1,018
    Positive Ratings:
    1,181
    I'm going to toss you this guide as well: http://www.nodraw.net/2009/12/optimization-in-source-a-practical-demonstration/

    From your vmf, I can immediately see you're making a fundamental optimization mistake: your map is just a giant box with very few world brushes sprinkled about. Because you've used so many func_details and displacements - which don't block visibility - you've essentially told the compiler that the entire map should render no matter where you are. So it will spend hours calculating visibility from every single visleaf to every other visleaf, and give you a map with terrible performance as a result. You need more world brushes in there to break up the map and lessen the load on both the compiler and player's computers.

    Also, try to "shrink-wrap" your skybox to cut out all that wasted space around the map and save the compiler from doing unnecessary calculations (specifically, you don't need a ceiling nearly that high, or any space below ground except where the lava pit is).

    This map is also extremely overscaled. In my honest opinion, it would benefit from a complete rework with smaller, more contained areas. Throw in a few prop_static playermodels or walk around the map in-game to see just how massive it is.
     
  5. worMatty

    aa worMatty Repacking Evangelist

    Messages:
    1,072
    Positive Ratings:
    836
    As killo says, the number of portals doesn't matter, because it's the amount that can see each other that has the biggest effect on compile time. Your compile log says VVIS is taking an hour and forty-three minutes. VVIS shouldn't last very long at all. It's the most common cause of inflated compile times.

    Don't focus on reducing numportals. Focus instead on reducing the possibility of them seeing each other. With visibility optimisation, your goal is to limit what the player can see, finding a balance between complexity and rendering performance. Players can 'see' through func_detail and displacements. Only world brushes will really block the line of sight between one numportal and the other. This is why many maps have areas separated by corridors or massive turns, or shrink-wrapped skybox brushes (which are also world brushes).

    IMO visibility optimisation begins with the initial planning of a map. Rather than start with a big rectangle, think about building a map into a giant S or C shape. Doing so will make it much easier to divide outdoor spaces. Also, put some thought into how large buildings can be placed to limit line of sight.

    <horizontal rule>

    Look at this
    [​IMG]
     
    • Funny Funny x 2
  6. DrSquishy

    DrSquishy L10: Glamorous Member

    Messages:
    751
    Positive Ratings:
    260
    Wow. As soon as possible, I have a lot of optimisation to do.
     
  7. Crowbar

    aa Crowbar perfektoberfest

    Messages:
    1,441
    Positive Ratings:
    1,196
    The amount of portals that can see each other out of the total amount of them matters in game but not while compiling, because each is going to be tested against each anyway, so what matters during compile is the total number. Do correct me if I'm being wrong.
     
  8. worMatty

    aa worMatty Repacking Evangelist

    Messages:
    1,072
    Positive Ratings:
    836
    I created a hollow box with dimensions of 10240 x 5184 x 10240 and filled it with 512HU-high wedges to create a 3D grid of visleafs with dimensions of 512x512x512. I performed two VBSP+VVIS-only compiles, one with the full hollow space and one with a floor dividing the chamber in half, cutting the number of portals that can see each other at once by roughly 50%.

    [​IMG]

    Here are the results:

    Starting a 'Normal' compile.
    Starting compilation of C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.vmf
    Valve Software - vbsp.exe (Jun 14 2017)
    8 threads
    materialPath: C:\Users\Matty\CloudStation\Steam\steamapps\common\Team Fortress 2\tf\materials
    Loading C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.vmf
    fixing up env_cubemap materials on brush sides...
    ProcessBlock_Thread: 0...1...2...3...4...5...6...7...8...9...10 (0)
    ProcessBlock_Thread: 0...1...2...3...4...5...6...7...8...9...10 (0)
    Processing areas...done (0)
    Building Faces...done (0)
    FixTjuncs...
    PruneNodes...
    WriteBSP...
    done (0)
    writing C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.prt...Building visibility clusters...
    done (0)
    *** Error: Skybox vtf files for skybox/sky_tf2_04 weren't compiled with the same size texture and/or same flags!
    Can't load skybox file skybox/sky_tf2_04 to build the default cubemap!
    *** Error: Skybox vtf files for skybox/sky_tf2_04 weren't compiled with the same size texture and/or same flags!
    Can't load skybox file skybox/sky_tf2_04 to build the default cubemap!
    Finding displacement neighbors...
    Finding lightmap sample positions...
    Displacement Alpha : 0...1...2...3...4...5...6...7...8...9...10
    Building Physics collision data...
    done (0) (2282 bytes)
    Placing detail props : 0...1...2...3...4...5...6...7...8...9...10
    Compacting texture/material tables...
    Reduced 6 texinfos to 4
    Reduced 2 texdatas to 2 (36 bytes to 36)
    Writing C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.bsp
    ZIP Output overshot buffer estimate: Estimated 0, actual 54
    0 seconds elapsed
    Valve Software - vvis.exe (Jun 14 2017)
    8 threads
    reading c:\users\matty\cloudstation\tf2 mapping\work\- experiments\portal_vis_calc.bsp
    reading c:\users\matty\cloudstation\tf2 mapping\work\- experiments\portal_vis_calc.prt
    1000 portalclusters
    2650 numportals
    BasePortalVis: 0...1...2...3...4...5...6...7...8...9...10 (0)
    PortalFlow: 0...1...2...3...4...5...6...7...8...9...10 (71)
    Optimized: 0 visible clusters (0.00%)
    Total clusters visible: 1000000
    Average clusters visible: 1000
    Building PAS...
    Average clusters audible: 1000
    visdatasize:258004 compressed from 256000
    writing c:\users\matty\cloudstation\tf2 mapping\work\- experiments\portal_vis_calc.bsp
    ZIP Output overshot buffer estimate: Estimated 0, actual 54
    1 minute, 11 seconds elapsed
    'Normal' compile finished in 00:01:12
    2 errors/warnings logged:
    (1) - Can't load skybox file [sub:1] to build the default cubemap! (Caution)
    (2) - Can't load skybox file [sub:1] to build the default cubemap! (Caution)

    Starting a 'Normal' compile.
    Starting compilation of C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.vmf
    Valve Software - vbsp.exe (Jun 14 2017)
    8 threads
    materialPath: C:\Users\Matty\CloudStation\Steam\steamapps\common\Team Fortress 2\tf\materials
    Loading C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.vmf
    fixing up env_cubemap materials on brush sides...
    ProcessBlock_Thread: 0...1...2...3...4...5...6...7...8...9...10 (0)
    ProcessBlock_Thread: 0...1...2...3...4...5...6...7...8...9...10 (0)
    Processing areas...done (0)
    Building Faces...done (0)
    FixTjuncs...
    PruneNodes...
    WriteBSP...
    done (0)
    writing C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.prt...Building visibility clusters...
    done (0)
    *** Error: Skybox vtf files for skybox/sky_tf2_04 weren't compiled with the same size texture and/or same flags!
    Can't load skybox file skybox/sky_tf2_04 to build the default cubemap!
    *** Error: Skybox vtf files for skybox/sky_tf2_04 weren't compiled with the same size texture and/or same flags!
    Can't load skybox file skybox/sky_tf2_04 to build the default cubemap!
    Finding displacement neighbors...
    Finding lightmap sample positions...
    Displacement Alpha : 0...1...2...3...4...5...6...7...8...9...10
    Building Physics collision data...
    done (0) (2674 bytes)
    Placing detail props : 0...1...2...3...4...5...6...7...8...9...10
    Compacting texture/material tables...
    Reduced 6 texinfos to 4
    Reduced 2 texdatas to 2 (36 bytes to 36)
    Writing C:\Users\Matty\CloudStation\TF2 mapping\Work\- Experiments\portal_vis_calc.bsp
    ZIP Output overshot buffer estimate: Estimated 0, actual 54
    0 seconds elapsed
    Valve Software - vvis.exe (Jun 14 2017)
    8 threads
    reading c:\users\matty\cloudstation\tf2 mapping\work\- experiments\portal_vis_calc.bsp
    reading c:\users\matty\cloudstation\tf2 mapping\work\- experiments\portal_vis_calc.prt
    1000 portalclusters
    2600 numportals
    BasePortalVis: 0...1...2...3...4...5...6...7...8...9...10 (0)
    PortalFlow: 0...1...2...3...4...5...6...7...8...9...10 (13)
    Optimized: 0 visible clusters (0.00%)
    Total clusters visible: 500000
    Average clusters visible: 500
    Building PAS...
    Average clusters audible: 500
    visdatasize:308004 compressed from 256000
    writing c:\users\matty\cloudstation\tf2 mapping\work\- experiments\portal_vis_calc.bsp
    ZIP Output overshot buffer estimate: Estimated 0, actual 54
    13 seconds elapsed
    'Normal' compile finished in 00:00:14
    2 errors/warnings logged:
    (1) - Can't load skybox file [sub:1] to build the default cubemap! (Caution)
    (2) - Can't load skybox file [sub:1] to build the default cubemap! (Caution)

    I just noticed that the one hour forty-three minute time in the OP's log is for the whole compile, but VRAD is missing (no lighting). @DrSquishy You have some other errors in your compile log you should take care of. IIRC, displacements with too many edges abutting can increase compile time, but it's been some time since I encountered the error. What it means is you have three or more displacement edges aligned along the same plane, touching each other. Normally you would only have two because you'd choose only the top faces of your ground brushes and convert them into displacements. I suspect you've selected the whole of a brush and turned all of its faces in to displacements, and the extra displacements are underground.

    Avoid using power 4 displacements, they use more memory and can rarely cause crashes. If you wish to obtain similar resolution, use multiple power 3 displacements instead.

    Are you familiar with the Interlopers compile log checker?
     
  9. henke37

    aa henke37

    Messages:
    1,945
    Positive Ratings:
    461
    VVIS is smart enough to use the portal data that VBSP compiled. It does a tree traversal from the visleaf and only compares against neighboring leafs. Not only that, it performs portal visibility checks along the way.
     
    • Thanks Thanks x 1
  10. DrSquishy

    DrSquishy L10: Glamorous Member

    Messages:
    751
    Positive Ratings:
    260
    I now use compilepal which highlights the errors and described in detail what they are
     
    • Like Like x 1
  11. MoonFox

    MoonFox L10: Glamorous Member

    Messages:
    737
    Positive Ratings:
    72
    besides the technical, using skybox textures only on visible areas helps, and nodrawing any non-visible surfaces helps. outside of that, these guys are better at it.
     
  12. DrSquishy

    DrSquishy L10: Glamorous Member

    Messages:
    751
    Positive Ratings:
    260
    Thanks