Repacking your maps

Discussion in 'Tutorials & Resources' started by Geit, Jul 5, 2015.

  1. Geit

    aa Geit 💜 I probably broke it 💜

    Messages:
    593
    Positive Ratings:
    1,100
    So, recently Valve have been pushing repacking tools for BSPs. These tools can typically reduce the size of the map by up to 70%. This means that your maps take less time to download (and it removes the need to BZ2 your maps) and take less space to store.

    Unfortunately, maps can't be repacked retroactively - any client playing a repacked version of the map will get a version mismatch if the server is running an un-repacked version, so this guide applies to anyone making new versions of their maps.

    How much space can I save?
    Below is a chart of a selection of maps I had installed when I wrote this, and their space savings after compression (higher is better).
    [​IMG]

    How does it work?

    A BSP file consists of a set of “lumps” that each contain information relevant to the map, including planes, entities and visibility data. The BSP Repacking utilities compress each of these lumps individually using LZMA compression. The exception to this is pakfile lump, which is a .zip file where each individual file instead undergoes its own compression.

    Make sure you've built cubemaps and whatnot before hand, otherwise they won't be compressed!

    BSPZip
    BSPZip is a command-line utility shipped with the TF2 SDK, it was originally designed to handle packing of custom content into a map’s Pakfile but since the Gun Mettle update also handles compressing a map’s lumps and pakfile.
    Here are some steps you can follow to compress your maps with it:

    1. Navigate to /SteamApps/common/Team Fortress 2/bin and shift-right-click (with nothing selected) in empty space and select “Open Command Window Here”.
    2. Type “bspzip -repack -compress ../tf/maps/<map_name>”
      • i.e. “bspzip -repack -compress ../tf/maps/cp_snowplow”
    3. Wait for the process to finish, and done!
    Alternatively you can use this batch file post to create a drag + drop compressor (Thanks YM!):
    To unpack a map, simply omit the -compress from the command.

    In-Game

    1. Open the developer console in-game
    2. Type “bsp_repack maps/<map_name>.bsp maps/<map_name>.bsp”
    3. Wait for the process to finish, and done!

    Do I need to BZ2 my map too?
    No! It doesn’t provide any benefit to BZ2 a repacked map, it’ll come out almost identically sized and just makes life difficult for people downloading your map.
     
    • Thanks Thanks x 44
    Last edited: Aug 29, 2015
  2. worMatty

    aa worMatty Repacking Evangelist

    Messages:
    1,029
    Positive Ratings:
    793
    Thanks for this information, Geit.

    It used to be the case that performing bsp_repack on a map would prevent its soundscript, soundscape, and some special types of material for working. I am happy to report that an example map I have used bsp_repack on today has working soundscript and soundscape, though I could not test an example broken material, because when I go in to spectator mode and fly around the map, or try to begin the game by joining a team and creating a puppet bot, TF2 crashes. The map I am testing with is a Deathrun map which has a lot of logic and is a large outdoor space, so perhaps there is something unusual that is the cause. I am not sure.

    I have also had mixed results with menu_photo images. I repacked two versions of my map, and on one of them the image was not loaded in to the map information screen. The version of the map that I have on the workshop, which has been there for some time, does display them, and does not crash.
     
  3. sevin

    aa sevin

    Messages:
    936
    Positive Ratings:
    616
    Thanks for writing this up Geit. Do you know if this feature was added only to TF2, or do other Source engine games have this capability now?
     
  4. Geit

    aa Geit 💜 I probably broke it 💜

    Messages:
    593
    Positive Ratings:
    1,100
    I believe it's built into all Source games, has been for a very long time (pak file compression might not be, however), no one's ever used it before though. It was originally used to reduce the size of the maps on Console releases of the engine.
     
  5. Egan

    aa Egan

    Messages:
    1,250
    Positive Ratings:
    1,364
    Yeah I just re-tested the bsp_repack command potentially breaking refracting materials in my surf_overlook map, however it does still break the refract material:

    Refract material works in uncompressed map:
    [​IMG]

    developer 2 prior to inputting bsp_repack command:
    Refract material broken after compressing map:
    [​IMG]

    After mat_reloadallmaterials it gives error message about refract material:
    [​IMG]

    Here's a quick copy of the 3 oceanic_glass_01 files in case someone doesn't want to redownload the entire maritime pack or doesn't have it (otherwise its in the location in the quote above): https://www.dropbox.com/s/i8uikek5h5vgccu/oceanic_refract_glass_materials.zip?dl=0

    Edit: and here's the VMT in case you also don't want to download the above to check it yourself:
    "refract"
    {
    "$model" "1"
    "$bluramount" "1"
    "$surfaceprop" "glass"
    "$refractamount" "0.2"
    "$scale" "[1 1]"
    "%keywords" "tf, bulletcrops"
    "$normalmap" "oceanic/glass/oceanic_glass_01_normal"
    //"$dudvmap" "oceanic/glass/oceanic_glass_01_dudv"
    "$REFRACTTINTTEXTURE" "oceanic/glass/oceanic_glass_01"

    "Proxies"
    {
    "TextureScroll"
    {
    "texturescrollvar" "$bumptransform"
    "texturescrollrate" .1
    "texturescrollangle" 45
    }
    }


    "Refract_dx90"
    {
    "$refractamount" "0.100"
    }

    "Refract_DX80"
    {
    "$refractamount" ".025"
    "$REFRACTTINTTEXTURE" "oceanic/glass/oceanic_glass_01"

    "Proxies"
    {
    "TextureScroll"
    {
    "texturescrollvar" "$bumptransform"
    "texturescrollrate" 0.1
    "texturescrollangle" -90
    }
    }
    }

    "UnlitGeneric_dx8"
    {
    "$basetexture" "oceanic/glass/oceanic_glass_01"
    "Proxies"
    {
    "Sine"
    {
    "resultVar" "$alpha"
    "sineperiod" 15
    "sinemin" 0.1
    "sinemax" 0.1
    }
    }

    "UnlitGeneric_dx6"
    {
    "$basetexture" "oceanic/glass/oceanic_glass_01"
    "Proxies"
    {
    "Sine"
    {
    "resultVar" "$alpha"
    "sineperiod" 15
    "sinemin" 0.1
    "sinemax" 0.1
    }
    }
    }
    }
    }
    // BulletCrops Project
    // http://www.minerality.fr/bulletcropsproject/

    Edit 2: I haven't sent this to valve yet because I was too busy to look into it any further. but maybe they know what's going on?
     
    • Thanks Thanks x 2
    Last edited: Jul 5, 2015
  6. Geit

    aa Geit 💜 I probably broke it 💜

    Messages:
    593
    Positive Ratings:
    1,100
    Spent a while debugging this tonight, and this happens because the refraction tint texture is very very small (342 bytes). Using a larger, but identical, texture (342KB) causes the issue to disappear, and gets compressed to <1KB anyway.

    The issue should probably be reported to Valve anyway though.
     
    • Thanks Thanks x 5
  7. YM

    aa YM LVL100 YM

    Messages:
    7,079
    Positive Ratings:
    5,680
    How exactly did you arrive at that conclusion? From a certain shader is broken to realising it's badly compressing small files. Impressive leaps.
     
  8. Geit

    aa Geit 💜 I probably broke it 💜

    Messages:
    593
    Positive Ratings:
    1,100
    I was trying to make a minimal test case to confirm it was the texture, so I had a simple map with just the texture and a prop behind it. The two errors printed out to console when loading the map were
    Code:
    LZMA file handle: Reached end of underlying stream before expected
    Error reading material data "materials/oceanic/glass/oceanic_glass_01.vtf"
    
    So thinking it was an issue with the shader I pulled up its documentation page and the VMT file, then I noticed that the shader was actually working fine - it was refracting correctly but the only problem was that it was tinted with the purple/black error checkers. So I decided to look at the actual tint texture and noticed it was only 16x16px with mipmaps (solid colour), so I decided to see if replacing it with a 512x512 texture would work, and it did.

    Honestly, considering the nature of the tint texture, I don't know why the VMT doesn't use $refracttint "[0.73 0.73 0.73]" instead.
     
    • Thanks Thanks x 3
  9. worMatty

    aa worMatty Repacking Evangelist

    Messages:
    1,029
    Positive Ratings:
    793
    The material I have been having problems with is a 16x16 white square.

    I didn't know you could specify a colour in a VMT without having to use a VTF. Thanks, Geit! And thanks for testing this, too, Egan!

    Question for you. Did this come up in your console on its own:
    or did you have to use a special mode to see that error?
     
  10. Geit

    aa Geit 💜 I probably broke it 💜

    Messages:
    593
    Positive Ratings:
    1,100
    Came up without any special modes.
     
  11. tyler

    aa tyler snail prince, master of a ruined tower

    Messages:
    5,015
    Positive Ratings:
    3,910
  12. worMatty

    aa worMatty Repacking Evangelist

    Messages:
    1,029
    Positive Ratings:
    793
    I never knew that form existed.
     
  13. tyler

    aa tyler snail prince, master of a ruined tower

    Messages:
    5,015
    Positive Ratings:
    3,910
  14. Geit

    aa Geit 💜 I probably broke it 💜

    Messages:
    593
    Positive Ratings:
    1,100
    Should do, most people tend to use BZ2s because it's faster to upload smaller files, but that's irrelevant if the map is repacked.
     
    • Thanks Thanks x 1
  15. worMatty

    aa worMatty Repacking Evangelist

    Messages:
    1,029
    Positive Ratings:
    793
    I'm neither of those things, ty, but the form is available to me. I have donated in the past but am not currently in that group. Maybe I have leftover permissions? The form doesn't look integrated with the site, maybe it's just an iframe or something?
     
  16. tyler

    aa tyler snail prince, master of a ruined tower

    Messages:
    5,015
    Positive Ratings:
    3,910
    There is a Google form on that page that is inserted, and I don't know why, especially since it's old. There's a different upload form that I'm fairly sure isn't displayed for non VIP/staff.
     
  17. Pocket

    aa Pocket spaalone babuguuscooties

    Messages:
    4,146
    Positive Ratings:
    1,915
    More to the point, doesn't the upload form for the Downloads section only accept files that have been zipped in some way? Seems redundant if BSP can now be compressed as far as it will go, but at the same time we probably don't want people wasting space posting uncompressed maps.
     
    • Thanks Thanks x 1
  18. YM

    aa YM LVL100 YM

    Messages:
    7,079
    Positive Ratings:
    5,680
    Put this in a batch file in your maps folder after moving gameinfo.txt and now you can just drag a bsp on to repack it.
     
    • Thanks Thanks x 4
  19. henke37

    aa henke37

    Messages:
    1,580
    Positive Ratings:
    320
    I think the issue is that the compression code isn't quite right. Bspsource has been reporting some strange lump sizes and such. Is it possible that the decompression slightly corrupts data by getting lump lengths subtly wrong?
     
  20. worMatty

    aa worMatty Repacking Evangelist

    Messages:
    1,029
    Positive Ratings:
    793
    For anyone reading this thread now, the problem affecting small VTFs was fixed on July 15th.
    Thanks, Vincentor.