Repacking your maps

Geit

💜 I probably broke it 💜
aa
May 28, 2009
598
1,161
So, recently Valve have been pushing repacking tools for BSPs. These tools can typically reduce the size of the map by up to 70%. This means that your maps take less time to download (and it removes the need to BZ2 your maps) and take less space to store.

Unfortunately, maps can't be repacked retroactively - any client playing a repacked version of the map will get a version mismatch if the server is running an un-repacked version, so this guide applies to anyone making new versions of their maps.

How much space can I save?
Below is a chart of a selection of maps I had installed when I wrote this, and their space savings after compression (higher is better).
X697u3.png


How does it work?

A BSP file consists of a set of “lumps” that each contain information relevant to the map, including planes, entities and visibility data. The BSP Repacking utilities compress each of these lumps individually using LZMA compression. The exception to this is pakfile lump, which is a .zip file where each individual file instead undergoes its own compression.

Make sure you've built cubemaps and whatnot before hand, otherwise they won't be compressed!

BSPZip
BSPZip is a command-line utility shipped with the TF2 SDK, it was originally designed to handle packing of custom content into a map’s Pakfile but since the Gun Mettle update also handles compressing a map’s lumps and pakfile.
Here are some steps you can follow to compress your maps with it:

  1. Navigate to /SteamApps/common/Team Fortress 2/bin and shift-right-click (with nothing selected) in empty space and select “Open Command Window Here”.
  2. Type “bspzip -repack -compress ../tf/maps/<map_name>”
    • i.e. “bspzip -repack -compress ../tf/maps/cp_snowplow”
  3. Wait for the process to finish, and done!
Alternatively you can use this batch file post to create a drag + drop compressor (Thanks YM!):
"..\..\bin\bspzip" -repack -compress %1
@pause
To unpack a map, simply omit the -compress from the command.

In-Game

  1. Open the developer console in-game
  2. Type “bsp_repack maps/<map_name>.bsp maps/<map_name>.bsp”
  3. Wait for the process to finish, and done!

Do I need to BZ2 my map too?
No! It doesn’t provide any benefit to BZ2 a repacked map, it’ll come out almost identically sized and just makes life difficult for people downloading your map.
 
Last edited:

worMatty

Repacking Evangelist
aa
Jul 22, 2014
1,257
999
Thanks for this information, Geit.

It used to be the case that performing bsp_repack on a map would prevent its soundscript, soundscape, and some special types of material for working. I am happy to report that an example map I have used bsp_repack on today has working soundscript and soundscape, though I could not test an example broken material, because when I go in to spectator mode and fly around the map, or try to begin the game by joining a team and creating a puppet bot, TF2 crashes. The map I am testing with is a Deathrun map which has a lot of logic and is a large outdoor space, so perhaps there is something unusual that is the cause. I am not sure.

I have also had mixed results with menu_photo images. I repacked two versions of my map, and on one of them the image was not loaded in to the map information screen. The version of the map that I have on the workshop, which has been there for some time, does display them, and does not crash.
 

Geit

&#128156; I probably broke it &#128156;
aa
May 28, 2009
598
1,161
Thanks for writing this up Geit. Do you know if this feature was added only to TF2, or do other Source engine games have this capability now?

I believe it's built into all Source games, has been for a very long time (pak file compression might not be, however), no one's ever used it before though. It was originally used to reduce the size of the maps on Console releases of the engine.
 

Egan

aa
Feb 14, 2010
1,375
1,721
It used to be the case that performing bsp_repack on a map would prevent its soundscript, soundscape, and some special types of material for working.

Yeah I just re-tested the bsp_repack command potentially breaking refracting materials in my surf_overlook map, however it does still break the refract material:

Refract material works in uncompressed map:
ehm4lSr.jpg


developer 2 prior to inputting bsp_repack command:
Repacking BSP: Created 'materials/oceanic/glass/oceanic_glass_01.vmt' in lump pak
Repacking BSP: Created 'materials/oceanic/glass/oceanic_glass_01_normal.vtf' in lump pak
Repacking BSP: Created 'materials/oceanic/glass/oceanic_glass_01.vtf' in lump pak

Refract material broken after compressing map:
QJQXI7n.jpg


After mat_reloadallmaterials it gives error message about refract material:
Et2OFav.jpg


Here's a quick copy of the 3 oceanic_glass_01 files in case someone doesn't want to redownload the entire maritime pack or doesn't have it (otherwise its in the location in the quote above): https://www.dropbox.com/s/i8uikek5h5vgccu/oceanic_refract_glass_materials.zip?dl=0

Edit: and here's the VMT in case you also don't want to download the above to check it yourself:
"refract"
{
"$model" "1"
"$bluramount" "1"
"$surfaceprop" "glass"
"$refractamount" "0.2"
"$scale" "[1 1]"
"%keywords" "tf, bulletcrops"
"$normalmap" "oceanic/glass/oceanic_glass_01_normal"
//"$dudvmap" "oceanic/glass/oceanic_glass_01_dudv"
"$REFRACTTINTTEXTURE" "oceanic/glass/oceanic_glass_01"

"Proxies"
{
"TextureScroll"
{
"texturescrollvar" "$bumptransform"
"texturescrollrate" .1
"texturescrollangle" 45
}
}


"Refract_dx90"
{
"$refractamount" "0.100"
}

"Refract_DX80"
{
"$refractamount" ".025"
"$REFRACTTINTTEXTURE" "oceanic/glass/oceanic_glass_01"

"Proxies"
{
"TextureScroll"
{
"texturescrollvar" "$bumptransform"
"texturescrollrate" 0.1
"texturescrollangle" -90
}
}
}

"UnlitGeneric_dx8"
{
"$basetexture" "oceanic/glass/oceanic_glass_01"
"Proxies"
{
"Sine"
{
"resultVar" "$alpha"
"sineperiod" 15
"sinemin" 0.1
"sinemax" 0.1
}
}

"UnlitGeneric_dx6"
{
"$basetexture" "oceanic/glass/oceanic_glass_01"
"Proxies"
{
"Sine"
{
"resultVar" "$alpha"
"sineperiod" 15
"sinemin" 0.1
"sinemax" 0.1
}
}
}
}
}
// BulletCrops Project
// http://www.minerality.fr/bulletcropsproject/

Edit 2: I haven't sent this to valve yet because I was too busy to look into it any further. but maybe they know what's going on?
 
Last edited:

Geit

&#128156; I probably broke it &#128156;
aa
May 28, 2009
598
1,161
Yeah I just re-tested the bsp_repack command potentially breaking refracting materials in my surf_overlook map, however it does still break the refract material:

Spent a while debugging this tonight, and this happens because the refraction tint texture is very very small (342 bytes). Using a larger, but identical, texture (342KB) causes the issue to disappear, and gets compressed to <1KB anyway.

The issue should probably be reported to Valve anyway though.
 

YM

LVL100 YM
aa
Dec 5, 2007
7,135
6,056
How exactly did you arrive at that conclusion? From a certain shader is broken to realising it's badly compressing small files. Impressive leaps.
 

Geit

&#128156; I probably broke it &#128156;
aa
May 28, 2009
598
1,161
How exactly did you arrive at that conclusion? From a certain shader is broken to realising it's badly compressing small files. Impressive leaps.

I was trying to make a minimal test case to confirm it was the texture, so I had a simple map with just the texture and a prop behind it. The two errors printed out to console when loading the map were
Code:
LZMA file handle: Reached end of underlying stream before expected
Error reading material data "materials/oceanic/glass/oceanic_glass_01.vtf"

So thinking it was an issue with the shader I pulled up its documentation page and the VMT file, then I noticed that the shader was actually working fine - it was refracting correctly but the only problem was that it was tinted with the purple/black error checkers. So I decided to look at the actual tint texture and noticed it was only 16x16px with mipmaps (solid colour), so I decided to see if replacing it with a 512x512 texture would work, and it did.

Honestly, considering the nature of the tint texture, I don't know why the VMT doesn't use $refracttint "[0.73 0.73 0.73]" instead.
 

worMatty

Repacking Evangelist
aa
Jul 22, 2014
1,257
999
The material I have been having problems with is a 16x16 white square.

I didn't know you could specify a colour in a VMT without having to use a VTF. Thanks, Geit! And thanks for testing this, too, Egan!

Question for you. Did this come up in your console on its own:
LZMA file handle: Reached end of underlying stream before expected
or did you have to use a special mode to see that error?
 

worMatty

Repacking Evangelist
aa
Jul 22, 2014
1,257
999
I never knew that form existed.
 

worMatty

Repacking Evangelist
aa
Jul 22, 2014
1,257
999
I'm neither of those things, ty, but the form is available to me. I have donated in the past but am not currently in that group. Maybe I have leftover permissions? The form doesn't look integrated with the site, maybe it's just an iframe or something?
 

tyler

aa
Sep 11, 2013
5,102
4,621
There is a Google form on that page that is inserted, and I don't know why, especially since it's old. There's a different upload form that I'm fairly sure isn't displayed for non VIP/staff.
 

Pocket

Half a Lambert is better than one.
aa
Nov 14, 2009
4,694
2,579
More to the point, doesn't the upload form for the Downloads section only accept files that have been zipped in some way? Seems redundant if BSP can now be compressed as far as it will go, but at the same time we probably don't want people wasting space posting uncompressed maps.
 

YM

LVL100 YM
aa
Dec 5, 2007
7,135
6,056
Put this in a batch file in your maps folder after moving gameinfo.txt and now you can just drag a bsp on to repack it.
"..\..\bin\bspzip" -repack -compress %1
@pause
 

henke37

aa
Sep 23, 2011
2,075
515
I think the issue is that the compression code isn't quite right. Bspsource has been reporting some strange lump sizes and such. Is it possible that the decompression slightly corrupts data by getting lump lengths subtly wrong?
 

worMatty

Repacking Evangelist
aa
Jul 22, 2014
1,257
999
For anyone reading this thread now, the problem affecting small VTFs was fixed on July 15th.
Fixed a case where map textures would fail to load properly in compressed maps (thanks to Egan and Geit for the report and test case)

Thanks, Vincentor.