Repacking your maps

[Vel0city]

How long should this process take? Mine's been chugging along for about a half hour for a 60 mb file.

The ingame method worked!

Does it really take that long? Or did you restart it and it was done much faster?

 

[tovilovan]

Does it really take that long? Or did you restart it and it was done much faster?

The cmd method never ended, but the ingame method was fine (maybe 1 min).

 

[bazooka]

Just want to point out that repacking and repacking with compression are distinct. We've conflated them.

bspzip -repack [ -compress ] <bspfile>
  Optimally repacks a BSP file, optionally using compressed BSP format.
  Using on a compressed BSP without -compress will effectively decompress
  a compressed BSP.

Repacking is probably reorganizing custom assets in the BSP so that they're arranged optimally for...something. I'm guessing for extraction speed of the custom assets.

Somewhat confusingly, bspzip.exe doesn't compress by default but bsp_repack does.

Repack without compression:
bspzip.exe -repack <input/output bsp>
bsp_repack -nocompress <input bsp> <output bsp>

These double as decompression commands when used on maps repacked with compression.

Repack with compression:
bspzip.exe -repack -compress <input/output bsp>
bsp_repack <input bsp> <output bsp>

Technically, you could do a repack without compression followed by a compression with bz2. You might do this as a last resort if repacking with compression was causing problems.

 

[Crowbar]

So if the goal is, as usually, reducing file size by as much as possible, it is best to use -compress unless having problems with it?

 

[bazooka]

Yep. LZMA compression, what bspzip -repack -compress uses, tends to produce smaller files and decompress faster than bz2.

Anyway, I was just trying to clear up the confusion about the seemingly different meanings of pack and repack (a few posts ago). Really, they're similar and mean pretty much what you'd expect. It's just that we've been using "repack" as a synonym for "repack with compression". Not a big deal, but it's the root of some confusion.

I should mention that I'm a big noob that shouldn't be considered an authority on anything.

 

[worMatty]

Hmm. I wonder if there's any way we can look at the difference between two uncompressed maps, one repacked, one not.

 

[Crowbar]

I'd rather still use pack as in pack content used and repack as in repack with compression. Clear, short, and everyone is used to it. Anyway, the more I know. Thanks!

 

[henke37]

LZMA is per chunk, while BZIP2 is for the entire file.

 

[Vel0city]

Just a small bump with a question: are all the stock maps that ship with the game repacked?

 

[LeSwordfish]

Not all. Valve is repacking and updating them a few at a time.

 

[Vel0city]

Well, I took the liberty of repacking each and every single map in tf/maps. It took a while, but nothing you can't solve with patience and Pink Floyd.

Size of tf/maps before repacking: 3.89GB. Size of tf/maps after repacking: 1.9GB. Reduction: 1.99GB. Space saved: 49%. That's not nothing.

E-mailing Valve on Monday to see if they can just take all the maps I compressed for them and have them just drop 'em in TF2 with the next update, just for the sake of saving 2GB of space. And I'll ask them to remove the HL2 folder while they're at it (since basically nothing apart from 2-3 models/materials is being used) since that adds another 2.65GB to the "space saved" list once it's gone. And I'll ask them if it's possible for vpk files to get some form of compression as well in the future. For giggles I took the 17 tf2_misc vpk files and compressed each individual file into a zip. File size of them combined before compression: 1750MB. After compressing them individually into a zip: 544 MB. That's about 31% of the original total file size. Imagine the space savings with compression on all the vpk files...

Oh and anybody who stumbles onto this thread: I made a couple of batch files (thanks to YM for the code which you can find in the OP of this thread) that make repacking/unpacking your map super easy. Like, drag-n-drop easy. Download that here: https://tf2maps.net/downloads/drag-n-drop-repack-unpack-batch-files.2451/

 

[Kube]

Well, I took the liberty of repacking each and every single map in tf/maps. It took a while, but nothing you can't solve with patience and Pink Floyd.

Size of tf/maps before repacking: 3.89GB. Size of tf/maps after repacking: 1.9GB. Reduction: 1.99GB. Space saved: 49%. That's not nothing.

E-mailing Valve on Monday to see if they can just take all the maps I compressed for them and have them just drop 'em in TF2 with the next update, just for the sake of saving 2GB of space. And I'll ask them to remove the HL2 folder while they're at it (since basically nothing apart from 2-3 models/materials is being used) since that adds another 2.65GB to the "space saved" list once it's gone. And I'll ask them if it's possible for vpk files to get some form of compression as well in the future. For giggles I took the 17 tf2_misc vpk files and compressed each individual file into a zip. File size of them combined before compression: 1750MB. After compressing them individually into a zip: 544 MB. That's about 31% of the original total file size. Imagine the space savings with compression on all the vpk files...

Ask for the repack. Don't ask for the removal of map content, for as much space as it could save, you will break many a deathrun map*.

*in all seriouesness, many, many non-stock-looking custom maps use HL2 content; removing this content and replacing it with checkerboards would result in utter community outrage.

 

[Vel0city]

Ask for the repack. Don't ask for the removal of map content, for as much space as it could save, you will break many a deathrun map*.

*in all seriouesness, many, many non-stock-looking custom maps use hl2 textures; removing them would result in an utter community outrage.

Not if you properly announce the change a week or 2 beforehand giving community servers running these maps enough time to do all their necessary work before you actually push the update out. But thanks for reminding me of that, I'll include it in my email.

 

[Crash]

Fun fact: Probed uses a HL2 material in the space ship, and a lot of the audio for the finale comes from HL2 as well!

 

[LeSwordfish]

Valve have been - quite sensibly I think - combining map fixes and repacks together. Sending them a repacked "pack" and saying "get to it" would just encourage them to skip the fixes.

 

[Kube]

Not if you properly announce the change a week or 2 beforehand giving community servers running these maps to do all their necessary work before you actually push the update out. But thanks for reminding me of that, I'll include it in my email.

To be blunt, that's remarkable unfeasible, especially for servers running maps without active authors. For the space it saves (for any space in that matter), it's really not worth it.

Valve have been - quite sensibly I think - combining map fixes and repacks together. Sending them a repacked "pack" and saying "get to it" would just encourage them to skip the fixes.

Hm, I didn't realize this. Still, with the relative effort it takes to push out a repacked version of a map, I don't see why a large-scale downsizing didn't happen in the first place (other than to not blow out Australians, of course).

 

[Vel0city]

I was also surprised to see all 3 maps shipped with Meet your Match to be also not repacked, something they were (with the exception being Swiftwater I think) before they went official. I read/heard somewhere that Valve recompiles the map anyway when they get it ready for a big update, but to not repack it afterwards is kinda strange this time around.

Guess they don't have my drag-n-drop repacking tools ¯\_(ツ)_/¯ (i'll send them that too)

 

[Powerlord]

And I'll ask them to remove the HL2 folder while they're at it (since basically nothing apart from 2-3 models/materials is being used) since that adds another 2.65GB to the "space saved" list once it's gone.

As a mapper, you should know why that's not possible. Since the HL2 content was shipped as part of all Orange Box games including TF2, Valve has to continue shipping them because they don't know what maps are using it. It's not just restricted to maps, either... these files can be used anywhere. This includes the HL2 voice files that they started shipping after the conversion to VPKs in 2013 (although why you'd want Alyx's voice and stuff, I have no idea).

The only reason Valve was able to get away with converting all the TF2 VO files from wav to mp3 was because Mappers are supposed to be using soundscripts instead (such as using the tf_gamerules PlayVO, PlayVORed, and PlayVOBlue inputs).

And I'll ask them if it's possible for vpk files to get some form of compression as well in the future. For giggles I took the 17 tf2_misc vpk files and compressed each individual file into a zip. File size of them combined before compression: 1750MB. After compressing them individually into a zip: 544 MB. That's about 31% of the original total file size. Imagine the space savings with compression on all the vpk files...

To my knowledge, neither VPK nor the game engine itself were built with compression in mind. This isn't just some switch you can throw and it suddenly works.

 

[worMatty]

• If the TF Team repacked all of the maps included with the game, every client and every server would suddenly need to download them all again, at once, which would be a massive surge of data transfer.
• There would be a risk of breaking something, as repacked maps have had problems in the past, so it's better to repack a small number of maps at a time and risk something small, manageable and unobtrusive for customers if it needs reversing.

 

[Pocket]

Out of curiosity, which of the official maps have been repacked so far?