Is there a limit to the number of items you can pack into a map?

Werewolf

Probably not a real Werewolf
aa
Apr 12, 2011
873
309
I compile the map with BspZip as part of the expert compiler (VBSP > VVIS > VRAD > BspZip > Run Game), and when TF2 opens if closes shortly after saying "Map with no textures". When I check the compile log, it stops mid way though packing an asset.

This is what I have at the bottom of my log:
Code:
Adding file: C:\Program Files (x86)\Steam\SteamApps\common\Team Fortress 2\tf\materials\custom\misc\flags\flag_uk.vmt
Adding file: C:\Program Files (x86)\Steam\SteamApps\common\Team Fortress 2\tf\materials\custom\misc\flags\flag_uk.vtf
Adding file: C:\Program Files (x86)\Steam\SteamApps\common\Team Fortress 2\tf\materials\custom\misc\flags\flag_ukraine.vmt
Adding file: C:\Program Files (x86)\Steam\SteamApps\common\Team Fortress 2\tf\materials\custom\misc\flags\flag_ukraine.vtf
Adding file: C:\Program Files (x86)\Steam\SteamApps\common\Team Fortress 2\tf\materials\custom\misc\flags\flag_usa.vmt
Adding file: C:\Program Files (x86)\Steam
Valve Software - bspzip.exe (Dec 30 2015)
You are out of memory!
Everything else compiles fine. If I run the map without running BspZip, the map loads.

Now I do have a very large list of custom assets in this particular map (my map_bspzip.txt file has over 2,000 lines) but I always assumed that so long as you were within the maps limits, you could pack in a lot more. Take props for example. My latest compile log says
Code:
Ready to Finish

Object names  Objects/Maxobjs  Memory / Maxmem  Fullness
------------  ---------------  ---------------  --------
models  655/1024  31440/49152  (64.0%)
Surely that should mean that if I wanted to I could add more custom props to my map, either 1 prop used lots of times or several hundred unique props each used once.

My question therefore is, as you might have guessed from the title, is there a limit to the number of items you can pack into a map? This is actually really important to me as I was about to get Propper installed again so that I can reduce my brushes and brushsides by turning various items into props.

If needed, I can post the whole compile log.
 

Geit

💜 I probably broke it 💜
aa
May 28, 2009
598
1,161
As far as I can tell, the size limit for the Pakfile is ~2GB. How much stuff are you trying to pack in?

EDIT: The maximum number of files in the Pakfile appears to be 65535
EDIT2: Check your task manager while building the Pakfile, this error sounds more like your actual system is running out of memory, rather than the BSP File.
 

Werewolf

Probably not a real Werewolf
aa
Apr 12, 2011
873
309
As far as I can tell, the size limit for the Pakfile is ~2GB. How much stuff are you trying to pack in?
I haven't counted it up yet, but most of the files are for models and their textures

EDIT2: Check your task manager while building the Pakfile, this error sounds more like your actual system is running out of memory, rather than the BSP File.
Are you talking about how much RAM I have? In that case I have 16gb. I didn't think that I would have filled that.
 

Bogdy

L4: Comfortable Member
Aug 13, 2014
187
208
If we're talking custom textures/models, it would be better to use Pakrat.
 

Vel0city

func_fish
aa
Dec 6, 2014
1,947
1,589
Are you talking about how much RAM I have? In that case I have 16gb. I didn't think that I would have filled that.
Keep in mind the entire SDK including BspZip is 32 bit, so we're actually looking at 3-4GB. Run the whole compile process again (including BspZip) and look in task manager for memory usage, particularly during the BspZip process. Usually if it goes above 2-2.3GB of RAM usage it starts to go down south very quickly which results in the command log saying you're out of memory and a broken map file.
 

henke37

aa
Sep 23, 2011
2,075
515
That's because only executables built with the magical "3gb ok" flag is allowed to use that much memory. All other 32 bit executables are limited to 2 GB.
 

Werewolf

Probably not a real Werewolf
aa
Apr 12, 2011
873
309
Well I just compiled the map without running BspZip, then re-compile doing only BspZip (so not re-doing vbsp/vvis/vrad again), and used Shadowplay to record my Task manager.

The highest I see it get to for memory usage is 537mb.

 

Werewolf

Probably not a real Werewolf
aa
Apr 12, 2011
873
309
So does anyone have any ideas why BspZip is failing at 537mb which is well under the 2048mb limit? For the record I have tried compiling the map then packing with VIDE which didn't work.
 

Geit

💜 I probably broke it 💜
aa
May 28, 2009
598
1,161
I forwarded this thread to Jill from Valve, who recently added the packing compression stuff to BSPZip - He suggested splitting your _bspzip.txt file into multiple runs. For example, split the file list into 4 parts and run bspzip 4 times. He also said that if you want to make a test case available (which would likely include an unpacked BSP, all the materials/models you're trying to pack, and the packing list) he'd take a look at it for you.
 

Werewolf

Probably not a real Werewolf
aa
Apr 12, 2011
873
309
I forwarded this thread to Jill from Valve, who recently added the packing compression stuff to BSPZip - He suggested splitting your _bspzip.txt file into multiple runs. For example, split the file list into 4 parts and run bspzip 4 times.
I'll give this a try and see what it does, though I'll do like I've done before. Compile the map, then run BspZip as a separate compile. I've backed up my original _bspzip.txt file and split it into 4 files, each with 500 lines of items.
He also said that if you want to make a test case available (which would likely include an unpacked BSP, all the materials/models you're trying to pack, and the packing list) he'd take a look at it for you.
Well this is for a custom map I have no intention of getting in the core TF2, would they still look at it?
 

Werewolf

Probably not a real Werewolf
aa
Apr 12, 2011
873
309
Still didn't work. It got though the first list fine, then nearly got to the end of the 2nd list before it ran out of memory. When it tried to run the 3rd and 4th lists it gave "ValidateLump: old version for lump 7 in map!" errors for both.
 

Geit

💜 I probably broke it 💜
aa
May 28, 2009
598
1,161
Step 2 likely corrupted the BSP File as it didn't finish correctly, I'm guessing. That's what's causing the lump 7 error in steps 3 and 4.

Also, just to confirm this is an definitely an issue with how BSPZip handles memory, you aren't running into any hard limits in the BSP format.
Well this is for a custom map I have no intention of getting in the core TF2, would they still look at it?

They've fixed bugs that have been triggered by other esoteric maps before (i.e. wubwub). So yeah, I'd guess they'd look at it.
 

Geit

💜 I probably broke it 💜
aa
May 28, 2009
598
1,161
@The Jill snuck this into the lastest update for you

TF2 Update 02/02/16 said:
- Improved bspzip tool stability when packing maps with large amounts of custom assets
 

Werewolf

Probably not a real Werewolf
aa
Apr 12, 2011
873
309
I did see that in the patch notes, however a couple hours before the update dropped my PC crapped out and is currently out of action. When I get it working again, I'll do a test compile and get back to you.
 

Werewolf

Probably not a real Werewolf
aa
Apr 12, 2011
873
309
OK so my PC is back in working order and I've managed to compile the map.

The good news is that I managed to do a full compile and pack the map with BspZip. The only downside has been that when I tried to build cubemaps, it then crashed with an 'Out of Memory' error. I suspect that might have because I used too many env_cubemap entities which I will look at reducing and see if things improve.