Is there a limit to the number of items you can pack into a map?

Discussion in 'Mapping Questions & Discussion' started by Werewolf, Jan 14, 2016.

  1. Werewolf

    aa Werewolf Probably not a real Werewolf

    Messages:
    723
    Positive Ratings:
    205
    I compile the map with BspZip as part of the expert compiler (VBSP > VVIS > VRAD > BspZip > Run Game), and when TF2 opens if closes shortly after saying "Map with no textures". When I check the compile log, it stops mid way though packing an asset.

    This is what I have at the bottom of my log:
    Code:
    Adding file: C:\Program Files (x86)\Steam\SteamApps\common\Team Fortress 2\tf\materials\custom\misc\flags\flag_uk.vmt
    Adding file: C:\Program Files (x86)\Steam\SteamApps\common\Team Fortress 2\tf\materials\custom\misc\flags\flag_uk.vtf
    Adding file: C:\Program Files (x86)\Steam\SteamApps\common\Team Fortress 2\tf\materials\custom\misc\flags\flag_ukraine.vmt
    Adding file: C:\Program Files (x86)\Steam\SteamApps\common\Team Fortress 2\tf\materials\custom\misc\flags\flag_ukraine.vtf
    Adding file: C:\Program Files (x86)\Steam\SteamApps\common\Team Fortress 2\tf\materials\custom\misc\flags\flag_usa.vmt
    Adding file: C:\Program Files (x86)\Steam
    Valve Software - bspzip.exe (Dec 30 2015)
    You are out of memory!
    Everything else compiles fine. If I run the map without running BspZip, the map loads.

    Now I do have a very large list of custom assets in this particular map (my map_bspzip.txt file has over 2,000 lines) but I always assumed that so long as you were within the maps limits, you could pack in a lot more. Take props for example. My latest compile log says
    Code:
    Ready to Finish
    
    Object names  Objects/Maxobjs  Memory / Maxmem  Fullness
    ------------  ---------------  ---------------  --------
    models  655/1024  31440/49152  (64.0%)
    Surely that should mean that if I wanted to I could add more custom props to my map, either 1 prop used lots of times or several hundred unique props each used once.

    My question therefore is, as you might have guessed from the title, is there a limit to the number of items you can pack into a map? This is actually really important to me as I was about to get Propper installed again so that I can reduce my brushes and brushsides by turning various items into props.

    If needed, I can post the whole compile log.
     
  2. Geit

    aa Geit 💜 I probably broke it 💜

    Messages:
    597
    Positive Ratings:
    1,110
    As far as I can tell, the size limit for the Pakfile is ~2GB. How much stuff are you trying to pack in?

    EDIT: The maximum number of files in the Pakfile appears to be 65535
    EDIT2: Check your task manager while building the Pakfile, this error sounds more like your actual system is running out of memory, rather than the BSP File.
     
  3. Werewolf

    aa Werewolf Probably not a real Werewolf

    Messages:
    723
    Positive Ratings:
    205
    I haven't counted it up yet, but most of the files are for models and their textures

    Are you talking about how much RAM I have? In that case I have 16gb. I didn't think that I would have filled that.
     
  4. Bogdy

    Bogdy L4: Comfortable Member

    Messages:
    187
    Positive Ratings:
    208
    If we're talking custom textures/models, it would be better to use Pakrat.
     
  5. Werewolf

    aa Werewolf Probably not a real Werewolf

    Messages:
    723
    Positive Ratings:
    205
    Yeah people keep saying use this or use that. BspZip works for me, with vide as a backup. And what I use to pack items is not the issue here.
     
  6. Vel0city

    aa Vel0city func_fish

    Messages:
    1,900
    Positive Ratings:
    1,512
    Keep in mind the entire SDK including BspZip is 32 bit, so we're actually looking at 3-4GB. Run the whole compile process again (including BspZip) and look in task manager for memory usage, particularly during the BspZip process. Usually if it goes above 2-2.3GB of RAM usage it starts to go down south very quickly which results in the command log saying you're out of memory and a broken map file.
     
  7. henke37

    aa henke37

    Messages:
    1,855
    Positive Ratings:
    430
    That's because only executables built with the magical "3gb ok" flag is allowed to use that much memory. All other 32 bit executables are limited to 2 GB.
     
  8. Werewolf

    aa Werewolf Probably not a real Werewolf

    Messages:
    723
    Positive Ratings:
    205
    Well I just compiled the map without running BspZip, then re-compile doing only BspZip (so not re-doing vbsp/vvis/vrad again), and used Shadowplay to record my Task manager.

    The highest I see it get to for memory usage is 537mb.

     
  9. Werewolf

    aa Werewolf Probably not a real Werewolf

    Messages:
    723
    Positive Ratings:
    205
    So does anyone have any ideas why BspZip is failing at 537mb which is well under the 2048mb limit? For the record I have tried compiling the map then packing with VIDE which didn't work.
     
  10. ics

    aa ics http://ics-base.net

    Messages:
    660
    Positive Ratings:
    411
    Have you even tried packing the content with pakrat, just to see if it works?
     
  11. Geit

    aa Geit 💜 I probably broke it 💜

    Messages:
    597
    Positive Ratings:
    1,110
    I forwarded this thread to Jill from Valve, who recently added the packing compression stuff to BSPZip - He suggested splitting your _bspzip.txt file into multiple runs. For example, split the file list into 4 parts and run bspzip 4 times. He also said that if you want to make a test case available (which would likely include an unpacked BSP, all the materials/models you're trying to pack, and the packing list) he'd take a look at it for you.
     
    • Like Like x 2
  12. Werewolf

    aa Werewolf Probably not a real Werewolf

    Messages:
    723
    Positive Ratings:
    205
    I'll give this a try and see what it does, though I'll do like I've done before. Compile the map, then run BspZip as a separate compile. I've backed up my original _bspzip.txt file and split it into 4 files, each with 500 lines of items.
    Well this is for a custom map I have no intention of getting in the core TF2, would they still look at it?
     
  13. Werewolf

    aa Werewolf Probably not a real Werewolf

    Messages:
    723
    Positive Ratings:
    205
    Still didn't work. It got though the first list fine, then nearly got to the end of the 2nd list before it ran out of memory. When it tried to run the 3rd and 4th lists it gave "ValidateLump: old version for lump 7 in map!" errors for both.
     
  14. Geit

    aa Geit 💜 I probably broke it 💜

    Messages:
    597
    Positive Ratings:
    1,110
    Step 2 likely corrupted the BSP File as it didn't finish correctly, I'm guessing. That's what's causing the lump 7 error in steps 3 and 4.

    Also, just to confirm this is an definitely an issue with how BSPZip handles memory, you aren't running into any hard limits in the BSP format.
    They've fixed bugs that have been triggered by other esoteric maps before (i.e. wubwub). So yeah, I'd guess they'd look at it.
     
  15. Geit

    aa Geit 💜 I probably broke it 💜

    Messages:
    597
    Positive Ratings:
    1,110
    @The Jill snuck this into the lastest update for you

     
  16. Werewolf

    aa Werewolf Probably not a real Werewolf

    Messages:
    723
    Positive Ratings:
    205
    I did see that in the patch notes, however a couple hours before the update dropped my PC crapped out and is currently out of action. When I get it working again, I'll do a test compile and get back to you.
     
  17. Werewolf

    aa Werewolf Probably not a real Werewolf

    Messages:
    723
    Positive Ratings:
    205
    OK so my PC is back in working order and I've managed to compile the map.

    The good news is that I managed to do a full compile and pack the map with BspZip. The only downside has been that when I tried to build cubemaps, it then crashed with an 'Out of Memory' error. I suspect that might have because I used too many env_cubemap entities which I will look at reducing and see if things improve.