Writing a Pakrat replacement

[EArkham]

It's just the vmt that matters far as I know.

I have models using the same specular map but different vtfs that aren't remotely named the same, for example.

And a vmt without a texture reference (or an incorrect one) just gives the purple checks in the texture browser. A vtf with no vmt is skipped by the material browser.

So I'd say the naming conventions for vtfs are just that, naming conventions. It obviously does make sorting which vtfs belong to which vmts easier, but it's not necessary to my knowledge.

Kep

 

[Terr]

Went ahead and rewrote the soundscape parser to handle non-quoted keys/values/labels/etc.

Having trouble actually extracting the data of a "file" inside a GCF/BSP with the HlLib code. I'd help if it was actually commented. And if I really knew C++. I can't tell if I'm getting null pointers because I'm not calling some set-up method, or because I'm doing the cross-language interface wrong. C'est la vie.

Edit: Doncha love it when you figure out part of the crazy problem five minutes after any public complaint about how tricky it is?

Edit2: Muwhahaa, I now have a Frankensteinian reflection-based way to use Java enums directly in place of the C/C++ enums through JNA. (OK, it's actually less Frankensteinian than the first attempt...)

 

[Dr. ROCKZO]

*looks up to see a post and any understanding that goes with it flying over my head*

 

[Terr]

Somewhere along the line I got sidetracked into a complete Java interoperability layer with HlLib. Do I really need to be able to find out how fragmented the GCF file is? Probably not. But I can get at that little hidden increases-every-time-you-hit-save version number for maps... Not that I know what I'd use it for.

 

[Terr]

Still having fun over-engineering the HlLib integration. How many projects offer an excuse to do Weak References, Dynamic Proxy Classes, and JNI? Be still, my heart! The interop library is going on 50k of Jave code, while the file-packing app is more like 15. (This is prior to any code-generation tools.)

Some of the over-engineering is doing things to protect against crashing the JVM and having "nice" function calls.


____

Edit, one day later: "O, parsing and extraction mixt! Reason in madness!"

Can now go from "Look! A BSP file!" to "That BSP file is using custom Skybox X." (At least in unit tests...)

 

[Terr]

And 19 delirious days later... Progress summary:

Important Things That Work:

• From a GCF/BSP file, list all contained files
• From a GCF/BSP file, extract one of the files inside it
• From a BSP file, determine the skybox
• From a BSP file, list the WAV files used
• From a BSP file, list the detail vbsp/sprites used
• From a BSP file, list all soundscapes used
• From a BSP file, list all particle system names used (Not useful unless I crack the PCF format)
• From a BSP file, list all models used
• From a BSP file, list all world-brush materials used
• From a Soundscape file, list all soundscape names
• From a Soundscape file, list all WAV files required
• From a Particle manifest, list all PCF files required

... and a lot of not-important-to-this-project HlLib integration features that work.

Don't Work / Not Yet Started:

• The basic workflow that ties everything together
• GUI access
• Command-line access
• From a VMF, list required VTFs
• Folder access/determination. (GCF location, mapsrc location, etc. all currently hardcoded.)
• A GCF cache to avoid re-reading huge files all the time
• Integration with BSPZIP, writing BSPZIP mapping files

_______

Edit: In the words of the Scout, it's frick'n hilarious (but pretty sensible) that the first VMT I come across is parsed perfectly by the same grammar/code I used for the soundscapes... 'Course, I've found a few other VMT examples which are problematic, but I still didn't expect it to just work on the first try.

 

[Terr]

Question: If you have a custom model and I can't automatically determine the right VMT files from Valve's weird binary proprietary MDL format, what would be a good solution?

Right now I'm thinking of having it look for an optional MODELNAME_materials.txt file in the same directory as the model, and interpreting to use that as a line-by-line list of VMT paths.

You'd need to update it if your recompile your model to use other skins, but it would allow automatic VMT/VTF packing to correspond with the model.

A similar solution could work for PCF files which reference custom sprites.

 

[Icarus]

CP Icons and Color Correction Tables?

Also, as far as I know intro movies don't work if they're packed. they MUST be a separate file.

 

[Terr]

CP Icons and Color Correction Tables?

Good catch. That's a lot of potential custom overlays/models/icons/sounds, but it should be extremely simple to parse them out like it currently does with the skybox. I just need to get a list together of the relevant properties.

Ditto for color_correction* entities.

BTW, do you know what the team_bodygroup_* properties on Team_control_point do? What's the difference between the model and the "model bodygroup"?

Also, as far as I know intro movies don't work if they're packed. they MUST be a separate file.

Can someone with working intro movies test this out? If true, it's too bad, but it does make for less work

 

[A Boojum Snark]

Bodygroup is what changes on the default model when the owner changes. (it's not a skin! /stabpeople)

 

[Terr]

So... it's a submodel or some variety of switchable whatsit that exists on a single model in multiple states?

 

[Terr]

Edit: Scratch that. I still need vbspinfo to get at the static props, since they don't actually create entities in the final output and I don't want to try parsing the binary lump data unless I absolutely have to.

It seems vbspinfo will also cover dynamic props, so there's no need to parse for those although I could.

Today added functionality to pull the determine the following from a BSP:

• func_smokevolume materials
• keyframe_rope material
• Control point / master models/icons/sounds
• env_sprite material
• Color_correction .raw table files

Also re-did a bunch of the parsing system to be more flexible. (Just one ANTLR grammar file now ;P)

 

[Terr]

Building the whole crazy File-X-Depends-On-File-Y part now, and need a quick sanity check. (Not a saving throw, but close )

Question: Are there any resources that are already inside the BSP that I'll need to actually extract and read? (Omitting the entity data.)

For example, is it safe to assume that every VMT that's already in the BSP also has the VTF it requires?

Or should I design with the idea that everything (esp. VMTs) that is already inside the BSP is subject to possible parsing for dependencies that might not yet be packaged as well?

 

[Terr]

It would do everything short of creating the VMF.

Just FYI, due to similarities with entdata/VMFs/soundscapes, I think I a pretty good ANTLR grammar to help with VMFs now.

So if you plan to do VMF stuff in Java/Python/C++ or something else on this page I can send you the .g file.

It should handle all the nasty blocks and quoting stuff so you can treat the VMF a bit like XML. Combine it with the FGD (requiring a differen grammar) and you could make an API for VMF editing for stuff like:

Node n = vmf.addPointEntity("prop_dynamic",0,0,0);
n.set("model","foo.mdl");

_____________________

Questions/Requests:

• I'm still interested if anyone has thoughts on how to get from "custom_bomb.mdl" to "custom_bomb_blueskin.vtf", other than specifying it manually.
• Under what circumstances should I care about mapsrc and materialsrc folders? Currently I'm assuming virtually all the resources to-be-inspected-or-packed are within the game directory (team fortress 2/tf/). The only exception is the map .bsp file itself, which could be sitting anywhere. Is not-scanning those folders a wrong assumption?
• I'm thinking the tool will not "repair" a half-packed BSP. I can't remove or replace stuff that is already in the BSP unless bspzip itself does it.

 

[Terr]

First off, if anybody has answers or thoughs regarding the questions in my last post...
Today working on GCF caching, so that I don't have to open up the GCFs and read them every time. Caches are currently written and read by XStream as XML. I plan to reuse this approach when it comes to config files that define what GCFs a given game requires and what folder paths are important.

_____________

Bleargh. This has turned into three projects at once. (Hell, perhaps four when the GUI comes along, if it seems easier to break it apart.) It highlights a personal failing of mine when it comes to personal projects: I never want to release anything because it's never "good enough". This can be handy as long as I don't exceed my time budget, but then I can't resist "just a small tweak"...

• JHlLib: A library (Maven .jar) for accessing HlLib DLL features
• HL2Parse: A library (Maven .jar) for parsing VMF/Entdata/PCF/Soundscape. Also has potential for VMF parsing. (This is generally the kind of "parsing" an "xml parser" gives you for XML, nodes, attributes, etc.)
• PackBSP: The glue that brings it together.

 

[Ravidge]

Looks like you're pretty much talking to yourself here

I dont have anything to add, but I'm looking forward to seeing this released. With all this effort it can't possibly be bad.

Go Go!

 

[Icarus]

Questions/Requests:[*]Under what circumstances should I care about mapsrc and materialsrc folders? Currently I'm assuming virtually all the resources to-be-inspected-or-packed are within the game directory (team fortress 2/tf/). The only exception is the map .bsp file itself, which could be sitting anywhere. Is not-scanning those folders a wrong assumption?

I never had to pack anything from the src folders and I can't imagine anything would.

 

[Terr]

I'm not dead. I'm still working on the project. It's just... embiggened.

• Using vconfig to find the currently-configured game (same thing SDK dropdown does):
  Not done, but should be easy.
• Reading gameinfo.txt files to find out what app-ids and paths and which SDK (ep1/orangebox) the game relies on:
  Done.
• Decoding the no-official-documentation binary(!) clientregistry.blob in order to find out what app-ids rely on what other app-ids or GCFS: In progress.

Hurray for self-inflicted scope creep. Y'all had better hope that there's no hope of decoding MDLs to get at those juicy VTF paths I see inside

 

[A Boojum Snark]

About the what-to-pack stuff. I thought of a few unusual things that you need to make sure about but I keep thinking there was a really odd one I can't remember now

- localized map descriptions
- fallback materials
- custom materials used in things like env_sprite, move_rope, env_screenoverlay, env_sun... all those odd places where you can specify something. oh and don't forget about the TF2 stuff, custom t_c_p models, custom hud icons...

 

[Terr]

Good catch, I'm missing:

• Localized descriptions
• env_screenoverlay
  --> Question: This has "materials/" and ".vmt" as the automatic prefix, suffix, right? Not "materials/effects/"?
• env_sun

Code teaser for map properties:

Set<String>	getAmbientSounds()    
Set<String>	getColorCorrectionTables() 
Set<String>	getCpMaterials() 
List<String>	getCpModels() 
Set<String>	getCpSounds() 
String		getDetailMaterial() 
String		getDetailVbsp() 
Set<String>	getParticleNames() 
Set<String>	getRopeMaterials() 
String		getSkybox()  
Set<String>	getSmokeMaterials() 
Set<String>	getSoundscapeNames() 
Set<String>	getSpriteMaterials()

As for materials, currently code looks for these things. It does a pretty basic check, so if you put a $reflecttexture that refers to a material inside a shader that doesn't support it, it'll still include it. It just assumes the material file was actually tested in-game already.

Texture attributes

• $basetexture
• $bumpmap
• $normalmap
• $detail
• $dudvmap
• $refracttinttexture
• $envmapmask
• $refracttexture
• $reflecttexture
• $envmap
• $parallaxmap
• $ambientoccltexture
• $hdrcompressedtexture
• $lightwarptexture
• $phongexponenttexture

Material attributes

• $fallbackmaterial
• $underwateroverlay
• $bottommaterial
• include (For patch shader)

Hl1 vortigaunt alyx-healing effects. Noted, but not yet done.

• $FleshInteriorTexture
• $FleshNormalTexture
• $FleshBorderTexture1D
• $FleshInteriorNoiseTexture
• $FleshSubsurfaceTexture
• $FleshCubeTexture (envmap)