New Python libraries: VMF-tools and FGD-tools

Discussion in 'Mapping Questions & Discussion' started by fubarFX, May 27, 2019.

  1. fubarFX

    aa fubarFX The "raw" in "nodraw"

    Messages:
    1,692
    Positive Ratings:
    1,884
    VMF-tools and FGD-tools
    Yep, here goes big daddy Fubar again...

    I've recently started working on some python libraries to help in handling source engine maps programmatically. Those are VMF-tools and FGD-tools.
    I've laid out a lot of the ground work but I'm far from done. This is my progress.

    Current Projects

    VMF-tools: A general purpose python library that enables you to parse, edit and write .VMF files.

    Project Link:
    Done:
    • Reading and writing .VMF files.
    Currently:
    • Adding functions to perform mirror symmetry on VMFs.
    • Leveraging Fgd-tools to identify entity properties to be geometrically modified.
    Todo:
    • Add functions to perform rotations on VMFs.
    • Add functions to mass replace textures and model skins.
    • Add functions to mass replace models.


    Finished projects:

    FGD-tools: A python library that can parse .fgd files so we have a schema for entities found in vmfs. FGD-tools will eventually assist VMF-tools when it comes to manipulating entities. They are seperate projects to keep things clean and self contained.

    Project Documentation:

    Codebase:
    Future projects:


    VMFlip: A Windows application leveraging VMF-tools and FGD-tools to automatically mirror TF2 maps by mirroring both brushes and entities. On top of doing the geometric transformations, it will also perform texture replacements, model replacements (and/or skin replacements)

    ...More snazzy stuff: to be revealed later​


    Why this thread?
    • To inform people in case they would be interested in contributing.
    • To inform people of new resources that are becoming available in case they'd like to create tools of their own in the future.
    • To discuss other useful features that should be added to these libraries

    Isn't most of this redundant with existing Hammer functionalities?

    Yeah! But that's exactly the point. Hammer's great, however, we can vastly improve our workflow by doing more stuff outside of the level editor. For that we need to replicate features.


    Want to contribute?

    Awesome! There's a lot of work to be done to pull this off. The good news is that not all of it involves programming; you don't necessarily need to know how to code to be a part of this. Get in contact with me and let's get that ball rolling!


    The ultimate goal is to make it easier for people to create better tools for source and eventually open up new possibilities for mappers.
     
    • Thanks Thanks x 6
    • Like Like x 2
    Last edited: Jul 27, 2019
  2. henke37

    aa henke37

    Messages:
    2,044
    Positive Ratings:
    490
    This is cool. Too bad that I despise python.
     
    • Respectfully Disagree Respectfully Disagree x 3
  3. fubarFX

    aa fubarFX The "raw" in "nodraw"

    Messages:
    1,692
    Positive Ratings:
    1,884
    Python is great for handling text files. It's a very good choice for vmfs and all the other text based stuff. aaaand python is also a lot more accessible for people who might be primarily level designers. When dealing with binary files like bsp tho, I'll admit python is a pretty poor choice. for that I lean towards c#.
     
  4. Zeus3005

    aa Zeus3005 Not a Krusty Krab

    Messages:
    134
    Positive Ratings:
    103
    Oooo this is super cool!

    I really like the idea of doing the mirroring / rotating, is there a way that could be integrated directly into hammer or maybe like compilepal?

    What things are you looking for contributions on? I've been a software engineer for 10 years so; id be happy to make some contributions

    Edit: Maybe a feature idea; something that looks for optmization suggestions. Applies nodraw to brushes with adjacent vertices, or even better if it could predict which things would be good as func details
     
  5. Idolon

    aa Idolon the worst admin

    Messages:
    1,886
    Positive Ratings:
    5,674
    If you want a precedent for integrating custom tools into Hammer, you might be interested in this auto-radar tool for CSGO. It inserts a new configuration into the expert compile window that runs the custom .exe. A similar process for a VMF flipper would be pretty great.
     
  6. Zeus3005

    aa Zeus3005 Not a Krusty Krab

    Messages:
    134
    Positive Ratings:
    103
    At a glance that seems incredibly easy to do; all we would need to do is compile our python into an exe and figure out what arguments the compiler would pass it.

    Though I think it would be way cooler as a 'plugin' for hammer itself, so you can just run mirroring directly from hammer
     
  7. Idolon

    aa Idolon the worst admin

    Messages:
    1,886
    Positive Ratings:
    5,674
    As far as I know, Hammer has no plugin support beyond directly editing the .exe.
     
  8. fubarFX

    aa fubarFX The "raw" in "nodraw"

    Messages:
    1,692
    Positive Ratings:
    1,884
    @Zeus3005
    Any standalone .exe can be added to either hammer's expert compiler or compilepal. So that's the goal yeah.

    And as far as contributions go, well... There's really a lot of stuff.

    for non-programmers:
    -We're going to need to catalog each prop and their respective axis of symmetry as well as their skin replacement (from red to blu).
    -We're also going to suggest common texture replacements.
    (the replacements will remain configurable so you still have control but it would be nice to have a default config that takes care of most things)

    for programmers:
    Well gee, where to start. There's really a ton of stuff to be done but specifically, there's many math/geometry things to be done. Those are small self-contained features that are easy to contribute. So far I only have basic brush and entity mirroring done.

    I'm planning to keep things pretty basic for now but if we want to get thorough, we'll need to be able to perform geometric operations based on any arbitrary point/plane, not just coordinate 0,0,0.

    You can find lists of things to be done for each libraries here:
    https://trello.com/b/u1TIWT2w/vmf-tools
    https://trello.com/b/BvHg6Dk8/fgd-tools
     
  9. fubarFX

    aa fubarFX The "raw" in "nodraw"

    Messages:
    1,692
    Positive Ratings:
    1,884
    Progress Report:

    FGD-tools is complete!
    if you are looking to create a Python application/tool that could use entity schemas normally provided by .fgd files, FGD-tools could be just the thing you need.

    The library is now available on pypi and can be installed with the command
    Code:
    pip install fgdtools
    View the full documentation here: https://maxdup.github.io/fgd-tools/

    VMF-tools is resuming development!
    With Fgd-tools out of the way, I'll now be able to identify entity properties that need to be geometrically modified.
     
    • Thanks Thanks x 4
    Last edited: Jul 27, 2019
  10. fubarFX

    aa fubarFX The "raw" in "nodraw"

    Messages:
    1,692
    Positive Ratings:
    1,884
    FGD-tools has updated!
    Its gone through a couple iterations over the last month and it's now officially done. It is feature complete, fully tested, stable and supported by all sensible versions of python. I'm really happy with how the project turned out and not only will I not have to deal with fgd files anymore, I also have a very good project template for the other future python libraries.

    Other future python libraries!
    Vmf-tools is next on the agenda but, since I also have to document every prop and their axis of symmetry before we can actually do vmf mirroring and such, I'm likely also going to have to make a "mdl-tools" to look at vertex data and establish symmetry (probably using statistics). So there we go, it's a huge project but it's moving forward. In the near future we have vmf-tools and mdl-tools but it's not going to be enough, there's many more puzzle pieces to come.
    "mdl-tools" is inevitable, "vmt-tools" is likely, "bsp-tools" is likely. (A cool side effect of this is that building a full level editor that can replace Hammer is going to become increasingly realistic. But don't get your hopes up, I'm not about to do that)

    Again, if people are interested to contribute, fgd-tools has laid down a pretty good workflow and now it's just a matter of replicating that to other areas of source, slowly chipping away at something impossibly large.
     
    • Thanks Thanks x 1
  11. Zeus3005

    aa Zeus3005 Not a Krusty Krab

    Messages:
    134
    Positive Ratings:
    103
    Do you have an example of what format you want this to be in? Like how do we find that axis of symmerty, and other metadata we need about models; then how do you want to capture that? Which props are in scope for this; I have a library of ~4gb's of custom models, so this could be an enormous amount of work; but we could start with props that ship with the game.

    Or even better; is there a method of reading the model files and finding that axis programically? (I don't know much about the format of source assets). If we could even get like 30% of models automatically then that would save us an enormous amount of time
     
  12. Zeus3005

    aa Zeus3005 Not a Krusty Krab

    Messages:
    134
    Positive Ratings:
    103
  13. fubarFX

    aa fubarFX The "raw" in "nodraw"

    Messages:
    1,692
    Positive Ratings:
    1,884
    Yeah, realizing how much work that would be. I pretty much came to the conclusion that it would have to be automated. I don't have a format yet to store that information but I'm thinking it would likely be a csv file. I'm hoping to make a library that can read mdl files and automatically establish symmetry through statistics, that way we also have a solution for all models and not just currently known ones. What I can't do automatically do tho is detecting mirrored versions of models. I can scan for for the _mirrored suffix in file names but I don't know how consistent this naming convention is across the entire library of props.

    So yeah, finding the best axis of symmetry for a given model could be automated. Model replacements however could require some amount of manual labor.


    Side note about sketchup: yes, someone could do that using vmf-tools once it's completed to convert anything into a map. I have no interest in pursuing this myself tho but this is precisely the point: it's about enabling other people to do crazy shit that would too large of an overtaking otherwise.
     
  14. Zeus3005

    aa Zeus3005 Not a Krusty Krab

    Messages:
    134
    Positive Ratings:
    103
    i'd prob recommend something like JSON, just so we could extend the functionality/add other metadata and not require positional data.

    We may not even have to do that though because if we can find it programatically then we don't need to capture it, just add that code to the library, unless that would have a high disk IO overhead or something