A Source engine hack I've always wanted to see

[Pocket]

The recent thread about Black Mesa and its various community-made engine upgrades reminded me of something I've always been curious if Source could do, given enough coding skill and access to the source code: Seamless map transitions.

Yes, yes, I know, Source is like the only engine on the planet that still stores its level data as discrete maps, yadda yadda... Don't care. My idea is, the engine would somehow (and this is the part where I'm the fuzziest on the hows or even ifs) be modded to hold two maps in memory at a time, and swap out the one the player isn't currently in once they reach a certain point. Then the maps could be linked using a pair of brush entities similar to world portals. To keep memory use from doubling, the maps themselves would be smaller — a manageable option since you would no longer have to keep the map count low to minimize loading screens.

Portal 2 is an example of a game that could be adapted to this system well, since it's linear and has points-of-no-return between every map. What I'd picture would mostly involve splitting the elevators off from the test chambers as separate maps, and triggering the transitions at the exact moment the doors close behind you. And then maybe tweaking which ones make you wait for debris to clear out of the tubes before the elevator arrives depending on which maps take longer to load. Obviously it's a system that would have worked better if it were working before they finished writing the game.

Anyone out there know enough about game engine tech to know if something like this would ever be feasible?

 

[Idolon]

I'm a little sketchy on how Source actually works under the hood, but here's my rough understanding.

It would mean rewriting map loading to be threaded, which may or may not be difficult. I don't actually know. It would also need to do so in a manner that doesn't noticeably slow down the game at runtime, which again, may or may not be difficult.

You would also probably be fighting with Source's memory limits. I believe the engine is more or less built to only load into memory what can possibly happen in a map, like materials and particles. Most likely you would need to somehow make the engine associate loaded content with a specific map so that packed content (cubemaps!) can't possibly conflict, but then you run into the issue of storing two maps' worth of data in memory, etc. The ideal solution probably involves figuring out what resources are already loaded from the existing map, not loading them again, and flagging them to make sure they don't get dropped when a map unloads.

There's also the issue of relative coordinates. Level transitions in Source usually* work by using a landmark entity that marks the same relative spot between two maps so that it can teleport entities between maps into the same relative positions. To make a seamless transition work, you would need to be able to load a map in at a relative position rather than the actual coordinates stored in the .bsp (OR you would need to build every map at it's "actual" position, which would be really dumb). I have no idea how much it would take to do this.

All of that aside, the real determining factor is going to come from how many things in the engine depend or don't depend on one another. Basically, code is sometimes written under the assumption that something will always be a certain way, like "only one map will be loaded at a given time." Those assumptions make it a lot faster and easier to get work done, and I imagine this is an assumption that exists within Source since it's based on code from eons ago when seamless loading probably would be dumb to try (memory limits). For all I know it would be relatively easy to make seamless loading work, but I'm gonna guess that it is actually really difficult.

PS: For the Portal 2 example, the elevators already do an infinite loop to let dialogue finish. That could be pretty easily repurposed into a faux-loading screen (a pretty common technique in seamless loading games that would need to exist anyways since PC games will have variable loading times).

*Portal 2 doesn't bother with landmarks. All of its level transitions are elevators or small rooms.

 

[Pocket]

I'd say the strongest piece of evidence that this can't be done is that in the 12 years Source has been around, no one has ever done it. Still, I'd be fascinated to hear exactly why not. Considering Valve themselves have been unsuccessful at even porting the engine to 64 bit, for some reason, there might be a lot of really weird, random limitations that aren't common knowledge.

There's also the issue of relative coordinates. Level transitions in Source usually* work by using a landmark entity that marks the same relative spot between two maps so that it can teleport entities between maps into the same relative positions. To make a seamless transition work, you would need to be able to load a map in at a relative position rather than the actual coordinates stored in the .bsp (OR you would need to build every map at it's "actual" position, which would be really dumb). I have no idea how much it would take to do this.

Presumably the use of world portals instead of teleporting would make that a moot point (and also just be a better way to do it in general since even with loading screens, transition-by-teleport can be noticeable via changes in lighting, HDR refreshing, physics objects spazzing out etc.)

 

[Kube]

I propose a solution.

In all seriousness, I hope something like you say is possible. Though I'm sure Source 2 already has this feature.

 

[fubarFX]

there's also the issue of entities of different maps conflicting with each other. But don't get me wrong, it's all really within the realm of possibility. it's more a question of is it worth doing?

I'd say the strongest piece of evidence that this can't be done is that in the 12 years Source has been around, no one has ever done it.

This argument is ass and I don't don't like hearing such things. Not seeing something being done in 12 years proves nothing. no one had figured out automated asset packing from reading bsps until recently, and it wasn't even that hard tbh. If the community has not managed to do something in 12 years, it's simply because they don't have the knowledge or resources to make it happen. Just about anything is possible you guys, it's just ones and zeros

 

[Pocket]

there's also the issue of entities of different maps conflicting with each other. But don't get me wrong, it's all really within the realm of possibility. it's more a question of is it worth doing?

The death of mid-game loading screens is its own reward, especially if you're working on a project as high profile as Black Mesa and the loading times are bordering on excruciating (which they are).

 

[fubarFX]

well, loading screens certainly didn't stop the game from selling and I doubt it would have sold more if they had found a way to remove them. that's kinda what I'm referring to when I ask if it's worth doing. sure it would make the game better but the game has no obligation to be the best it can be in order to be successful. At the end of the day, video games are a business and if a feature is not essential, you probably have better things to focus on. But say it's essential to your game, it can definitely be done. however, the question then becomes "would be more cost efficient to do it in an other engine?"

 

[Osspial]

As hard as getting full seamless loading would be, I wonder what kind of optimizations would be possible that stop just short of actually having two fully functional maps loaded and running at the same time. It would probably be easier (although still not necessarily easy) to begin reading map/model data off of the disk into RAM, unprocessed, before the actual map transition starts to help negate the cost of disk read speeds. However, even if that could be accomplished there's still the issue of 32-bit RAM limitations. However, that may or may not be an issue depending on how much memory is used by the game when running normally, which would vary wildly depending on the game.

 

[Vel0city]

Maybe @Fr0Z3nR can tell us more.

Not that loading times are a problem for me, really. If you have a recent system with a half-decent SSD, loading times really shouldn't be that long. I have the pleasure of having a PCIe-based SSD doing 800MB/sec reads, the longest I need to wait in games like Half Life 2 or Black Mesa is like a second or two at most.

I think the biggest roadblock one will have if he/she would decide to implement this would be Source's 32-bit nature. 3GB of total addressable memory seems enough for two maps to be loaded in RAM at once and enable seamless map transitions but that's also counting VRAM (if I'm not mistaken) and so you're suddenly left with maybe 1.7-2GB of actual addressable memory to store maps in RAM. Subtract from that things like AI code, sound, physics data, the whole lot and you're left with even less RAM to store maps in.

 

[Pocket]

I did say the maps would be cut down in size; does anyone know if current Source games come close to maxing out addressable memory as is?

 

[Werewolf]

Well, here's hoping this is something they bring to Source2, and games like HL3 and L4D3 make use of it.

 

[Fruity Snacks]

Maybe @Fr0Z3nR can tell us more.

Not that loading times are a problem for me, really. If you have a recent system with a half-decent SSD, loading times really shouldn't be that long. I have the pleasure of having a PCIe-based SSD doing 800MB/sec reads, the longest I need to wait in games like Half Life 2 or Black Mesa is like a second or two at most.

I think the biggest roadblock one will have if he/she would decide to implement this would be Source's 32-bit nature. 3GB of total addressable memory seems enough for two maps to be loaded in RAM at once and enable seamless map transitions but that's also counting VRAM (if I'm not mistaken) and so you're suddenly left with maybe 1.7-2GB of actual addressable memory to store maps in RAM. Subtract from that things like AI code, sound, physics data, the whole lot and you're left with even less RAM to store maps in.

I probably can't without you being specific. I was an MP designer for the most part, didn't do much with SP.

I did say the maps would be cut down in size; does anyone know if current Source games come close to maxing out addressable memory as is?

Why cut down the size when you can just double the ent and displacement limits?

Also, on topic. Seamless map transitions in Source 1.

HA. Ha ha. Ha. haaaaaaa

 

[Vel0city]

I probably can't without you being specific. I was an MP designer for the most part, didn't do much with SP.

Well, more about how Source could be able to hold two whole maps in memory and enable a no-map-loading-screens Source game or if that's not possible but I think the

Also, on topic. Seamless map transitions in Source 1.

HA. Ha ha. Ha. haaaaaaa

part kinda explained pretty well that it probably can't be done on Source 1.

 

[Idolon]

Portal 2 is actually a nice example for memory limits for two reasons: One, it was built to run on consoles. Two, Portal Stories: Mel tries to cram as much as it possibly can into every .bsp, with 3 or 4 test chambers per map, which says to me that Portal 2 as-is could probably hold multiple chambers in memory at the same time.

Presumably the use of world portals instead of teleporting would make that a moot point (and also just be a better way to do it in general since even with loading screens, transition-by-teleport can be noticeable via changes in lighting, HDR refreshing, physics objects spazzing out etc.)

World portals are a total hack and can only really be used in situations with really controlled lighting or doorways since lighting doesn't pass through them. In Portal 2's commentary they talk about how they used world portals a lot to stitch the game together in development, but the released game only uses them once. (I don't actually know where.)

That also raises the question of how level transition seams would even happen, since lighting would either somehow need to pass between levels (doable... probably) or transitions would always have to happen in the same cases where world portals would be acceptable. I guess that's just the price you pay for baked lighting.

 

[Kube]

Portal 2 is actually a nice example for memory limits for two reasons: One, it was built to run on consoles. Two, Portal Stories: Mel tries to cram as much as it possibly can into every .bsp, with 3 or 4 test chambers per map, which says to me that Portal 2 as-is could probably hold multiple chambers in memory at the same time.


World portals are a total hack and can only really be used in situations with really controlled lighting or doorways since lighting doesn't pass through them. In Portal 2's commentary they talk about how they used world portals a lot to stitch the game together in development, but the released game only uses them once. (I don't actually know where.)

That also raises the question of how level transition seams would even happen, since lighting would either somehow need to pass between levels (doable... probably) or transitions would always have to happen in the same cases where world portals would be acceptable. I guess that's just the price you pay for baked lighting.

For reference, this is the place with the world portal. It helps that you never really get a good look through the portal, as it's covered with a grate.

 

[Pocket]

World portals are a total hack and can only really be used in situations with really controlled lighting or doorways since lighting doesn't pass through them.

This should be easy enough to deal with, provided they only appear at spots where the floor texture changes. Ever play The Stanley Parable? That game is full of world portals you'd never spot if it weren't obvious that the layouts are nonlinear. All you need to do is have near-identical lighting coming from the (inaccessible) other side of the brush. Consider the overlapping areas Valve puts at the edges of maps in Half-Life games to ensure you can see what's ahead of you just before you hit a teleport trigger. That would easily be enough as long as there's something to mask the slight jump between the two lightmaps, like only putting them in doorframes and using a separate texture for the threshold (like that metal strip TF2 is so fond of).

 

[Tacoman_]

Anyone out there know enough about game engine tech to know if something like this would ever be feasible?

Look no further than the antique-source-engine-like-frankenstein-patchwork of an engine called gamebryo/creation. More specifically the version used in Fallout 4.

There are few elevators that will allow you to replace the loading screen with a calm elevator ride. You can even interact with your companions and possibly manage your inventory while the elevator is moving. (I have never tried checking my inventory, as that "pauses" the game and I don't want to risk it)
So its totally possible to accomplish this assuming the programmers are capable.

 

[Pocket]

Update: I finally broke down and asked the Black Mesa team about this directly. I only got one response, but this is what it was:

This was something our programmers investigated extensively back when we were considering VR as a potential project. It's literally impossible without rewriting the entire rendering code. And rewriting the entire rendering code amounts to basically rewriting the whole engine. Which amounts to a whole waste of time for very little payoff (why not just use a modern engine at that point?)