Idea for how to do dynamic baked-in lighting

[Pocket]

I recently watched a video of a presentation Mark Ferrari did at GDC last year. If you're not familiar, he's the bloke wot did these amazing 256-color animations. Turns out he also did a series of pictures that let you cycle through different times of the day by cycling through different 256-color palettes. And you can try those out here. His presentation explains better how he managed it.

Anyway, it got me to thinking: What if you could achieve the same sort of thing in a modern game, using special lightmaps? Obviously you'd need more than 256 colors even just for a lightmap, but what if your hypothetical next-gen lighting compiler could dynamically generate a palette of 256 "primary" colors, and then use a 24-bit colorspace where the first 8 bits are one palette index, the next 8 bits are another, and the third value is where on the spectrum between the two the actual texel's color falls.

Why would this be useful? Well, right now complex deferred lighting like VRAD produces is incompatible with proper day-night cycles. Presumably every engine that offers them relies exclusively on real-time lighting, which looks good for most cases but has limitations. You're stuck using generic, monochromatic, non-directional soft shadows and hoping it still looks good in conjunction with what your dynamic lights can do in 1/60th of a second. There's a reason Source 2 is still going to be making use of precomputed lighting. But imagine if you could have lighting outside that looks as good as CS:GO. No compromises, no need for fancy hardware; as far as the engine can tell, it is the same system as CS:GO, just with a different method of translating lightmap data into real colors. Your cascading shadow that crawls across the sky will still be doing the heavy lifting of convincing players that time is passing, but now everything else looks just a little more natural.

Granted, this is still only useful for natural light. I daresay all artificial lighting in your map would need its own separate lightmap because you could run out of color indexes real quick otherwise depending on how crazy you get with it. But that's not a big deal.

Of course, the operative question is, what kind of game could benefit from something like that? I couldn't tell you, honestly. The only games I can think of that actually use day-night cycles right now* are open world games, and those practically need real-time lighting because nobody wants to spend all that time compiling every chunk of a gigantic world nor wasting disk space on that many lightmaps.

*OK, technically you could say Dota 2 does, I guess? I still have no idea what the point of that is.

 

[Idolon]

Color cycling is effectively a cheaper way of doing certain types of animation. Instead of doing the manual labor of drawing all the individual frames and spending the memory on storing them, you just cycle some colors around. Hiding a cityscape in the same image as some trees is a neat trick, but it's important to note for your lightmap idea that nothing has been generated here - he drew the city, he drew the trees, and then overlaid the two and kept track of the colors.

One crucial problem with the lightmap idea is that I think it's a lot more like the second example than the first. Shifting the color palette around creates a time of day effect because its a flat image without any cast shadows. The artist is manually defining the colors of everything and how they fade between colors and what fades at what time. Mappers don't manually define any of that, VRAD does.

Plus, VRAD casts shadows - where shadow lands on the ground is an image, not just a color. if you want a shadow to actually crawl along, we're back to frame-by-frame animation. This means VRAD would need to do lighting calculations for every surface hit by sun for every single possible moment of the day. Yikes!

We can learn something from the city/forest example, in that we may be able to save on memory by hiding multiple images in the same image with some clever use of color space. For example, if we plot out a curve between the sun and ambient color, we can make a guess that all sun-affected luxels will be on that curve, and we could use just one color channel to describe where they are on that curve. In theory this gives us two channels for free (plus any extra if we want) to describe how that value animates over the course of a day cycle, but I have no idea how you would actually do that.

tl;dr: VRAD would suffer big time but memory/storage could be helped with some clever techniques

 

[fubarFX]

yeah I think you'd have to make separate lightmaps for dynamic and static lights and composite them back together within shader code. You could then manipulate the dynamic set of lightmaps in a global way. That's all very much within reach and would take too much time to hack into source. As Idolon points out tho, you're not going to be able to change angles so that puts day/night cycles out of the question. Because lights cannot move, the only practical application to this would be an eclipse or something like that. or maybe your sun changes color because... narrative reasons

 

[worMatty]

What if you changed the perceived length of shadows by making all lightmaps caused by directional environment lighting smooth gradients, with the same direction as the sun's movement arc, and changing their weighting depending on the time of day?

 

[Idolon]

What if you changed the perceived length of shadows by making all lightmaps caused by directional environment lighting smooth gradients, with the same direction as the sun's movement arc, and changing their weighting depending on the time of day?

The problem with that idea is that it relies on the animation for every luxel to be fairly regular, which in many cases it won't be. Here's an example:

As the sun passes, our luxel in this little pit will be start dark, be lit as the sun rises, become dark again as the object above blocks the sun, become bright again, and then darken as the sun sets. This requires at least four pieces of data to represent accurately. Once you put something like a garden trellis in your map (or even just a window with mullions), you could have twenty pieces of data as shadows casted by different pieces of geometry all hit the same luxels one at a time.

 

[Dr. Spud]

Putting direct lighting (shadows from the sun, spotlights, etc) into dynamic lightmaps isn't going to be practical. It's simpler and cheaper to have real-time lights.

For bounce lighting, this seems more plausible. If you sacrifice a lot of color-accuracy and can live with that, maybe you could blend between a handful of colors over the course of 24 hours. Blending between 2 seems too course, maybe 4-5. And then all the luxels share the same pallet. It's a neat idea.

For more accurate colors though, you're probably better-off computing new sets of lightmaps for your map and blending between them. There was a recent GDC talk about Mirror's Edge Catalyst that you can view here: http://www.gdcvault.com/play/1023284/Lighting-the-City-of-Glass

• All direct lighting is real-time.
• The bounce lighting is pre-computed in lightmaps. There are sets of lightmaps computed in 4 minute intervals over 24 hours.
• They can get away with low-res lightmaps because it's only bounce lighting, so the memory required doesn't get too crazy.
• The lightmaps are streamed-in, and the current lightmap is blended into the next lightmap as time-of-day advances.

 

[Pocket]

Plus, VRAD casts shadows - where shadow lands on the ground is an image, not just a color. if you want a shadow to actually crawl along, we're back to frame-by-frame animation. This means VRAD would need to do lighting calculations for every surface hit by sun for every single possible moment of the day. Yikes!

Dr. Spud had the right idea by suggesting that it could fade between different palettes, but the cascading shadows would do the heavy lifting of making it look like the sun is moving across the sky.

Having VRAD only commit bounced light, as in the Mirror's Edge example, is how I would have designed a hybrid system to work, and how I thought CS:GO did work, until I had it explained to me recently. The idea is, they needed something less detailed to fade to after you exceed the threshold for rendering the cascading shadows. Some engines just have the dynamic shadows disappear abruptly (Sonic 2006, most conspicuous in the Soleanna Forest area) while others either do that if you're on low settings or just keep going at the expense of performance if you're on high (Yooka-Laylee). I guess Valve designed their system to provide an ideal balance of looks and performance.