[Tutorial] REMOVING UGLY SHADOWS

  • Site Migration: See bugs? Report them here. Want something changed or have an idea? Suggest it here.

YM

LVL100 YM
aa
Dec 5, 2007
7,098
5,955
Tired of stupid shadows?

EDIT: rewritten - http://www.nodraw.net/2010/12/lighting-compile-options/

Ever had a prop that gives you damn ugly shadows that just don't look right, sort of like this fence here:

blocky_shadow.jpg


First off I should probably explain what is going on here. RAD uses collision meshes of props to calculate lighting, for the vast majority of props this is absolutely fine since their collision mesh closely matches its shape. But for some their collision mesh doesn't represent the shape of the opaque part of the model so the transparent parts appear to be fully opaque based upon the shadow. Some models don't have collision meshes like the mine track models which means RAD doesn't really know what to cast the shadow around and forms a rubbish little bounding box type thing, the result is a horrible shadow.

There are a few commands we can give to VRAD that make it use the static prop's outline for shadows and another that gives alpha textures the right transparency as well. But they come at a cost so I recommend you imagine their effects for all but the final few compiles of your map.

You're going to need to know which props you want to cast proper shadows of, the path and all, then open up GCFscape and extract the lights.rad file from the team fortress content GCF and put it in your /tf/ folder and rename it whatever you like (I called mine lights_youme.rad, inventive aren't I?)

Next open the rad file in your text editor of choice and add some lines at the bottom (or at the top, doesn’t really matter where)

Code:
forcetextureshadow models/props_gameplay/security_fence256.mdl
forcetextureshadow models/props_gameplay/security_fence128.mdl
Those two lines make VRAD cast proper shadows based on the alpha channel of any texture on the two fence models. If you find another model casting bad shadows that has an alpha channel texture add it into there. (if you don't want a load of warnings appearing in the console delete everything that was inside the rad file before adding your new stuff in, the duplication isnt needed but doesn't cause problems if you leave them in)

Now in hammer load up your map and hit run and hit the expert button at the bottom
expert-button.jpg


you'll see this:
Untitled-1.jpg


make a new compile command thing in the left hand box and paste the following into the parameters bit on the right

-lights lights_youme.rad -both -staticproppolys -staticproplighting -textureshadows -final -game $gamedir $path\$file

Then add $light_exe to the command box just above it.

make sure you check it's box (and move it up to just underneath the other $lights_exe compile command group and uncheck the original one's box so it doesn't run.

Then run your map and wait (the compile will now take a phenomenal amount of time and at the end you will end up with this:
textureshadowtest0000fn5.jpg


Although for that your lightmap scale has to be at 1 which as we all know is a bit OTT so leave it at 16 and it will look like the following, which compared with the original image is far more acceptable, don’t you think?

texture_shadow_test0001.jpg


Further explaination of -staticproplighting:

OK, so I've mocked up a little scene that might happen in any TF2 map, I've used the corrugated metal sheets because they illustrate what -staticproplighting does well because they have a high poly desnsity for their size as well as the fact they are a common problem for lighting and something people often get wrong when finding a fix.
My scene looks like this:
3170966392_0329fcf328_o.jpg



So here's what it looks like ingame, not very good is it?
3170965956_e55da70483_o.jpg

So why does it look so terrible?
well, The two metal sheets on the sides are both facing outwards and lighting is taken from their origin. These two things combined makes sure they're black in this instance. The front one's origin is right behind the wooden beam so the light is being blocked, making it dark. The back one's origin is facing away from the light source and since it's only a one sided prop it gets no light on it.

So what can we do about it?
Well the first thing to do is to make sure all the origins are facing the lightsource so that they get light falling on them. Here it is again with the far sheet rotated 180 so the origin is facing the other way (you can tell in hammer which way it's facing because there is a little yellow line once you select the prop, the direction of that line tells you which way it's facing)
3170965890_55156dc688_o.jpg


But you can see there that whilst we've fixed the back sheet the front sheet is still dark even though its origin is facing the light. We could use an info_lighting (info_target actually since info_lighting is broken) to 'move' it's origin so that it is in the light, but we can do one better than that.

In the first post there are three extra commands, -staticproppolys -staticproplighting and -textureshadows. The first and the last deal with taking shadows from the props exact outline rather than the collision mesh (great for props with no collision mesh like the payload tracks!) and taking shadows from props with alpha textures(great for chainlink fences). The middle one, -staticproplighting works out the lighting on props based on each vertex instead of the origin, so one side of a prop might be in light and lit nicely whilst the other side might have the light blocked to it so it will be dark.

Here is what it looks like with -staticproplighting on:
3170136301_a1cce75411_o.jpg

As you can see that looks much better because the metal sheets are now light where the light hits them and darker where the wood is blocking the light and casting shadows. Admittedly the resolution is a little low because the points at which lighting is done are fairly far apart and the dark patches are quite dark (that's because I don't have much ambient light in this scene) but I think you'll agree it looks much better than the first or second compile.

Here it is again showing the back of the front panel, it shows how the wood blocks the light from both sides of the sheet. and how the panel on the floor has a shadow across it in the same place the ground would if the metal sheet were not there.
3170966092_a276ee4568_o.jpg


This method will also help any props that have their origin embedded in geometry, since all the prop that is outside of the geometry will be lit properly - great for those large rocks which usually have their origin below a displacement.
 
Last edited:

Snipergen

L13: Stunning Member
Nov 16, 2007
1,052
150
Nicely done sir, about time somebody cleared that up.
I believe this is ep2 engine only yes?
 

roninmodern

L3: Member
Feb 5, 2008
124
5
if it is ep2 engine only, it won't affect those of us working on tf2 maps.
 

YM

LVL100 YM
aa
Dec 5, 2007
7,098
5,955
Nope I just did this in tf2 (see the screenshots?) It works.

if it was in EP2 I wouldnt have to mess about with the lights.rad, the ep2 props have something build in or perhaps the lights.rad is has the stuff in already and TF2 doesn't. for EP2 you can just stick in the commands and it works.
 

roninmodern

L3: Member
Feb 5, 2008
124
5
youme, he meant the ep2 version of source, which tf2 runs in. different from source in ep1, and thats different from source in hl2.
 

YM

LVL100 YM
aa
Dec 5, 2007
7,098
5,955
If you mean its only orange box stuff compatible then yes, I'm pretty sure you can't do this in older versions of the engine, but don't hold me to it. All I know is it works for TF2.
 

grazr

Old Man Mutant Ninja Turtle
aa
Mar 4, 2008
5,442
3,810
Seems brighter too, or did you increase the brightness?

Lighting/shading seems to have improved on the model as well. But maybe that's just my eyes/angle of the ss?

edit: How does this effect regular props? Does this change the lighting solution on all props? It sounds like it does, but if it was better than your standard RAD compile then i would have thought Valve would have put it in your compile options as defualt.

You believe/know this is how Valve render lights on a final compile?
 
Last edited:

YM

LVL100 YM
aa
Dec 5, 2007
7,098
5,955
Could just have been the fact I moved the props and light around inbetweenscreenshots, or it might actually be another benefit, I'm not entirely sure. (probably just the first though)
and its brighter because the wall behind has more light on it so is bouncing more.
 

Snipergen

L13: Stunning Member
Nov 16, 2007
1,052
150
I believe valve uses this in ep2 at some points in the maps to create dramatic shadows.
Would be brillant for a zombie map/left 4 dead too.
 

YM

LVL100 YM
aa
Dec 5, 2007
7,098
5,955
Yeah I don't doubt that they'll use plenty of this sort of stuff in L4D, I can't actually see why they wouldn't use this in everything now they've got it
(they used it in all the tf2 maps, the specific examples I was confused about before I looked this up was the track in goldrush and the fencing in badlands, both didnt have shadows disabled and yet both cast proper shadows)
 

marnamai

L1: Registered
Dec 7, 2007
48
2
Thanks a lot for this tutorial :wow:

This has caused many frustration for me in the past, I 'll certainly be using this nice feature in my map.
 

marnamai

L1: Registered
Dec 7, 2007
48
2
:cursing: Still getting fullbright maps after following this tutorial to the smallest points.

It seems as if vrad is completly skipped
Here is the compile log
Code:
** Executing...
** Command: "c:program filessteamsteamappsmaartenfrooninckxsourcesdkbinorangeboxbinvbsp.exe"
** Parameters: 

Valve Software - vbsp.exe (Mar 11 2008)
Command line: "c:program filessteamsteamappsmaartenfrooninckxsourcesdkbinorangeboxbinvbsp.exe" 

usage  : vbsp [options...] mapfile
example: vbsp -onlyents c:hl2hl2mapstest

Common options (use -v to see all options):

  -v (or -verbose): Turn on verbose output (also shows more command
                    line options).

  -onlyents   : This option causes vbsp only import the entities from the .vmf
                file. -onlyents won't reimport brush models.
  -onlyprops  : Only update the static props and detail props.
  -glview     : Writes .gl files in the current directory that can be viewed
                with glview.exe. If you use -tmpout, it will write the files
                into the tmp folder.
  -nodetail   : Get rid of all detail geometry. The geometry left over is
                what affects visibility.
  -nowater    : Get rid of water brushes.
  -low        : Run as an idle-priority process.

  -vproject <directory> : Override the VPROJECT environment variable.
  -game <directory>     : Same as -vproject.


** Executing...
** Command: "c:program filessteamsteamappsmaartenfrooninckxsourcesdkbinorangeboxbinvvis.exe"
** Parameters:  

Valve Software - vvis.exe (Mar 11 2008)
Command line: "c:program filessteamsteamappsmaartenfrooninckxsourcesdkbinorangeboxbinvvis.exe" 

usage  : vvis [options...] bspfile
example: vvis -fast c:hl2hl2mapstest

Common options:

  -v (or -verbose): Turn on verbose output (also shows more command
  -fast           : Only do first quick pass on vis calculations.
  -mpi            : Use VMPI to distribute computations.
  -low            : Run as an idle-priority process.
                    env_fog_controller specifies one.

  -vproject <directory> : Override the VPROJECT environment variable.
  -game <directory>     : Same as -vproject.

Other options:
  -novconfig      : Don't bring up graphical UI on vproject errors.
  -radius_override: Force a vis radius, regardless of whether an
  -mpi_pw <pw>    : Use a password to choose a specific set of VMPI workers.
  -threads        : Control the number of threads vbsp uses (defaults to the #
                    or processors on your machine).
  -nosort         : Don't sort portals (sorting is an optimization).
  -tmpin          : Make portals come from tmp<mapname>.
  -tmpout         : Make portals come from tmp<mapname>.
  -trace <start cluster> <end cluster> : Writes a linefile that traces the vis from one cluster to another for debugging map vis.
  -FullMinidumps  : Write large minidumps on crash.
  -x360		   : Generate Xbox360 version of vsp
  -nox360		   : Disable generation Xbox360 version of vsp (default)


** Executing...
** Command: "c:program filessteamsteamappsmaartenfrooninckxsourcesdkbinorangeboxbinvrad.exe"
** Parameters: -lights lights_port.rad -both -staticproppolys -staticproplighting -textureshadows -final -game

Valve Software - vrad.exe SSE (Mar 11 2008)

      Valve Radiosity Simulator     
Command line: "c:program filessteamsteamappsmaartenfrooninckxsourcesdkbinorangeboxbinvrad.exe" "-lights" "lights_port.rad" "-ldr" "-staticproppolys" "-staticproplighting" "-textureshadows" "-final" "-game" 

usage  : vrad [options...] bspfile
example: vrad c:hl2hl2mapstest

Common options:

  -v (or -verbose): Turn on verbose output (also shows more command
  -bounce #       : Set max number of bounces (default: 100).
  -fast           : Quick and dirty lighting.
  -fastambient    : Per-leaf ambient sampling is lower quality to save compute time.
  -final          : High quality processing. equivalent to -extrasky 16.
  -extrasky n     : trace N times as many rays for indirect light and sky ambient.
  -low            : Run as an idle-priority process.
  -mpi            : Use VMPI to distribute computations.
  -rederror       : Show errors in red.

  -vproject <directory> : Override the VPROJECT environment variable.
  -game <directory>     : Same as -vproject.

Other options:
  -novconfig      : Don't bring up graphical UI on vproject errors.
  -dump           : Write debugging .txt files.
  -dumpnormals    : Write normals to debug files.
  -dumptrace      : Write ray-tracing environment to debug files.
  -threads        : Control the number of threads vbsp uses (defaults to the #
                    or processors on your machine).
  -lights <file>  : Load a lights file in addition to lights.rad and the
                    level lights file.
  -noextra        : Disable supersampling.
  -debugextra     : Places debugging data in lightmaps to visualize
                    supersampling.
  -smooth #       : Set the threshold for smoothing groups, in degrees
                    (default 45).
  -dlightmap      : Force direct lighting into different lightmap than
                    radiosity.
  -stoponexit	   : Wait for a keypress on exit.
  -mpi_pw <pw>    : Use a password to choose a specific set of VMPI workers.
  -nodetaillight  : Don't light detail props.
  -centersamples  : Move sample centers.
  -luxeldensity # : Rescale all luxels by the specified amount (default: 1.0).
                    The number specified must be less than 1.0 or it will be
                    ignored.
  -loghash        : Log the sample hash table to samplehash.txt.
  -onlydetail     : Only light detail props and per-leaf lighting.
  -maxdispsamplesize #: Set max displacement sample size (default: 512).
  -softsun <n>    : Treat the sun as an area light source of size <n> degrees.                    Produces soft shadows.
                    Recommended values are between 0 and 5. Default is 0.
  -FullMinidumps  : Write large minidumps on crash.
  -chop           : Smallest number of luxel widths for a bounce patch, used on edges
  -maxchop		   : Coarsest allowed number of luxel widths for a patch, used in face interiors

  -LargeDispSampleRadius: This can be used if there are splotches of bounced light
                          on terrain. The compile will take longer, but it will gather
                          light across a wider area.
  -StaticPropLighting   : generate backed static prop vertex lighting
  -StaticPropPolys   : Perform shadow tests of static props at polygon precision
  -OnlyStaticProps   : Only perform direct static prop lighting (vrad debug option)
  -StaticPropNormals : when lighting static props, just show their normal vector
  -textureshadows : Allows texture alpha channels to block light - rays intersecting alpha surfaces will sample the texture
  -noskyboxrecurse : Turn off recursion into 3d skybox (skybox shadows on world)
  -nossprops      : Globally disable self-shadowing on static props


** Executing...
** Command: Copy File
** Parameters: "c:program filessteamsteamappsmaartenfrooninckxsourcesdk_contenttfmapsrcalphashadowtest.bsp" "c:program filessteamsteamappsmaartenfrooninckxteam fortress 2tfmapsalphashadowtest.bsp"


** Executing...
** Command: "c:program filessteamsteamappsmaartenfrooninckxteam fortress 2hl2.exe"
** Parameters: -dev -console -allowdebug -game "c:program filessteamsteamappsmaartenfrooninckxteam fortress 2tf" +map "alphashadowtest"
 

YM

LVL100 YM
aa
Dec 5, 2007
7,098
5,955
Just spent quite a while with marnamai trying to fix a load of problems, we managed to fix them thankfully and the tutorial has been ammended (I missed a bit when I was copying stuff over) so that no one else will run into the same mess.
 

efciem

L69: Deviant Member
Mar 31, 2008
69
6
Thanks for this bit of info,

I've been editing rad files for various self-lit texture stuff for a while, but this is a nice addition
 

YM

LVL100 YM
aa
Dec 5, 2007
7,098
5,955
Just spent quite a while with marnamai trying to fix a load of problems, we managed to fix them thankfully and the tutorial has been ammended (I missed a bit when I was copying stuff over) so that no one else will run into the same mess.

eh somehow my edits got removed, if they do again this is what I edited:
Code:
-lights lights_youme.rad -both -staticproppolys -staticproplighting -textureshadows -final -game $gamedir $path\$file
(notice how its got $gamedir $path\$file at the end)
 

Ida

deer
aa
Jan 6, 2008
2,289
1,372
Wow, this tutorial is great! I have a feeling I'm really going to need this when I start prettying up things.
 
Dec 25, 2007
567
438
Next open the rad file in your text editor of choice and add some lines at the bottom (or at the top, doesn’t really matter where)

When going through this, I gound that your own lights-whatever.rad should start entirely blank, and not as a copy of the original lights.rad. As Youme said, it won't break if you do copy it, you'll just get a lot of warnings about duplicate lights.

Code:
forcetextureshadow models/props_gameplay/security_fence256.mdl
forcetextureshadow models/props_gameplay/security_fence128.mdl
Those two lines make VRAD cast proper shadows based on the alpha channel of any texture on the two fence models. If you find another model casting bad shadows that has an alpha channel texture add it into there.

I went through every tf2 model, and identified all (I believe) the models that use alpha textures that you'd want these rules for, so here's a complete custom lights-tf2.rad:

Code:
forcetextureshadow models/props_gameplay/security_fence_big01.mdl
forcetextureshadow models/props_gameplay/security_fence_big02.mdl
forcetextureshadow models/props_gameplay/security_fence512.mdl
forcetextureshadow models/props_gameplay/security_fence512_lower.mdl
forcetextureshadow models/props_gameplay/security_fence512_skybox.mdl
forcetextureshadow models/props_gameplay/security_fence256.mdl
forcetextureshadow models/props_gameplay/security_fence32.mdl
forcetextureshadow models/props_gameplay/security_fence64.mdl
forcetextureshadow models/props_gameplay/security_fence64_lower.mdl
forcetextureshadow models/props_gameplay/security_fence80.mdl
forcetextureshadow models/props_gameplay/security_fence_section01.mdl
forcetextureshadow models/props_gameplay/security_fence128.mdl
forcetextureshadow models/props_gameplay/security_fence128_lower.mdl
forcetextureshadow models/props_gameplay/security_fence256_gate01.mdl
forcetextureshadow models/props_gameplay/security_fence256_gate02.mdl
forcetextureshadow models/props_gameplay/security_fence256_lower.mdl
forcetextureshadow models/props_farm/haypile001.mdl
forcetextureshadow models/props_farm/tree001.mdl
forcetextureshadow models/props_farm/tree001_skybox.mdl
forcetextureshadow models/props_gameplay/door_grate001_bottom.mdl
forcetextureshadow models/props_gameplay/door_grate001_top.mdl
forcetextureshadow models/props_gameplay/door_grate002_bottom.mdl
forcetextureshadow models/props_gameplay/door_grate002_top.mdl
forcetextureshadow models/props_gameplay/door_grate003_bottom.mdl
forcetextureshadow models/props_gameplay/door_grate003_top.mdl
forcetextureshadow models/props_foliage/bramble001a.mdl 
forcetextureshadow models/props_foliage/bramble001a_skybox.mdl 
forcetextureshadow models/props_foliage/corn_plant01.mdl
forcetextureshadow models/props_foliage/shrub_01a.mdl
forcetextureshadow models/props_foliage/shrub_02a.mdl
forcetextureshadow models/props_foliage/shrub_03_card.mdl
forcetextureshadow models/props_foliage/shrub_03_card_skybox.mdl
forcetextureshadow models/props_foliage/shrub_03_cluster.mdl
forcetextureshadow models/props_foliage/shrub_03_cluster02.mdl
forcetextureshadow models/props_foliage/shrub_03_cluster_skybox.mdl
forcetextureshadow models/props_foliage/shrub_03a.mdl
forcetextureshadow models/props_foliage/shrub_03b.mdl
forcetextureshadow models/props_foliage/shrub_03c.mdl
forcetextureshadow models/props_foliage/shrub_04a.mdl
forcetextureshadow models/props_foliage/shrub_04b.mdl
forcetextureshadow models/props_foliage/tree01.mdl
forcetextureshadow models/props_foliage/tree02.mdl
forcetextureshadow models/props_foliage/tree_pine01.mdl
forcetextureshadow models/props_foliage/tree_pine01_4cluster.mdl
forcetextureshadow models/props_foliage/tree_pine01_8cluster.mdl
forcetextureshadow models/props_foliage/tree_pine01_8cluster.mdl
forcetextureshadow models/props_forest/shrub_03_cluster.mdl
forcetextureshadow models/props_forest/shrub_03b.mdl
forcetextureshadow models/props_forest/shrub_03c.mdl

If you come across any I've missed, let me know and I'll add them here.