[Tutorial] REMOVING UGLY SHADOWS

Discussion in 'Tutorials & Resources' started by YM, Jul 28, 2008.

  1. YM

    aa YM LVL100 YM

    Messages:
    7,099
    Positive Ratings:
    5,744
    Tired of stupid shadows?

    EDIT: rewritten - http://www.nodraw.net/2010/12/lighting-compile-options/

    Ever had a prop that gives you damn ugly shadows that just don't look right, sort of like this fence here:

    [​IMG]

    First off I should probably explain what is going on here. RAD uses collision meshes of props to calculate lighting, for the vast majority of props this is absolutely fine since their collision mesh closely matches its shape. But for some their collision mesh doesn't represent the shape of the opaque part of the model so the transparent parts appear to be fully opaque based upon the shadow. Some models don't have collision meshes like the mine track models which means RAD doesn't really know what to cast the shadow around and forms a rubbish little bounding box type thing, the result is a horrible shadow.

    There are a few commands we can give to VRAD that make it use the static prop's outline for shadows and another that gives alpha textures the right transparency as well. But they come at a cost so I recommend you imagine their effects for all but the final few compiles of your map.

    You're going to need to know which props you want to cast proper shadows of, the path and all, then open up GCFscape and extract the lights.rad file from the team fortress content GCF and put it in your /tf/ folder and rename it whatever you like (I called mine lights_youme.rad, inventive aren't I?)

    Next open the rad file in your text editor of choice and add some lines at the bottom (or at the top, doesn’t really matter where)

    Code:
    forcetextureshadow models/props_gameplay/security_fence256.mdl
    forcetextureshadow models/props_gameplay/security_fence128.mdl
    Those two lines make VRAD cast proper shadows based on the alpha channel of any texture on the two fence models. If you find another model casting bad shadows that has an alpha channel texture add it into there. (if you don't want a load of warnings appearing in the console delete everything that was inside the rad file before adding your new stuff in, the duplication isnt needed but doesn't cause problems if you leave them in)

    Now in hammer load up your map and hit run and hit the expert button at the bottom
    [​IMG]

    you'll see this:
    [​IMG]

    make a new compile command thing in the left hand box and paste the following into the parameters bit on the right

    -lights lights_youme.rad -both -staticproppolys -staticproplighting -textureshadows -final -game $gamedir $path\$file

    Then add $light_exe to the command box just above it.

    make sure you check it's box (and move it up to just underneath the other $lights_exe compile command group and uncheck the original one's box so it doesn't run.

    Then run your map and wait (the compile will now take a phenomenal amount of time and at the end you will end up with this:
    [​IMG]

    Although for that your lightmap scale has to be at 1 which as we all know is a bit OTT so leave it at 16 and it will look like the following, which compared with the original image is far more acceptable, don’t you think?

    [​IMG]

    Further explaination of -staticproplighting:

    OK, so I've mocked up a little scene that might happen in any TF2 map, I've used the corrugated metal sheets because they illustrate what -staticproplighting does well because they have a high poly desnsity for their size as well as the fact they are a common problem for lighting and something people often get wrong when finding a fix.
    My scene looks like this:
    [​IMG]


    So here's what it looks like ingame, not very good is it?
    [​IMG]
    So why does it look so terrible?
    well, The two metal sheets on the sides are both facing outwards and lighting is taken from their origin. These two things combined makes sure they're black in this instance. The front one's origin is right behind the wooden beam so the light is being blocked, making it dark. The back one's origin is facing away from the light source and since it's only a one sided prop it gets no light on it.

    So what can we do about it?
    Well the first thing to do is to make sure all the origins are facing the lightsource so that they get light falling on them. Here it is again with the far sheet rotated 180 so the origin is facing the other way (you can tell in hammer which way it's facing because there is a little yellow line once you select the prop, the direction of that line tells you which way it's facing)
    [​IMG]

    But you can see there that whilst we've fixed the back sheet the front sheet is still dark even though its origin is facing the light. We could use an info_lighting (info_target actually since info_lighting is broken) to 'move' it's origin so that it is in the light, but we can do one better than that.

    In the first post there are three extra commands, -staticproppolys -staticproplighting and -textureshadows. The first and the last deal with taking shadows from the props exact outline rather than the collision mesh (great for props with no collision mesh like the payload tracks!) and taking shadows from props with alpha textures(great for chainlink fences). The middle one, -staticproplighting works out the lighting on props based on each vertex instead of the origin, so one side of a prop might be in light and lit nicely whilst the other side might have the light blocked to it so it will be dark.

    Here is what it looks like with -staticproplighting on:
    [​IMG]
    As you can see that looks much better because the metal sheets are now light where the light hits them and darker where the wood is blocking the light and casting shadows. Admittedly the resolution is a little low because the points at which lighting is done are fairly far apart and the dark patches are quite dark (that's because I don't have much ambient light in this scene) but I think you'll agree it looks much better than the first or second compile.

    Here it is again showing the back of the front panel, it shows how the wood blocks the light from both sides of the sheet. and how the panel on the floor has a shadow across it in the same place the ground would if the metal sheet were not there.
    [​IMG]

    This method will also help any props that have their origin embedded in geometry, since all the prop that is outside of the geometry will be lit properly - great for those large rocks which usually have their origin below a displacement.
     
    • Thanks Thanks x 83
    Last edited: Dec 2, 2010
  2. Snipergen

    Snipergen L13: Stunning Member

    Messages:
    1,053
    Positive Ratings:
    150
    Nicely done sir, about time somebody cleared that up.
    I believe this is ep2 engine only yes?
     
  3. roninmodern

    roninmodern L3: Member

    Messages:
    124
    Positive Ratings:
    5
    if it is ep2 engine only, it won't affect those of us working on tf2 maps.
     
  4. YM

    aa YM LVL100 YM

    Messages:
    7,099
    Positive Ratings:
    5,744
    Nope I just did this in tf2 (see the screenshots?) It works.

    if it was in EP2 I wouldnt have to mess about with the lights.rad, the ep2 props have something build in or perhaps the lights.rad is has the stuff in already and TF2 doesn't. for EP2 you can just stick in the commands and it works.
     
    • Thanks Thanks x 1
  5. Snipergen

    Snipergen L13: Stunning Member

    Messages:
    1,053
    Positive Ratings:
    150
    Yes but.. I still want to know it ^^
     
  6. roninmodern

    roninmodern L3: Member

    Messages:
    124
    Positive Ratings:
    5
    youme, he meant the ep2 version of source, which tf2 runs in. different from source in ep1, and thats different from source in hl2.
     
  7. YM

    aa YM LVL100 YM

    Messages:
    7,099
    Positive Ratings:
    5,744
    If you mean its only orange box stuff compatible then yes, I'm pretty sure you can't do this in older versions of the engine, but don't hold me to it. All I know is it works for TF2.
     
  8. grazr

    aa grazr Old Man Mutant Ninja Turtle

    Messages:
    5,436
    Positive Ratings:
    3,570
    Seems brighter too, or did you increase the brightness?

    Lighting/shading seems to have improved on the model as well. But maybe that's just my eyes/angle of the ss?

    edit: How does this effect regular props? Does this change the lighting solution on all props? It sounds like it does, but if it was better than your standard RAD compile then i would have thought Valve would have put it in your compile options as defualt.

    You believe/know this is how Valve render lights on a final compile?
     
    Last edited: Jul 28, 2008
  9. YM

    aa YM LVL100 YM

    Messages:
    7,099
    Positive Ratings:
    5,744
    Could just have been the fact I moved the props and light around inbetweenscreenshots, or it might actually be another benefit, I'm not entirely sure. (probably just the first though)
    and its brighter because the wall behind has more light on it so is bouncing more.
     
  10. Snipergen

    Snipergen L13: Stunning Member

    Messages:
    1,053
    Positive Ratings:
    150
    I believe valve uses this in ep2 at some points in the maps to create dramatic shadows.
    Would be brillant for a zombie map/left 4 dead too.
     
  11. YM

    aa YM LVL100 YM

    Messages:
    7,099
    Positive Ratings:
    5,744
    Yeah I don't doubt that they'll use plenty of this sort of stuff in L4D, I can't actually see why they wouldn't use this in everything now they've got it
    (they used it in all the tf2 maps, the specific examples I was confused about before I looked this up was the track in goldrush and the fencing in badlands, both didnt have shadows disabled and yet both cast proper shadows)
     
  12. marnamai

    marnamai L1: Registered

    Messages:
    48
    Positive Ratings:
    2
    Thanks a lot for this tutorial :wow:

    This has caused many frustration for me in the past, I 'll certainly be using this nice feature in my map.
     
  13. YM

    aa YM LVL100 YM

    Messages:
    7,099
    Positive Ratings:
    5,744
    Just remember the amount this adds to compile time is incredible and it uses loads more CPU power too so make sure you only do it for pretty much final compile only.
     
  14. marnamai

    marnamai L1: Registered

    Messages:
    48
    Positive Ratings:
    2
    :cursing: Still getting fullbright maps after following this tutorial to the smallest points.

    It seems as if vrad is completly skipped
    Here is the compile log
    Code:
    ** Executing...
    ** Command: "c:program filessteamsteamappsmaartenfrooninckxsourcesdkbinorangeboxbinvbsp.exe"
    ** Parameters: 
    
    Valve Software - vbsp.exe (Mar 11 2008)
    Command line: "c:program filessteamsteamappsmaartenfrooninckxsourcesdkbinorangeboxbinvbsp.exe" 
    
    usage  : vbsp [options...] mapfile
    example: vbsp -onlyents c:hl2hl2mapstest
    
    Common options (use -v to see all options):
    
      -v (or -verbose): Turn on verbose output (also shows more command
                        line options).
    
      -onlyents   : This option causes vbsp only import the entities from the .vmf
                    file. -onlyents won't reimport brush models.
      -onlyprops  : Only update the static props and detail props.
      -glview     : Writes .gl files in the current directory that can be viewed
                    with glview.exe. If you use -tmpout, it will write the files
                    into the tmp folder.
      -nodetail   : Get rid of all detail geometry. The geometry left over is
                    what affects visibility.
      -nowater    : Get rid of water brushes.
      -low        : Run as an idle-priority process.
    
      -vproject <directory> : Override the VPROJECT environment variable.
      -game <directory>     : Same as -vproject.
    
    
    ** Executing...
    ** Command: "c:program filessteamsteamappsmaartenfrooninckxsourcesdkbinorangeboxbinvvis.exe"
    ** Parameters:  
    
    Valve Software - vvis.exe (Mar 11 2008)
    Command line: "c:program filessteamsteamappsmaartenfrooninckxsourcesdkbinorangeboxbinvvis.exe" 
    
    usage  : vvis [options...] bspfile
    example: vvis -fast c:hl2hl2mapstest
    
    Common options:
    
      -v (or -verbose): Turn on verbose output (also shows more command
      -fast           : Only do first quick pass on vis calculations.
      -mpi            : Use VMPI to distribute computations.
      -low            : Run as an idle-priority process.
                        env_fog_controller specifies one.
    
      -vproject <directory> : Override the VPROJECT environment variable.
      -game <directory>     : Same as -vproject.
    
    Other options:
      -novconfig      : Don't bring up graphical UI on vproject errors.
      -radius_override: Force a vis radius, regardless of whether an
      -mpi_pw <pw>    : Use a password to choose a specific set of VMPI workers.
      -threads        : Control the number of threads vbsp uses (defaults to the #
                        or processors on your machine).
      -nosort         : Don't sort portals (sorting is an optimization).
      -tmpin          : Make portals come from tmp<mapname>.
      -tmpout         : Make portals come from tmp<mapname>.
      -trace <start cluster> <end cluster> : Writes a linefile that traces the vis from one cluster to another for debugging map vis.
      -FullMinidumps  : Write large minidumps on crash.
      -x360		   : Generate Xbox360 version of vsp
      -nox360		   : Disable generation Xbox360 version of vsp (default)
    
    
    ** Executing...
    ** Command: "c:program filessteamsteamappsmaartenfrooninckxsourcesdkbinorangeboxbinvrad.exe"
    ** Parameters: -lights lights_port.rad -both -staticproppolys -staticproplighting -textureshadows -final -game
    
    Valve Software - vrad.exe SSE (Mar 11 2008)
    
          Valve Radiosity Simulator     
    Command line: "c:program filessteamsteamappsmaartenfrooninckxsourcesdkbinorangeboxbinvrad.exe" "-lights" "lights_port.rad" "-ldr" "-staticproppolys" "-staticproplighting" "-textureshadows" "-final" "-game" 
    
    usage  : vrad [options...] bspfile
    example: vrad c:hl2hl2mapstest
    
    Common options:
    
      -v (or -verbose): Turn on verbose output (also shows more command
      -bounce #       : Set max number of bounces (default: 100).
      -fast           : Quick and dirty lighting.
      -fastambient    : Per-leaf ambient sampling is lower quality to save compute time.
      -final          : High quality processing. equivalent to -extrasky 16.
      -extrasky n     : trace N times as many rays for indirect light and sky ambient.
      -low            : Run as an idle-priority process.
      -mpi            : Use VMPI to distribute computations.
      -rederror       : Show errors in red.
    
      -vproject <directory> : Override the VPROJECT environment variable.
      -game <directory>     : Same as -vproject.
    
    Other options:
      -novconfig      : Don't bring up graphical UI on vproject errors.
      -dump           : Write debugging .txt files.
      -dumpnormals    : Write normals to debug files.
      -dumptrace      : Write ray-tracing environment to debug files.
      -threads        : Control the number of threads vbsp uses (defaults to the #
                        or processors on your machine).
      -lights <file>  : Load a lights file in addition to lights.rad and the
                        level lights file.
      -noextra        : Disable supersampling.
      -debugextra     : Places debugging data in lightmaps to visualize
                        supersampling.
      -smooth #       : Set the threshold for smoothing groups, in degrees
                        (default 45).
      -dlightmap      : Force direct lighting into different lightmap than
                        radiosity.
      -stoponexit	   : Wait for a keypress on exit.
      -mpi_pw <pw>    : Use a password to choose a specific set of VMPI workers.
      -nodetaillight  : Don't light detail props.
      -centersamples  : Move sample centers.
      -luxeldensity # : Rescale all luxels by the specified amount (default: 1.0).
                        The number specified must be less than 1.0 or it will be
                        ignored.
      -loghash        : Log the sample hash table to samplehash.txt.
      -onlydetail     : Only light detail props and per-leaf lighting.
      -maxdispsamplesize #: Set max displacement sample size (default: 512).
      -softsun <n>    : Treat the sun as an area light source of size <n> degrees.                    Produces soft shadows.
                        Recommended values are between 0 and 5. Default is 0.
      -FullMinidumps  : Write large minidumps on crash.
      -chop           : Smallest number of luxel widths for a bounce patch, used on edges
      -maxchop		   : Coarsest allowed number of luxel widths for a patch, used in face interiors
    
      -LargeDispSampleRadius: This can be used if there are splotches of bounced light
                              on terrain. The compile will take longer, but it will gather
                              light across a wider area.
      -StaticPropLighting   : generate backed static prop vertex lighting
      -StaticPropPolys   : Perform shadow tests of static props at polygon precision
      -OnlyStaticProps   : Only perform direct static prop lighting (vrad debug option)
      -StaticPropNormals : when lighting static props, just show their normal vector
      -textureshadows : Allows texture alpha channels to block light - rays intersecting alpha surfaces will sample the texture
      -noskyboxrecurse : Turn off recursion into 3d skybox (skybox shadows on world)
      -nossprops      : Globally disable self-shadowing on static props
    
    
    ** Executing...
    ** Command: Copy File
    ** Parameters: "c:program filessteamsteamappsmaartenfrooninckxsourcesdk_contenttfmapsrcalphashadowtest.bsp" "c:program filessteamsteamappsmaartenfrooninckxteam fortress 2tfmapsalphashadowtest.bsp"
    
    
    ** Executing...
    ** Command: "c:program filessteamsteamappsmaartenfrooninckxteam fortress 2hl2.exe"
    ** Parameters: -dev -console -allowdebug -game "c:program filessteamsteamappsmaartenfrooninckxteam fortress 2tf" +map "alphashadowtest"
     
    • Thanks Thanks x 1
  15. YM

    aa YM LVL100 YM

    Messages:
    7,099
    Positive Ratings:
    5,744
    what are you doing to get it to print out all those commands?
     
  16. YM

    aa YM LVL100 YM

    Messages:
    7,099
    Positive Ratings:
    5,744
    Just spent quite a while with marnamai trying to fix a load of problems, we managed to fix them thankfully and the tutorial has been ammended (I missed a bit when I was copying stuff over) so that no one else will run into the same mess.
     
  17. efciem

    efciem L69: Deviant Member

    Messages:
    69
    Positive Ratings:
    6
    Thanks for this bit of info,

    I've been editing rad files for various self-lit texture stuff for a while, but this is a nice addition
     
  18. YM

    aa YM LVL100 YM

    Messages:
    7,099
    Positive Ratings:
    5,744
    eh somehow my edits got removed, if they do again this is what I edited:
    Code:
    -lights lights_youme.rad -both -staticproppolys -staticproplighting -textureshadows -final -game $gamedir $path\$file 
    (notice how its got $gamedir $path\$file at the end)
     
  19. luxatile

    aa luxatile deer

    Messages:
    2,289
    Positive Ratings:
    1,349
    Wow, this tutorial is great! I have a feeling I'm really going to need this when I start prettying up things.
     
  20. VelvetFistIronGlove

    aa VelvetFistIronGlove

    Messages:
    567
    Positive Ratings:
    437
    When going through this, I gound that your own lights-whatever.rad should start entirely blank, and not as a copy of the original lights.rad. As Youme said, it won't break if you do copy it, you'll just get a lot of warnings about duplicate lights.

    I went through every tf2 model, and identified all (I believe) the models that use alpha textures that you'd want these rules for, so here's a complete custom lights-tf2.rad:

    Code:
    forcetextureshadow models/props_gameplay/security_fence_big01.mdl
    forcetextureshadow models/props_gameplay/security_fence_big02.mdl
    forcetextureshadow models/props_gameplay/security_fence512.mdl
    forcetextureshadow models/props_gameplay/security_fence512_lower.mdl
    forcetextureshadow models/props_gameplay/security_fence512_skybox.mdl
    forcetextureshadow models/props_gameplay/security_fence256.mdl
    forcetextureshadow models/props_gameplay/security_fence32.mdl
    forcetextureshadow models/props_gameplay/security_fence64.mdl
    forcetextureshadow models/props_gameplay/security_fence64_lower.mdl
    forcetextureshadow models/props_gameplay/security_fence80.mdl
    forcetextureshadow models/props_gameplay/security_fence_section01.mdl
    forcetextureshadow models/props_gameplay/security_fence128.mdl
    forcetextureshadow models/props_gameplay/security_fence128_lower.mdl
    forcetextureshadow models/props_gameplay/security_fence256_gate01.mdl
    forcetextureshadow models/props_gameplay/security_fence256_gate02.mdl
    forcetextureshadow models/props_gameplay/security_fence256_lower.mdl
    forcetextureshadow models/props_farm/haypile001.mdl
    forcetextureshadow models/props_farm/tree001.mdl
    forcetextureshadow models/props_farm/tree001_skybox.mdl
    forcetextureshadow models/props_gameplay/door_grate001_bottom.mdl
    forcetextureshadow models/props_gameplay/door_grate001_top.mdl
    forcetextureshadow models/props_gameplay/door_grate002_bottom.mdl
    forcetextureshadow models/props_gameplay/door_grate002_top.mdl
    forcetextureshadow models/props_gameplay/door_grate003_bottom.mdl
    forcetextureshadow models/props_gameplay/door_grate003_top.mdl
    forcetextureshadow models/props_foliage/bramble001a.mdl 
    forcetextureshadow models/props_foliage/bramble001a_skybox.mdl 
    forcetextureshadow models/props_foliage/corn_plant01.mdl
    forcetextureshadow models/props_foliage/shrub_01a.mdl
    forcetextureshadow models/props_foliage/shrub_02a.mdl
    forcetextureshadow models/props_foliage/shrub_03_card.mdl
    forcetextureshadow models/props_foliage/shrub_03_card_skybox.mdl
    forcetextureshadow models/props_foliage/shrub_03_cluster.mdl
    forcetextureshadow models/props_foliage/shrub_03_cluster02.mdl
    forcetextureshadow models/props_foliage/shrub_03_cluster_skybox.mdl
    forcetextureshadow models/props_foliage/shrub_03a.mdl
    forcetextureshadow models/props_foliage/shrub_03b.mdl
    forcetextureshadow models/props_foliage/shrub_03c.mdl
    forcetextureshadow models/props_foliage/shrub_04a.mdl
    forcetextureshadow models/props_foliage/shrub_04b.mdl
    forcetextureshadow models/props_foliage/tree01.mdl
    forcetextureshadow models/props_foliage/tree02.mdl
    forcetextureshadow models/props_foliage/tree_pine01.mdl
    forcetextureshadow models/props_foliage/tree_pine01_4cluster.mdl
    forcetextureshadow models/props_foliage/tree_pine01_8cluster.mdl
    forcetextureshadow models/props_foliage/tree_pine01_8cluster.mdl
    forcetextureshadow models/props_forest/shrub_03_cluster.mdl
    forcetextureshadow models/props_forest/shrub_03b.mdl
    forcetextureshadow models/props_forest/shrub_03c.mdl
    
    If you come across any I've missed, let me know and I'll add them here.
     
    • Thanks Thanks x 11