KotH Heatmap

Discussion in 'Map Factory' started by RubbishyUser, Oct 26, 2014.

  1. RubbishyUser

    RubbishyUser L7: Fancy Member

    Messages:
    414
    Positive Ratings:
    242
    This is an experimental map, designed to test the actual design strength of the heatmaps.tf tool.

    So this is fruition of a discussion I had with Frozen on this thread. The discussion was about whether heatmaps compiled by Geit's heatmaps.tf actually made valid evidence when deciding what direction to take a map. Make no mistake: people do use the heatmap tool to evaluate their map and it is very useful for compiling data on sniper or engineer locations. My fear was that mappers could examine the heatmaps whichever way they liked and come to the conclusion that the heatmap supported the decision they already favoured: effectively a form a confirmation bias. For example: a red splodge on a heatmap indicates both a region which stalemates and is too chokepointy, or a valid hold location that should be reinforced with health and ammo, depending on what the mapper already believed by the time the heatmap was available.

    This is made worse by the fact that I do not know of a mapper who hasn't played his own map before studying the data: if the round went poorly, he's already formed strong opinions long before the accompanying data is available, and if by then he's tested several times, then he's able to reinforce his belief and will expect the heatmap to agree with him.

    The end result is that heatmaps become effectively useless except in very large data samples where you are unable to attend or watch every match and miss a large proportion of the "sniper/engineer spots".

    My testing method: prove that heatmaps can accurately describe gameplay without interference from personal bias by iterating on a map based on heatmap data alone: I will not attend testing sessions; I will not read in-game feedback and I won't even do bot testing.

    I plan to iterate a handful of times before taking the blindfold off and examining my handiwork first hand. I'm also getting a friend to test the map technically without spoiling the experience, but please let me know if there are any technical issues as long as you do not reveal the effect they had on the game. I plan to submit each version for playtesting at least 3 times over the course of a month or so and will start examining heatmaps with more than 1000 deaths. I'll try and mix it up between US and EU as well.

    TL;DR: Please do not leave feedback on the play as they will be considered spoilers and ruin the experiment: either post your opinions using the spoiler tag (knowing that I won't read them until much later) or wait until I declare the end.

    Also, I guess I should apologize for probably inflicting a terrible map on you guys, forcing you to play it a bunch, and then not even fix the issues for the next version. :cool1:
     
    Last edited: Oct 28, 2014
  2. wareya

    wareya L7: Fancy Member

    Messages:
    493
    Positive Ratings:
    173
    Fight that good fight.
     
    • Thanks Thanks x 1
  3. Trotim

    aa Trotim

    Messages:
    1,180
    Positive Ratings:
    997
    Does this include making changes to one side of the map only, breaking symmetry when the heatmap tells you to?
     
    • Thanks Thanks x 1
  4. Sergis

    aa Sergis L666: ])oo]v[

    Messages:
    1,871
    Positive Ratings:
    1,131
    but its koth, a standart viaduct scheme koth at that

    you might go rc now and itd be half decent

    make something that actually needs srs testing data
     
  5. TyeZenneth

    TyeZenneth L6: Sharp Member

    Messages:
    340
    Positive Ratings:
    279
    Please let this happen.
     
  6. RubbishyUser

    RubbishyUser L7: Fancy Member

    Messages:
    414
    Positive Ratings:
    242
    Hmmm maybe. Since the map is symmetrical atm, any imbalance from sides would clearly be random chance so trying to design based on that would be foolish. If I really don't learn anything from this excercise, I might move onto something more challenging like A/D.

    And Sergis, even saying the map wasn't terrible is totally a spoiler. :) That's how strict I'd like to be on this matter. Also, not every koth map is good or even likely to be good. I know you're probably tired of seeing koth maps on gamedays, and I'm sure the reason is that we tell everyone to make a koth map their first - as they should, it's by far the easiest gamemode to design for.

    Four your information, this is why I chose to make a koth map:
    • Smaller map means more focused death count which reduces noise and makes heatmap more accurate. Having it symmetric even means I can effectively double my sample size by looking at whether something occurred on the other side.
    • On that note, a singular objective means all 1000 of my deaths relate to that objective - making my job easier and increasing accuracy of the heatmaps.
    • Making koth IS quick - I have other things to map, not just pump out a weird experiment thing.
    • Koth IS easier to design - that means that I'm less likely to subject you guys to a terrible map multiple times while your complaints fall on deaf ears. I'm looking out for you guys!
     
    Last edited: Oct 30, 2014
  7. takabuschik

    aa takabuschik

    Messages:
    663
    Positive Ratings:
    316
    tc_heatmap now!
     
  8. HQDefault

    aa HQDefault ...what

    Messages:
    1,058
    Positive Ratings:
    437
    Ironically, this map looks like fun :3
     
    • Thanks Thanks x 1
  9. RubbishyUser

    RubbishyUser L7: Fancy Member

    Messages:
    414
    Positive Ratings:
    242
    Uninformed praise is allowed.
     
  10. Zed

    aa Zed Certified Most Crunk™

    Messages:
    1,249
    Positive Ratings:
    1,015
    Looks too much like a map.
     
  11. Sergis

    aa Sergis L666: ])oo]v[

    Messages:
    1,871
    Positive Ratings:
    1,131
    i havent played it :p
     
    • Thanks Thanks x 1
  12. RubbishyUser

    RubbishyUser L7: Fancy Member

    Messages:
    414
    Positive Ratings:
    242
    I have now run a few tests with koth_heatmap_a1 and, with 1047 deaths visible on heatmaps.tf, I think it's time to move to the next version. The way I plan to do this is to write a quick post-mortem of the previous version before I look at the heatmap, then a first impressions and further examination of the heatmap as I design the next release. I'm hoping to get some of my friends, who have played the map during the game day, to write up their own opinions WITHOUT seeing what I had already written, suggesting their own avenues for further design. The plan is to hide all of these behind spoiler tags so you guys can look for your own amusement, or you can play along and keep yourself in the dark if you want to. Let me know if you also want to only look at the heatmap and avoid testing and other spoilers. For those interested, the code is (spoiler) and (/spoiler), but with square brackets.

    koth_heatmap_a1: Pre-Analysis

    Let's get right to it. I entirely improvised the layout over the map in the 4 or so hours it took for me to make it. I suspect this probably led to some poor design choices that I'm hoping to root out by examining the heatmaps. So I'm sorry if you guys had to put up with unfairly one sided rounds, and then had to put up with it again as I did nothing to improve the matter. Mentally, I picture this map as very similar to Process middle, using a similar industrial theme and introducing the two teams in opposite corners with strong flanking paths either side. The point is loosely inspired by Viaduct, of all maps, offering a small height advantage to the team on the point. To counteract this, Viaduct offers strong alternative routes to the point, punishing teams that push too far down one avenue - instead, I decided to add the landmark houses either side of the bridge.

    I feel like these houses could be one of the most controversial parts of the map: in order to stop the map growing too large and cutting off the flank, the house couldn't have a large base, and yet it still needed to provide enough cover and elevation to look down onto the point and provide a platform of attack. The compromise is that the ramp that allows you to get onto the roof is rather thin and steep, promoting spam from explosive classes, especially given the vertical nature. I also decided to add a small hole in the roof, allowing heavies to stick their heads out and shoot onto the point, or snipers to take potshots from a decent position. Although the holes are different shapes because I needed to match up to the differing size of the tiles, both allow players to crouch their way into the building or jump out onto the roof, and are roughly the same size.

    I'd like to note here that recently, both Valve and custom mappers have shied away from bunker-like features in Team Fortress 2 in general, which is interesting. Both Dustbowl and Goldrush feature rooms pressed into the ground with small window slits to shoot out of, and gone are the days of 128 unit wide doorways or hallways to anywhere. Some maps, like the aforementioned Process, tend to go no thinner than 256 all around, even on the flanks. Now while I am in favor of widening doorways I'm not convinced these "bunkers" have to go with them - I've always felt I hated Dustbowl and Goldrush for other reasons.

    I am fully aware that it is almost impossible to get these shipping containers into these buildings. I don't know why I did that. It seems like a bad idea.

    I've always had a soft spot for Granary and it's middle point but I fear the double stacked containers were a bad idea, as it allows Soldiers to get a very powerful angle on the three main entry doors. I put those containers there to block the sightline to the two middle doors from the point, and vice versa, to prevent demomen spamming the point from the safety of the inside. First thing I will look at on the heatmap will be Killer>Soldier locations for sure.

    I placed the two healthkits on the flanks as a nod to Viaduct's straightforward health and ammo placement and to encourage players to take those hopefully safer but longer routes. An additional medium healthkit went underneath the point to encourage classes to flank and dodge round the containers and buildings as opposed to staying above.

    The spawns are simple. Two doors out, three doors to pick what side of the point you want to enter on. I added a shutter door to the middle to prevent a potential sightline from near middle back to the spawns, and added barrels and other clutter to cut down the sightline from the flanks and to add geometry to play on, similar to the back field in Viaduct.

    Thank you to everyone who has tested the map so far.
     
    Last edited: Dec 1, 2014
  13. RubbishyUser

    RubbishyUser L7: Fancy Member

    Messages:
    414
    Positive Ratings:
    242
    Now for the heatmap analysis for koth_heatmap_a1. The heatmap can be found here: http://heatmaps.tf/koth_heatmap_a1. Note that I had not looked at the heatmap at all until it broke 1000 deaths and that these are my initial impressions.


    First impressions for koth_heatmap_a1:
    Thank goodness the majority of the deaths occur in the point area and not inside the houses or at the spawns. That suggests that at no point did one team completely roll. There are a couple exceptions: the turbine room and, for one team, the connector behind the house.

    On another note, I see that the heatmap is not as symmetrical as the map itself is. That could indicate one of two things: that there is still a large variance in the heatmap caused by the performance of the teams, on the basis that the sample size wasn't large enough, or that my map has a large "variance" of deaths. There is little that I can do about the former - I don't want to waste any more of people's time, I want to keep death counts consistent across all versions, and I don't want to do additional testing after I have already examined the heatmap.

    If the problem is the latter though, then I have a bigger issue. What I mean by a larger death variance is that if a team is stronger than average, they cause a larger number of deaths and send the other team further back to spawn. This is symptomatic of entire games where a single team has taken control and dictated the course of the round.

    At the top of the map, you can see where what was hopefully a single round of unfair teams resulted in far more victims closer to the spawn. That may be another indicator that not enough rounds were played, and as a result, outliers like this are clearly present in the data. Or at least, I hope it was an outlier.

    Another point that I noticed as I was getting this map tested: this stuff takes serious time in order to get a halfway decent heatmap. According to the upload, I started this in the last week of October and I'm now writing this post on the first week of December, having submitted the map to three gamedays. Admittedly, I could have forced more playtesting at a faster rate, but at once a week-ish most people playing in the TF2 maps group that regular play will have seen this map once or twice (based on personal experience of average time I play in Gamedays per month.) Yet I only got 1000 deaths out of it: I would frankly like to double that to really be certain with what the heatmap was telling me. That would mean coming out with a new version every couple of months - really too slow at this early stage of development - or testing twice as frequently, completely driving people up the wall. I fear the first thing we have to chalk up against heatmaps.tf is that it only works when the map in question is "played to death" - if you pardon my pun.

    And finally, a little psychological phenomenon. After a few rounds of testing I decided to show my TF2-savvy housemates the heatmaps - without looking, of course, or letting them tell me about the results. That didn't stop my brain working in overtime to try and glean some information about the map based on their reaction - even though I knew it was bad practice. I really started seeing hints that weren't there - that brief conversation with Sergis above is an example. Once I'd found a "hint" I immediately subconsciously set about establishing that information inside my world view about the map - even though I had no world view of the map. I think this is probably a great example of how humans have such strong selection biases that they can misread or reinterpret information based on their existing beliefs.
     
  14. RubbishyUser

    RubbishyUser L7: Fancy Member

    Messages:
    414
    Positive Ratings:
    242
    So having had a couple of days to go over the heatmap, I have come to one conclusion: I cannot continue without more deaths. When 1000 deaths are visible, the heatmap is filled out rather nicely, but applying a single filter - where I can actually extract information about sniper locations and so on - cuts down the number of deaths to such a degree that the information is frankly no longer reliable. In it's current state, the heatmap would have me believe that soldiers only stand on one of the shipping containers when both sides are equally effective. This means I'm left with no choice but to continue testing - apologies to those who are tired of it. I will do a more detailed analysis when I reach, say, 2000 deaths.