Heatmaps

Discussion in 'Site Discussion' started by Geit, Jul 10, 2014.

  1. Geit

    aa Geit 💜 I probably broke it 💜

    Messages:
    597
    Positive Ratings:
    1,114
    Hi all,

    This a thread for my attempt at a heat mapping application for map developers. If you've been in chat over the past few weeks, you'll have already seen this in action a few times, this thread is for collecting feedback and ideas for further development.

    What's a heat map?

    A heat map is a visualization for allowing a designer to infer patterns from various kinds of spatial data according to the relative density of the data. A higher density results in a "hotter" map.

    So, how's it progressing?
    Currently, I have a working demo available at http://heatmaps.tf if you'd like to have a quick play. We're currently collecting data from all TF2Maps servers and another server host that's allowed us to collect data from them. This application is very much WIP and is in a development environment, so it may not be functioning at all at points.

    Great, but I want the raw data!
    Raw data is available using the API documented at http://heatmaps.tf/api.html

    ---


    Notice
    This is the new thread for Geit's attempt at Heatmaps: the old thread, for InstantMuffin's attempt, is here: http://forums.tf2maps.net/showthread.php?t=22563
     
    • Thanks Thanks x 4
    Last edited: Aug 5, 2014
  2. Geit

    aa Geit 💜 I probably broke it 💜

    Messages:
    597
    Positive Ratings:
    1,114
    So I've been working on this a fair bit over the last few days, I've sorted out a very early WIP page here: http://geit.co.uk:62923 (may not always be up, or working correctly, and it doesn't work on smaller screens).

    I need to rewrite parts of what I have (they were written on very little sleep, and are very hacky) then work on the filters implementation. I'm sort of away for the weekend, so may not work on it till next week some time, but it shouldn't take too long after that.
     
    • Thanks Thanks x 4
    Last edited: Jul 19, 2014
  3. InstantMuffin

    InstantMuffin L2: Junior Member

    Messages:
    64
    Positive Ratings:
    48
    That's one way to put it. Precisely I disassociated myself from you and frozen, not from the community. I created a group which people could join to see how much interest there actually is, and with 6 people in it and different stuff I'm working on, it is not my priority. If people want it, they can still have it from me.
     
  4. Sergis

    aa Sergis L666: ])oo]v[

    Messages:
    1,871
    Positive Ratings:
    1,131
    thanks but we now have a sun that does not give light to the moon assuming the moon is going to owe it one
     
    • Thanks Thanks x 1
  5. Bakscratch

    aa Bakscratch Finisher of Maps

    Messages:
    683
    Positive Ratings:
    1,175
    But in the end It doesn't even matter I had to fall to lose it all but in the end It doesn't even matter
     
    • Thanks Thanks x 3
  6. TyeZenneth

    TyeZenneth L6: Sharp Member

    Messages:
    340
    Positive Ratings:
    278
    Except I haven't even heard about this group until now, as I'm sure many people haven't.
     
  7. InstantMuffin

    InstantMuffin L2: Junior Member

    Messages:
    64
    Positive Ratings:
    48
  8. Crash

    aa Crash func_nerd

    Messages:
    3,182
    Positive Ratings:
    4,878
    Oh man that looks amazing. Is it all maps tested right now showing up on there or does it have to be "turned on"?
     
  9. Another Bad Pun

    Server Staff Another Bad Pun over my dead body

    Messages:
    585
    Positive Ratings:
    1,212
    Jeez, this is really neat. So you've been stalking play-tests this whole time?
    Layl pointed out in chat that the engi party on koth_amazing is visible in the heatmaps:
    [​IMG]
    Edit: I really really wish you could zoom into the map somehow
     
    Last edited: Jul 19, 2014
  10. Teddy Schnapps

    Teddy Schnapps L2: Junior Member

    Messages:
    51
    Positive Ratings:
    22
    It would be very nice to have the separation of self-suicide deaths and regular gameplay deaths. I think that would be more helpful then just any death willy-nilly.

    Also I would personally like to see in real time the players moving from spawn throughout the map. This will give me a better understanding of how players react to their environment and how confused they are with the layout.

    P.S. Hope this thing comes out soon! I really, really need it, bad!

    Love
    Teddy
     
    • Thanks Thanks x 2
  11. Geit

    aa Geit 💜 I probably broke it 💜

    Messages:
    597
    Positive Ratings:
    1,114
    Data is collected automatically - I have a daemon that collects the overview images that I currently have to run manually before the data will appear - the /data/ interface is open at all times if you want to download your kill data.

    Map zooming is planned - the code for it is already written down in my notes, it's a low-priority at the moment though.

    Self Suicide is completely ignored by the data logger (they give no value, imo). There are plans to record player movements, my main gripe at the moment is recording that data without destroying the server's FPS and bandwidth. I'd need to store the player's individual movement points and then run a line simplification algorithm... tl;dr It's planned but will not be a first-release feature.
     
    • Thanks Thanks x 1
    Last edited: Aug 29, 2015
  12. wareya

    wareya L7: Fancy Member

    Messages:
    493
    Positive Ratings:
    173
    Save their position every second or half second, collect it into deltas, save to disk on a thread when they die or periodically? Can you even do threading?
     
  13. InstantMuffin

    InstantMuffin L2: Junior Member

    Messages:
    64
    Positive Ratings:
    48
    You're actually over-thinking this.
    I'll see what I can do once I'm available again.
     
  14. Geit

    aa Geit 💜 I probably broke it 💜

    Messages:
    597
    Positive Ratings:
    1,114
    If you wanted a suitable resolution in the data you'd need to collect positions, as wareya said, every few seconds at most: let's say every 2 seconds. Now let's say the server is full and there are 24 people online. A coordinate trio takes 12 bytes of network/storage at least, so that's 144 bytes per second before network overhead.

    Assuming a 30 minute map, that's 0.25 MB per map, per playthrough - In terms of data storage that's nothing: easy peasy to store and process - but now let's say a map accumulates 24 hours of playtime with 24 players: that's 12MB of data that now has to be delivered to the client before they can begin plotting the data, or in real terms: around 16 seconds before the client can begin rendering the heatmap. (at 6mbit and assuming no other delays)

    Now if you actually processed your data with a line simplification algorithm such as https://hydra.hull.ac.uk/resources/hull:8338 - pre or post storage - you can reduce the data size by up to 80% while still maintaining virtually the same resolution, which means faster rendering, less storage space and benefits all around.

    Personally, I'd rather overthink it. Wasting some server cycles is a pittance compared to getting the heatmaps delivered fast.

    Also, I'm hardly one to talk but: don't make promises you can't fulfill.
     
  15. InstantMuffin

    InstantMuffin L2: Junior Member

    Messages:
    64
    Positive Ratings:
    48
    EDIT:
    nvm, you're right.

    I'm not making any promisses here. I have no obligations. I came into this by looking for something worthwhile and interesting to do and asking Frozen how I could help mappers. And he brought up heatmaps, so I went for it.
    Now while things happened between us two and Frozen another project conveniently came up which is high priority for me now as I can reach/help more people with it and I like to work with the people involved. If I'm available again and I think there's a better way of helping mappers analyze their data (compared to what you do) I wouldn't decline doing them a favor and provide them with an alternative.
    If you personally don't want my help (I wouldnt be surprised after what and how you said it back then), feel free to ignore it. Remember, I am not doing this for you, frozen, or the heads of tf2maps. I like to help people who bust their asses trying to accomplish something I recognize as art. If I can help them with a bit of science, why not go for it?
     
    Last edited: Jul 21, 2014
  16. wareya

    wareya L7: Fancy Member

    Messages:
    493
    Positive Ratings:
    173
    TF2 has a speed limit (per axis?), so if anything suddenly goes significantly faster than that for a single frame (teleports) you can just make a new line.

    The problem is really that there is just SO MUCH FUCKING DATA. You would need to filter it dramatically for the client: Only paths before mid, only paths carrying intel, only paths up to 10 seconds out of <given spawn>, etc.
     
  17. henke37

    aa henke37

    Messages:
    1,910
    Positive Ratings:
    446
    The trick then would be to load data on demand. Tag each path to allow quick filtering and then let the server do the filtering when sending the data. Index the data for extra fast retrieval. Partion the data in separate files for extra quick filtering on very common criteria like time.
     
  18. Geit

    aa Geit 💜 I probably broke it 💜

    Messages:
    597
    Positive Ratings:
    1,114
    So while I haven't got much work done over the past week because I've been at a friend's house, I did manage to pick up http://heatmaps.tf/ and move the development site to it. Yey :3
     
  19. YM

    aa YM LVL100 YM

    Messages:
    7,099
    Positive Ratings:
    5,742
    These just going off our servers?
     
  20. Fr0Z3nR

    aa Fr0Z3nR Creator of blackholes & memes. Destroyer of forums

    Messages:
    6,391
    Positive Ratings:
    4,906
    A popular payload server also has it linked, I believe Geit has said.
     
    Last edited by a moderator: Jul 24, 2014