How to make structured map testing work for everyone

Discussion in 'Site Discussion' started by Yrr, Sep 10, 2016.

  1. Yrr

    aa Yrr An Actual Deer

    Messages:
    1,028
    Positive Ratings:
    1,998
    Improving The Impromptu or How To Make Gamedays Great Again

    There's been a lot of discussion about how tests are done in the steam chat, and there are disagreements on whether Impromptu tests should be run the current way or the "old-school" way.

    With that in mind I would like to list all the methods of testing available and suggest how I think things could work, as well as inviting discussion on the issue.

    Methods of Testing

    Current "Bot" Impromptus
    People submit their maps to the bot, the maps are played in first-come-first-serve order, ensuring anyone can get testing.
    Downsides:
    - The queue builds up faster than it can be played, queuing a map when you are available could see the map being played days later while you're busy
    - Without the mapper there, discussion about the map quickly goes off course


    Old-style Impromptus
    The maps of those present are played, ensuring there can be actual discussion about the maps.
    Downsides:
    - Some people need testing but aren't available during tests
    - Can sometimes be hard to source maps to host one


    Gameday
    Formerly the more "open" form of test to people who could not be around to imp, this method is now somewhat obselete in the face of the easy-access bot, but is still filled first-come-first-served the same way.
    Downsides:
    - Hard to get a map in if you aren't around when the thread goes up
    - The more focused testing doesn't necessarily go to the maps that could use it

    The Proposal

    So here's the suggestion that's been floating around in the chat but being buried line-by-line in the discussion, up where everyone can read and criticise it:

    Step 1. Split Impromptus into Old- and New- style Impromptus (and probably rename one)
    Each style serves a specific purpose, and eliminating one in the face of the other was arguably unnecessary and isn't ideal for some people but, likewise, the new style definitely makes testing more accessible and is a necessity in the current era of mapping.
    I propose alternating impromptus based on demand between working the list and playing the maps of people who are available. This way, people who can wait can wait, people who can't don't have to.

    Step 2. Rework Gamedays
    The purpose of a gameday in the past was to allow testing for people who were unable to attend imps. As feedback in gamedays improved compared to impromptus, and as the bot replaced gamedays as the accessible test, the gameday's primary purpose has become the scheduled, "event" nature, and the more focused testing. With the first-come-first-served system the maps that could make best use of the testing don't necessarily make it in.
    I propose letting gameday hosts pick the submitted maps based on which maps they think could make the best use of the testing, whether it be because the map has potential or because the mapper is attending. This wouldn't come at the cost of accessibility as the bot is always available, and instead would make Gamedays more of an event and something for mappers to aim for.

    As stated above, the purpose of this thread is to provide a full explanation of the issue and its proposed resolutions, and to allow discussion and criticism that wouldn't be possible through me just trying to explain it in steam chat.
     
    • Like Like x 3
    • Thanks Thanks x 2
    • Agree Agree x 2
    Last edited: Sep 10, 2016
  2. Egan

    aa Egan

    Messages:
    1,318
    Positive Ratings:
    1,512
    @Step 1, of course, why isn't it done already. give priority to mappers online, and if you 'run out of maps' use the bot's list.

    @Step 2, I feel like giving the host free reign over what is being played can be unfair, and while staff can say "well, we'll keep an eye on that" , no, nobody except the host does on a regular basis, let's be real. Unless you wanted to do a combination of the two again, like in step 1, where the host can select one or two maps from the list they think should get priority, but then otherwise fills the list with the first-come-first-serve submitted maps. Might also be more fun for the host!
     
  3. Yrr

    aa Yrr An Actual Deer

    Messages:
    1,028
    Positive Ratings:
    1,998
    Of course it'll always be up to the host, these are simply suggested as possibilities.

    Although I'd also suggest that while the host picking maps may have been unfair in the past, the bot provides the accessibility that gamedays used to, and so FCFS may no longer be ideal.

    I think about it like "what does the Gameday offer that Impromptus don't" and for now that seems to be focused testing in an environment where someone is able to decide what maps would benefit most from it, even if they don't do that currently.
     
  4. DrLambda

    aa DrLambda L69: Teeheehee, Member

    Messages:
    458
    Positive Ratings:
    454
    Aside from the suggestions you made - I always considered Gameday a much more focused way of testing. While you can get good results with imps, gamedays usually give you way more feedback than imps do. I can't really put my finger on why it is the way it is, but looking at my numbers, i usually get more than the double amount of feedback on Gamedays than i get during imps. Sure, you can analyse demos the same way (and you might even say that demos are often preferable to feedback, as they show statistics rather than anecdotal information,) but i consider knowing what the players think of the map as absolutely vital. While i am thankful that i can add maps to imps and am happy that they will get played at some time, i absolutely WANT my maps to be in the game days.

    The problem with having the host choose which map will get played, you:
    A: Basically deny new mappers the chance to get their map tested in that way. I know some of the maps have been hellholes, but usually getting annihilated in the feedback helps these mappers improve a lot.
    B: Could kill off some promising maps with big weaknesses at that point in development. Let's be serious here. Within the forum, you'll get feedback if either your map has glaring weaknesses you can see on the screenshots or you're a known veteran (or however you'd like to call it.) The biggest step you have to take as an aspiring mapper on TF2M is from intermediate to actually kinda good, and this is where testing is vital even though your maps might not be considered worthy by the host right now.

    I understand that the way it's currently done is suboptimal and might even be unfair to those people who aren't around when the gameday thread gets posted, but following your suggestion might harm the influx of new promising mappers if not done very, very carefully.

    I don't think that the problem is that the way it's currently done isn't working, it's just that after the 72h jam, which worked exceptionally well in bringing a lot of new mappers over to TF2M, the sheer amount of maps is way bigger than it was a few months ago. Combined with a lot of 7.2h testing over the last few days, both the hosts and the testers got burned out, and the queue got longer and longer.
    I'd suggest actively trying to get the word out that TF2M is always looking for testers, because the number of people that actually play in imps or game days didn't increase in the same way the amount of mappers increased. If we get more testers, that'll probably mean that we'll get more potential hosts, and with that might even go back to running two gamedays (one on Saturday, one on Sunday, one on US, one on EU) which even could use the same queue/thread, so the second one could start where the first one stopped.
     
  5. Yrr

    aa Yrr An Actual Deer

    Messages:
    1,028
    Positive Ratings:
    1,998
    I agree with a lot of those points but I feel like they would be the considerations the hosts would be making with regards to which maps would benefit from that feedback.

    The suggestion is just to not always default to FCFS, how the host decides to choose the maps is up to them, whether that's giving new mappers the full treatment or getting some not-quite-there maps the feedback they need to continue.

    Like FCFS is still an option this is rly just a reminder that it doesn't have to be and that always doing FCFS isn't necessarily the best idea.
     
    Last edited: Sep 10, 2016
  6. Turnip

    aa Turnip The 80s Vegetable

    Messages:
    1,432
    Positive Ratings:
    804
    Usually announcements are ran through the steam group, which is on the home page, but I dont see any other way aside from making a notification on the website itself or having a bot somehow send a message to every registered steam account on the site (that would be a bad idea but its a suggestion).

    I would love to see more players on the imps, and personally I haven't been attending them because I always see 23/24 and then 24/24 every time I check the server in my favorites tab, if I had known it was an issue I would pay much closer attention to when imps and game-days are announced, but whenever I check the server its usually always full.

    Perhaps a solution to the lack of hosts that get burnt out from the extreme amount of maps, is to have another way to have people host games. Rather then only staff and donators, maybe there could be another method for people to be able to host on the servers?
    Ex: Veterans could host (just an example this wouldn't be a really good idea, but you get the concept of the idea)
     
  7. DrLambda

    aa DrLambda L69: Teeheehee, Member

    Messages:
    458
    Positive Ratings:
    454
    I didn't want to say that imps don't fill up the server or that the group announcements are ineffective. I meant that we try to get more people into the group that, while maybe not mapping a lot themselves, are interested in the testing or creation process of maps or just like killing other players in unknown areas.

    Like i said, the 72h jam was really successful in raising the interest of aspiring mappers. If we could get something like that going on for players who just want to help out or have a good time while being an active part of the map creation process, that would help in getting more imps going, which would help with the current queue overflow. Or like i said, would possible make TF2M able to host two gamedays on the weekend.
     
    • Thanks Thanks x 1
  8. Fantasma

    aa Fantasma

    Messages:
    888
    Positive Ratings:
    984
    The problem isn't about getting as many maps run as possible. It's about quality of feedback and the inability to get map testing instantly like the old days. Running more imps or gamedays is not the solution because it's simply not sustainable as a solution, mappers will just simply get faster and rush out half assed maps to get them tested as much as possible,and doesn't solve the issue regarding quality feedback
     
    • Agree Agree x 1
  9. DrLambda

    aa DrLambda L69: Teeheehee, Member

    Messages:
    458
    Positive Ratings:
    454
    The issue is differencing between which maps are genuinely half-assed and which ones are just bad because of the the mapper not knowing any better. When i started mapping for TF2, i already had used Hammer/Worldcraft for 10+ years, and i still made quite a few maps that didn't play very well. Having to learn about both Hammer and layout in TF2 at the same time is a very steep learning curve. I'm also sure that the possibility of running the same map through multiple testing sessions once you solved all the obvious problems isn't a bad thing and something i WILL do once i get another map on the verge of beta after the problems i had with Hierarch.
     
    • Agree Agree x 1
  10. Sergis

    aa Sergis L666: ])oo]v[

    Messages:
    1,871
    Positive Ratings:
    1,131
    i for one dont much care for quality of feedback, i just want the STV with one-two quick opinions in it

    so just getting the maps ran works for me
     
    • Agree Agree x 2
  11. Fr0Z3nR

    aa Fr0Z3nR Creator of blackholes & memes. Destroyer of forums

    Messages:
    6,391
    Positive Ratings:
    4,907
    For those who know how to analyze STV's, this works well. For those who don't however, it's not the best solution.
     
    • Agree Agree x 2
  12. Yrr

    aa Yrr An Actual Deer

    Messages:
    1,028
    Positive Ratings:
    1,998
    Some people especially prefer a dialogue with the players, which is not available through demos, and isn't always available from the current style of impromptus.
     
  13. Fantasma

    aa Fantasma

    Messages:
    888
    Positive Ratings:
    984
    A suggestion/idea to help out with the prevalent issue of mappers not being at imps is the idea of adding an additional command to the bot: !prioritymaps
    When used it will return only maps of users who are currently online (maybe even only in the chatroom) , (and possibly even show the most recently added maps first)

    However, a major purpose of the Bot is to allow mappers who can't make it to imps 100% out of circumstance to still get their map tested, so i've considered adding a whitelist. Users on this whitelist will automatically be returned (in a separate manner) when !prioritymaps is called. the command may look like: Koth_map Koth_othermap //Extra: koth_australianmapper

    The ability to get added to this Whitelist is up to the discretion of admins, and I strongly urge them to make the list as small as possible. At most I can only picture 5 users being added.

    If an individual map needs to be added to get onto the whitelist, admins could use a !priorityadd command, or !givepriority <mapname>

    SO my question is, does this alleviate the issues many have regarding the bot-testing? It'd encourage users to join imps more often, and join in discussions whilst not acting as a detriment to users who /need/ to use the bot to get their map tested. Furthermore it encourages imps to be spaced out across time-zones. Imp-hosters can have more discretion towards wanting to host an imp with hosts within the imp or churn down the list.
     
  14. LeSwordfish

    aa LeSwordfish semi-trained quasi-professional

    Messages:
    4,113
    Positive Ratings:
    6,085
    I very much dislike the idea of having a cosy club of people who can get their maps tested whenever, and forcing everyone else to stay in chat or miss out.
     
    • Agree Agree x 2
  15. Fantasma

    aa Fantasma

    Messages:
    888
    Positive Ratings:
    984
    As I said, I encourage that cosy club of people to be an extremely limited pool. It would consist of for example Australians who can't stay up till 2:00am for an EU imp or play tf2 at work during 10:00am. It wouldn't mean any and all Australians can use it, staff could decide on who constitutes that usage. I do understand however if the staff are against having to make that decision due to bias.
     
  16. Yrr

    aa Yrr An Actual Deer

    Messages:
    1,028
    Positive Ratings:
    1,998
    I very much dislike people not wanting to queue their maps for imps because they can't guarantee being around when the imp finally reaches their map in 3 days

    the new system is a necessity for a lot of people but so was the old one, but we shouldn't just pick one and scrap the other.

    prioritising present mappers will never outnumber the queue, and the benefits of ignoring present mappers for a first-come-first-served system are few if any
     
    • Agree Agree x 2
  17. Idolon

    aa Idolon the worst admin

    Messages:
    1,557
    Positive Ratings:
    4,480
    A lot of people running imp tests are already trying to prioritize maps by authors who are present, might as well just build that into the bot.

    Someone mentioned that running more tests isn't realistic, which makes sense. However, that's not the only way to work through a large volume of maps. Giving maps less playtime would have the same effect without increasing how much time is spent testing overall.

    I usually have a decent idea of what I want to change with my map about 15-20 minutes into my map's 30 minute test (sometimes even sooner), and bad maps usually have all of their most glaring flaws pointed out within the first 5-15 minutes. That's not to say that the full playtime of a map isn't useful, but the longer a test goes on for, the more nitpicky and specific the feedback tends to get.

    I'd be interested in seeing impromptu tests with shorter map playtimes, and having specialized tests with longer play times reserved for more developed alphas that need the extra time for more refined testing/feedback.
     
    • Agree Agree x 2
  18. Sergis

    aa Sergis L666: ])oo]v[

    Messages:
    1,871
    Positive Ratings:
    1,131
    why tho

    apart from wanting to be present in every test that is

    surely theres enough people in the group to fill two full testing crews
     
  19. LeSwordfish

    aa LeSwordfish semi-trained quasi-professional

    Messages:
    4,113
    Positive Ratings:
    6,085
    You'd be surprised - even at peak times (during 72hr, US and EU both peaking) we could rarely get more than a server and a half full.

    Personally, I would very much like to run more gamedays: giving people a definite schedule for when their maps will be played allows them to plan ahead: to try and make specific times. If we had, say, three gamedays a week, people would be able to choose which to enter and which to sit out of, and could be sure not to wait more than a week for their map to be played.

    An alternative might be a more structured approach to imping - perhaps ensure every imp is six maps and we do one every non-gameday day, for example, so we can reasonably reliably guess when each map will be played.
     
    • Like Like x 1
    Last edited: Sep 11, 2016
  20. Yrr

    aa Yrr An Actual Deer

    Messages:
    1,028
    Positive Ratings:
    1,998
    This is the point where I'd suggest splitting impromptus. Keep regular "impromptus" for people who are there at the time and want their maps played while they're available, but use the planned ones (would need a different name) to play through the list with.
     
    • Agree Agree x 1