How to make structured map testing work for everyone

Yrr

An Actual Deer
aa
Sep 20, 2015
1,308
2,743
Improving The Impromptu or How To Make Gamedays Great Again

There's been a lot of discussion about how tests are done in the steam chat, and there are disagreements on whether Impromptu tests should be run the current way or the "old-school" way.

With that in mind I would like to list all the methods of testing available and suggest how I think things could work, as well as inviting discussion on the issue.

Methods of Testing

Current "Bot" Impromptus
People submit their maps to the bot, the maps are played in first-come-first-serve order, ensuring anyone can get testing.
Downsides:
- The queue builds up faster than it can be played, queuing a map when you are available could see the map being played days later while you're busy
- Without the mapper there, discussion about the map quickly goes off course


Old-style Impromptus
The maps of those present are played, ensuring there can be actual discussion about the maps.
Downsides:
- Some people need testing but aren't available during tests
- Can sometimes be hard to source maps to host one


Gameday
Formerly the more "open" form of test to people who could not be around to imp, this method is now somewhat obselete in the face of the easy-access bot, but is still filled first-come-first-served the same way.
Downsides:
- Hard to get a map in if you aren't around when the thread goes up
- The more focused testing doesn't necessarily go to the maps that could use it

The Proposal

So here's the suggestion that's been floating around in the chat but being buried line-by-line in the discussion, up where everyone can read and criticise it:

Step 1. Split Impromptus into Old- and New- style Impromptus (and probably rename one)
Each style serves a specific purpose, and eliminating one in the face of the other was arguably unnecessary and isn't ideal for some people but, likewise, the new style definitely makes testing more accessible and is a necessity in the current era of mapping.
I propose alternating impromptus based on demand between working the list and playing the maps of people who are available. This way, people who can wait can wait, people who can't don't have to.

Step 2. Rework Gamedays
The purpose of a gameday in the past was to allow testing for people who were unable to attend imps. As feedback in gamedays improved compared to impromptus, and as the bot replaced gamedays as the accessible test, the gameday's primary purpose has become the scheduled, "event" nature, and the more focused testing. With the first-come-first-served system the maps that could make best use of the testing don't necessarily make it in.
I propose letting gameday hosts pick the submitted maps based on which maps they think could make the best use of the testing, whether it be because the map has potential or because the mapper is attending. This wouldn't come at the cost of accessibility as the bot is always available, and instead would make Gamedays more of an event and something for mappers to aim for.

As stated above, the purpose of this thread is to provide a full explanation of the issue and its proposed resolutions, and to allow discussion and criticism that wouldn't be possible through me just trying to explain it in steam chat.
 
Last edited:

Egan

aa
Feb 14, 2010
1,375
1,720
@Step 1, of course, why isn't it done already. give priority to mappers online, and if you 'run out of maps' use the bot's list.

@Step 2, I feel like giving the host free reign over what is being played can be unfair, and while staff can say "well, we'll keep an eye on that" , no, nobody except the host does on a regular basis, let's be real. Unless you wanted to do a combination of the two again, like in step 1, where the host can select one or two maps from the list they think should get priority, but then otherwise fills the list with the first-come-first-serve submitted maps. Might also be more fun for the host!
 

Yrr

An Actual Deer
aa
Sep 20, 2015
1,308
2,743
Of course it'll always be up to the host, these are simply suggested as possibilities.

Although I'd also suggest that while the host picking maps may have been unfair in the past, the bot provides the accessibility that gamedays used to, and so FCFS may no longer be ideal.

I think about it like "what does the Gameday offer that Impromptus don't" and for now that seems to be focused testing in an environment where someone is able to decide what maps would benefit most from it, even if they don't do that currently.
 

DrLambda

L69: Teeheehee, Member
aa
Feb 18, 2015
458
475
Aside from the suggestions you made - I always considered Gameday a much more focused way of testing. While you can get good results with imps, gamedays usually give you way more feedback than imps do. I can't really put my finger on why it is the way it is, but looking at my numbers, i usually get more than the double amount of feedback on Gamedays than i get during imps. Sure, you can analyse demos the same way (and you might even say that demos are often preferable to feedback, as they show statistics rather than anecdotal information,) but i consider knowing what the players think of the map as absolutely vital. While i am thankful that i can add maps to imps and am happy that they will get played at some time, i absolutely WANT my maps to be in the game days.

The problem with having the host choose which map will get played, you:
A: Basically deny new mappers the chance to get their map tested in that way. I know some of the maps have been hellholes, but usually getting annihilated in the feedback helps these mappers improve a lot.
B: Could kill off some promising maps with big weaknesses at that point in development. Let's be serious here. Within the forum, you'll get feedback if either your map has glaring weaknesses you can see on the screenshots or you're a known veteran (or however you'd like to call it.) The biggest step you have to take as an aspiring mapper on TF2M is from intermediate to actually kinda good, and this is where testing is vital even though your maps might not be considered worthy by the host right now.

I understand that the way it's currently done is suboptimal and might even be unfair to those people who aren't around when the gameday thread gets posted, but following your suggestion might harm the influx of new promising mappers if not done very, very carefully.

I don't think that the problem is that the way it's currently done isn't working, it's just that after the 72h jam, which worked exceptionally well in bringing a lot of new mappers over to TF2M, the sheer amount of maps is way bigger than it was a few months ago. Combined with a lot of 7.2h testing over the last few days, both the hosts and the testers got burned out, and the queue got longer and longer.
I'd suggest actively trying to get the word out that TF2M is always looking for testers, because the number of people that actually play in imps or game days didn't increase in the same way the amount of mappers increased. If we get more testers, that'll probably mean that we'll get more potential hosts, and with that might even go back to running two gamedays (one on Saturday, one on Sunday, one on US, one on EU) which even could use the same queue/thread, so the second one could start where the first one stopped.
 

Yrr

An Actual Deer
aa
Sep 20, 2015
1,308
2,743
I agree with a lot of those points but I feel like they would be the considerations the hosts would be making with regards to which maps would benefit from that feedback.

The suggestion is just to not always default to FCFS, how the host decides to choose the maps is up to them, whether that's giving new mappers the full treatment or getting some not-quite-there maps the feedback they need to continue.

Like FCFS is still an option this is rly just a reminder that it doesn't have to be and that always doing FCFS isn't necessarily the best idea.
 
Last edited:
Jul 6, 2015
1,425
819
I'd suggest actively trying to get the word out that TF2M is always looking for testers, because the number of people that actually play in imps or game days didn't increase in the same way the amount of mappers increased.

Usually announcements are ran through the steam group, which is on the home page, but I dont see any other way aside from making a notification on the website itself or having a bot somehow send a message to every registered steam account on the site (that would be a bad idea but its a suggestion).

I would love to see more players on the imps, and personally I haven't been attending them because I always see 23/24 and then 24/24 every time I check the server in my favorites tab, if I had known it was an issue I would pay much closer attention to when imps and game-days are announced, but whenever I check the server its usually always full.

Perhaps a solution to the lack of hosts that get burnt out from the extreme amount of maps, is to have another way to have people host games. Rather then only staff and donators, maybe there could be another method for people to be able to host on the servers?
Ex: Veterans could host (just an example this wouldn't be a really good idea, but you get the concept of the idea)
 

DrLambda

L69: Teeheehee, Member
aa
Feb 18, 2015
458
475
Usually announcements are ran through the steam group, which is on the home page, but I dont see any other way aside from making a notification on the website itself or having a bot somehow send a message to every registered steam account on the site (that would be a bad idea but its a suggestion).

I would love to see more players on the imps, and personally I haven't been attending them because I always see 23/24 and then 24/24 every time I check the server in my favorites tab, if I had known it was an issue I would pay much closer attention to when imps and game-days are announced, but whenever I check the server its usually always full.

Perhaps a solution to the lack of hosts that get burnt out from the extreme amount of maps, is to have another way to have people host games. Rather then only staff and donators, maybe there could be another method for people to be able to host on the servers?
Ex: Veterans could host (just an example this wouldn't be a really good idea, but you get the concept of the idea)

I didn't want to say that imps don't fill up the server or that the group announcements are ineffective. I meant that we try to get more people into the group that, while maybe not mapping a lot themselves, are interested in the testing or creation process of maps or just like killing other players in unknown areas.

Like i said, the 72h jam was really successful in raising the interest of aspiring mappers. If we could get something like that going on for players who just want to help out or have a good time while being an active part of the map creation process, that would help in getting more imps going, which would help with the current queue overflow. Or like i said, would possible make TF2M able to host two gamedays on the weekend.
 

Fantaboi

Gone and one day forgotten
aa
Mar 11, 2013
892
1,050
The problem isn't about getting as many maps run as possible. It's about quality of feedback and the inability to get map testing instantly like the old days. Running more imps or gamedays is not the solution because it's simply not sustainable as a solution, mappers will just simply get faster and rush out half assed maps to get them tested as much as possible,and doesn't solve the issue regarding quality feedback
 

DrLambda

L69: Teeheehee, Member
aa
Feb 18, 2015
458
475
The problem isn't about getting as many maps run as possible. It's about quality of feedback and the inability to get map testing instantly like the old days. Running more imps or gamedays is not the solution because it's simply not sustainable as a solution, mappers will just simply get faster and rush out half assed maps to get them tested as much as possible,and doesn't solve the issue regarding quality feedback

The issue is differencing between which maps are genuinely half-assed and which ones are just bad because of the the mapper not knowing any better. When i started mapping for TF2, i already had used Hammer/Worldcraft for 10+ years, and i still made quite a few maps that didn't play very well. Having to learn about both Hammer and layout in TF2 at the same time is a very steep learning curve. I'm also sure that the possibility of running the same map through multiple testing sessions once you solved all the obvious problems isn't a bad thing and something i WILL do once i get another map on the verge of beta after the problems i had with Hierarch.
 

Fruity Snacks

Creator of blackholes & memes. Destroyer of forums
aa
Sep 5, 2010
6,394
5,571
i for one dont much care for quality of feedback, i just want the STV with one-two quick opinions in it

so just getting the maps ran works for me

For those who know how to analyze STV's, this works well. For those who don't however, it's not the best solution.
 

Yrr

An Actual Deer
aa
Sep 20, 2015
1,308
2,743
Some people especially prefer a dialogue with the players, which is not available through demos, and isn't always available from the current style of impromptus.
 

Fantaboi

Gone and one day forgotten
aa
Mar 11, 2013
892
1,050
A suggestion/idea to help out with the prevalent issue of mappers not being at imps is the idea of adding an additional command to the bot: !prioritymaps
When used it will return only maps of users who are currently online (maybe even only in the chatroom) , (and possibly even show the most recently added maps first)

However, a major purpose of the Bot is to allow mappers who can't make it to imps 100% out of circumstance to still get their map tested, so i've considered adding a whitelist. Users on this whitelist will automatically be returned (in a separate manner) when !prioritymaps is called. the command may look like: Koth_map Koth_othermap //Extra: koth_australianmapper

The ability to get added to this Whitelist is up to the discretion of admins, and I strongly urge them to make the list as small as possible. At most I can only picture 5 users being added.

If an individual map needs to be added to get onto the whitelist, admins could use a !priorityadd command, or !givepriority <mapname>

SO my question is, does this alleviate the issues many have regarding the bot-testing? It'd encourage users to join imps more often, and join in discussions whilst not acting as a detriment to users who /need/ to use the bot to get their map tested. Furthermore it encourages imps to be spaced out across time-zones. Imp-hosters can have more discretion towards wanting to host an imp with hosts within the imp or churn down the list.
 

Fantaboi

Gone and one day forgotten
aa
Mar 11, 2013
892
1,050
I very much dislike the idea of having a cosy club of people who can get their maps tested whenever, and forcing everyone else to stay in chat or miss out.
As I said, I encourage that cosy club of people to be an extremely limited pool. It would consist of for example Australians who can't stay up till 2:00am for an EU imp or play tf2 at work during 10:00am. It wouldn't mean any and all Australians can use it, staff could decide on who constitutes that usage. I do understand however if the staff are against having to make that decision due to bias.
 

Yrr

An Actual Deer
aa
Sep 20, 2015
1,308
2,743
I very much dislike the idea of having a cosy club of people who can get their maps tested whenever, and forcing everyone else to stay in chat or miss out.
I very much dislike people not wanting to queue their maps for imps because they can't guarantee being around when the imp finally reaches their map in 3 days

the new system is a necessity for a lot of people but so was the old one, but we shouldn't just pick one and scrap the other.

prioritising present mappers will never outnumber the queue, and the benefits of ignoring present mappers for a first-come-first-served system are few if any
 

Idolon

they/them
aa
Feb 7, 2008
2,105
6,106
A lot of people running imp tests are already trying to prioritize maps by authors who are present, might as well just build that into the bot.

Someone mentioned that running more tests isn't realistic, which makes sense. However, that's not the only way to work through a large volume of maps. Giving maps less playtime would have the same effect without increasing how much time is spent testing overall.

I usually have a decent idea of what I want to change with my map about 15-20 minutes into my map's 30 minute test (sometimes even sooner), and bad maps usually have all of their most glaring flaws pointed out within the first 5-15 minutes. That's not to say that the full playtime of a map isn't useful, but the longer a test goes on for, the more nitpicky and specific the feedback tends to get.

I'd be interested in seeing impromptu tests with shorter map playtimes, and having specialized tests with longer play times reserved for more developed alphas that need the extra time for more refined testing/feedback.
 

LeSwordfish

semi-trained quasi-professional
aa
Aug 8, 2010
4,102
6,597
why tho

apart from wanting to be present in every test that is

surely theres enough people in the group to fill two full testing crews

You'd be surprised - even at peak times (during 72hr, US and EU both peaking) we could rarely get more than a server and a half full.

Personally, I would very much like to run more gamedays: giving people a definite schedule for when their maps will be played allows them to plan ahead: to try and make specific times. If we had, say, three gamedays a week, people would be able to choose which to enter and which to sit out of, and could be sure not to wait more than a week for their map to be played.

An alternative might be a more structured approach to imping - perhaps ensure every imp is six maps and we do one every non-gameday day, for example, so we can reasonably reliably guess when each map will be played.
 
Last edited:

Yrr

An Actual Deer
aa
Sep 20, 2015
1,308
2,743
An alternative might be a more structured approach to imping - perhaps ensure every imp is six maps and we do one every non-gameday day, for example, so we can reasonably reliably guess when each map will be played.
This is the point where I'd suggest splitting impromptus. Keep regular "impromptus" for people who are there at the time and want their maps played while they're available, but use the planned ones (would need a different name) to play through the list with.