Discussion - Playtests, Iteration, Frustration

  • If you're asking a question make sure to set the thread type to be a question!

DrLambda

L69: Teeheehee, Member
aa
Feb 18, 2015
458
475
Good Evening. First of all, please take everything i say with a grain of salt. It's not random chance that i'm creating this thread now rather than earlier, and it will directly involve the last tests of my current main project plr_hierarch. It's a map i've worked long and hard on, and i'm a bit proud of it, but i still try to be as objective as somehow possible.
Also, while i've been mapping for FPS for the better part of my life and built my first room in Hammer (or rather Worldcraft) in a time where at least a part of the users of this forum wasn't born, let me also say that everything i'm describing here are luxury problems - I wish i had such a great community and active testing group as everyone in this forum has back in the day. I'm really happy for that, especially considering that i don't even have enough people in my friend list that play TF2 to even test 6v6 on a regular base. I really love what you guys are doing - Keep up the good work!

But let me tell you about the problems i have right now. The last game day and the imp before that (even though i still don't know why Hierarch was even in that imp,) Hierarch was RTVed after about 20 minutes in each one. I watched every replay multiple times. Problems were basically that the stronger team who could control the high ground in the middle was so dominant that the other team wasn't able to play much.

While watching the imp replay, i came to the conclusion that the losing team was not really using the spawn door that can't be as easily controlled from the hill and brings some interesting flanking options without forcing you to fight from the low ground. But i'm not one to say "You're playing my map all wrong!" Because if you are playing my map in an unintended way, that's probably my fault - I should've made that path more obvious to use.

Before i get to one of the core problems, i want to say it again - Yes, i saw problems during those tests, and the fact that the third exit is underused is only one of them. There will be some big changes incoming before i ever release a new beta.

But there are also some problems that are outside of my reach, and i want to discuss those:

1. Disfunctional / Unbalanced teams
This one was glaring obvious to me in the last two tests, and i had the same problem when joining imps as a player. One team works together really well, the other team is trying to break that by running in there one-by-one rather than combining their efforts. Usually that problem is solved by scrambling, but that didn't really work on those last two tests or it took so long to scramble that everyone was already burned out from being slaughtered or simply shooting at the spawn door for three minutes. Another side of that problem is simply team constellations that wouldn't even work if both teams were of equal strength. For at least the last three rounds before the map was RTVed on the game day, a quickfixed Brass Beast Heavy was holding down the hill, supported by multiple soldiers, but the combined number of Soldiers, Demomen or Heavies in the other team was usually 0, but never higher than 1 (maybe two, but i can't recall that situation.) Even if the four pyros during the last round would try to take the ramp from the Heavy and his medic together, i'd guess they'd still be mown down. I literally wanted to scream at my screen while watching them getting slaughtered over and over.
My suggestion would be quicker or more frequent scrambles from the host(s), and a better team composition as a player. I get it, i love Pyro quite a lot, even called it my "main" for the longest time i've been playing TF2, but you're not taking the high ground from a bunch of soldiers and a quickfixed Heavy, no matter how often you try or how good your airblast technique is.

2. Iteration - Or, you're testing the map too often.
The basic layout of Hierarch hasn't changed by more than a few hammer units in any place since Alpha 3. During both tests, i heard both voices that made jokes about how often i'm adding this map to tests, and that the map should've stayed in alpha for a longer time, solving the problems the map currently has. But those two points are basically mutually exclusive - It took 7 tests (combining everything from A3 to B2) before the problem that showed during those tests showed up for the first time. All the tests before that were very positive, although with a fair share of understandable negative feedback, which i tried to address in later versions. Should i have tested the same map in alpha 7 times? Even after 2 game days and 3 imps since Hierarch hit beta, i can tell you that the amount of written feedback you get gets to hilariously low levels very fast. I got way less written feedback in the 3 imps and the second gameday combined than in the first game day, and that was on the AU server with way less players, and the map thread feels like writing into my diary, because noone is reading or replying to that as well.
If i wouldn't have put versions with minor changes into the diferent imps, i would never have seen the problems that are arising now before way later in the development cycle. If i tested every version of the map only once, i would still think that everyone thinks my map is fun, but a little cramped. Still, most of the discussion on the later tests was about how often my map was already tested.
On the other hand, if i kept in Alpha for longer for additional tests, literally noone would play my map anymore by the time i get into Beta.
I still don't know whats a good amount of testing - Do you guys have experience with that? And what are your numbers? I mean, i'd love to get the map into different test groups to get a lot of data without burning anyone out, but right now, i don't have any other groups to look to. I'd guess even putting it into EU / US imps would change enough of the playerbase to get that result, but that's kinda hard to do right now, considering that US imps are kinda slow right now and usually won't start before 3-4AM over here.

And lastly, i have a plea to the players - Please be more outspoken about what you like and what you didn't like, maybe even visit the map thread. I have and always will watch every available replay of tests, and previously i even tried to react to things that happened in said tests inside my map thread, but there is practically no reaction in that forum. Especially if i can't play in the test myself (like i sadly wasn't able to do on this gameday,) that makes me feel really dissociated from the group i want to make happy with my creations.

Thanks for reading all that stuff, feel free to voice your opinion - I'm looking forward to it.
 

Moonfixer

L5: Dapper Member
Aug 23, 2014
229
81
I had a similar problem a while ago. Some players found a way to get out of my map and this left the other players on their team struggling to capture the objective. After I took note that I needed to fix the problem, players continued to exploit this and players kept complaining about the map feeling imbalanced. That's not to say that my map doesn't have several more issues (which were pointed out by way of extensive and helpful feedback going beyond the feedback plug in) seeing as this is my first one but still, I can see how that can be frustrating.
 
Dec 28, 2014
330
307
Should i have tested the same map in alpha 7 times? Even after 2 game days and 3 imps since Hierarch hit beta, i can tell you that the amount of written feedback you get gets to hilariously low levels very fast. I got way less written feedback in the 3 imps and the second gameday combined than in the first game day, and that was on the AU server with way less players, and the map thread feels like writing into my diary, because noone is reading or replying to that as well.
If i wouldn't have put versions with minor changes into the diferent imps, i would never have seen the problems that are arising now before way later in the development cycle. If i tested every version of the map only once, i would still think that everyone thinks my map is fun, but a little cramped. Still, most of the discussion on the later tests was about how often my map was already tested.
Don't get discouraged if you don't get too much written feedback. I know as someone who's not a mapper but just wants to help playtest maps to help map makers sometimes it can be difficult to put a lot of my feedback into words unless something is very obviously right or wrong.

I do think doing at least 2-3 playtests if possible is a good idea just so if the teams are horribly imbalanced in one playtest that doesn't lead you down the wrong path for doing new versions. It might be difficult to get people interested in doing more playtesting if you've done 6 or 7 of the same version.
 

Yrr

An Actual Deer
aa
Sep 20, 2015
1,308
2,743
generally people get irritated if maps with problems barely change, it feels like "why should we test this if the feedback doesnt get used"

i think the map has core problems that have been pointed out for the last week or so but the map keeps getting tested with no changes made to address them
 

Lain

lobotomy success story
aa
Jan 8, 2015
724
757
The basic layout of Hierarch hasn't changed by more than a few hammer units in any place since Alpha 3.

Then why test? I seriously do not understand the logic in updating a map with literally no changes that address problems with the level, and then get mad and make a wall of text when people won't take your single staged payload race seriously.

As early as alpha 1 Berry bought up problems with the layout, massive, insane and glaring problems with the layout that ring true today. I feel as if you're too connected to original layouts, you need to make your levels to be torn apart and built back up. When we tested temper all that changed over several versions where tiny tiny changes, and they never addressed the ingrained issues people had with the level.

I have been CONSISTENTLY getting testing done on Uptown, almost every single night, and people never say 'oh its this map again >.>' because they all know that whatever changes i have made have been massive overhauls. The Alpha 1 of Uptown looks nothing like the Alpha 7 version I have right now, completely different flow and layout, just to adress issues.

You shouldn't expect playtesters to want to play your level over and over if they do not feel like their feedback is being heard. I think at the point you can list precisely what amount of units you've moved existing geometry is the point at which you're not changing your level enough. Nobody wants to play the exact same map week after week that doesn't improve on gameplay. That's not how things work.

And honestly, if you want more cooperative and balanced teams, you should look for external testing. While Tf2maps is great for testing levels overall, I agree that the playtesting is very hit and miss. You get the true 'pub experience' when playing tf2m, with a lot more complaining about error models. Most of the amazing levels gameplay-wise are playtested externally.
 

DrLambda

L69: Teeheehee, Member
aa
Feb 18, 2015
458
475
Then why test? I seriously do not understand the logic in updating a map with literally no changes that address problems with the level, and then get mad and make a wall of text when people won't take your single staged payload race seriously.

As early as alpha 1 Berry bought up problems with the layout, massive, insane and glaring problems with the layout that ring true today. I feel as if you're too connected to original layouts, you need to make your levels to be torn apart and built back up.
The layout actually changed "a lot" from A1 to A3 - then i tested A3 two times successfully, and went over to beta (half a year later) and it's gonna change a lot from B2 to B3 because of the problems we've seen in the tests.
My primary concerns when testing B1 and B2 was visual stuff, because the gameplay seemed to be working fine, and you guys found and i fixed quite a few visual problems over those tests. The only reason why i had B2, B2a and B2b was because of some minor graphical glitches, and then suddenly, the gameplay went horribly wrong over the last two tests. And i wanted to make it known that i'm only fixing minor things right now, but somehow i didn't manage to reach the players.
The reason why i tested the map multiple times was because i had some bugs in both version B2 and B2a (okay, actually i just forgot to pack textures) which were the primary concerns of the feedback, and i wanted to test a "finished" version of B2 before actually reaching conclusions. A side effect of that was the gameplay/layout problems emerging, which showed me that i have to do way more than just finishing up the art pass.

I know that i'm not exactly hitting the zeitgeist when i release a single-stage-PLR map, but when i started Hierarch, i was absolutely in love with that gamemode. My priorities changed over the last half-year, but i still want to create a map every player that doesn't instantly say "Ugh, PLR" can enjoy.

And the guys who played took the map seriously, and the tests were really helpful - I never intended to make you think otherwise. I just wanted to show up some problems (obviously also because this project is close to my heart,) which made me post this wall of text (And the one in this post.)

When we tested temper all that changed over several versions where tiny tiny changes, and they never addressed the ingrained issues people had with the level.
Temper was a completely different issue. I was completely aware of the existing problems, but i just wanted to finish and polish it with those problems because:
1. I built it to be played on our local LAN in a 5v5, max 6v6 environment, where the problem of the size wouldn't matter that much.
2. I knew that fixing everything with Temper would basically force me to do the whole map over, in which case i'd rather create a new map. i actually have newer versions of Temper on my pc (together with the hordes of other alpha maps that have glaring problems that every mapper has,) but i'm never going to release them to the public because nothing except a complete overhaul is making this map playable on a pub server.

For Hierach, all of this isn't true. I want to create a map that is fun to play in a pub environment, and i want to do everything necessary to achieve this. The problems that arose during the last two tests just weren't obvious to me before those tests.

You shouldn't expect playtesters to want to play your level over and over if they do not feel like their feedback is being heard. I think at the point you can list precisely what amount of units you've moved existing geometry is the point at which you're not changing your level enough. Nobody wants to play the exact same map week after week that doesn't improve on gameplay. That's not how things work.
Like i said before, i wanted to get tests in with the finished product, that's why i tested B1, B2, B2a and B2b. I'm not confident enough in my abilities as a mapper for the Source engine (GoldSrc is a completely different matter) to think that i can do an artpass without having some problems, and i wanted the players to find these visual problems. The gameplay was never in question until the last two tests.
I also wanted to get feedback from as many voices as possible, although i have to admit that the playerbase in the EU doesn't change enough over imps to get that result and i'm just burning the same people out. I understand that now.
Also, the map was tested an additional time without my knowledge. I only saw that by accident after adding the map to the gameday, which helped burning the players out even more because they didn't see any changes (because there weren't any.)

And honestly, if you want more cooperative and balanced teams, you should look for external testing. While Tf2maps is great for testing levels overall, I agree that the playtesting is very hit and miss. You get the true 'pub experience' when playing tf2m, with a lot more complaining about error models. Most of the amazing levels gameplay-wise are playtested externally.
I'd love to run it in external playtests, but the problems are that i don't feel like i have neither the reputation nor the social contacts in the TF community to get this done.
 

Fruity Snacks

Creator of blackholes & memes. Destroyer of forums
aa
Sep 5, 2010
6,394
5,571
Then why test? I seriously do not understand the logic in updating a map with literally no changes that address problems with the level, and then get mad and make a wall of text when people won't take your single staged payload race seriously.

Sometimes, small things have BIG ramifications. I know that with Rust I didn't change the layout too much, literally just moved a few things around a little bit and it was a big impact. Same thing with Respawn times and ammo placements.

Onto the main point:
So, @DrLambda I read through this whole post and there's a few things I want to touch on. You point out some counter arguements already to what i'm going to say, but I'm going to reiterate them and change things up a little bit from my perspective.

1. Disfunctional / Unbalanced teams
This one was glaring obvious to me in the last two tests, and i had the same problem when joining imps as a player. One team works together really well, the other team is trying to break that by running in there one-by-one rather than combining their efforts. Usually that problem is solved by scrambling, but that didn't really work on those last two tests or it took so long to scramble that everyone was already burned out from being slaughtered or simply shooting at the spawn door for three minutes. Another side of that problem is simply team constellations that wouldn't even work if both teams were of equal strength. For at least the last three rounds before the map was RTVed on the game day, a quickfixed Brass Beast Heavy was holding down the hill, supported by multiple soldiers, but the combined number of Soldiers, Demomen or Heavies in the other team was usually 0, but never higher than 1 (maybe two, but i can't recall that situation.) Even if the four pyros during the last round would try to take the ramp from the Heavy and his medic together, i'd guess they'd still be mown down. I literally wanted to scream at my screen while watching them getting slaughtered over and over.
My suggestion would be quicker or more frequent scrambles from the host(s), and a better team composition as a player. I get it, i love Pyro quite a lot, even called it my "main" for the longest time i've been playing TF2, but you're not taking the high ground from a bunch of soldiers and a quickfixed Heavy, no matter how often you try or how good your airblast technique is.

So, the short of how this all read to me was "Inbalanced teams means that my map doesn't work!" ... Totally,. Usually inbalanced teams means that the map is not going to play the way that you feel it should when the teams are balanced. You, as the designer need to ercognize that. Inablanced teams will break maps. You will get MANY instances, here and elsewhere where players will play on inbalanced maps. You need to include design changes/features/things in your map that helps the teams keep balance. That is, if You have a Red team who's stacked, you want to make sure that your design can help Blue out in those situations. A little inbalance shouldn't stop your map. You need to design your map so that it's inbalance tolerance is high, so that only really inbalanced teams can hinder your map, not stop it. A medic/Heavy just camping is a dinky inbalance and highlights that this is a problem. Look at your design again, find the problem, and solve the problem. Without looking at the map, I can probably guess a lot of the problems you're pegging on inbalanced teams... aren't really problems with inbalanced teams. Some are, yes, but not as many as I think you may think. We can't force team composition, and we won't. Thats bad testing. We could scramble more, but you want to at least play 2 rounds with the same teams so you can identify if it is infact a problem with teams, or the map. (usually both, for aforementioend reasons).

2. Iteration - Or, you're testing the map too often.
The basic layout of Hierarch hasn't changed by more than a few hammer units in any place since Alpha 3. During both tests, i heard both voices that made jokes about how often i'm adding this map to tests, and that the map should've stayed in alpha for a longer time, solving the problems the map currently has. But those two points are basically mutually exclusive - It took 7 tests (combining everything from A3 to B2) before the problem that showed during those tests showed up for the first time. All the tests before that were very positive, although with a fair share of understandable negative feedback, which i tried to address in later versions. Should i have tested the same map in alpha 7 times? Even after 2 game days and 3 imps since Hierarch hit beta, i can tell you that the amount of written feedback you get gets to hilariously low levels very fast. I got way less written feedback in the 3 imps and the second gameday combined than in the first game day, and that was on the AU server with way less players, and the map thread feels like writing into my diary, because noone is reading or replying to that as well.
If i wouldn't have put versions with minor changes into the diferent imps, i would never have seen the problems that are arising now before way later in the development cycle. If i tested every version of the map only once, i would still think that everyone thinks my map is fun, but a little cramped. Still, most of the discussion on the later tests was about how often my map was already tested.
On the other hand, if i kept in Alpha for longer for additional tests, literally noone would play my map anymore by the time i get into Beta.

Playing your map over and over is fine, thats testing. Unless there are hyper-glaring problems, you shouldn't really use only one 30 minute test for feedback. So testing over and over again (I'd say like, 2 hours worth?) isn't bad. If you need less, even better. If you need more, thats fine. People in this community though (at least, some vocal members? I haven't pin-pointed the issue yet, to be honest) think that if a map is being tested then the changes should be noticable. It's on you to be in the tests to say "This is what I changed" and say why. It's super important to be able to tell people "I'm looking for this type of feedback over this change." It helps tremendously to be actively part of your playtest, saying I'm lookig to see how this plays" without directing people like traffic. You want to say "I want to see how this new flank works" but you don't want to say "I want people to use this flank." Why? because if no one uses the flanks, then thats something. Directing people like traffic will always ensure you see what you want to see... which is bad. You want to see what the level says, not you.

And lastly, i have a plea to the players - Please be more outspoken about what you like and what you didn't like, maybe even visit the map thread. I have and always will watch every available replay of tests, and previously i even tried to react to things that happened in said tests inside my map thread, but there is practically no reaction in that forum. Especially if i can't play in the test myself (like i sadly wasn't able to do on this gameday,) that makes me feel really dissociated from the group i want to make happy with my creations.

And yes, everyone should be posting in the threads with feedback. Take notes as you play! I have a notebook for all the notes I have when I test. I can then come back to them later. They're useful! If you want to do this type of seriously, I would highly suggest getting some paper and note-take!

Hope it helps, yo.
 

SnickerPuffs

(*single chuckle*)
aa
Apr 10, 2014
1,315
1,857
Honestly, it feels like the adverage feedback I got changed drastically with the last version or two.

Before the latest version on Cruise, the layout suggestions were only one or two per playtest, and it seemed most people focused on health packs or detailing. But since beta 1b, everyone has exclusively been pointing out huge flaws in the layout that I've vastly overlooked since alpha 8.
 

fubarFX

The "raw" in "nodraw"
aa
Jun 1, 2009
1,720
1,978
I'm with frozen on designing a better map. Think of it this way, A good level has only 2 critical things to do, 1. present an objective, 2. provide tools to fulfill that objective. If the pyro filled team didn't have any good option presented to them, this may be have been a shortcoming of the map. that's really all there is to it. As for the frequency of testing, you really want to get rid of the obvious stuff quick, being exposed to the same big problems over and over is just going to put a strain on the testing group and is going to degrade the opinion of your map. People may become less favorable to your map over time. If things are okay for the most part and you simply don't have enough data to make a call on what needs to be addressed or what could be improved, keep testing. no harm no foul.

As testers we certainly could do better as well. sometimes I'm under the impression that tf2m players tend to play nice and don't want to break each other's maps and/or ruin everyone's fun. I'd like to see more devious plays and people actively trying to break maps. No need to be a dick about it ofc, just expose a problem convincingly and move on.
 

Fruity Snacks

Creator of blackholes & memes. Destroyer of forums
aa
Sep 5, 2010
6,394
5,571
I'm with frozen on designing a better map. Think of it this way, A good level has only 2 critical things to do, 1. present an objective, 2. provide tools to fulfill that objective. If the pyro filled team didn't have any good option presented to them, this may be have been a shortcoming of the map. that's really all there is to it. As for the frequency of testing, you really want to get rid of the obvious stuff quick, being exposed to the same big problems over and over is just going to put a strain on the testing group and is going to degrade the opinion of your map. People may become less favorable to your map over time. If things are okay for the most part and you simply don't have enough data to make a call on what needs to be addressed or what could be improved, keep testing. no harm no foul.

As testers we certainly could do better as well. sometimes I'm under the impression that tf2m players tend to play nice and don't want to break each other's maps and/or ruin everyone's fun. I'd like to see more devious plays and people actively trying to break maps. No need to be a dick about it ofc, just expose a problem convincingly and move on.

(should note, we don't mind you breaking maps. We do encourage it... but breaking, THEN EXPLOITING (which would mean, continuing to do the break) is grounds for being a jerk.)
 

Berry

resident homo
aa
Dec 27, 2012
1,056
1,898
As testers we certainly could do better as well. sometimes I'm under the impression that tf2m players tend to play nice and don't want to break each other's maps and/or ruin everyone's fun. I'd like to see more devious plays and people actively trying to break maps. No need to be a dick about it ofc, just expose a problem convincingly and move on.

The thing about people actively trying to break others' maps is that sure it is useful, but just about everyone who does that does it in the server and when they do they're finding ways to get out of the map then sitting there and congaing so people go "woah how did you do that" after seeing them on spec cams instead of feedbacking it and moving on.

If you don't think it's that much of an issue either, on the imp earlier I had to enforce class limits (among other things like a warning) because quite literally half of the server decided to go Jumper Demo and Soldier when someone found a very minor clipping issue that meant you could get on to a cliff on the spawn of each side.
 

Muddy

Muddy
aa
Sep 5, 2014
2,574
4,592
In the last few tests on Kemptown I saw one guy spending the entire match stickyjumping around the place trying to find clipping exploits, and then standing there and taunting for the rest of the round when he found one.

Protip: please don't do that.
 

fubarFX

The "raw" in "nodraw"
aa
Jun 1, 2009
1,720
1,978
The thing about people actively trying to break others' maps is that sure it is useful, but just about everyone who does that does it in the server and when they do they're finding ways to get out of the map then sitting there and congaing so people go "woah how did you do that" after seeing them on spec cams instead of feedbacking it and moving on.
When I say breaking a map, I'm not really talking about petty exploits and such. It's more of what you'd do from a comp perspective. Comp players are really good at identifying things that give them an edge over the other team and exploiting that. While your map may be /fine/ and not have any exploits, a well coordinated team can and will completely break your map when given the opportunity, and by break I mean completely locking down an area and shutting the whole map down for example. This can take the form of sentry placement, abusing a certain choke, sightlines etc.
 

Berry

resident homo
aa
Dec 27, 2012
1,056
1,898
When I say breaking a map, I'm not really talking about petty exploits and such. It's more of what you'd do from a comp perspective. Comp players are really good at identifying things that give them an edge over the other team and exploiting that. While your map may be /fine/ and not have any exploits, a well coordinated team can and will completely break your map when given the opportunity, and by break I mean completely locking down an area and shutting the whole map down for example. This can take the form of sentry placement, abusing a certain choke, sightlines etc.

Oh yeah that sorta thing is fantastic feedback-wise, for instance I had Maple Ridge tested a ton of times and everyone was like "great". I then took it into competitive once and snipers could rip the map up totally once the CP was held with some stuff nobody else really realised (and coordination helps, we don't have to think about that when mapping for pub play really).