[PSA] Don't Test With Bots

[henke37]

The automatic mesh generator is rather conservative when it comes to marking drops. If you want bots to take a drop then you either need to force them into nav ignoring mode by not having any connections at all or manually mark the link between nav areas.

Note: 4 years later and the bots are still dumb. And it's still a bad idea to test with bots.

 

[Luigi1000]

Ya I have no idea why people get the idea to use bots. personally just grab 12 ppl I know to hop on a dedicated server to just playtest and the occasional person for feedback in between playtests to both get back on changes as well as do some testing of elements to fix bugs.

 

[hutty]

4 years late to the party, but, I disagree. I test with bots all the time, one of the reasons I dont' make payload maps is because they are not bot compatible.

I understand that bots do not mimic human behavior very well, but they give me something to focus on when I am designing play spaces, its hard to tell if an area will be fun fight in until you fight in it, and a quick easy way to do that is to turn on bots.

 

[Muddy]

one of the reasons I dont' make payload maps is because they are not bot compatible.

Yes they are

 

[hutty]

 

[Lain]

You guys made me sad, i thought Tyler was back.

 

[killohurtz]

4 years late to the party, but, I disagree. I test with bots all the time, one of the reasons I dont' make payload maps is because they are not bot compatible.

I understand that bots do not mimic human behavior very well, but they give me something to focus on when I am designing play spaces, its hard to tell if an area will be fun fight in until you fight in it, and a quick easy way to do that is to turn on bots.

I think the main takeaway from this PSA is not "don't test with bots ever;" rather, it's "don't test with and iterate based on bot behavior thinking it's a good substitute for human players." It's good that you personally understand the stupidity of bots and can exclude them from gameplay feedback, but new mappers may not yet realize the difference.

 

[EArkham]

People freaking love lists. Bots are indifferent.

Things bots are fine for:

• Seeing how long it takes to reach points of interest for each class (assuming good navigation -- though this is something you personally should be timing from the start of an A1)
• Seeing if areas are large enough to handle 24 players' worth of spam
• Recognizing spots that the bots think are good places for sentries or snipers that likely don't match up to the spots you intended
• Spotting awkward areas of the map
• Giving yourself something to do when walking around the map (eg, changing your state of mind from "look what I made" to "ok, time to fight here, how does this area feel when I'm preoccupied?").

Things bots are HORRIBLE for:

• Seeing if your map paths are intuitive
• Seeing how strong the sentry spots are (bots struggle with sentries no matter where they are)
• Checking sniper lines (bots are either horrible snipers or insane aimbots)
• General balance (bots don't work as a team very well)
• Class balance (bots don't work as a team very well)
• Broken areas of the map due to clipping or weapon unlocks
• Checking how fun your map is

Result: There is no substitute for real players.

 

[Blade x64]

Your mind can simulate how areas and the map as a whole will play much better than any bots. You understand how classes play. You can time how long it takes to get from spawn to the objective.

If you spend enough time doing mental simulation, you can get a reasonably close prediction of how the map will play. It'll save you development time in the long run.

 

[UKCS-Alias]

Things bots are fine for:

• MvM
• Screenshot making

Anything involving players already shouldnt require bots. Even to mimic dumb players. Dumb players are dumb for other reasons, not because their code tells them to respond 1 second too late. And awkward areas, they might only exist because their nav tells them to take a weird direction. That awkward area might only be like that because its supossed to contain only 3 players instead of 10. They sometimes might find a nice sentry spot, but a player in that case would most likely do the same, and if they dont. Might it only look nice because the bots are just stupid?

Even in MvM you cant blindly rely on bots. You must tell them how to behave by giving hints.

 

[EArkham]

If you spend enough time doing mental simulation, you can get a reasonably close prediction of how the map will play. It'll save you development time in the long run.

True, but that can be a trap. Doing mental simulations can end up reinforcing bad choices, or creating areas that have only one strategy to deal with. I spend 75% of my time in Hammer flying around and asking myself what any particular class should or would do in a given area, and I have to make a conscious effort to step back from my assumptions once I get a demo/gameday feedback and see that what I would do in a situation doesn't necessarily translate to what most (or even some) people would do in that situation.

If a bot builds a sentry in a spot that seems bizarre, you should ask yourself why the bot thought that area was good. Examine it and compare to your intended sentry spots. It's about challenging your viewpoints, not looking for new sentry spots.

Again, better to have this tested with actual players.

And awkward areas, they might only exist because their nav tells them to take a weird direction. That awkward area might only be like that because its supossed to contain only 3 players instead of 10.

That's not what I meant by "awkward areas." I mean areas that feel fine when you walk through them and looked fine scale-wise in Hammer, but when you see what's happening there, when you see the various classes moving through it, it looks clumsy and poorly designed. Just seeing bodies in motion in the area can change your perspective compared to seeing a scale reference in Hammer in the same spot.

By the way, I don't count MvM as "testing with bots". You're not trying to simulate players in that mode because the bots are a feature there, not a method.

In response to both, you should note that I said very clearly player testing is better in every way at the end of the post. But for someone who maybe can't get in a game day or impromptu, and is still very early in the design process, you can make use of bots as long as it's only for very specific and limited testing.

It's a tool, like anything else, but one that is not as useful as others.

 

[LeSwordfish]

If a bot builds a sentry in a spot that seems bizarre, you should ask yourself why the bot thought that area was good. Examine it and compare to your intended sentry spots. It's about challenging your viewpoints, not looking for new sentry spots.

I disagree: If a player builds a sentry in a spot that seems bizarre, you should ask yourself why the player thought that area was good. If a bot does it you CAN do that, but its likely the answer is "because the bot is broken."

 

[Blade x64]

I agree that player testing is far superior, but the testing that we get here doesn't go as deep as I'd like it to. Learning how players adapt to an unfamiliar layout is important, but so is how a team of experienced players play on a map they know well.

The former is easily testable, the latter not so much. I design the best I can for both, but I must rely on simulation for that depth for the first half of development. I may even be unable to get that depth testing until the RC stage.