chickenm4n
L1: Registered
- Nov 27, 2009
- 49
- 6
the judginator?
Major Mapping Contest #4
- Thread starter YM
- Start date
- Status
the judginator?
And here I had hoped to get through this contest w/out learning anything new 
Something else to keep in mind is that if we do full scoring & judges review for all 26 maps, we're looking at 2, even 3&1/2 months of extra time needed. On the other hand, the point about authors getting eliminated without even a review for all their hard work, is also valid. I'll need to give this some more thought, and talk over w/ the other judges.
Something else to keep in mind is that if we do full scoring & judges review for all 26 maps, we're looking at 2, even 3&1/2 months of extra time needed. On the other hand, the point about authors getting eliminated without even a review for all their hard work, is also valid. I'll need to give this some more thought, and talk over w/ the other judges.
Just a note to Mangy first: When I first met you, you came off as an arrogant, know-it-all teenager. You still do kind of come off like an arrogant know-it-all, but it seems to be like me and my sarcasm. Since I've known you for longer, you aren't as much of an arrogant know-it-all as you used to be, but you did come off as one first.
I think asking us to do at least 13 maps is fine. But I think asking the judges to do detailed reviews of twenty six is not fine. I'll give you an analogy in italics with what we could do in normal face.
There are twenty six entries in an art contest. The first round of judging is done strictly by the public. All the art is hung up at the local mall, and people who see it can write down which one they like the most.
There are twenty six entries in a CTF contest. The first round of judging is done strictly by the public. All the maps are played in the servers for a week and a half in a random rotation, and people then vote (in a poll on the forums) for which map they like the most (May be reversed to which maps they don't like, or may be five maps they like, or whatever).
Once the votes are tallied, the eleven pieces of art that get the least amount of votes are dropped.
Once the voting has ended (maybe a week?), the eleven maps with the least amount of votes get dropped.
The final round of judging starts. The public may rate each piece of art on a scale of 1 to 7 on a couple different categories, but must rate at least 75% of the total pieces for any of their votes to count. Their votes are worth 40% of the total outcome. A panel of judges who are well taught in the field of art then rate each piece of art on the same categories, but their votes are worth 60% of the total outcome.
The final round of judging starts. The public may rate each map on a scale of 1 to 7 on four categories, but must rate 75% of the total maps for their votes to count (That's 11 maps). Their votes are worth 40% (unless the rules on the OP had different numbers...). A panel of judges who are well taught in the fields of Competitive play or Level design then rate each map on the same categories, but their votes are worth 60% (again, if the OP said differently...)
Averages are then figured out and the one with the highest average vote wins.
Averages are then figured out and the one with the highest average vote wins.
But then again, I would love tog o with my original idea a few pages ago and just make everybody say a sentence or two about why they think a map should or should not make it through prelims as they vote on which eleven to drop (or however many). That way it proves that they played each map.
I think asking us to do at least 13 maps is fine. But I think asking the judges to do detailed reviews of twenty six is not fine. I'll give you an analogy in italics with what we could do in normal face.
There are twenty six entries in an art contest. The first round of judging is done strictly by the public. All the art is hung up at the local mall, and people who see it can write down which one they like the most.
There are twenty six entries in a CTF contest. The first round of judging is done strictly by the public. All the maps are played in the servers for a week and a half in a random rotation, and people then vote (in a poll on the forums) for which map they like the most (May be reversed to which maps they don't like, or may be five maps they like, or whatever).
Once the votes are tallied, the eleven pieces of art that get the least amount of votes are dropped.
Once the voting has ended (maybe a week?), the eleven maps with the least amount of votes get dropped.
The final round of judging starts. The public may rate each piece of art on a scale of 1 to 7 on a couple different categories, but must rate at least 75% of the total pieces for any of their votes to count. Their votes are worth 40% of the total outcome. A panel of judges who are well taught in the field of art then rate each piece of art on the same categories, but their votes are worth 60% of the total outcome.
The final round of judging starts. The public may rate each map on a scale of 1 to 7 on four categories, but must rate 75% of the total maps for their votes to count (That's 11 maps). Their votes are worth 40% (unless the rules on the OP had different numbers...). A panel of judges who are well taught in the fields of Competitive play or Level design then rate each map on the same categories, but their votes are worth 60% (again, if the OP said differently...)
Averages are then figured out and the one with the highest average vote wins.
Averages are then figured out and the one with the highest average vote wins.
But then again, I would love tog o with my original idea a few pages ago and just make everybody say a sentence or two about why they think a map should or should not make it through prelims as they vote on which eleven to drop (or however many). That way it proves that they played each map.
Geal
L4: Comfortable Member
- Aug 16, 2009
- 162
- 56
That's pretty cold isn't it? Besides, aren't the people who get eliminated the ones who really NEED the feedback, so they know how to improve? Otherwise it's essentially just saying "oh you're a bad mapper, so we won't even bother with you."
Really the only way this could work would be if people voted on their least favorite maps. I would say vote on which 15 they think should stay, but at that point it'd be easier to vote on which ones they didn't like. Also, only a week or two seems like far too short amount of time for this, maybe a month would be more appropriate? Immortal-D is mentioning extending the judging time by as much as 2-3+ months. An extra month of preliminary judging wouldn't be out of the question, I think.
Last edited:
I would rather pull my admittedly bad map out of the contest, than delay the next one by three months, and not be able to show off my talent then.
- Nov 12, 2007
- 1,128
- 746
@littleedge: The problems with the logistics and accuracy of a preliminary vote are not really any different between public and judges. In either case, there's not enough time to do a full evaluation of the maps to rate them. Even at 20 minutes a map, that's almost 9 hours of playtime just to play them all once.
I would actually rather the judges make that call, because when we have members making ridiculous claims like "12v12 can be competitive" and "the competitive side of this contest has been a farce from the beginning", I have no faith that their public ratings won't be biased towards maps that play great in 12v12 but might be crap in 6v6, particularly if we go with your idea of putting them on the normal server for a week as a means of evaluation.
Let me make this analogy, because it really happened:
There are a few thousand entries into an art contest that spans many different kinds of media. This contest is to create creatures in a particular style to fit a particular movie, and the final judge will be picked by the director of that movie. Rather than qualified judges picking the finalists, a handful of unassociated people choose the 25 finalists. The resulting finalists are almost exclusively from the first few hundred entrants (even though there are thousands, likely because the choosers ran into serious time constraints), and many of them look nothing like the art direction in the movie. A few of these are even from the same artist, even though there were thousands of participants.
The movie in question was 9, and it was a Deviant Art contest. About half the finalists were an insult to people who had seen the movie, done the art research, and submitted something in the last day or two, because it was the unfitting "oh hey this is pretty!" entries that were posted in the first few days (before the movie had even opened in theaters) that made it instead.
So, given that I speak from experience about how frustrating and insulting that kind of thing is, I really don't want to see whoever is responsible for deciding the finalists to do so based off of anything but 6v6 gameplay.
I would actually rather the judges make that call, because when we have members making ridiculous claims like "12v12 can be competitive" and "the competitive side of this contest has been a farce from the beginning", I have no faith that their public ratings won't be biased towards maps that play great in 12v12 but might be crap in 6v6, particularly if we go with your idea of putting them on the normal server for a week as a means of evaluation.
Let me make this analogy, because it really happened:
There are a few thousand entries into an art contest that spans many different kinds of media. This contest is to create creatures in a particular style to fit a particular movie, and the final judge will be picked by the director of that movie. Rather than qualified judges picking the finalists, a handful of unassociated people choose the 25 finalists. The resulting finalists are almost exclusively from the first few hundred entrants (even though there are thousands, likely because the choosers ran into serious time constraints), and many of them look nothing like the art direction in the movie. A few of these are even from the same artist, even though there were thousands of participants.
The movie in question was 9, and it was a Deviant Art contest. About half the finalists were an insult to people who had seen the movie, done the art research, and submitted something in the last day or two, because it was the unfitting "oh hey this is pretty!" entries that were posted in the first few days (before the movie had even opened in theaters) that made it instead.
So, given that I speak from experience about how frustrating and insulting that kind of thing is, I really don't want to see whoever is responsible for deciding the finalists to do so based off of anything but 6v6 gameplay.
Flame
aa
- Jul 19, 2009
- 368
- 865
And here I had hoped to get through this contest w/out learning anything new
three and a half months?
if the next contest isnt running by mid-late june we did it wrong
Ezekel
L11: Posh Member
- Dec 16, 2008
- 818
- 245
maps that receive significantly more votes (of any value... so lots of low votes or lots of high votes or a mixture of itor whatever) could perhaps be made to appear less ofter on the rotation?
that way maps with less exposure will be played more, thus bringing about a balance once more.
else: after a map has received X number of votings, it cannot receive anymore till the average number of votes for all maps is 75% of the number that it has received.
In the mean time, Let me direct everyones attention to this: http://forums.tf2maps.net/showthread.php?t=12665
I was told pages upon pages ago that maps would be judged in 6v6. I'm happy with that answer for now. I'm gonna run some pugs, and as I see it. we have atleast 26 contest mappers, and a bunch of other talented players around that can participate. We only need 12 for each game, this is gonna be easy!
I was told pages upon pages ago that maps would be judged in 6v6. I'm happy with that answer for now. I'm gonna run some pugs, and as I see it. we have atleast 26 contest mappers, and a bunch of other talented players around that can participate. We only need 12 for each game, this is gonna be easy!
- Apr 19, 2009
- 4,460
- 1,722
Yeah about that...
Honestly, I just want to play the competition maps in their intended environment for fun. I'm not a judge, I will care about the details of the scoring and such later, when it's announced how it's gonna be handled.
Now you can either join me in having fun, or sit here and not have fun grumbling over numbers.
Now you can either join me in having fun, or sit here and not have fun grumbling over numbers.
Who are the judges?
not sure how many judges there is but I think it'd be wise to split them up to do the preliminaries. say you make 4 groups, each responsible for 6.5 maps they look at the maps and quickly evaluate them, then everyone regroups to discuss the maps they played, look at some demos and choose which maps move on to the next step. of course some will argue that you can't make 4 equal/fair teams. but imo I couldn't care less, I'd prefer to see the judges being able to concentrate on the good maps and give an accurate judging THERE.
of course the judges will be giving feedback and say why certain map didn't make it.
-everyone gets what he wanted
-preliminaries gets an adequate judging (not perfect, adequate. which is fair enough for preliminaries)
-judges can concentrate on their top 10 or 12 or w/e they choose and provide an accurate judging
of course the judges will be giving feedback and say why certain map didn't make it.
-everyone gets what he wanted
-preliminaries gets an adequate judging (not perfect, adequate. which is fair enough for preliminaries)
-judges can concentrate on their top 10 or 12 or w/e they choose and provide an accurate judging
You could have just asked. I already chatted with YM. Any fixes have been made by the authors, and we agreed upon the preliminary measures. You guys really don't have much faith in me anymore
- Status