Forum: Author Hangout

The design flaw means that first round opponents of the high scorers will have artificially low scores etc.

How do you call that a design flaw?

Rounds 1-3 are all match ups with in one of several categories. IIRC: Classic Detectives, Modern Detectives, International Detectives, and Next Generation.

Yes, within any one person's rankings, the first round losers will score very low. However there's no particular reason to think that wouldn't balance out over many different people's rankings.

Whoever is the first round opponent of the top scorer will get a very low score when they could possibly have been the second best detective if given a fair chance.

Your assumption of bias only holds if the top scorer has a huge margin over 2nd place. If it's more even, you can't assume those who put second place over all on top didn't rank the over all leader's first round match up against the over all leader.

Since you have access to the overall scores, could you list them alongside each competitors first round opponent? It should then be clear.

I only had access because I went through it. The only way I know of to get back to that would be to go through it again. I might do that, but no matter the result, that doesn't necessarily prove what you are claiming.

Different first round match ups wouldn't necessarily change the final results.

Okay, I've had a look at the data. I don't see a systemic bias over all from the first round match ups.

While the first round match ups of the top 3 seemingly support your claim at first glance (1 v 29. 2 v 31, 3 v 26), you then have ranks 4 and 5 whose first round opponents also finished in the top 10 (4 v 8 and 5 v 9).

Hercule Poirot (4th) 1st round vs Miss Marple (8th)

Vera Stanhope (5th) 1st round vs Maisie Dobbs (9th)

Also, one first round match up where both finished in the bottom 5.

Coffin Ed Johnson and Grave Digger Jones (28th) vs Alan Grant (32nd)

The evidence is quite clear. Gamache and Holmes were so far ahead of the rest in terms of score that their first round opponents had to be near the bottom - actually 29th and 31st.

Of course that might happen no matter who Gamache and Holmes were paired against.

And it is not in any way evidence their their first round opponents would have performed better if matched against someone else.

This is fairly common in championship tournaments in team sports. For post season championship tournaments very frequently the team with the best record gets paired against the team that just barely qualified for post season play.

Yes, but that's the way such single elimination bracket tournaments work for championships in team sports, or chess, or anything else.

For round 1, they put the team with the best record against the qualifying team with the worst record in the regular season. They never put the #1 team against the #2 team in round one.

Ultimately your problem is not really with the design of the single elimination bracket tournament format, but with how the detectives are seeded for round 1.

This is not a design flaw. It is how all single elimination bracket tournaments work. It doesn't matter if it's sports, chess, poker or this.

Someone has to go up against #1 in the first round and whoever that is will be first out unless they pull off an upset.

1. That's still a seeding even if it is arbitrary. Try investigating how they do the seeding in the National NCAA invitational tournaments for post season college play in the US. A big part of the seeding is to my understanding subjective assessments by a select group of sports reporters selected by the NCAA. Even if the round 1 bracket assignment had been determined by a random number generator, that would still be a seeding as that term is used with single elimination bracket tournaments.

2. No. I've been through the whole thing all the way to the end more than once. There are four round 1 brackets in Each sub category (these are not genres in the traditional sense). you don't get to inter category brackets until round 4.

Round two puts for each category winner of bracket 1 against winner of bracket 2 and winner of bracket 3 against winner of bracket 4.

Round 3 puts the surviving pair from each category against each other.

Round 4 puts the winner of one category against the winner of another. Without going back to check, IIRC, Classic Sleuth V Modern Detective and International V Next Generation.

Round 5 is just the final 2 to select your over all winner.

But there's no seeding, other than the designer picking their eight best in each sub-category and trying initially to match them like-for-like.

But that is still a seeding.

And every time Holmes is advanced from the 'classic detectives' sub-category, Poirot and Marple can at best score 3. Without that built-in bias, it's likely they would have scored higher.

Yes, but that's inherent to the single elimination bracket tournament structure that is used for a lot of competitive activities.

Without changing the base structure, anything you try to do to get Poirot or Marple out from under Holmes' shadow will put another detective under it. Again, it doesn't matter how you seed the initial brackets. Someone ends up with the short stick.

The only way to eliminate that bias would be to switch to a double elimination bracket setup.

https://en.wikipedia.org/wiki/Double-elimination_tournament

Each person playing the tournament could have been presented with randomised sets of detective pairings. Then relative overall positions in the points table would have had meaning.

I think a valid argument can be made that with random pairings, the individual results are no longer comparable and the over all results would have even less meaning.

I think a valid argument can be made that with random pairings, the individual results are no longer comparable and the over all results would have even less meaning.

You stated in an earlier post;

For round 1, they put the team with the best record against the qualifying team with the worst record in the regular season. They never put the #1 team against the #2 team in round one.

By setting the best against the worst in round one, how does that make the result more meaningful?

Wouldn't putting the No1 v No2, No3 v No4, etc be a less biased and thus more meaningful result?

Granted far more high seeded players/teams would be out earlier, but isn't that fairer than pitting the best against the worst in round one?

The again, maybe 'fair' isn't the prime or even a high consideration?

Personally I enjoy competitions where the players/teams are close in ability far more than watching the farce of a 'walkover'.

By setting the best against the worst in round one, how does that make the result more meaningful?

It's not about specifically setting the best against the worst, but having the same round one match ups for all users.

If they did one randomized pass at the round 1 brackets that was then used by everyone running the ranking exercise that would be fine.

But randomizing the round 1 brackets for each individual users makes the aggregate results less meaningful because the individual iterations aren't starting with equal conditions.

Wouldn't putting the No1 v No2, No3 v No4, etc be a less biased and thus more meaningful result?

Arguably that's even more biased, because essentially the whole tournament gets decided in round 1.

That really only works well if all the teams and/or competitors are evenly matched.

Either #1 knocks out #2 right away and then has weaker opponents for the rest tournament. Or #2 gets to knock the only competitor with a high chance of beating them out in round 1.

Yes, the way you suggest, all the round 1 brackets are evenly matched, but then the winner of bracket 1 faces weaker and weaker opponents in each subsequent round.

Assume 4 brackets in round 1 and 3 rounds. Assume the higher ranked team wins each bracket.

Round 1

Bracket 1: #1 v #2

Bracket 2: #3 v #4

Bracket 3: #5 v #6

Bracket 4: #7 v #8

Round 2

Bracket 5: #1 v #3

Bracket 6: #5 v #7

Round 3

Bracket 7 #1 v #5

Without a whole string of major upsets #1 is almost guaranteed to win the entire tournament.

What is generally done with in sports is to arrange the brackets so that the #1 and #2 teams can't meet before the final match.

More like:

Bracket 1: #1 v #8

Bracket 2: #3 v #6 Bracket 5: W1 v W2 Bracket 7: W5 v W6

Bracket 3: #4 v #5 Bracket 6: W3 v W4

Bracket 4: #2 v #7

Barring upsets, #1 and # 2 will each face progressively stronger opponents and will meet each other in the final round.

Without a whole string of major upsets #1 is almost guaranteed to win the entire tournament.

Barring upsets, #1 and # 2 will each face progressively stronger opponents and will meet each other in the final round.

In both of your arguments the #1 team is in the final round.

The fact is that the #1 team isn't always in the final, upsets happen, in your version they are rare because the draw is designed to ensure the top teams get to the semi-finals.

What is generally done with in sports is to arrange the brackets so that the #1 and #2 teams can't meet before the final match.

Which a more acceptable way of saying the competition is rigged to weed out the lower ranked teams early and get higher ranked teams more games, thus more exposure and of course more money.

Cynical. But true.

The fact is that the #1 team isn't always in the final, upsets happen, in your version they are rare because the draw is designed to ensure the top teams get to the semi-finals.

Yes upsets do happen, But like I explained, in your arrangement, upsets become progressively less likely in subsequent rounds because the match ups in subsequent rounds become necessarily less even as the tournament progresses.

A random seeding of round 1 brackets might give a less biased result. but your idea of ensuring all the round 1 brackets are as even as possible doesn't do it.

To demonstrate I will repeat the previous exercise assuming all round 1 matches are upsets.

1 v 2
3 v 4 : 2 v 4
5 v 6 : 6 v 8 : (2 or 4) v (6 or 8)
7 v 8

Upsets or not, it guarantees that the final round will be the least even match up of the tournament.

Upsets or not, it guarantees that the final round will be the least even match up of the tournament.

It also guarantees more close matches in the early rounds and fewer high ranking teams in the later rounds. Not what the owners want to see.

Using only 8 starting teams, thus fewer rounds isn't realistic, or does it allow for sufficient rounds to really highlight the differences in possible permutations.

Neither extreme is entirely fair or unbiased, so claiming one is "more meaningful" than the other just isn't true.

so claiming one is "more meaningful" than the other just isn't true.

Which is a straw man because no one claimed that.

What the link In the OP is doing is ranking fictional detectives by having many users each simulate a single elimination tournament between the detectives by picking winners for each bracket and then aggregating the results of all the simulations.

The argument between me and AJ is not about this method of seeding the 1st round brackets vs that method.

AJ wants each iteration of the simulation to start with unique conditions by having the brackets randomly seeded individually for each user. He thinks that will make the aggregated results more meaningful.

My position is that it doesn't particularly matter how the initial bracket assignments are done, but having unique starting conditions (bracket seeding) for each iteration of the simulation makes it harder to compare results across iterations and thus makes the aggregated results less meaningful/significant.

I would be fine with a random seeding, if it was done once and all iterations of the simulation start with identical conditions.

Which is a straw man because no one claimed that.

Actually, you claimed that..!!

But randomizing the round 1 brackets for each individual users makes the aggregate results less meaningful because the individual iterations aren't starting with equal conditions.

I think a valid argument can be made that with random pairings, the individual results are no longer comparable and the over all results would have even less meaning.

My position is that it doesn't particularly matter how the initial bracket assignments are done, but having unique starting conditions (bracket seeding) for each iteration of the simulation make it harder to compare results across iterations and thus makes the aggregated results less meaningful/significant.

By stating the alternative is "less meaningful", you are claiming your option is more meaningful.

By stating the alternative is "less meaningful", you are claiming your option is more meaningful.

Yes, but my option is agnostic about how the brackets are seeded. It's not the actual bracket seeding that's at issue.

AJs suggested method pretty much requires random seeding, but my doesn't rule out random seeding, I just rule out doing the seeding individually for each user.

You are emphasizing the wrong part.

Going back to what I said previously:

But randomizing the round 1 brackets for each individual users makes the aggregate results less meaningful because the individual iterations aren't starting with equal conditions.

I think a valid argument can be made that with random pairings, the individual results are no longer comparable and the over all results would have even less meaning.

My position is that it doesn't particularly matter how the initial bracket assignments are done, but having unique starting conditions (bracket seeding) for each iteration of the simulation make it harder to compare results across iterations and thus makes the aggregated results less meaningful/significant.

The dispute is not about how the brackets get assigned/seeded, but when.

My position is that it doesn't particularly matter how the initial bracket assignments are done, but having unique starting conditions (bracket seeding) for each iteration of the simulation makes it harder to compare results across iterations and thus makes the aggregated results less meaningful/significant.

Is the point of the tournament to enable the easy comparison, or is it to ascertain the most popular detective across a wide range of readers? You have previously explained how the #1 author is most likely to win, why would repeated randomising change that? After all, you also explained why pitting closer seeded authors results in more uneven matches in later rounds, so again, given your explanations, why should repeated randomising matter?

ps

AJ. Don't get used to it. :)

Is the point of the tournament to enable the comparison, or is it to ascertain the most popular detective across a wide range of readers?

Well that depends. Is the purpose just to determine the #1 most popular detective or is it to rank the full set of detectives.

My personal opinion is that if the purpose is the later and not the former, yes, comparability of individual runs matters.

And I have nowhere put that up as anything more than my personal opinion.

As I see it variations in the overall ranking with individually randomized brackets may be affected as much by variation in the initial bracket assignments as it is by reader preferences.

The final aggregate results are driven by two variables rather than just one. It's not that likely to have a strong impact who is at the very top and who is at the very bottom, but it would likely make a big difference in the middle.

Well that depends. Is the purpose just to determine the #1 most popular detective or is it to rank the full set of detectives.

My personal opinion is that if the purpose is the later and not the former, yes, comparability of individual runs matters.

If the purpose is to rank them in order, then the most accurate method is to have each reader rank them in order and then compile the results.

If the purpose is to rank them in order, then the most accurate method is to have each reader rank them in order and then compile the results.

Agreed.

That said, AJ was all peeved over how the bracket assignments were done.

If you are going to stick with the simulated single elimination tournament then individually randomizing the bracket assignments is just adding an extra variable lowering the signal/noise ratio.

How dare you introduce common sense into the argument.

It's a female thing, you wouldn't understand...

:)

By having every user faced with the same initial set of pairings, biases are inevitable.

By not having every user faced with the same initial set of pairings, you are adding an extra variable and therefore introduce noise to whatever signal you are looking for.

adding an extra variable and therefore introduce noise to whatever signal you are looking for.

Sometimes adding noise is the best way to get better accuracy on the results - in a quantised system, adding noise and averaging the result of N trials allows you to increase the (post-processed) measurement accuracy by (up to) sqrt(N), which is useful where the quantisation is coarse and the number of trials is high. Here, the quantisation is very coarse (binary), and the number of trials is potentially very large, so adding noise (randomising each quiz) should be considered. If the ultimate outcome desired is simply a 'best', then that's even more quantised and non-randomising is acceptable, and a lot less work for the programmer. If, however, an order ranking is desired, this is a lot more fine-grained than the individual trials so additional accuracy is required and the trials should be randomised and the results averaged.

I would have preferred to have an I've not heard of this one button for each, which besides removing random noise/bias in the results, also allows a direct measurement of market penetration or brand recognition.

AJ wants each iteration of the simulation to start with unique conditions by having the brackets randomly seeded individually for each user. He thinks that will make the aggregated results more meaningful.

Why respond to me when the quote is from DS..??

It's shocking that 'Cracker' wasn't in that list, though plenty of others I had never even heard of, so it must be fairly regional

Good point.

Lazeez added a plea to the Clitorides not to vote for stories you haven't read.

The same ought to apply to the tournament in question, but how many taking part in the voting will have read at least one book by every named author?

Bosch came out on top for me. Not too meaningful, even to me, because I'd never heard of nearly half the detectives in the list, and many of my favorites were not in the competition, Cadfael, for example.

And there lies the big bias in something like this. Which detectives get "invited" to the tournament in the first place.

Why not just have people list their five favorites? Probably Harry Potter would win.

If someone did that and Harry Potter did win, I'd discount the whole thing. Harry Potter was/is not a detective.

What are the ingredients to a great fictional detective?

Well, first they actually have to be a detective, and in my mind, that means solving mysteries for other people as a career or a serious hobby.

A police detective ultimately works for "the people" so that counts.

There are some 'curious' early pairings. Holmes/Marlowe comes to mind. Here you have arguably the most famous 'consulting detective' paired against arguably the most famous hardboiled detective in the first round.

Well, to be fair, there weren't a lot of lightweights in the Classic Sleuths. And why was Nancy Drew in that group?

There are some 'curious' early pairings. Holmes/Marlowe comes to mind. Here you have arguably the most famous 'consulting detective' paired against arguably the most famous hardboiled detective in the first round.

Well, to be fair, there weren't a lot of lightweights in the Classic Sleuths. And why was Nancy Drew in that group?

There are some 'curious' early pairings. Holmes/Marlowe comes to mind. Here you have arguably the most famous 'consulting detective' paired against arguably the most famous hardboiled detective in the first round.

Well, to be fair, there weren't a lot of lightweights in the Classic Sleuths. And why was Nancy Drew in that group?

Exactly the system used for the Clitorides.

Whilst the result would in actuality be the three most popular, not 'best' it would arguably be a more meaningful result.

A nice idea but then there should also be categories because every protagonist has it's own specific good and bad characteristics. I'll leave it up to the the great people here to decide on a few categories :D