Camestros Felapton

FELAPTONL

Hugo 2023 Noms: Mind the Gap

Here (after the fold) is a more accessible table for the Best Novel EPH stats and some free floating discussion of its oddness.

Place Finalist N % 9 8 7 6 5 4 3 2 1
1 Legends & Lattes 831 51% 202.35 203.56 206.9 206.9 207.78 210.45 210.45 211.45 214.28
2 Nettle & Bone 815 50% 193.5 195.75 198.92 198.92 202.58 207.5 207.5 207.67 218.17
3 Babel * 810 49% 164.93 164.93 164.93 164.93 164.93 164.93 164.93 164.93 164.93
4 Nona the Ninth 795 49% 189.48 190.61 192.11 192.11 194.86 196.78 196.78 197.28 204.53
5 The Spare Man 778 48% 184.32 185.4 186.73 186.73 188.07 194.32 194.32 195.65 201.82
6 The Kaiju Preservation Society 765 47% 173.25 173.8 175.05 175.05 176.1 178.68 178.68 178.85 183.18
7 The Daughter of Doctor Moreau 767 47% 172.16 174.16 174.75 174.75 176.75 177.25 177.25 177.25 181.08
8 Age of the Godmakers 150 9% 79.82 83.45 87.78 87.78 97 101.58 101.58 102.92
9 The Mountain in the Sea 115 7% 61.08 62.33 62.58 70.08 70.58 71.5 109.5

10 The Red Stone 113 7% 60 60 60 75 75 75


11 A Half-Built Garden 78 5% 38.78 42.28 43.2 43.2 44.33



12 We Live in Nanjing 77 5% 41.78 44.53 46.53 46.53




13 Residual Light 65 4% 29.5 29.5 29.5





14 The Prophet Machine 61 4% 32.92 33.67






15 Stories Bygone on Mars 59 4% 28.12







The top 7 finalists, as others have noted, received way more votes than expected based on the total number of nominations and past statistics from previous years. Also, the top 4 are all close to 50%. That is doubly unusual. Typically, there is a drop between each placed nominee. A second-place nominee might be half the percentage of the top-placed nominee. It is a kind of fuzzy power-law like distribution usually. Here they are tightly clustered and the drop looks more linear than usual.

The sixth and seventh places are also very close and then there is a sharp drop from 7th to 8th. Imagine, we had a more organic set of figures with a more typical drop off from first downward. For example:

Place Finalist Scenario 1
1 Legends & Lattes 231
2 Nettle & Bone 215
3 Babel 210
4 Nona the Ninth 195
5 The Spare Man 178
6 The Kaiju Preservation Society 165
7 The Daughter of Doctor Moreau 167

It’s still not entirely consistent with the past but the drops are more consistent. These figures were calculated by uniformly subtracting 600 from the original figures. Spliced with the original data for 8th+ place, you’d get a pattern like this:

Place Finalist Scenario01
1 Legends & Lattes 231
2 Nettle & Bone 215
3 Babel * 210
4 Nona the Ninth 195
5 The Spare Man 178
6 The Kaiju Preservation Society 165
7 The Daughter of Doctor Moreau 167
8 Age of the Godmakers 150
9 The Mountain in the Sea 115
10 The Red Stone 113
11 A Half-Built Garden 78
12 We Live in Nanjing 77
13 Residual Light 65
14 The Prophet Machine 61
15 Stories Bygone on Mars 59

I’m not saying there was a magic 600+ votes added, just that if you take a chunk off the top set of nominees, you end up with a pattern that looks more like a normal year. To be honest, in a “normal” year I’d have guessed The Daughter of Doctor Moreau and The Mountain in the Sea to have been close and maybe The Kaiju Preservation Society also (Scalzi has reliable fans but I don’t think the book had big support beyond that, so again, a plausible edge-of-the-ballot nominee).

There is another unusual thing. If you take the raw votes and divide it by the score in a given EPH round it should give you the average number of other nominees currently still in play that the nominee has in common. The higher the number the more the work shares voters with the other nominees still in play.

For example, here is 2022 top 15. By this point it is about an average of 2 works per ballot.

Finalist N Ratio round 147
A Desolation Called Peace Arkady Martine 242 2.1 113.8
A Master of Djinn P. Djèlí Clark 178 2.1 84.5
Project Hail Mary Andy Weir 139 1.7 79.72
She Who Became the Sun Shelley Parker-Chan 111 2.2 51.58
The Galaxy, and the Ground Within Becky Chambers 124 2.3 54.77
Light from Uncommon Stars Ryka Aoki 117 2.1 55.98
Perhaps the Stars Ada Palmer 93 1.6 57.6
The Witness for the Dead Katherine Addison 92 2.1 43.33
The Last Graduate Naomi Novik 81 2.0 41.52
Black Water Sister Zen Cho 91 2.1 42.68
Leviathan Falls James S.A. Corey 75 1.9 40.32
Shards of Earth Adrian Tchaikovsky 74 2.0 36.5
The Jasmine Throne Tasha Suri 69 2.0 33.78
When Sorrows Come Seanan McGuire 49 1.3 37.83
Machinehood S.B. Divya 57 1.6 35.08

Here is 2023

Place Finalist N Ratio 9
1 Legends & Lattes 831 4.1 202.35
2 Nettle & Bone 815 4.2 193.5
3 Babel * 810 4.9 164.93
4 Nona the Ninth 795 4.2 189.48
5 The Spare Man 778 4.2 184.32
6 The Kaiju Preservation Society 765 4.4 173.25
7 The Daughter of Doctor Moreau 767 4.5 172.16
8 Age of the Godmakers 150 1.9 79.82
9 The Mountain in the Sea 115 1.9 61.08
10 The Red Stone 113 1.9 60
11 A Half-Built Garden 78 2.0 38.78
12 We Live in Nanjing 77 1.8 41.78
13 Residual Light 65 2.2 29.5
14 The Prophet Machine 61 1.9 32.92
15 Stories Bygone on Mars 59 2.1 28.12

From 8th place downward it looks pretty usual and not just for the Chinese language finalists but also for The Mountain in the Sea. The top 7, those same finalists with whopping totals, it is 4 or nearly 5 in the case of Babel. These are very correlated ballots.

Isn’t EPH proof against such things? No, not if the correlated ballots have big numbers. In that case they win the day.

So am I saying there was a slate? Um…I’m not not saying there was a slate. OK, there could be other ways this could happen. For example, if data got duplicated by accident during cleaning or through flaws in the algorithm, you might get similar effect. Also, it would be a weird sort of slate and note that it includes Babel so it would be a seperate shenanigan from the Babel-ileligibility shenanigan and if there is one thing we do on this blog is not multiply entities unless they are cats or motives.

Could it organic? I mean people do all sorts of things but this would be a very weird degree of organic action in common for Hugo voters. The actual slates we know about from the Puppy years look more organic than this but they were smaller in scale than if this was a hypothetical slate.

Then we get to The Mountain in the Sea and Babel. I know there are people who vote for both and yet when one is eliminated it made no change to the other.

Honestly, I don’t think I can get more out of these figures. They simply do not look right to me.

ETA: See also Heather Rose Jones’s analysis here: https://alpennia.com/blog/comparison-hugo-nomination-distribution-statistics The graphs really demonstrate what I was trying to handwave at.

camestrosfelapton

, , ,

24 responses to “Hugo 2023 Noms: Mind the Gap”

  4. Thanks once again, but I would much appreciate if you could find a way not to force sparser tables to width:100% with all that whitespace. Having to scan left to right and back to see which number corresponds to which heading, is bad for usability ant the eyes.

    Like

  5. A question about the EPH rules. Are the rules unambiguous in the case of a work being ruled ineligible?

    I would have expected that if a work was ruled ineligible ballots nominating that work would be treated as ballots with one fewer nominees (i.e. as if those ballots did not nominate that work). However I see a possible ambiguity – when calculating the division between nominees of a ballot’s nominations does one use the number of nominations or the number of eligible nominations?

    What I might have expected to see with ineligible works is either they are not in the tables at all (just a footnote that they were ruled ineligible), or at the bottom of the table with nomination value “0 (x)”, where x is the number of nominations.

    The other possible ambiguity is whether eligibility restrictions are applied before or after executing the EPH algorithm. (Before, surely, but is this explicit in the rules?)

    Similar questions arise when nominations are declined.

    Like

    • EPH is always run completely, before questions of eligibility are even looked at. And you never kill something in the first round of EPH.

      The Admin first has to determine the longlist by running EPH. The Admin then goes down the list, determining what’s ineligible, and removing those from the list, then takes the remaining Top 6 as Finalists. As Finalists decline, they are crossed off, and the next one in line is notified.

      EPH is never re-run after its initial run — that would drastically change all of the results, if those entries were removed from the calculations.

      It’s not necessarily a perfect method (IIRC Tommy Arnold missed out being a Finalist in 2017 because of it) – but I do agree that things quickly become very messy if you don’t just immediately freeze the EPH results as based solely on raw ballot nominations. Imagine removing an entry after the creator declines or is ruled ineligible and re-running EPH — only to discover that one of the Finalists you’ve already notified is no longer a Finalist! Which is absolutely a possibility, which is why EPH is only run once and the ranking is never recalculated.

      Liked by 2 people

    • Heather’s excellent analysis not only highlights just how anomalous the 2023 nomination numbers are (I do love me a good graph) but also answers a question raised by Cam’s “Mind the gaps” post: is it possible to distinguish between JJ’s suggestion that a whole block of nominees had simply been excised and some more direct manipulation** – Heather’s hypothesis #3 – of the vote totals (e.g., adding 600 votes arbitrarily to the top 7 placings, not that I’m suggesting that’s what was done). I was fairly confident, based on the abnormally high correlation among the top placers that Cam noted, that it was the latter and not the former, but Heather’s analysis seems quite conclusive.

      I’m not sure who to feel worse for, the people like Paul who were simply excluded from the ballot because someone wanted them gone, or the winners, because no one is ever going to believe that these are legitimate results, and it’s quite clear that neither Dave McCarty nor anyone else connected with the Chengdu Worldcon have any intention of ever answering any of the questions raised by all this. The 2023 results are going to be permanently asterisked just like the Puppy years.

      **A massive slate would fall into this category, although as already pointed out it would be very weird to slate Babel into the nominations and then declare it ineligible.

      Liked by 2 people

      • My hypothesis was that a middle section of nominees had been whacked out of the longlist (which would cause the steep cliff effect) and that their nomination totals had been re-allocated amongst the top part of the longlist (which would further enhance the cliff effect AND cause the much-higher-than-normal percentages of the total number of nominations); this would have had the effect of moving a lot of Chinese works up the longlist far enough to appear in the top 15 — with the bonus of enabling works considered “problematic” to be disappeared from the list entirely.

        The problem with Babel, because it was so massively popular and already multi-awarded, is the Hugo Admin knew if he just removed that one completely, that people would know that the data had been doctored.

        Which in retrospect was just ludicrous thinking on his part. If he was going to doctor the stats as badly as he did, it was going to be blindingly obvious to all of us anyway.

        I mean, he had 6 fricking months to massage numbers and remove works to his heart’s content, double-check all the arithmetic, and create a document that might have looked vaguely suspicious but not obviously so. And maybe that’s what he intended, but he procrastinated so long that he ended up having to do a half-assed version of it and not having time to polish the numbers it to look plausible.

        There’s a loooooong history of half-assing things from that source, and this is just the latest and most egregious example.

        Liked by 2 people

      • Those of us who aren’t great at statistics (though any of us here are better than LC) but felt something hinky was going on appreciate the analysis. I really appreciated Heather’s graphs because you can simply look at the pretty colors and go “that clearly ain’t right”.

        I hope the permanent floating Worldcon committee has someone else lined up to run things after this fuster cluck.

        I am leaning towards (for US-locations) instituting a rule of “If you can’t legally smoke pot and be openly gay in your state, you don’t get a Worldcon”. And nowhere where there’s book-banning, either.

        Like

  7. and that their nomination totals had been re-allocated amongst the top part of the longlist

    JJ, my apologies – I somehow managed to forget that was part of your hypothesis.

    You’d think he would have realized that the kind of people who actually want to see the nomination stats each year would immediately spot how unbeliveable these are, but maybe he figured the unjustified “not eligible” removals were such blatant red flags that he wasn’t going to be able to sneak anything by anyone anyway. Or maybe he just doesn’t understand anything about statistics, and believes that’s true of everyone.

    Liked by 2 people

    • Based on the previously-mentioned looooooong history, I’m going with the “he meant to doctor them properly before release, but put it off too long and ended up half-assing it” hypothesis.

      Liked by 2 people

      • I don’t believe I’ve ever disagreed with JJ on anything of this nature, and the shaving implement of Mr. Ockham agrees too, so this is now my head canon.

        Really don’t know why Babel got so particularly shafted — it wasn’t for me at all, but lots of people did like it. Personal grudge?

        The people who should have been nominated on eligibility (like for Astounding) doesn’t make any sense either. Not sure whether to chalk that up to malice or incompetence.

        In any case, I’m glad I didn’t even bother to nominate, which I could have. It wouldn’t have made a difference, and now the Chinese government doesn’t have any extra data on me.

        (I am NOT blaming the fans from China — they can’t help their government’s repression, and they demonstrably genuinely like SF)

        Like

  8. To add a little bit to Heather’s analysis: Taking the total number of nomination ballots as 100% in each year. Best Novella 2014 – 2022: between 44% and 63% of ballots nominated in this category. Best Novella 2023: 75%Best Novelette 2014 – 2022: between 30% and 49% vs. 2023: 57%Best Short story: 36% – 61% vs. 81%Best Series: 43% – 58% vs. 76%.The difference between the previous and this year’s value looks like about 20%, ca. 350 additional nominations in all four cases. My guess for the reason would be simple addition “errors” (aka ballot stuffing) making the finalists look more impressive, but guessing is not proof. (Best Editor Long Form: 13% – 44% vs. 51%, but 44% was also an outlier. I’m not sure about this.)However, this phenomenon does NOT show up in best novel:Best Novel: 82% – 91% vs. 88%.

    Like

  10. Cam, would you be willing to poll people on what they nominated for Series? Because looking at past years, it looks to me as though what got whacked out of the middle of Series would have included Incryptid, Liaden, Dominion of the Fallen, Dresden Files, and Maradaine, at the very least.

    Like

    • Yes, I was thinking of doing that for novel but series is a good choice as well.

      I’d quite like to do a smaller category as well but BDPs are too messy, Fan Writer is too personal and many of the others people skip.

      Like

Blog at WordPress.com.