Camestros Felapton

FELAPTONL

Hugo stats: where are we today

Oh boy, it simply hasn’t got better has it.

The data does not look organic, there are confirmed (but maybe superificial) errors in the first version of the results and there are poorly explained disqualification of nominee who may have won in the final stage (or affected the outcome) and there is poor transparency/explanation from Chengdu Worldcon.

  1. there could be a whole bunch of genuine mistakes in the data e.g. I think the simplest explanation for the totals not adding up right that Marshall has documented is that the published totals have typos in them. Maybe that explains Babel’s weird stats as well. There were confirmed errors in the stats (duplicate names). If there are lots of transcription/cut-and-paste errors then we are all working off faulty information and drawing wrong conclusions about reality. The released data looks rushed and I’m not confident that it reflects the actual data. Conclusion 1: The released data has definite transcription/copy-paste errors
  2. Even so, to explain how the data look requires a lot of errors and the patterns Heather Rose Jones showed do not look like errors from publishing the data but rather errors in the data itself. The strange ratios in the EPH data for Best Novel make little sense as a typo or transcription error. Conclusion 2: As well as errors in how the data is presented, the underlying data appears odd.
  3. Maybe EPH was just run wrong. Without access to raw data it is hard to check. Still, it doesn’t look like that. Access to the raw data is unlikely. The Hugo Committee for 2023 hasn’t been transparent at the best of times but sharing data from China outwards is legally complex and sharing identified data between Worldcons is legally complex under EU law (and similar legislation) Conclusion 3: Without access to the raw data we will never know for sure what all the issues are.
  4. The simplest, hardest-to-check way to skew the results is to stuff the ballot. That could be done externally to the Hugo committee and there’s no way to tell. That would explain some of the patterns but not other issues (eg Babel’s EPH values not changing). Conclusion 4: Even with access to the raw data, statistics/auditing can only tell us that the data looks strange and that the strangeness is in the ballots themselves it can’t confirm a slate or ballot stuffing.
  5. However…one thing a Hugo committee could do if they thought (but couldn’t prove) that the nominations were being ballot stuffed is disqualify a bunch of stuff they thought was dodgy without saying why…except the stuff they did disqualify makes no sense in that way. Conclusion 5: I think the weird disqualifications are a separate issue to the other irregularities.
  6. The ballot data oddities and the disqualification oddities do not work in tandem eg Babel has weird numbers and was disqualified but all the Best Novel finalists had weird numbers and weren’t disqualified. Paul’s numbers weren’t weird and he was disqualified. Conclusion 6: There are two unrelated things going on which suggests that no single person or cohesive group did all these things.
  7. The Hugo nomination process was particularly complex this year, with works in two different languages, many new voters unfamiliar with eligibility rules, and technical/legal challenges with dealing with data across China’s internet borders. Seperate from the nomination stats, there were delays at each stage of the process and other mistakes (e.g. the release of an incorrect set of finalists). Conclusion 7: circumstances with the Worldcon itself made errors more likely but also made it likely that there would be weird (but legitimate) data that did not follow previous patterns.
  8. The process was complex but also the Hugo committee was inexperienced. Putting aside Dave McCarty’s personality, he was clearly busy with many other aspects of the convention (with all its scale and complexity) and much of the work must have had to be delegated. In particular, I suspect the data cleaning stage and eligibility checks would have been challenging (point 7). Major stuff ups with the data may have occurred such as accidental duplication of data, accidental deletion of data or accidental disordering of data. These three kinds of issues do happen when dealing with a data-heavy project (trust me – I’ve made all three kinds of errors in the past at work but luckily either I spotted it in time or somebody else did). Time pressure, inexperience, weak supervision, novel issues and unsystematic QA checks are all circumstances that will lead to substantial mistakes that poison the whole process. The solution when that happens is to go back to the raw data and start all over again…but people (in my experience) try to “fix” things instead. Conclusion 8: Cock-up and conspiracy aren’t alternate explanations. Sometimes, the conspiracy is an attempt to fix a cock-up. We can’t tell from the data but the Hugo Committee can and if this is what happened they can tell us. We also know for certain that errors of various kinds were made and have been admitted to (point 2).
  9. The weird eligibility decisions are, regardless of everything else, overt decisions made by somebody who had actual reasons for those decisions. Conclusion 9: We can’t know everything that went on but these decisions can and should be explained by the people who made them or by officials of the Chengdu Worldcon committee.
  10. I suspect there may be explosive aspects to those decisions and maybe explaining those decisions create personal or legal liability issues for officials or the Worldcon. Conclusion 10: We still need to know if only to plan to avoid such issues in the future.
  11. Conclusion 11: I do not trust the data and, as awful it is to say this, I don’t trust the results of the 2023 Hugo Awards.

camestrosfelapton

,
, ,

62 responses to “Hugo stats: where are we today”

  3. Has there been comments from any of the oddly disqualified people, other than Paul Weimer? In particular R. F. Kuang?

    I think I’ve noticed that at least some previous Worldcons have used the same website system to run the voting. I didn’t vote this year, so I don’t know what the site looked like – did they use their own voting solution, or an existing one?

    Liked by 1 person

    • The data is not out that long and today is sunday. I would exspect that some of the disqualified people are not aware of it/have no time today or want to wait before the answer.

      Like

    • Yes, R. F. Kuang did post about this issue on Bluesky.

      https://bsky.app/profile/rfkuang.bsky.social/post/3kjljukyow42o

      Quoting it in full here, which I hope is acceptable:

      “I initially planned to say nothing about Babel’s inexplicable disqualification from the Hugo Awards. But I believe that these cases thrive on ambiguities, the lingering question marks, the answers that aren’t answers. I wish to clarify that no reason for Babel’s ineligibility was given to me or my team. I did not decline a nomination, as no nomination was offered.

      “Until one is provided that explains why the book was eligible for the Nebula and Locus awards, which it won, and not the Hugos, I assume this was a matter of undesirability rather than ineligibility. Excluding “undesirable” work is not only embarrassing for all involved parties, but renders the entire process and organization illegitimate. Pity.

      “That’s all from me. I have books to write.”

      Like

  5. Exposing my ignorance for all to marvel at, I wonder if the conjecture that the Chinese government may have caused at least some of this issue is justified? It seems to have taken a front seat without any evidence other than an awkward clause in the short, uninformative statement about why the works were disqualified. Issues like this, involving China, are tough for me to parse because I don’t know all that much about just how repressive the government is. We get a ton of anti-China propaganda here in the US and between that and simple racism it’s hard to know what’s real and what’s Red Dawn.

    Liked by 3 people

    • I have to admit that I am a bit confused by this as well. Obviously, the PRC government is authoritarian and certainly engaged in a lot of censorship, but it also seems like Babel was published in China as well. This seems like a bit of a contradiction. It’s certainly possible that this is one of a long line of decisions on a part of an authoritarian government that isn’t particularly consistent or that different authorities made different arbitrary decisions, but its still kind of stands out.

      Like

      • There was a very plausible (to me) hypothesis in comments, I think in File 770 and not here. To summarise my understanding (and, sorry, not quote and attribute):1) Babel itself is anti-colonialist and explicitly condemns the opium wars – so there was no principal reason not to publish it in China, in October 2023.

        2) However, Kuang is outspokenly critical of the Communist regime (and so, reportedly, were her previous books in some respect and degree).

        3) She can also speak Chinese, directly to the audience without the approved interpreters having any chance to intervene.So, either to punish her for 2) and/or proactively prevent 3), the votes were suppressed. No nomination – no problem!

        Liked by 6 people

        • I had not seen that in File 770 (and its possible that I missed it, a lot of conversations go on there), but that does sound like a pretty plausible argument. Thanks!

          Liked by 1 person

          • You could have done that, but it really would have stuck out given the traditions of the awards and would made people suspicious and angry at the time of the awards, tarnishing the publicity that the government wanted. This may have broken the rules, but the awards went smoothly and the conflict is considerably separated from the awards. (This is obviously pure speculation.)

            Liked by 2 people

        • Babel is also 1) about a student revolution, 2) about the importance of allowing people to know and use their indigenous languages, rather than forcing them to assimilate, and 3) the (magical) advantages of having access to a multitude of languages compared to a monolingual society that becomes increasingly moribund and hidebound.

          Liked by 2 people

          • Though as the bad guys are British, there is plausible deniability (and my experience of government censors is that they often lack the imagination to realise that a book officially about something Not Them is in fact about Them).

            Liked by 1 person

            • Conversely they’ll sometimes see stuff that is Not About Them as Totally About Them because they can’t conceive things aren’t about them, either for or against.

              Like

              • I don’t know about government censors, but my experience of individuals who see some things as Totally About Them is that they see it if it lets them claim to be a victim of someone tarnishing them personally, but will also unironically cite things that are absolutely about them or people like them as if they were criticizing the people/positions they hate. (Like an utterly toxic personality posting comments about how to identify and remove toxic people from your life…) I wonder if that applies to censors, too, and their apparent contradictions?

                Like

                • I’ve read several accounts of someone writing a scathing depiction of someone in their orbit and the subject doesn’t see it: Walter Winchell is clearly the target of “Sweet Smell of Success” but he was convinced it was one of his rival columnists.

                  Liked by 1 person

      • Indeed and an international award giving a book a prize is not the same as publishing a book. Now, I can see, at a stretch, the government being worried about speeches at the ceremony but there are other ways of handling that.

        Liked by 1 person

    • I’m still sceptical of the actual government in Beijing caring at all but maybe it was somebody local or people within Worldcon trying to apply arbitary regulations over zealously? Maybe I should write to the Chinese Embassey in Canberra: “Hi guys, this may sound weird but did your government mess around with the Hugo Awards?” 🙂

      Liked by 2 people

      • Even a much smaller state than China would break down completely if every single decisions had to come from a high-level official in Beijing – but I’d say that some kind of regional official is still “the government”.

        My experience with authoritarian regimes is mostly from fiction, but (like most people, I assume) I have some experience with bureaucracies in general – and I can absolutely imagine both a local party official breathing down the neck of people in the Worldcon committee, and people in the committee walking on eggshells and being overzealous in an attempt to not get a party official breathing down their neck. And that can lead to behavior that aren’t necessarily easy to interpret from outside.

        Liked by 3 people

        • I doubt people get to be regional officials without some kind of purity test. They may be “more X than national X” to CYA and aspire to upper levels. Or just drunk on power/zealotry, which happens with petty officials in any government.

          Liked by 1 person

    • Given that the two most prominent books that were deemed ineligible were by well known published authors, had very high numbers of votes that placed them as finalists, and most importantly, were both written by Chinese authors who were born in Mainland China and have included subject matter relating to China in their works and have publicly made criticisms of the CCP, it is extremely likely the disqualifications were due to interference by the party. Otherwise it’s just too coincidental given the disqualifications are otherwise completely inexplicable.

      Given that the Hugo awards are THE most famous and prominent SF awards and Worldcon is an international event, the central goverment would definitely have been involved in its operation etc.

      Its important to remember that the CCP is extremely petty. They recently pressured a low-profile under 20s international Volleyball championship to be cancelled because it was held in Taiwan. The surname Xi is censored everywhere on the Chinese internet even though it is a common last name. They would definitely push for books to be excluded the authors had written things in them or previous works that the they didn’t like. Or it could have been to punish the authors for criticising the Party in the past.

      Of course the books may have been disqualified for another reason, but if so, why haven’t the organisers fronted up and explained the situation?

      Liked by 2 people

        • Neil Gaiman has done a good bit of human rights work in his spare time. He also has a habit of showing up places where he has won awards, and giving speeches.

          Like

        • If someone was making politically motivated disqualifications they might also add some random disqualifications to muddy the waters.

          Liked by 1 person

          • As someone else pointed out, the Sandman episode (and in fact the entire series) features deities, ghosts, supernatural concepts, the occult, superstitions, and magical powers — all of which are on the PRC’s list of forbidden elements.

            Like

  6. Yes, I’m pretty much at 11 on your list too. This is much worse than 2021’s horrible stats which simply seemed to be not knowing how to put the data into a table correctly.

    Liked by 4 people

  7. I would say that there is enough circumstantial evidence now to proceed to the explicit conclusion that the bulk of disqualifications are people/works that did, or just in the opinion of someone high enough in the concom could conceivably do (no, today’s PRC is not a monolithic, perfectly centralised panopticon), something that the Chinese censors and wider authorities frown upon, also called “the rules we must follow”. (Sorry, I am very little acquainted with Paul Weimer’s work, but I recall seeing at least one hypothesis that seemed plausible at first sight.)I also have some doubts about #3 – or possibly not; by now I suppose it is quite conceivable that they would refuse to adopt and run the counting software that the previous Worldcons used, and instead develop one of their own / emulate it in a spreadsheet with a lot of manual work etc. (and include some unwanted bugs).

    Liked by 3 people

    • Jan Vaněk jr.: <em> I suppose it is quite conceivable that they would refuse to adopt and run the counting software that the previous Worldcons used, and instead develop one of their own / emulate it in a spreadsheet with a lot of manual work etc. (and include some unwanted bugs).</em>

      I think you’ve gotten a mistaken impression. Yes, the main convention committee being taken over by representatives for the Chinese government, the change of location, and the change of date were being done by Chinese reps.

      But the Hugo Award division operates independently of the main committee. Administration of the Hugo Awards was done by a non-Chinese SMOF. Yes, the website listed a Chinese committee member as well — but the person making the decisions and creating the POS statistics document we’ve just been given was not Chinese. He was no doubt taking advice from Chinese committee members on which works and creators should not be allowed to appear on the Hugo ballot, but make no mistake, he — and not the Chinese members — is the one who made the executive decisions.

      Liked by 3 people

      • I do not quite understand where you think me mistaken. I am roughly aware of how Worldcon concoms and divisions operate. But I was indeed under the impression that the final calculations are done onsite — is it not still possible to use snail-mail, and some stalwarts keeping to it? In which case, both the problems with getting the data out of China, as mentioned originally, and other reasons might cause Chengdu to implement their own EPH software, which might be conceivably be buggy.

        But if I am wrong somewhere in these (so would that mean that Dave McCarty did the calculations at his home, sending just the output tables to the appropriate other divisions?), we revert to my original point that the conclusion #3 “Maybe EPH was just run wrong.” does not seem likely then.

        Liked by 1 person

        • My point is that it isn’t anyone Chinese who has done this. It’s McCarty. No doubt he ran all the works and creators past some Chinese committee members, but make no mistake, he’s the one who’s run EPH and decided what to do with the results. Nobody “made” him do this. As Ada Palmer has pointed out, they didn’t have to; this was a choice he made based on either external pressure or what he believed the external pressure would be.

          Liked by 1 person

        • (And, I will add, that doesn’t let him off the hook for ultimate responsibility. He chose to promote the Chengdu bid despite knowing that the Hugos would have to doctor the results. He chose to be Hugo Admin despite knowing that he would have to be the one to doctor the results.)

          Like

  8. I was going to ask if the EPH process has made any potential shadiness more obvious than it would be without EPH, but there were so many issues in this case that extra help wasn’t really needed, right?

    Liked by 1 person

    • Oh it definitely highlights the issue with the gap. My suspicion is that what we are seeing in the stats is an attempt to overcome EPH using ballots.

      The ratios imply each of the top 7 works appear with, on average, 4 others of the top 7 works. Babel is closer to 5, which explains why its numbers don’t change – it appears on no ballots that get eliminated. Except we KNOW that in the real ballot people did include it in ballots without the top 5. QED – it ain’t right.

      I have to also say that while there are many issues here all of the major ones apply to Babel, the gap, weird EPH figures and unexplained disqualification.

      Liked by 6 people

  10. I tend to go with an Occam’s Razor approach here. I don’t think the Chinese government would particularly interest itself in manipulating data at a SFF convention award or with the convention itself at all, unless there was someone there they didn’t want to be or an incident at the convention, etc., none of which we have heard about. And if they did, I don’t think they’d then throw most of the Chinese authors off the finalist lists.

    I think this is more a matter of the Chengdu Worldcon con-runners not knowing what it’s doing on the data/voting, since it’s a first time for them, first time in China, the convention runners’ communication skills have had problems, the convention had issues, etc. — which is why there was a longer delay and discrepancies, and very possible them having lost data, and yes, possibly covering up problems or not realizing they messed up, plus typos that might have caused errors.

    So it’s probably going to be a long process of untangleing what errors there are and how they happened and yes, possibly astericks and/or changes to some results. Which nobody would like to really deal with, but there it is. It’s part of the growing pains of widening WorldCon to the world, which is not very comforting to the nominees affected by the discrepancies and possible errors, but hopefully they can fix the major problems and it will help the process and coordination with WorldCon & its host cons in the future so it doesn’t happen again.

    I hope that WorldCon has a process for challenges/claims; I assume they do, so it will be, as always, a matter of time. It may be a matter of one person messing up or several, but I would suggest not rushing to attach elaborate conspiracies to it just yet.

    Liked by 2 people

    • It’s my understanding that the nomination ballots have already been destroyed, as is usual practice. There isn’t going to be any do-over or correction. This is what we are stuck with. And I do not expect anyone who e-mails the provided hugoteam e-mail address will ever get a response. We will be told “Oh, gee, your enquiry must have gotten caught in a spam filter, or they (hugoteam) would have written you back.”

      Liked by 2 people

      • I wouldn’t be expecting a do-over — that’s not possible as a chunk of the voting is done at the convention. But this is the Hugos — some sort of explanation eventually has to be given, rules discussions come into effect, corrections have happened before. WorldCon has business meetings at which people scream loudly and one of the folks who is pissed off is Neil Gaiman. So the email got caught in the spam filter won’t hold forever. Exact answers may not be possible on all things, but the issue is not going quietly into the night.

        Like

        • I don’t think any voting for the Hugos is done at the convention – the deadline is weeks before; at the con there is last minute voting for Site Selection, though. 

          Like

        • Kat Goodwin: some sort of explanation eventually has to be given

          Not with this Hugo Administrator it doesn’t — and I can pretty much guarantee that we are never going to get one. This is a guy with an ego the size of the entire U.S. who gets off on taking power trips. He considers himself a supreme being who answers to no one.

          Although there is a slight someone on the inside may unofficially leak more info down the road. The problem is there isn’t really anyone else on the inside except anyone in whom he may have confided. Usually the Hugo team is composed of several non-Chinese members doing the various parts. But it looks as though the Hugo Admin deliberately kept any western fans from being on the team.

          I do agree that there is going to be something done at the Business Meeting about making a Hugo Administrator accountable. And if we’re very lucky, this has made him into so much kryptonite that no future Worldcon will be willing to allow him to participate in the Hugo process again.

          * sidescript: no Hugo voting is done at the convention. It’s all done and dusted well before then; the deadline is usually 3-4 weeks in advance of the convention so that the trophy plaques can be engraved and the slideshow with the winners created.

          Liked by 1 person

  13. Conclusion 8: Cock-up and conspiracy aren’t alternate explanations. Sometimes, the conspiracy is an attempt to fix a cock-up.

    See also: the UK Post Office/ Fujitsu scandal (though that’s obviously considerably more serious).

    Liked by 1 person

Blog at WordPress.com.