Inside TikTok owner’s dystopian Chinese censorship machine

Comically affronted parents, drunken piano concerts and viral conspiracy theories about trafficked children hidden in wardrobes: it's a fairly ordinary evening on TikTok, the video-sharing app enjoyed by more than 500m people across the Western world.

Things do not seem so different on Douyin, the Chinese version of TikTok, where a live streaming boom has minted new social media millionaires. Behind the scenes, however, Chinese streamers are subject to an elaborate regime of automated surveillance and censorship.

One system can use facial recognition to scan live streamers' broadcasts and guess their age, reporting them to a human moderator if they appear under 16. 

Another checks whether users' faces match their state ID cards before letting them stream, automatically excluding foreigners and people from Hong Kong.

Another system assigns streamers, who are expected to uphold "public order and good customs", a "safety rating", similar to a "credit score. If the score dips below a certain level, they are punished automatically.

Meanwhile, speech and text recognition is used to ferret out sins such as "feudal superstition", defamation of the Communist Party and even ASMR, which is banned because  it has become too "pornographic". 

On Wednesday, a British man called Joshua Dummer fell afoul of this apparatus when his wife, who is Chinese, tried to feature him in a stream from their Beijing apartment. Somehow – potentially via facial recognition – Douyin detected his presence and killed the broadcast, telling him that foreigners without government "permission" were banned.

All these methods are laid bare in a little-known document from TikTok and Douyin's parent company, ByteDance, and unearthed by New York City journalist Izzy Niu, which explains how the apps have adapted China's strict internet censorship laws to the unprecedented speed and chaos of live streaming.

The document raises difficult questions for TikTok, which faces privacy probes in the US and UK and has already been banned in India, about whether and how it applies the same technologies to its Western users.

Those were questions that TikTok declined to answer.

Foreigners banned from streaming in China

All Joshua Dummer had to do to anger the internet censors was sit on his sofa and talk about his day. 

It was just before 11pm on Wednesday night when his wife, who is Chinese, sat down beside him and pointed the camera in his direction.

"She urged me to speak, and I just started talking to the four or five people watching about the boring things I had done that day, in Mandarin with some English sprinkled in," Mr Dummer told the Telegraph. "After about one or two minutes, the warning popped up."

The warning said that foreigners were not allowed to live stream on Douyin "without permission". Surprised, the couple pressed on – but within a few minutes their broadcast was cut, and Mr Dummer's wife was temporarily blocked from streaming.

Mr Dummer is no stranger to censorship, which is a universal fact of life for the 850m or so internet users who browse the Chinese mainland's home-grown apps from behind its famous "great firewall". 

So he was not fazed by the hoops his wife had to jump through to begin streaming, which included submitting her ID card (not available to foreign nationals) and a picture of her blinking face. 

"All Chinese apps, websites and media have to follow the country’s rules on what content is acceptable, which vary from medium to medium," Mr Dummer said. 

But he went on: "This is the first time that the fact I’m a foreigner has been used as a reason for preventing me from using a platform, or requiring me to fulfil extra registration procedures."

That is not surprising. China's Ministry of Culture officially banned foreigners from live streaming without a government permit in 2017, but the rule does not appear to have been enforced until recently.

This year – perhaps accelerated by coronavirus lockdowns, which have supercharged China's streaming culture – reports began to surface of foreigners having their streams unexpectedly cancelled.

Douyin has also reportedly suspended users who tried to stream in Cantonese, a branch of the Chinese language that is dominant in Hong Kong but which the Communist Party has often sought to sideline.

'A complete monitoring system'

On Thursday, a spokeswoman for ByteDance confirmed to the Telegraph that it had censored Mr Dummer, saying: "As all live streaming moderation on Douyin is conducted by human moderators in real time, we can confirm that the action in this case was carried out by a person."

But that was not the full story. According to a white paper from ByteDance itself, said human moderator was almost certainly tipped off by one of Douyin's sophisticated automated sentinels.

The document that Niu spotted was originally published in 2019 but has not been previously reported by English-language media. It details ByteDance's response to the huge increase in streaming in China, which has drawn in hundreds of millions of viewers and made many "anchors" – the Chinese term for both streamers and TV newsreaders – into stars. 

"But while the live streaming industry is booming, various problems have emerged," says the paper, which covers ByteDance's Toutiao ("Headlines") news app and two video apps as well as Douyin.

"A lot of content that violates laws, regulations, public order and good customs has appeared on a large number of live streaming platforms and has caused adverse social impact...

"In order to prevent the risks arising from all aspects of live streaming, [ByteDance apps] have established a complete monitoring process."

It goes on to boast of "no-dead-end, full-coverage monitoring", using artificial intelligence (AI) to analyse every detail of each stream including the behaviour of viewers). Suspicious streams are flagged to human moderators, who can limit their spread or stop them cold.

Some of these methods are common in the West, too. Both Facebook and YouTube use AI to police their services, and have massively expanded their censorship during the pandemic as they crack down on hoax cures and coronavirus misinformation. Their critics might say they are just as dystopian as ByteDance in principle.

Facebook, for example, perpetually watches users' content for signs that they are selling drugs, spreading fake news, praising banned hate groups or expressing genuine suicidal intentions. It had to make its own assurances to Western governments after the Christchurch terror attack in 2019, which was live-streamed by its perpetrator. 

Many of the companies' rules are also the same. Both Facebook and ByteDance frown on pornography, nudity, sexually suggestive content, harassment, scams, copyright infringement and minors who pretend to be older than they are.

Other rules, however, are rather different.

No short skirts, no ASMR and no Western propaganda

ByteDance bans any live broadcast of "anti-Party and anti-government" sentiments. "Feudal superstition" is certainly taboo, as is "smoking and drinking", "money worship" and "insulting women".

More strangely, the company has found reason to forbid many popular trends. ASMR videos – whispering monologues intended to stimulate the brain with phantom sensations – are out, since their anchors sometimes "attract the intention of the audience by means of strong sexual  cues" such as "ear-licking".

The slime craze beloved of British children is also banned, because the slime contains potentially toxic borax. ByteDance even cites a case in Manchester in 2017, when a YouTube-obsessed British child received severe chemical burns after making "unicorn slime".

The celebrity competitive eaters known in China as "big stomach kings" are branded a bad example to society. "Some anchors are already obese, but they still go in desperately", the white paper sniffs – although it does concede that their appetite are "staggering".  

And, of course, there are detailed rules about how much clothing female anchors can wear. The dress code for Douyin is not unlike that of Microsoft's doomed video game streaming site Mixer, which was attacked for its prudishness. But ByteDance's version is even stricter, and comes with a helpful diagram.

As for Mr Dummer, it is unclear what system flagged him. A US journalist in China who tested the app found little evidence that Douyin was ethnically profiling streamers' faces.

One alternative is that it simply checks streamers' faces against their ID cards, freezing out foreigners by default; another possibility is speech recognition AI, which might have reacted to Mr Dummer's sprinkling of English.

Either way, ByteDance claims its system gets results. Between 2018 and 2019, live streaming violations dropped by 58pc, with 93pc of problems caught by the monitoring system before users could report them. 

That compares favourably with TikTok's most recent transparency report, which includes non-live videos and put that rate at 98.2pc.

Questions now loom over TikTok itself

What does all this mean for TikTok? The app is separate from Douyin in some ways, and runs on different networks. However, both apps were designed in China and share the same  core features, with Douyin often seen as a testing ground for ideas that later migrate to TikTok.

ByteDance is still trying to pry the two apps apart, hiring former Disney executive Kevin Mayer as TikTok's chief executive and reportedly founding a new TikTok-only engineering office in the US.

That has not stopped US officials from suggesting that the Chinese government has access to all the personal data it collects. TikTok has consistently claimed that it has never given data to the Chinese state and will not do so in future.

Douyin's use of facial recognition and ID cards to police its users also goes far beyond most Western apps. Although its youth-detection systems were still being "tested" when the white paper was written, they are likely to have been rolled out more widely since then.

TikTok is already under investigation by US regulators for allegedly violating children's privacy, and is being sued in the state of Illinois by four teenagers who claim that it collected their face data without asking for their consent.

Nor does censorship appear to have stayed in China. Investigative journalists have unearthed internal documents suggesting that TikTok told its moderators throughout 2019 to suppress videos by users with an "abnormal body shape", "too many wrinkles" or "eye disorders".

The point, reportedly, was to make sure that new users were sucked into the app by beautiful people, rather than being turned off by "ugly" or "low quality" features.

Another investigation found that TikTok had a policy of systematically suppressing videos featuring disabled people (such as those with "autism", "Down's syndrome" or "facial disfigurement"), ostensibly to stop them from being bullied. One user in Germany, who had been branded a "special user", called the practice "discriminatory" and "inhuman".

Then there were the political guidelines, which reportedly censored Western videos mentioning the Tienanmen Square massacre, Tibetan independence or the anti-Communist Falun Gong religious sect.

TikTok responded said that most of these documents were out of date or no longer in force. In a few cases it claimed to have no record of them. It described the politics and disability guidelines as being "blunt" or early attempts to limit conflict on the service, which later gave way to more "nuanced" solutions. 

So how much of Douyin's censorship and monitoring carries over into TikTok? Does TikTok also use facial recognition? If so, in what ways? In other words, how worried should Western users be?

A spokesman for TikTok declined to answer those specific questions, only saying: "TikTok takes the safety of our younger users seriously. We have strict rules against users under 16 participating in live streaming. 

"In addition to age gating, we also rely on human moderators who evaluate factors that clearly indicate users are under 16, such as information users put in their public bios that may contradict what they have provided when they signed up.

"TikTok has never provided user data to the Chinese government, nor would we if asked to do so."