The Strange Zoom Pile-On

I have been in the technology industry a long time, and have rarely seen a sequence of events similar to that which happened to Zoom last week. By the end of the week, Zoom's reputation was completely trashed in the media, New York City Public Schools had banned the product, and the stock had dropped $30 per share. Why? Negative security story after negative security story, in the largest, most coordinated corporate pile-on I have ever seen.

The security stories were weird. While there was one high-profile issue that deserved mainstream coverage, most of this was stuff that wouldn't normally go beyond the security research community. Nothing was greater than medium severity. I'll unpack these a bit so we can begin to make sense of them.

Zoom Bombing

Of all of the reported problems, "Zoom bombing" is the one I consider newsworthy to a broad audience. In case you're not spooled up on this problem, by default, Zoom allows screen sharing, and it allows users to re-join conferences when they have been removed. Online pranksters have taken advantage of this to share inappropriate content in conferences. This is exacerbated by the fact that, by default, Zoom conference links allow anyone with the link to join.

There is some missing context, though. Practically overnight, Zoom went from 10 million users, largely in corporate environments hosting internal conference calls, to 200 million users representing essentially every possible use case. Zoom simply chose and configured its application security defaults based on a different (and much smaller) audience than it has today. That isn't Zoom's fault; ultimately, it's the responsibility of the user to know how to use the product correctly. As a Zoom host, you can lock down Zoom conferences with a password, you can prevent users from re-joining, and you can disallow screen sharing. That just wasn't the default (Zoom has announced this is changing to reflect the changing landscape of their user base).

Is this a security issue, though? Strictly speaking, no. It's a user education issue, and that's what makes it newsworthy.

0-day Exploits

A trio of 0-day vulnerabilities, all dropped at once, really stood out to me as unusual. It's highly unusual anyway for security researchers to drop what's called a "0-day" vulnerability. Generally speaking, security researchers practice "responsible disclosure" whereby they confidentially notify the vendor of an issue, the vendor fixes the issue, and then the security researcher is acknowledged when a patch is delivered. Researchers generally reserve disclosure of "0-day" vulnerabilities for the most egregiously recalcitrant of vendors; companies that drag their feet repeatedly and don't work with the research community in good faith. Now, in all fairness, Zoom hasn't been a great company for the security research community to work with, but dropping 0-days in the middle of a pandemic where Zoom literally scaled 20x in a month and when there are much bigger problems to solve (see above) is, at best, incredibly tone deaf. And these weren't even serious vulnerabilities. Two were low severity because they only worked on a Mac and required local access. One essentially amounted to "clickable links are clickable." Yes, input should be sanitized, but this was medium-low severity.

It's very rare that any of this would end up in the press, and yet it did.

Routing Data Through China

Like essentially every other online service with a presence in China, Zoom has a data center in China. If you're not in China, you generally wouldn't want to send any traffic there; performance is terrible when transiting the Great Firewall. If you're running an enterprise version of Zoom, you can even specifically target whether the client targets the Chinese data center or not (and FedRAMP-certified deployments never use data centers outside the US).

As you might imagine, scaling 20x in a month is a hard engineering problem. Engineers distribute traffic across data centers using a "load balancing" service. This is very tricky engineering. It's easy to get wrong, especially when you're going fast. Accordingly, configuration errors are very common in load balancing deployments. Ask any network engineer: this sort of human error happens all the time. And someone at Zoom made a mistake: the load balancing configuration for connections originating from outside China, for a short time, failed to exclude the Chinese data centers (except for enterprise deployments where China was excluded, and for FedRAMP deployments).

What does this mean in practice? It didn't mean that all traffic went through China. It meant that on rare occasions, completely random Zoom calls would accidentally be routed through a Chinese data center. Bear in mind, though, that performance is generally terrible when doing this, because the Great Firewall--by design--slows everything down. It's so bad that most people would simply disconnect the call and try again. And given the nature of the configuration error, there would be no way to force specific, sensitive calls to be routed via China. In fact, the most sensitive calls Zoom handles (in FedRAMP certified environments) have separate infrastructure.

Press articles spun this as everything from "Zoom enabled Chinese surveillance" to "Zoom sent encryption keys to China." While this may be technically true, context matters a lot here. Did the encryption keys persist beyond a single Zoom call? No. Did the Chinese government even know it had this capability before a security researcher discovered it? Unknown. Would the Chinese army find intelligence value in monitoring, say, a random 4th grade math lesson versus other stuff they could be doing with their resources? Probably not. In practical terms, would it matter at all, to anyone, even if they did? Again, probably not. We'll never know what the Chinese government did or didn't do, but again, configuration errors like this happen all the time at pretty much every company with an online presence, and they are usually caught internally by companies without anyone outside ever noticing because of the performance issues they cause. Occasionally, massive volumes of Internet traffic are misrouted due to similar configuration errors in a protocol called BGP because humans configure networks and they occasionally make mistakes.

Zoom published a blog post explaining what happened. I personally am giving them a pass on this, because I have enough operations experience to know that people who live in glass houses should not throw stones.

OMFG Chinese People!

A typical scaremongering article will say something like "Zoom, led by a Chinese-born CEO, has a Chinese engineering team" leaving the reader to conclude that Zoom must be some sort of red flag waving, wholly owned subsidiary of the Chinese government. In fact, many people now seem to think that Zoom is a Chinese company.

At some point, I think this is really just racist dog whistling. Zoom's CEO is a US citizen. He's as American as any of us. Zoom is an American company. And Zoom isn't the only American company to have engineering teams in China. Microsoft, Intel, Apple and many other companies also do. I have no inside knowledge of Zoom, but I did live and work for 3 years in Beijing at Microsoft Research Asia. If I were going to build a product like Zoom, I'd go where the talent is: China. Every country seems to specialize in different areas of computer science, and China has invested very heavily in machine learning, artificial intelligence, facial recognition and video streaming R&D. Beijing is home to some of the world's top researchers working in these fields. Sure, other countries also have talent in this area, but in the US, there's a ton of competition for a limited talent base. It's pretty tough for a relatively small company like Zoom to compete for talent with the likes of Google and Facebook.

"Malware" Allegations

I have seen several articles alleging that Zoom uses "malware techniques" or "malware tactics." Most of these involve the Mac platform. There was one actual, really bad thing Zoom was caught doing last year: they silently installed a server process on Macs, because Apple didn't provide Zoom another reasonable way for their product to work. This obviously wasn't ideal, but neither was Apple's design decision that led to this unfortunate choice. Last year, Apple and Zoom agreed upon a solution and the world moved on.

More recently, Zoom is up to its old tricks again--of getting its product to work on the Mac platform. Zoom optimizes for being easy to use and this is why their product has gained popularity. One thing that makes their product easier to use is that you can install it with one click. However, it takes some hacks to do this on the Mac platform, because the way Apple does installers requires multiple clicks. Is this really "malware?" No. Intent matters. Software with malicious intent is malware. There is no malicious intent with Zoom, and the user did, in fact, click to install. They just didn't click multiple times. In any event, Zoom was successfully pressured to make their installer harder to use, and I wonder what the world actually gained here. I don't think it's especially responsible of the information security community (or the technology media) to use a scary word like "malware" when what we're really talking about is bypassing clicks in an installer the user has already consented to run.

End to End Encryption

We have all read a technical document written by marketing, and facepalmed at it. Zoom's technical documentation has traditionally had a very strong marketing voice. One such case was in promoting "end to end encryption."

I won't sugar-coat it: Zoom's documentation was misleading. While chat in Zoom has end-to-end encryption capability, audio and video conferences do not. However, no competent security architect should ever have believed that this existed. How could real-time facial recognition or audio transcription possibly work without a centralized server? Zoom works like pretty much every service doing the same thing: they encrypt connections between the server and each client, but decrypt the traffic in the middle (on the server itself) to allow the service to function.

Does this represent a security risk? Possibly, but in practice, probably not, if Zoom is to be believed. They claim that conversations aren't accessible by Zoom staff. That can't be strictly true, because if decrypted traffic exists on the server, an administrator could potentially dump it to disk and reconstruct it. In practical terms, is this is a risk? Probably not, because given the way the Zoom service works, calls can be routed through any data center via any server. Unless you knew exactly which call you were targeting and could instantly compromise the server it was routed through, and do so undetected, it'd be really hard to exploit this.

Zoom responded by updating their documentation to be technically accurate, and they published a blog post apologizing for the error and explaining how it happened. I am calling this a case of "marketing writing security documentation" and leaving it there.

Short Sellers?

So, something funny happened. An independent, Seattle-based "tech journalist," who wrote a 7,000 word screed tearing down Zoom, asked whether I owned Zoom stock assuming that this was my motivation for questioning the narrative. I don't own the stock, never have, and also have never held any position in $ZM either short or long. But curiously, when I asked the same question, the silence was deafening.

This could be an echo chamber of lazy journalism feeding on scare tactics, but it could also have been a coordinated action to tank the stock.

Where Do We Go From Here?

I'm not sure what is going on, but what I do know is that Zoom has been hyper-responsive. They updated their privacy policy three times in March in response to criticism. They have fixed every bug and design issue that has been raised, and have done so almost immediately in the midst of a pandemic where demand for their services has grown more than 20-fold. Yes, they have done shady stuff in the past, and their engagement with the information security community has been anywhere from "mediocre" to "hostile." However, they have learned fast, have improved fast, and from where I sit, they seem to be doing all of the right things.

In the information security industry, we represent confidentiality, integrity and availability. The "Availability" corner of the triad is the most important one for most organizations right now. There is a very real impact of forcing an IT team, students, and teachers, all of whom are working remotely and none of whom have done this before, to all switch to entirely new tooling because of #infosec fear-mongering. It's easy for you to just fire up a Docker container with a Jitsi instance and host it yourself. Ask a second grade teacher to, without any physical presence, individually help thirty 8-year-olds on multiple platforms migrate to whatever open source tool of choice you're promoting.

Look: If Zoom really is shady, I'll be right there with you burning them with fire. But I'm just not seeing that. Not here, not now. I'm seeing a company being torn down--maybe for money, for politics, or for sport, but for no good security reason that I can see. And this is causing a very real impact.

It needs to stop.

The first principle right now needs to be keeping the lights on.

Nothing else matters without availability.