Inside the NIH’s controversial decision to stop its big remdesivir study

The drug maker Gilead Sciences released a bombshell two weeks ago: A study conducted by a U.S. government agency had found that the company’s experimental drug, remdesivir, was the first treatment shown to have even a small effect against Covid-19.

Behind that ray of hope, though, was one of the toughest quandaries in medicine: how to balance the need to rigorously test a new medicine for safety and effectiveness with the moral imperative to get patients a treatment that works as quickly as possible. At the heart of the decision about when to end the trial was a process that was — as is often in the case in clinical trials — by turns secretive and bureaucratic.

The National Institute of Allergy and Infectious Diseases has described to STAT in new detail how it made its fateful decision: to start giving remdesivir to patients who had been assigned to receive a placebo in the study, essentially limiting researchers’ ability to collect more data about whether the drug saves lives — something the study, called ACTT-1, suggests but does not prove. In the trial, 8% of the participants given remdesivir died, compared with 11.6% of the placebo group, a difference that was not statistically significant. 

A top NIAID official said he had no regrets about the decision. 

“There certainly was unanimity within the institute that this was the right thing to do,” said H. Clifford Lane, NIAID’s clinical director. “While I think there might’ve been some discussion, [because] everyone always tries to play devil’s advocate in these discussions, I think there was a pretty uniform opinion that this was what we should do.”

From the standpoint of the agency, he said, the study had answered the question it was designed to answer: The median time that hospitalized Covid-19 patients on remdesivir took to stop needing oxygen or exit the hospital was, at 11 days, four days shorter than those who were on placebo. “How many patients would we want to put at risk of dying,” he asked, for that last little bit of proof? Remdesivir, he noted, was not a home run, but is probably better than nothing.

Steven Nissen, a veteran trialist and cardiologist at the Cleveland Clinic, disagreed that giving placebo patients remdesivir was the right call. “I believe it is in society’s best interest to determine whether remdesivir can reduce mortality, and with the release of this information doing a placebo-controlled trial to determine if there is a mortality benefit will be very difficult,” he said. “The question is: Was there a route, or is there a route, to determine if the drug can prevent death?” The decision is “a lost opportunity,” he said.

Peter Bach, the director of the Center for Health Policy and Outcomes at Memorial Sloan Kettering Cancer Center, agreed with Nissen. “The core understanding of clinical research participation and clinical research conduct is we run the trial rigorously to provide the most accurate information about the right treatment,” he said. And that answer, he argued, should ideally have determined whether remdesivir saves lives.

The reason we have shut our whole society down, Bach said, is not to prevent Covid-19 patients from spending a few more days in the hospital. It is to prevent patients from dying. “Mortality is the right endpoint,” he said.

Most experts contacted by STAT expressed opinions that fell between Nissen and Lane, believing that the decision was a difficult case, with several defending the NIAID. 

“I think it was a really tough call,” said Janet Wittes, a prominent statistician and the president of Statistics Collaborative. 

When the remdesivir results were announced, the NIH said the data came from an “interim” analysis. This means that a study was stopped early because a drug’s benefit was so undeniable that it would be unethical to continue the study. But Lane said this was incorrect. The data come from a preliminary final analysis, a point at which the study would normally end.

The ACTT study (short for Adaptive Covid-19 Treatment Trial) began in late February. The first patient dosed in the study was an American repatriated from the Diamond Princess, a British cruise ship where there was an outbreak of more than 800 Covid-19 cases. By the terms of the study, hospitalized patients were randomly assigned to receive either intravenous remdesivir or a placebo. On day 15, the study would score patients on a scale from 1 (dead) to 8 (not hospitalized, with no restrictions on activities).

As results from other Covid-19 studies conducted in China started to trickle in, Lane and his team began to worry that looking at the outcome on only the 15th day could lead the study to fail even if the drug was effective. On March 22, with only 77 patients enrolled in the study, members of the NIAID team had a conference call on which they decided to change the measure that would be used. Instead of measuring patients on an eight-point scale on one day, the study would measure the time until the patients scored one of the best three outcomes on the scale. This decision was finalized on April 2; it was posted to clinicaltrials.gov, a government registry of clinical trials, on April 16.

Ironically, Lane said, the study would still have been positive if the change had not been made. But the change in the study’s main goal also changed the way the study would be analyzed. Now, the NIAID decided, the analysis would be calculated when 400 patients out of the 1,063 patients the study enrolled had recovered. If remdesivir turned out to be much more effective than expected, “interim” analyses would be conducted at a third and two-thirds that number.

The job of reviewing these analyses would fall to a committee of outside experts on what is known as an independent data and safety monitoring board, or DSMB. Though they generally go unseen, DSMBs are among the most important and powerful forces in medical research. They are allowed to analyze the data from a trial while it’s ongoing, even as drug companies, doctors, and patients are kept from knowing who is getting the medicine and who is getting placebo. These boards have two jobs: to make sure that patients aren’t being harmed by the experimental drug, and to ensure that it’s not already clear beyond a doubt that a medicine is effective.

Those decisions bring moments of triumph, despair, and, occasionally, confusion.

When Merck decided to withdraw the painkiller Vioxx in 2004, it was because a DSMB had recommended stopping a study of the drug when it became clear the medicine increased the risk of heart attacks and strokes. In 2014, when a study of the cancer immunotherapy Opdivo first proved that drug extended survival in melanoma, it was because a DSMB had found the result incontrovertible and recommended stopping the study.

But the DSMB for the remdesivir study did not ever meet for an interim efficacy analysis, Lane said. All patients had been enrolled by April 20. The data for a DSMB meeting was cut off on April 22. The DSMB met and, on April 27, it made a recommendation to the NIAID.

That recommendation was not about whether the patients on placebo should receive remdesivir. Instead, the DSMB recommended that in the next phase of the study, testing Eli Lilly’s arthritis drug Olumiant against remdesivir, there was no need for a placebo-only group. 

That decision, Lane said, led the NIAID to conclude that patients who had been given placebo should be offered remdesivir, something that started happening after April 28.

This is where Nissen and Bach disagree. There were 1,063 patients in the study, but only 480 had recovered at the time of the analysis. Researchers could have collected more data, they argue, and perhaps have learned if remdesivir saves lives. They were already close, both note. Results are considered “significant” if a measure called a p-value is less than 0.05; the value for mortality in the preliminary analysis was 0.059. “How many patients would we want to put at risk of dying to get that 0.01 on the p-value,” Lane retorted.

Marc Pfeffer, a cardiologist at the Brigham and Women’s Hospital in Boston, said he believes NIAID made the right call. He said that he was “very sympathetic” to the fact that researchers were getting this study done during a pandemic. “If you make the decision that remdesivir should be part of everybody’s therapy in the next phase, then those volunteers taking the risks in the current trial should be switched to the active therapy now considered effective,” he said.

Should this decision have been left to the DSMB, not the NIAID? DSMBs are technically only advisory panels, said Richard Chaisson, a professor at the Johns Hopkins Bloomberg School of Public Health. 

Chaisson remembers running an NIH-funded study of a preventative treatment for tuberculosis. The DSMB recommended continuing the trial, but he decided not to, because it was putting patients at too much risk. “The NIH had no problem with me not following the DSMB’s advice, and were even relieved I made the decision I did,” he said.

Wittes, of Statistics Collaborative, said she is glad she wasn’t on this DSMB, adding, “I don’t know where I would have come out.” And she said that when full results of the study are available, she would be “shocked” if the NIAID had not done things properly.

“I think there are groups of people who you’d really respect who would not have stopped a study like this without a mortality benefit,” Wittes said. “And I think you can argue that both ways.”

But she also worried that the evidence might not be strong enough to make the decision society is now making: that every new Covid-19 treatment must be given with or compared to remdesivir. 

“The danger is now it’s the treatment for everybody,” she said. “Now this is the base drug and everything is going to be that plus something or the control. I think we don’t know if it’s strong enough for it to be the standard of care. I don’t think we know who should be treated.”

Steven Joffe, an ethics expert at the University of Pennsylvania, said he believes the NIAID likely took the right steps in making its decision to give remdesivir to the placebo patients. But he worries about deciding to use time to improvement, not death, as the measure of success, in the first place.

“I don’t find this endpoint very compelling, and to me the real issue is the decision to design the trial around the endpoint of time to recovery defined in the way they defined recovery,” Joffe said. “To me, the decisions that are this weighty ought to be based on clinically important endpoints.”

All of this would normally wait until the full results were published, at which point the roster of the DSMB may be revealed. (Lane would not share their names.) But what is unusual in this case is that, before the data are even fully analyzed, the FDA has authorized remdesivir’s use. A Chinese study, meanwhile, failed to show remdesivir had a benefit. Several more studies of the drug are expected to read out soon.

Ethan Weiss, a cardiologist at the University of California, San Francisco, who traveled to New York two weeks ago to treat Covid-19 patients, said that he does worry that we have missed “a fleeting opportunity” to understand how well remdesivir works. “It is sad to me that we’re not going to get a complete answer about it.”  But he said he also thinks the issue is “inside baseball.” Remdesivir, as several experts have pointed out, is not a game changer.

The real problem, Weiss said, is not the handling of this particular study but that there aren’t more like it.  He said he wished the U.S. had built the infrastructure needed to do more studies like this when the pandemic in New York was at its height. He wished there were more studies, with more DSMBs.

“We’ve squandered an incredible opportunity to do good science,” Weiss said. “If we could ever go back and do something all over, it would be the infrastructure to actually learn something. Because we’re not learning enough.”