After several days of rumors of ever-growing hype, the Trump administration announced on Sunday that the Food and Drug Administration has granted an Emergency Use Authorization (EUA) for a COVID-19 treatment. The move was controversial from the start, with reports indicating the EUA was opposed by a number of health experts, including the director of the National Institutes of Health Francis Collins and the director of the National Institute of Allergy and Infectious Diseases Anthony Fauci. The press conference did not settle matters, with a growing chorus of scientists saying the data presented in support of the EUA had been misrepresented.
On Monday night, FDA Commissioner Stephen Hahn acknowledged that: he made a big mistake in presenting the benefits of the treatment, and he followed that statement with an apology Tuesday. But Hahn resisted evidence that the approval of the treatment on the eve of the Republican National Convention was motivated by political pressure.
Wrong kind of risk
The treatment in question involves taking the antibody-containing plasma from those who have recovered from SARS-CoV-2 (repairing plasma) infection and giving it to those currently suffering from COVID-19 symptoms. At Sunday’s press conference, the main justification for allowing this treatment under an EUA was a 35 percent reduction in mortality for those who received plasma in the first three days of treatment — notably, Hahn said 35 out of 100 people “would have been saved” by this treatment.
There were some obvious problems with this statement, which was supposedly based on a draft paper from a study of people receiving this restorative plasma treatment. The most obvious problem is that the newspaper has no control group; everyone received restorative plasma, but there were differences in how quickly treatment was started. Without control, it is impossible to make statements about how many people would have died without treatment.
While it was possible to find some numbers quoted in the paper that could be interpreted as a 35 percent difference between different types of treatments, things got more complicated as the press conference progressed and attendees added additional caveats to the number: it applied on people under the age of 80 who did not require a ventilator, and it required the identification of plasma with high levels of antibodies specific to the virus. Since these numbers were not part of the authors’ analysis of the concept, it was clear that the 35 percent figure had been generated by working with the raw data discussed in the study. So it was impossible to say how it was derived.
With Hahn’s acknowledgment, things are now a bit clearer. Hahn was talking about a figure called relative risk that — as the name implies — compares the risks faced by two different groups. He then discussed that risk as if it were an absolute risk to produce his number of 35 out of 100 — a very basic statistical error, and science journalists are often warned not to make them.
To see why this is a problem, imagine a case where only three in 100 people are affected, and an experimental condition changes that number to one in 100. The relative risk differs by a factor of three – a difference of 300 percent. But the absolute risk is the small change from three percent to one percent.
The actual difference in absolute risk seen in the study was 4.8 percent, not 35 percent. That means the treatment could save an additional 4.8 lives per 100 COVID-19 patients.
The politics of it all
Hahn made a rookie mistake that made a treatment sound much better than it actually was, and that mistake contributed to the perception that politics was a major driver of the EUA. At the same time that he admitted his mistake, Hahn . said pushed back against suspicion of political interference powerful. He stated that the decision to allow the EUA was made by an FDA career scientist rather than a political appointee.
That statement is actually fully consistent with the previous reporting, which suggested the FDA was ready to issue the EUA, but health authorities outside the agency had urged the decision to be delayed until a randomized clinical trial could be completed. According to the New York Times, Francis Collins, Anthony Fauci and Clifford Lane of the National Institute of Allergy and Infectious Diseases were among those who urged caution. According to a report last week, their opposition had put the EUA on hold.
But Trump seemed blame the FDA for unspecified delays, accuses the staff of having sabotaged his reelection shortly before the EUA was approved. If there had been political pressure, it would not have been the original FDA approval decision; instead, it would have overcome resistance to the move from outside the agency.
According to one source, Trump’s finger-pointing at the FDA was simply the product of his confusion about the precise roles of the FDA and the NIH.
In any event, the limited evidence we currently have is sufficient to meet the legal threshold for an EUA. But this mistake, the EUA’s timing and its presentation in a press conference laden with praise for President Trump, has damaged the credibility of the FDA. And with several vaccines in the later stages of testing right now, it’s not an ideal time for that damage.