Today, a study from Imperial College in London was all over the news. It reportedly found evidence that antibody immunity to COVID-19 decreased over time. Again, this is an unpublished study simply placed on the BioRX website. Therefore, it lacked any peer review or scientific vetting before it was released to the press.
The study looked at COVID-19 antibody prevalence (the frequency of antibody) in the British population at three time points; essentially May through June, July and then September. Subjects were asked if they had COVID-19, then were asked to test themselves for antibodies. They were then classified as either positive or negative for having COVID-19 antibody according to the self-administered test.
The lead headline in the news release was that the frequency of individuals with antibody declined over the three time points, and was significantly lower from the first to time points. The authors conclusion was that because the number of people with antibody decreased over time, then immunity to the virus must be decreasing over time.
There are significant problems with the studies conducted in this paper and the scientific analysis the investigators performed. One cannot conclude there was a decrease in immunity on the basis of this work. Let me elaborate.
First, the same individuals were not sampled in each of the three cross-sectional studies done of the population. The investigators simply took random individuals who volunteered each time. This induces a selection bias, since individuals who knew they were infected after the first time point had no personal incentive to participate in the later time points. Thus, you are de facto selecting for non-infected, antibody negative individuals at later time points. Most importantly, since you aren’t following antibody in the same people over time you don’t know that antibody in these individuals is actually dropping.
There are also a number of significant technical problems with this manuscript. The investigators us two different assays to analyze for COVID-19 antibodies and use them inappropriately. Continuous data from these assays are converted to positive vs. negative results using criterial that make no sense.
The investigators use a Fortress lateral flow assay as their primary method to detect antibody. This assay (which is not approved for any use in the United States) is only approved for use in Europe in laboratories on stored blood serum (fluid) samples. The investigators actually use this assay for self-administered, in-home testing with finger stick obtained whole blood. The participants obtain the good and do the test themselves after a video training program. The validity of this testing is therefore not defined.
In some analysis the investigators compare finger stick samples to blood serum control samples, further confusing the data and making it less consistent. It may be that differences in the way the assays were distributed, or how the video instructions were given or variability in the actual assay performance from time to time played into the results since none of these variables were controlled.
In addition, the sensitivity of the Fortress lateral flow assay seemed extremely low. The authors admit that the “lateral flow test used for this study detects antibodies against the spike protein (anti-S), but is qualitative rather than quantitative and the threshold of detection is not stated in manufacturer’s instructions.” So they don’t actually know how many people they are missing with this approach.
To attempt to address this, the authors included a pilot sample from 49 health care workers who had been demonstrated to be COVID-19 PCR positive. Only 40 of 49 PCR positive health care workers had antibody with their assay; 80%! This assay would never gain approval in the US, where the sensitivity cutoff is 95%.
Nine of 49 COVID-19 PCR positive subjects were negative on the Fortress Assay; a sensitivity of 81%; so almost 1 in 5 infected people were missed in this prevalence study.They analyzed these same individuals with a functional assay that measured the ability of the subject’s serum to neutralize virus. This is a high bar as lots of antibody is often needed to accomplish neutralization (for example, only about 40% of individuals gain this activity against influenza virus after flu vaccine). Even with this insensitive assay 80% of these individuals were positive. Together, this suggests that the Fortress assay is not an appropriate tool to use when determining prevalence as it is too insensitive.
Finally, the authors themselves point out that there is no way to know whether or not the antibodies are protective in these patients, or in fact when and if their antibody levels fall they will lose their protection. So the findings in this study do not, in and of themselves, suggest that the overall population immunity is actually waning.
I could continue about at least five other problems, but by now I hope you get the picture.
I continue to caution against unpublished data that is simply put up on websites. There are a number of problems with this study that would have be brought up in peer review. The fact that this press release was pasted all over the news the past two days shows what a real problem we have with “broadcast science” and why people don’t trust what scientists say anymore.
Pandemic Pondering 