One of the articles Mickey sent me was from the World Health Organization (WHO), Field Evaluation of Vaccine Efficacy, written in 1985 by Orenstein, et al, and published in the Bulletin of the WHO. This article was, in fact, listed in the bibliography, but I didn't notice it. Thanks again, Mickey.
In general, Vaccine Efficacy (VE) is the difference between the incidence or attack rate of disease among the unvaccinated (ARU) and vaccinated (ARV), divided by the ARU, and multiplied by 100.
VE=(ARU-ARV)/ARU x 100.
So, for a perfect vaccine, the ARV would be zero, and then
VE= (ARU-0)/ARU x 100
= ARU/ARU x 100
= 100%
For a vaccine that didn't work at all, the ARU would equal the ARV, and then
VE=(ARU-ARU)/ARU x 100
= 0/ARU x 100
= 0%
In the study, we have the following data:
The ARU is the number of those who were unvaccinated and flu positive divided by the total number of unvaccinated.
ARU= 11/65 = 0.169
Similarly,
ARV= 6/104 = 0.058
Therefore, the VE= (0.169-0.058)/0.169 x 100 = 0.111/0.169 x 100 = 66%
It's close, but it's not 71%, and the reason for this is that this is the general formula for VE. The study had a case-control design with unmatched pair analysis, in which case,
VE= (1-RR) x 100,
where RR = relative risk, which is roughly equal to the Odds Ratio (OR) in this case.
OR = (Flu+, Vaccinated)(Flu-, Unvaccinated)/(Flu-, Vaccinated)(Flu+, Unvaccinated)
= 6x54/98x11
= 324/1078
= 0.301
So VE= (1-0.301) x 100
= 0.700 x 100
= 70%
The study got 71%, but I'm assuming they had a better calculation of the OR, so 70% is close enough.
Okay, now we know how they determined that the flu vaccine effectiveness was 71%. So I'm going to act like an analyst and ask, "What does this really MEAN?"
The article claims it means that there was a 71% reduction in flu-related hospitalizations in patients who have been vaccinated against flu, vs. those who haven't.
But I don't think that's correct, and it was one of the things I went back and forth about with Mickey.
They looked at patients' vaccination statuses, and at which patients tested positive for flu. The appropriate conclusion to draw from this data is that vaccination resulted in a 71% reduction in flu INFECTION, in this population.
They did NOT look at patients' vaccination status, which patients tested positive for flu, AND which patients ended up hospitalized. They couldn't possibly look at that, because the entire population was hospitalized. So they can't logically draw any conclusions about whether vaccination reduced hospitalization or not.
For example, let's say they looked at 3000 people in the community, 1000 of whom were vaccinated against flu, and 2000 of whom were unvaccinated. And let's say they checked to see who was hospitalized with an illness that looked like flu, and it turned out that 104 vaccinated patients and 65 unvaccinated patients were hospitalized. These are the same numbers as in the study.
Now let's say they checked to see which of the hospitalized patients were flu+, and it turned out that 6 of the vaccinated, and 11 of the unvaccinated patients were flu+. Again, same numbers.
Then 6/1000 = 0.60% of the vaccinated patients were flu+ and hospitalized,
And 11/2000 = 0.55% of the unvaccinated patients were flu+ and hospitalized.
So how would the vaccination have reduced flu-related hospitalizations by 71%, when the rate of flu-related hospitalization is lower for the unvaccinated patients?
Obviously, I just made up the 1000 and 2000 figures, but my point is you can't know whether vaccination reduced flu-related hospitalizations without knowing how many were NOT hospitalized.
The thing is, I really know very little about statistics. So I suspect I'm missing something here. But I can't figure out what. And in case I'm not missing something, it's a pretty big deal that the CDC is using this result to support their recommendation for universal flu vaccination.
The truth is, a vaccine efficacy of 71% is not so great. By comparison, the inactivated polio vaccine has an efficacy of 90% after 2 doses, and 99% after 3 doses (link). This doesn't mean there isn't good reason to recommend universal flu vaccination. For one thing, older people, who stand to benefit greatly from not getting flu, don't have a good serologic response to the flu vaccine, simply by virtue of age. The best way to protect them, then, is herd immunity, which you can get from having younger adults vaccinated.
I would really appreciate comments on this post. In particular, comments from people who know some statistics and have taken a look at the article. I'd like to know what I'm not seeing correctly, or if perhaps I am seeing things correctly.
Thanks for your help, especially Mickey.