Thank You, Mickey! Part II

Picking up where I left off in Thank You, Mickey! Part I, I was about to describe how the article I'm examining, Effectiveness of influenza vaccine for preventing laboratory-confirmed influenza hospitalizations in adults, 2011-2012 influenza season, figured out that there is a 71% reduction in flu-related hospitalizations in patients who have been vaccinated against flu, vs. those who haven't.

One of the articles Mickey sent me was from the World Health Organization (WHO), Field Evaluation of Vaccine Efficacy, written in 1985 by Orenstein, et al, and published in the Bulletin of the WHO. This article was, in fact, listed in the bibliography, but I didn't notice it. Thanks again, Mickey.

In general, Vaccine Efficacy (VE) is the difference between the incidence or attack rate of disease among the unvaccinated (ARU) and vaccinated (ARV), divided by the ARU, and multiplied by 100.

VE=(ARU-ARV)/ARU x 100.

So, for a perfect vaccine, the ARV would be zero, and then

VE= (ARU-0)/ARU x 100
     = ARU/ARU x 100
     = 100%

For a vaccine that didn't work at all, the ARU would equal the ARV, and then

VE=(ARU-ARU)/ARU x 100
     = 0/ARU x 100
     = 0%

In the study, we have the following data:



The ARU is the number of those who were unvaccinated and flu positive divided by the total number of unvaccinated.

ARU= 11/65 = 0.169

Similarly,

ARV= 6/104 = 0.058

Therefore, the VE= (0.169-0.058)/0.169 x 100 = 0.111/0.169 x 100 = 66%

It's close, but it's not 71%, and the reason for this is that this is the general formula for VE. The study had a case-control design with unmatched pair analysis, in which case,

VE= (1-RR) x 100,

where RR = relative risk, which is roughly equal to the Odds Ratio (OR) in this case.

OR = (Flu+, Vaccinated)(Flu-, Unvaccinated)/(Flu-, Vaccinated)(Flu+, Unvaccinated)
      = 6x54/98x11
      = 324/1078
      = 0.301

So VE= (1-0.301) x 100
           = 0.700 x 100
           = 70%

The study got 71%, but I'm assuming they had a better calculation of the OR, so 70% is close enough.

Okay, now we know how they determined that the flu vaccine effectiveness was 71%. So I'm going to act like an analyst and ask, "What does this really MEAN?"

The article claims it means that there was a 71% reduction in flu-related hospitalizations in patients who have been vaccinated against flu, vs. those who haven't.

But I don't think that's correct, and it was one of the things I went back and forth about with Mickey.

They looked at patients' vaccination statuses, and at which patients tested positive for flu. The appropriate conclusion to draw from this data is that vaccination resulted in a 71% reduction in flu INFECTION, in this population.

They did NOT look at patients' vaccination status, which patients tested positive for flu, AND which patients ended up hospitalized. They couldn't possibly look at that, because the entire population was hospitalized. So they can't logically draw any conclusions about whether vaccination reduced hospitalization or not.

For example, let's say they looked at 3000 people in the community, 1000 of whom were vaccinated against flu, and 2000 of whom were unvaccinated. And let's say they checked to see who was hospitalized with an illness that looked like flu, and it turned out that 104 vaccinated patients and 65 unvaccinated patients were hospitalized. These are the same numbers as in the study.

Now let's say they checked to see which of the hospitalized patients were flu+, and it turned out that 6 of the vaccinated, and 11 of the unvaccinated patients were flu+.  Again, same numbers.

Then 6/1000 = 0.60% of the vaccinated patients were flu+ and hospitalized,

And 11/2000 = 0.55% of the unvaccinated patients were flu+ and hospitalized.

So how would the vaccination have reduced flu-related hospitalizations by 71%, when the rate of flu-related hospitalization is lower for the unvaccinated patients?

Obviously, I just made up the 1000 and 2000 figures, but my point is you can't know whether vaccination reduced flu-related hospitalizations without knowing how many were NOT hospitalized.

The thing is, I really know very little about statistics. So I suspect I'm missing something here. But I can't figure out what. And in case I'm not missing something, it's a pretty big deal that the CDC is using this result to support their recommendation for universal flu vaccination.

The truth is, a vaccine efficacy of 71% is not so great. By comparison, the inactivated polio vaccine has an efficacy of 90% after 2 doses, and 99% after 3 doses (link). This doesn't mean there isn't good reason to recommend universal flu vaccination. For one thing, older people, who stand to benefit greatly from not getting flu, don't have a good serologic response to the flu vaccine, simply by virtue of age. The best way to protect them, then, is herd immunity, which you can get from having younger adults vaccinated.

I would really appreciate comments on this post. In particular, comments from people who know some statistics and have taken a look at the article. I'd like to know what I'm not seeing correctly, or if perhaps I am seeing things correctly.

Thanks for your help, especially Mickey.

Thank You, Mickey! Part I

Where to begin? I got an email with a letter from the commissioner of the NY DOH that covered three topics:

1. Flu Season
2. Synthetic Cannabinoids
3. NY's Medical Marijuana Program

The bit about Synthetic Cannabinoids started with, "The surging popularity of these man-made drugs has created a serious and sustained public health problem in New York State..."

It went on to describe the problem, and ended the next paragraph with, "Be on the lookout for the use of these drugs by your patients. We need to work together to stop this scourge."

This was immediately followed by:

If you have patients with medical conditions that may benefit from the use of medical marijuana, I would also like to remind you that next month is the launch of New York’s Medical Marijuana Program. I encourage you to enroll in the online course to become a registered physician, so you can certify eligible patients to receive medical marijuana. For more information, please access at: https://www.health.ny.gov/regulations/medical_marijuana/practitioner/.

Granted, synthetic cannabinoids are not the same thing as medical marijuana, and the letter even remarks on this point. But you'd think someone would have noticed the irony in the juxtaposition, and the complete absence of any comment about the problems with medical marijuana.

However, this post is really about flu vaccination. I know I've written about this topic before, and it's not a psychiatric topic, but the interpretation of research findings is a psychiatric and clinically relevant topic, and this turned into an exercise in understanding the literature.

 The commissioner's comment on Flu includes the following:

It’s impossible to predict the severity and timing of any flu season. Every year, however, flu causes widespread illness. Last year’s flu sickened approximately 51,000 people in New York, causing more than 11,000 hospitalizations and six pediatric deaths. The Centers for Disease Control and Prevention recommends an annual vaccine for everyone over six months of age... One study showed that flu vaccination reduced flu-related hospitalizations among adults of all ages by 71 percent.

No reference was given for the "one study".

This topic holds personal relevance for me because I don't get the flu vaccination, and as a result, I have to agree that I will not be around patients in my affiliated hospital without wearing a mask.

The reason I don't get the flu vaccine is the Cochrane Review (the link goes to the summary of the review, from which you can access the article). Briefly, they looked at a total of 90 studies, 20% of which had a high risk of bias, and 10% of which had good quality methodology. The evidence is current through May of 2013.

They found that the Number Needed to Vaccinate (NNV) to prevent one case of influenza-like illness (ILI) was 40 (CI: 26-128), and 71 (CI: 64-80) to prevent one case of confirmed influenza, in the case of parenteral inactivated vaccine. For live aerosol vaccine the NNV for ILI was 46 (CI: 29-115).

In addition, "Vaccination had a modest effect on time off work and had no effect on hospital admissions or complication rates." I view time off work as a good measure of severity of illness, because if you're really sick, you don't go to work.

That's why I don't get the flu vaccine.

But I was curious about this statement from the DOH letter that, "One study showed that flu vaccination reduced flu-related hospitalizations among adults of all ages by 71 percent." I mean, according to Cochrane, vaccination has no effect on hospitalization. And according to this study, it reduces flu-related hospitalization by 71%.

The first thing I looked at were the CDC recommendations, and none of their references seemed relevant. Please note that I was only looking for information about adults. There may be some stuff there about kids.

So then I looked for the 71% study, and I found it through a link on NPR, of all places. The study is entitled,
Effectiveness of influenza vaccine for preventing laboratory-confirmed influenza
hospitalizations in adults, 2011-2012 influenza season, by Talbot et al. It was funded by the CDC,  and ultimately published in Clinical Infectious Diseases.

Talbot, et al did a case-positive control-negative analysis of 169 adult patients admitted to a hospital for something that looked like flu. That is, after eliminating patients who didn't meet eligibility criteria, they had 169 left. They were able to track down the vaccination status for these patients, and they tested them for flu.

It turned out that 11 of 65 (17%) non-vaccinated patients were positive for flu, and 6 of 104 patients (6%) were positive for flu. There were confounding factors, e.g. the vaccinated group was older, and the non-vaccinated group smoked more.

They concluded:

"Unadjusted vaccine effectiveness was 71.1% (95%CI: 17.3%, 89.9%) for all adults and 76.8% (24.1%, 92.9%) for adults ≥50 years. Adjusted vaccine effectiveness for preventing influenza associated hospitalizations was 71.4% (95%CI: 17.1%, 94.9%) for all adults and 76.8% (24.0%, 97.9%) for adults ≥50 years."

This is where I got stuck. There was a table at the end, but it didn't help all that much, except to give me the 11 and 6 numbers above. I didn't know how they got their 71%, and I didn't know how to reconcile their findings with Cochrane's conclusions.

So I asked for help. I emailed Mickey, who writes the 1boringoldman blog. It's a great psychiatry blog-you'll see a link listed on my blog roll, to the right. Please visit it early and often. An embarrassing fact is that when I first started reading it, I thought Mickey's last name must be, "Goldman". Or, "Oldman". It's not.

Three of Mickey's recent posts, In the Land of Sometimes 1, In the Land of Sometimes 2, and In the Land of Sometimes 3, are excellent statistics tutorials. Mickey is also one of the people responsible for republishing Paxil Study 329, which involved quite a bit of statistical knowledge. So I thought he'd probably be able to help.

That I thought he'd be interested in and willing to help is probably a testament to his decency, which comes through in his blog posts. We'd had a couple of email exchanges in the past, so I wasn't contacting him completely cold, but he really didn't need to help at all. And within a few hours of emailing him, he got back to me with an explanation, computation, and references, and I am close to tears over his generosity as I type this.

I'm going to stop here, and continue in the next post with how to compute Vaccine Efficacy, how the 71% statistic came to be, and my discussion with Mickey about the conclusions.

Thanks, Mickey!

Flu Season

It's that time of year, again. Time to get a flu vaccine. Well, time to decide whether or not to get a flu vaccine.

I never do. Why?

1. You can get vaccinated and still get the flu, it's specific for certain strains.
2. I'm always already sick by the time I think of getting vaccinated.
3. I know the chance of getting Guillian-Barré is, literally, one in a million. But I happen to know someone who was that one. That doesn't change the odds, but it makes it feel like it does, and I'd rather have the flu.
4. I don't work in a hospital setting, so I'm not at a higher risk than most "healthy adults".
5. I don't treat geriatric patients, so I'm not putting my patients at higher risk.

However, I decided to be a little more responsible this year, so I looked it up in The Cochrane Reviews. They have this really nice section called, PEARLS - Practical Evidence About Real Life Situations (it's at the lower half of the linked page).

Here's the conclusion:

Bottom line: 

There is insufficient evidence to decide whether routine vaccination to prevent influenza in healthy adults is effective. Influenza vaccination did not affect the number of people needing to go to hospital or to take time off work (the follow up period was up to 3 months post vaccine).

Caveat: 

Vaccination against influenza avoided 80% of cases at best (in those confirmed by laboratory tests, and using vaccines directed against circulating strains), but only 50% when the vaccine did not match, and 30% against influenza-like illness. Some vaccines cause pain and redness at the injection site (NNH* 1), muscle ache (NNH 27), and other very rare serious harms such as transient paralysis. *NNH = number needed to treat to cause harm in one individual.

Incidentally, The Cochrane Reviews Site is awesome. Check it early and often. The Cochrane Collaboration was established in 1993, and is named after the British epidemiologist and medical researcher, Archie Cochrane (1909-1988).

Archie Cochrane

I'm always disappointed that it isn't named after Doc Cochran from Deadwood, SD-apparently there were several doctors in the camp, but the name is fictional.

Brad Dourif as Doc Cochran

I guess this is a peculiar post for a psych blog, but I'm a doctor, and I treat patients in my office, and flu vaccination is an important question. And who says lifelong learning has to be restricted to psychiatry?