COVID-19 vaccine “95% effective”: It doesn’t mean what you think it means!


COVID-19 has the world more than ever in its grip – but there is hope: several vaccines have been developed which promise to deliver “95% efficacy”.

When people read this many assume that it means that 95% of vaccinated persons will be protected from infection – but that is not true. Even many (science) journalists get it wrong! If you want to understand what it really means, read on!

Let us take the example of the German biotech company BioNTech which developed its mRNA vaccine BNT162b2 in cooperation with pharmaceutical giant Pfizer. First of all, let me make it very clear that this is an extraordinary achievement and I am hopeful that it will prove to be a real game-changer in the fight against this horrible virus. I stress this because nowadays you cannot be cautious enough not to be instrumentalized by all kinds of strange people who have an interest in downplaying this devastating pandemic.

Now, what does a so-called efficacy rate of 95% mean? Let me start by stating what it doesn’t mean: it doesn’t mean that 95 out of 100 vaccinated persons will be protected from COVID-19, nor does it mean that it will reduce the severity of the illness in case you contract the virus despite being vaccinated.

To understand the real meaning let us go through the press release of Pfizer:

Primary efficacy analysis demonstrates BNT162b2 to be 95% effective against COVID-19 beginning 28 days after the first dose;170 confirmed cases of COVID-19 were evaluated, with 162 observed in the placebo group versus 8 in the vaccine group

What Pfizer did was to select about 43,000 voluntary participants where about half received the vaccine and the other half (= the control group) received only a placebo, without any active substance. After about a month after the first dose (i.e. one week after the second dose), they started to count the number of confirmed COVID-19 cases for each group: in the placebo group 162 cases were confirmed, whereas in the vaccine group only 8 cases appeared.

To get to the 95%-number the following calculation was performed:

(1 - 8/162) * 100
## [1] 95.06173

So the 95% is the relative risk-reduction in infections, it doesn’t tell us the absolute probability of not getting infected despite being vaccinated!

We already covered relative vs. absolute risk reductions (and its dangers) in this blog. There we saw that personograph plots (also called Kuiper-Marshall plots) are an excellent way to communicate risks.

Let us start by illustrating the confirmed COVID-19 cases in the placebo (= control) group for a better manageable group size of 2,500…

library(personograph) # first install from CRAN
## Loading required package: grImport
## Loading required package: grid
## Loading required package: XML

n <- 2500
inf_wo_vac <- 20 / n

data <- list(first = inf_wo_vac, second = 1-inf_wo_vac)
personograph(data,  colors = list(first = "red", second = "lightgrey"),
             fig.title = "20 of 2500 infected without vaccine",
             draw.legend = FALSE, n.icons = n, dimensions = c(25, 100), 
             plot.width = 0.97)

…and now for the vaccine group of the same size:

inf_w_vac <- 1 / n

data <- list(first = inf_w_vac, second = 1-inf_w_vac)
personograph(data,  colors = list(first = "red", second = "lightgrey"),
             fig.title = "1 of 2500 infected despite of vaccine",
             draw.legend = FALSE, n.icons = n, dimensions = c(25, 100), 
             plot.width = 0.97)

I think that those plots really put matters in perspective.

The main problem is that vaccine studies cannot directly measure what we really want to know: the effectiveness of the vaccine in the real world, i.e. how well it protects us from contracting the disease. Therefore they use efficacy as a proxy instead, i.e. relative risk-reduction of infections in the two study groups. This can give a good indication of the order of magnitude of the real-world effect but is not the same!

So while “95% effective” indeed does mean something different from what most people think it means let us hope that it translates into a real turning point for the better.

6 thoughts on “COVID-19 vaccine “95% effective”: It doesn’t mean what you think it means!”

  1. Thanks for these thoughts, that are quite related to similar problems of risk communication, e.g. in cancer screening, and prevention in general.

    Still, I don’t completely agree with your conclusion.

    Formally, you are right in that the 95% are the relative risk reduction as outlined in the publication (Polack et al. New England Journal of Medicine; December 10, 2020).

    On the contrary, in my view the personograph does not represent what people seek for, either, including the arbitrary choice of a n=2.500 base. My experience is that people would interpret s.th. like the “95% effective” phrase, like that the vaccine would be “working” in 95% of the vaccinated individuals. “Working” would mean not gettig infected if exposed in a relevant way.

    Although, we can’t really say s.th. about individuals “exposed in a relevant way” from the Pfizer phase 3 trial, the randomized design would practically allow for using the number of infected persons in the placebo arm as a fairly representative proxy.

    Best, Michael
    If so, the n=162 from the placebo group would be best guess of a number also applying to the vaccinated group and therefore you could well communicate that the vaccine hypothetically worked in n=154 of those hypothetical 162 vaccinated individuals exposed to SARS-Cov-2, which again would mean that “it worked” in 95.1% of them.

    1. Dear Michael, thank you for your comment!

      I agree that people are not looking for that but I still think that the above plots represent the outcome of those trials in a proper way. The base n = 2500 is not that arbitrary but is the smallest whole number which allows for the representation of just one person in the second plot.

      I also agree that the outcome of those trials is a good proxy for what we really want to know, I guess even the best one can come up with without employing unethical study designs.

      best h
      P.S.: Sent you a LinkedIn invite!

Leave a Reply

Your email address will not be published. Required fields are marked *

I accept that my given data and my IP address is sent to a server in the USA only for the purpose of spam prevention through the Akismet program.More information on Akismet and GDPR.

This site uses Akismet to reduce spam. Learn how your comment data is processed.