Base Rate Fallacy – or why No One is justified to believe that Jesus rose


In this post we are talking about one of the most unintuitive results in statistics: the so called false positive paradox which is an example of the so called base rate fallacy. It describes a situation where a positive test result of a very sensitive medical test shows that you have the respective disease… yet you are most probably healthy!

The reason for this is that the disease itself is so rare that even with a very sensitive test the result is most probably false positive: it shows that you have the disease yet this result is false, you are healthy.

The key to understanding this result is to understand the difference between two conditional probabilities: the probability that you have a positive test result when you are sick and the probability that you are sick in case you got a positive test result – you are interested in the last (am I really sick?) but you only know the first.

Now for some notation (the vertical dash means “under the condition that”, P stands for probability):

  • P(B \mid A): if you are sick (A) you will probably have a positive test result (B) – this is what we know
  • P(A \mid B): if you have a positive test result (B) you are probably not sick (A) – this is what we want to know

To calculate one conditional probability from the other we use the famous Bayes’ theorem:

    \[P(A\mid B) = \frac{P(B \mid A) \, P(A)}{P(B)}\]

In the following example we assume a disease with an infection rate of 1 in 1000 and a test to detect this disease with a sensitivity of 99%. Have a look at the following code which illustrates the situation with Euler diagrams, first the big picture, then a zoomed-in version:

library(eulerr)

A <- 0.001 # prevalence of disease
BlA <- 0.99 # sensitivity of test

B <- A * BlA + (1 - A) * (1 - BlA) # positive test (specificity same as sensitivity)
AnB <- BlA * A
AlB <- BlA * A / B # Bayes's theorem
#AnB / B # Bayes's theorem in different form

C <- 1 # the whole population
main <- paste0("P(B|A) = ", round(BlA, 2), ", but P(A|B) = ", round(AlB, 2))

set.seed(123)
fit1 <- euler(c("A" = A, "B" = B, "C" = C, "A&B" = AnB, "A&C" = A, "B&C" = B, "A&B&C" = AnB), input = "union")
plot(fit1, main = main, fill = c("red", "green", "gray90"))

fit2 <- euler(c("A" = A, "B" = B, "A&B" = AnB), input = "union")
plot(fit2, main = main, fill = c("red", "green"))

As you can see although this test is very sensitive when you get a positive test result the probability of you being infected is only 9%!

In the diagrams C is the whole population and A are the infected individuals. B shows the people with a positive test result and you can see in the second diagram that almost all of the infected A are also part of B (the brown area = true positive), but still most ob B are outside of A (the green area), so although they are not infected they have a positive test result! They are false positive.

The red area shows the people that are infected (A) but get a negative test result, stating that they are healthy. This is called false negative. The grey area shows the people who are healthy and get a negative test result, they are true negative.

Due to the occasion we are now coming to an even more extreme example: did Jesus rise from the dead? It is inspired by the very good essay “A drop in the sea”: Don’t believe in miracles until you’ve done the math.

Let us assume that we had very, very reliable witnesses (as a side note what is strange though is that the gospels cannot even agree on how many men or angels appeared at the tomb: it is one angel in Matthew, a young man in Mark, two men in Luke and two angels in John… but anyway), yet the big problem is that not many people so far have been able to defy death. I have only heard of two cases: supposedly the King of Kings (Jesus) but also of course the King himself (Elvis!), whereby sightings of Elvis after his death are much more numerous than of Jesus (just saying… 😉 )

Have a look at the following code (source for the number of people who have ever lived: WolframAlpha)

A <- 2/108500000000 # probability of coming back from the dead (The King = Elvis and the King of Kings = Jesus)
BlA <- 0.9999999 # sensitivity of test -> very, very reliable witnesses (many more in case of Elvis 😉

B <- A * BlA + (1 - A) * (1 - BlA) # positive test = witnesses say He rose
AnB <- BlA * A
AlB <- BlA * A / B # Bayes's theorem

C <- 1 # all people
main <- paste0("P(B|A) = ", round(BlA, 2), ", but P(A|B) = ", round(AlB, 2))

fit1 <- euler(c("A" = A, "B" = B, "C" = C, "A&B" = AnB, "A&C" = A, "B&C" = B, "A&B&C" = AnB), input = "union")
plot(fit1, main = main, fill = c("red", "green", "gray90"))

fit2 <- euler(c("A" = A, "B" = B, "A&B" = AnB), input = "union")
plot(fit2, main = main, fill = c("red", "green"))

So, in this case C is the unfortunate group of people who have to go for good… it is us. 🙁 As you can see although the witnesses are super reliable when they claim that somebody rose it is almost certain that they are wrong:

  • P(B \mid A): if Jesus rose (A) the very, very reliable witnesses would with a very high probability say so (B)
  • P(A \mid B): if the very, very reliable witnesses said that Jesus rose (B) Jesus would still almost surely have stayed dead

Or in the words of the above mentioned essay:

No one is justified in believing in Jesus’s resurrection. The numbers simply don’t justify the conclusion.

But this chimes well with a famous Christian saying “I believe because it is absurd” (or in Latin “Credo quia absurdum”) – you can find out more about that in another highly interesting essay: ‘I believe because it is absurd’: Christianity’s first meme

Unfortunately this devastating conclusion is also true in the case of Elvis…

26 thoughts on “Base Rate Fallacy – or why No One is justified to believe that Jesus rose”

  1. This seems like a really fun case to demonstrate the difference between Bayesian vs. Frequentist analysis. I’d be fun to see an analysis on how strong a prior would be needed to generate P(A|B) > epsilon .

  2. It’s an overstatement that nobody is justified to believe that Jesus rose. I ran your code for the resurrection part; P(A|B) = 0.0001843. Suppose the probability of rising again is what you gave. All that I need to make P(A|B) larger is to make the witnesses more reliable. When I set P(B|A) = 1 – 1e-11, then P(A|B) = 0.6483. Not bad, right? When I set P(B|A) = 1 – 1e-12, P(A|B) = 0.9485.

    Another comment about the base rate fallacy: it does not mean that doctors screw up most of the time when diagnosing relatively rare diseases or that you shouldn’t believe in diagnoses you receive, because the textbook example about diagnostics does not happen in real life. In real life, when it comes to the prevalence of the disease, i.e. P(A) or the prior, it depends on which population you are talking about. People don’t go to hospitals for medical tests for no reason, and few if anyone would test for every single disease that can be named, so the prevalence shouldn’t be among the general population. Normally, those who seek the medical test are already not feeling well and suspecting the disease of interest to begin with. Which means the prevalence of the disease among those who actually get the medical test is much higher than in the general population. Which means P(A|B) will be larger. For instance, suppose 1 in 10 people suspecting the disease in your example has the disease (same prevalence in general population and same accuracy of test), i.e. P(A) = 0.1, then P(A|B) = 0.9167, so you have good reason to believe that you have the disease. Don’t let the base rate fallacy delay your treatments!

    Now come back to if it’s justified to believe that Jesus rose again. Again, it’s related to the prior. One thing to note about miracles: not every rare event can count as miracle, as miracles need theological significance to count as miracles. In orthodox Christianity, Jesus is no random person, so for Christians, the prior is stronger. When he walked on earth, he had a mission to die for our sins and rise again to conquer our death in sins, so we can be justified by grace, not by law, and be reconciled with God the Father. For Christians, it makes sense to believe that someone with a special theological mission is more likely than some random person to rise from the dead if doing so is relevant to the mission, and the number of people with special theological missions are rare. Of course, one also needs to believe that rising from the dead is possible, however unlikely it is, to have that prior. So with the prior, Christians can be justified to believe that Jesus rose again, and that’s why, in Acts, the apostles — who also heard Jesus saying that he would rise again in 3 days (Matthew 20:17-19) — accepted the resurrection more readily than random people who didn’t consider Jesus special. In contrast, for Christians, the prior for Elvis rising from the dead is much lower as Elvis had no theological mission, so Christians are not as likely to believe in witness accounts of Elvis rising again (except for Elvis fanatics, but presumably those aren’t good Christians anyway).

    This on its own is not apologetics (and I’m not doing apologetics, just to highlight the importance of priors), since of course, you may dispute that prior. There’re all sorts of reasons why people lean towards Christianity. In sum, all that your post amounts to is that we need pretty much infallible witnesses (like the probability of being wrong is 1e-11) to convince someone who does not believe that Jesus had a theological mission relevant to rising again to believe that he rose again. So for Christians, still, happy Easter!

    1. Thank you for your thoughtful and balanced comment.

      I think it is a good first step to challenge your own beliefs and check on your priors. Of course the historicity of the resurrection is of the utmost importance for Christians: Paul writes in 1 Corinthians 15:17: “And if Christ has not been raised, your faith is futile; you are still in your sins.” And to be honest, the quality of the witnesses doesn’t seem to be that good considering the different resurrection accounts…

      Another point: it is always important to keep your priors below 1 or above 0. When you start with 100% certainty (like some fundamentalists, religious or otherwise), no amount of evidence could ever persuade you!

  3. Dear blogger,
    The miracles are up from the logic in most cases
    and always there is a small probability to happen something important, like life itself.

    Christ is risen!
    and life is given!

    1. Thank you for your comment.

      “The miracles are up from the logic in most cases and always there is a small probability to happen something important”: is this equally true for all “miracles” of all religions or only for Christian ones?

      1. A miracle is a form the God’s will that can happen to all the people that believe in some religion.

        Confusion arises when we equate the apocalypse of Christ (religion) with the human-created religions.
        So the answer is that miracle can happen to everyone but from the only One God!

          1. “All Non-Christians are just confused”: no and yes.

            No, because all the people can leave a miracle and
            Yes, because they attribute miracles in human/creatures and not in the One God.

            If we want to speak more precise the God’s miracles are signs that don’t impose His Will to mankind but give us the opportunity to believe or not!

    1. I personally don’t find this very persuasive because it presupposes that the Christian God is the real God which the miracle should establish (or at least strengthen) in the first place. On top of that it is a symmetric argument: every other religion could argue the exact same way and nothing would be gained.

      But again, thank you for your contribution.

      1. Yes, you have right every person who believes somewhere has his own “truth” evidence for the God(s) that believe.

        The trip for the Truth is personal and apart from the only “listened miracles” someone eventually will believe easier if a “miracle” happened to himself.

        Thank you for hosting my opinion-beliefs!

  4. I do not think I follow the logic. You claim the a priori probability of raising from the dead (A) is minuscule because only two persons are claimed to have done so out of many million. Let’s get this down to one of many million.

    Now, let us think you are stabbed in the back while leisurely walking. Reliable witnesses say in court that Mr. X, whom they happen to know well, was the person who attacked you.

    There are millions of would-be attackers close to you and the a priori probability that Mr. X is the one, is minuscule: one in several million. Would you, as a member of the jury, discard the testimony of the witnesses against Mr. X based on the fact that the a priori probability of him being the attacker is minuscule?

    1. Thank you for your request for clarification!

      No, it is not just the prior but the process of Bayesian updating.

      Concerning your example, let us take the biggest German city, Berlin, with about 3.5 million inhabitants: when you run the numbers with the same witnesses as above you get a probability of nearly 3/4 so it is quite probable that Mr. X is the perpetrator!

      Does this clarify things?

      1. Not quite, I am afraid; if one person taken at random from the population of Berlin is shown to the witness and the witness identifies him as the attacker, I would agree that the probability of being so is what you claim (I did not check the computations).

        But this is not quite what happens here: the witness is telling you that ONE person, Mr. X, is the attacker. He is not looking to anybody else, there is no chance of false positives. I think you would have to judge the testimony with no reference to how many people is around. Failing to do so means that the same evidence might be enough for a conviction in Berlin, but not in New York or Shanghai. Which seems puzzling.

        Thank you for your effort in clarifying this, may be it is just that I am a little too dense.

        1. Why should this be puzzling? If the murderer is from a group of 2 people (where there is a fifty-fifty chance already) less evidence would be needed than if he came from a group of 2 million people… seems intuitive enough, don’t you think?

  5. If the witness is presented two people, one of which is certain to be the murderer, I would agree that his/her choice of one as the culprit might be more convincing than if he/she has to choose with the same probabilities of error from 3.5 million. But I do not see this as being the setup here. The witness is telling you that Mr. X, who he/she knows well, is the murderer. How many other people might have been murderers, seems to me of no consequence.

  6. But let’s now turn to your original argument. I see bayesian updating kind of paradoxical here. If you think that the a priori probability of rising, P(A), is zero, then that’s it: end of the argument.

    But you say that P(A) = 2/108500000000, hence you admit that it is possible for a person to rise from the dead. Then, you say that about the only person for which such claim has been put forward (I was unaware of a similar claim for Elvis) is very unlikely to have risen. Who else, then, since from your setting of P(A) there have been two?

    For your argument to be convincing, the witnesses should have had a look to 108500000000 persons. That, with a probability 0.01 of false positive, would have produced about 1085000000 false claims and then, yes, the probability that precisely Jesus of Nazareth had risen out of that many would be very small. This is not the case: not nearly that many claims of persons rising from the dead have been put forward. The witnesses did not see that many people –or else their probability or claiming that some one had risen when he/she did not was very nearly zero.

    1. The Elvis thing was supposed to be a joke, I wouldn’t have expected that I had to explain this…

      Concerning the prior: I see your point but as you said, setting it to zero wouldn’t make any sense. Yet it has to be very minuscule otherwise even the Christian message wouldn’t be so strong (if risen people were to roam the streets on a regular basis Jesus’ resurrection would be met with a shrug of the shoulder). You can see the prior as a first best guess.

      Also the number assigned as the quality of the witnesses is debatable (why 0.9999999 and not 0.99999999 or 0.99999?). I gave you the code, you can play around with the numbers yourself.

      The whole point of this exercise is to illustrate this point:

      “Extraordinary claims require extraordinary evidence” (Carl Sagan)

      In the case of Christianity we sure have a very extraordinary claim, yet the quality of the evidence is wanting (to say the least) in comparison.

  7. I have gone through the discussions and your responses all the way. Just as you challenged Christian to reconsider their belief in the Resurrection of CHRIST, you also need to challenge your belief in weak witnesses! I will attempt to explain some issues in my understanding and leave you to conclude as you would.

    1. The Bible, even though an open book that all can read, only those with the ability to understand its teaching can get a meaning out of it. It is like some chemical equations being read by an Historian or a text in Chinese of Japanese being read by and English! If indeed the letters are understood, the essence would be difficult to comprehend. That is why it is said that “the scriptures is spirit-breathed”, and only those who have the spirit understands it.

    2. No one had control over the documents presented in the Bible. Otherwise, David’s escapade with Bathsheba would have been excluded from a ‘holy book’! Only in later times when varied documents were being paraded as ‘the word of God’ that certain criteria were agreed upon to be met by any document that is ‘the word of God’. Even then we still have differences. For example, the Roman Catholic Church and or the Methodist Church still regard some other documents as in the Bible.

    3. Your doubts as to how many angels or people were found at the tomb of Jesus is just one of the dynamics of a book that was not tailored along ‘positive and corrected’ stories but to try and give various shades of opinion about what people either saw or experienced or heard or all. With this in view, let me pull certain facts out of the stories that were reported about the Resurrection:
    The Jews were apprehensive of the Disciples of Christ and so heavily guarded the tomb so that the error at the end will not be worse than the beginning. Matthew 27: 62-66.
    Only the guards were at the tomb when the actual Resurrection took place but Matthew reported there was an earthquake (Matthew 28 : 2) before it happened. Of course the guards were as dead men (Matthew 28: 4) when it happen and immediately fled to the Jewish leaders who now worked out a corrected story to suite their mischief (Matthew 28 : 11- 15).
    As regards to how many Angels were there, it is all a matter of how we now put the whole events together. But three angels were on assignment on this Resurrection work. One rolled away the heavy stone put against the entrance to the tomb which actually is contrary to the popular beliefs of the Jews of that time. The tomb is covered in such a way that if the dead come back to life (the Jews had the belief that a dead person can come back to life within four days John 11 : 39), he could come out by himself. The other two sat at the position where Jesus’s body was laid. So those who reported one saw the angel standing outside after rolling the stone (Matthew 27 : 2 – 3, Mark 16 : 5) while those who saw two did not report the sight of the angel standing outside but the two who were by his body position (John 20 : 11 – 12, Luke 24 : 4).
    The last point does not negate the super-naturalness of the Resurrection, at least Jesus Himself predicted that He would died and be raised after three days.
    The women were therefore at the tomb to freshen the body since it was not four days yet!
    In conclusion, and in this instance, the most credible witnesses were the guards but were compromised. The account by John is from eye witness account while that of Matthew, Mark and Luke were reported speech. The three women, Peter and John are the witness of the events after the guards fled and took bribes!

    1. Dear colleague, Dear Job,

      Thank you for your comment.

      We are both in the teaching business. I don’t know you but I would suppose that you are a committed and dedicated colleague. You want your students to succeed and understand the topics that you present to them. So you will go out of your way to give them all the help they need “to get it”, right!?!

      Concerning your first point: What I don’t understand is the didactics of God. We are His children and I hope that He is interested in saving His children. So all of His children need to understand His message. Therefore I have a fundamental problem with your “The Bible, even though an open book that all can read, only those with the ability to understand its teaching can get a meaning out of it.”

      My question for you: Is God not willing to write a book where everybody “can get a meaning out of it” – or is He not able to? In either case, why?

      Thank you

      best
      h

  8. Thanks for recognising my little contribution to the debate.
    While I am incapable of defending God is His Chosen methods of revealing Himself to mankind, what I alluded to in the scripture, which seems to be a point of concern, is, who is able to understand the Bible?

    To my mind, the matter is simple. You are a data scientist today because you went through some courses of training. In the same vein, a lawyer, a medical doctor, an engineer etc. did not qualify on the first day of enrollment but after passing series of examinations. So God loves us that he would want to save us, but only those who actually ‘enrol’ and ‘pass’ all prescribed ‘examinations’ get to understand Him through His written word, the Bible. And that process begins by being born again just as Jesus Himself told Nicodemus in John Chapter three which I quote inter alia “Now there was a man of the Pharisees, named Nicodemus, a ruler of the Jews; this man came to Jesus by night and said to Him, “Rabbi, we know that You have come from God as a teacher; for no one can do these signs that You do unless God is with him.” Jesus answered and said to him, “Truly, truly, I say to you, unless one is born again he cannot see the kingdom of God.” Nicodemus *said to Him, “How can a man be born when he is old? He cannot enter a second time into his mother’s womb and be born, can he?” Jesus answered, “Truly, truly, I say to you, unless one is born of water and the Spirit he cannot enter into the kingdom of God. That which is born of the flesh is flesh, and that which is born of the Spirit is spirit. Do not be amazed that I said to you, ‘You must be born again.’ The wind blows where it wishes and you hear the sound of it, but do not know where it comes from and where it is going; so is everyone who is born of the Spirit.” Nicodemus said to Him, “How can these things be?” Jesus answered and said to him, “Are you the teacher of Israel and do not understand these things? Truly, truly, I say to you, we speak of what we know and testify of what we have seen, and you do not accept our testimony. If I told you earthly things and you do not believe, how will you believe if I tell you heavenly things? No one has ascended into heaven, but He who descended from heaven: the Son of Man. As Moses lifted up the serpent in the wilderness, even so must the Son of Man be lifted up; so that whoever believes will in Him have eternal life. “For God so loved the world, that He gave His only begotten Son, that whoever believes in Him shall not perish, but have eternal life. For God did not send the Son into the world to judge the world, but that the world might be saved through Him. He who believes in Him is not judged; he who does not believe has been judged already, because he has not believed in the name of the only begotten Son of God. This is the judgment, that the Light has come into the world, and men loved the darkness rather than the Light, for their deeds were evil. For everyone who does evil hates the Light, and does not come to the Light for fear that his deeds will be exposed. But he who practices the truth comes to the Light, so that his deeds may be manifested as having been wrought in God.”

    Thanks.

    1. I can relate to your parable of “passing examinations”. Unfortunately He as a teacher has a failure rate of way over 70% (assuming that all two billion Christians will be saved and the 5.5 billion Non-Christians won’t). Possibly it is much higher because not everybody who calls him- or herself Christian really deserves the title, as we both know.

      So, would you call a teacher with a failure rate of way over 70% the best teacher ever? I seriously doubt it! I guess it would be very implausible to only blame the students, right? Even for us meek humans there are so many possibilites to help our students. Just writing a big, convoluted script and hiding oneself, on the other hand, would be a good strategy to keep and increase that failure rate even further.

      As you correctly acknowledged, I am a data scientist, I hold a PhD and I am a professor… yet obviously even I am too stupid to understand the Bible and do fail the exams!

      Now, as they say “With great power comes great responsibility”, but you tell me that “With absolute power comes no responsibility”? It is all our fault and He gets a free pass? About 99% of all the books I have ever read (and I have read a lot!) are better at communicating what they want to tell me than the Bible (and I have also read the Bible). Isn’t that strange that an omnipotent being is not able to communicate as clearly as most of His creatures… please think about that!

      Thank you

  9. God is:
    Sovereign
    Omniscient
    Omnipotent
    Gracious
    Loving.

    He loves all but has given the power to choose to each individual. Those who choose to accept His plan of Salvation consummated through the death and resurrection of Jesus gets to hear Him and understand His Word. According to Jesus, “My sheep hears my voice and would not hear the voice of strangers”. The strangers are thieves and robbers and try to gain access to the sheep fold through the ‘window’. God’s open arm is anxiously waiting for all those who have decided to accept His plan of Salvation.

    1. “Those who choose to accept His plan of Salvation […] understand His Word.” – do you think that this is a especially clever communication strategy? Don’t you think that more people would be saved if He communicated more clearly, so that everybody understands His word? And on what basis would you decide to choose to accept His plan anyway if you don’t understand the guy? Tossing a coin? (Speaking of: coin tosses will be the starting point of one of my upcoming posts on statistical tests… so stay tuned!)

      “God’s open arm is anxiously waiting […].” Perhaps He should not just sit and wait? Doesn’t seem to be working too well…

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.