An Introduction to Bayes' Theorem

Bayes' Theorem is a theorem of probability theory originally stated by the Reverend Thomas
Bayes. It can be seen as a way of understanding how the probability that a theory is true is
affected by a new piece of evidence. It has been used in a wide variety of contexts, ranging from
marine biology to the development of "Bayesian" spam blockers for email systems. In the
philosophy of science, it has been used to try to clarify the relationship between theory and
evidence. Many insights in the philosophy of science involving confirmation, falsification, the
relation between science and pseudosience, and other topics can be made more precise, and
sometimes extended or corrected, by using Bayes' Theorem. These pages will introduce the
theorem and its use in the philosophy of science.

Begin by having a look at the theorem, displayed below. Then we'll look at the notation and
terminology involved.

In this formula, T stands for a theory or hypothesis that we are interested in testing, and E
represents a new piece of evidence that seems to confirm or disconfirm the theory. For any
proposition S, we will use P(S) to stand for our degree of belief, or "subjective probability," that
S is true. In particular, P(T) represents our best estimate of the probability of the theory we are
considering, prior to consideration of the new piece of evidence. It is known as the prior
probability of T.

What we want to discover is the probability that T is true supposing that our new piece of
evidence is true. This is a conditional probability, the probability that one proposition is true
provided that another proposition is true. For instance, suppose you draw a card from a deck of
52, without showing it to me. Assuming the deck has been well shuffled, I should believe that the
probability that the card is a jack, P(J), is 4/52, or 1/13, since there are four jacks in the deck.
But now suppose you tell me that the card is a face card. The probability that the card is a jack,
given that it is a face card, is 4/12, or 1/3, since there are 12 face cards in the deck. We represent
this conditional probability as P(J|F), meaning the probability that the card is a jack given that
it is a face card.
(We don't need to take conditional probability as a primitive notion; we can define it in terms of
absolute probabilities: P(A|B) = P(A and B) / P(B), that is, the probability that A and B are both
true divided by the probability that B is true.)

Using this idea of conditional probability to express what we want to use Bayes' Theorem to
discover, we say that P(T|E), the probability that T is true given that E is true, is the posterior
probability of T. The idea is that P(T|E) represents the probability assigned to T after taking
into account the new piece of evidence, E. To calculate this we need, in addition to the prior
probability P(T), two further conditional probabilities indicating how probable our piece of
evidence is depending on whether our theory is or is not true. We can represent these as P(E|T)
and P(E|~T), where ~T is the negation of T, i.e. the proposition that T is false.

Example

A desk lamp produced by The Luminar Company was found to be defective (D). There are three
factories (A, B, C) where such desk lamps are manufactured. A Quality Control Manager (QCM)
is responsible for investigating the source of found defects. This is what the QCM knows about
the company's desk lamp production and the possible source of defects:

Factory % of total production Probability of defective lamps

A 0.35 = P(A) 0.015 = P(D | A)

B 0.35 = P(B) 0.010 = P(D | B)

C 0.30 = P(C) 0.020 = P(D | C)

The QCM would like to answer the following question: If a randomly selected lamp is defective,
what is the probability that the lamp was manufactured in factory C?
Now, if a randomly selected lamp is defective, what is the probability that the lamp was
manufactured in factory A? And, if a randomly selected lamp is defective, what is the probability
that the lamp was manufactured in factory B?

Solution. In our previous work, we determined that P(D), the probability that a lamp
manufactured by The Luminar Company is defective, is 0.01475. In order to find P(A | D) and
P(B | D) as we are asked to find here, we need to perform a similar calculation to the one we
used in finding P(C | D). Our work here will be simpler, though, since we've already done the
hard work of finding P(D). The probability that a lamp was manufactured in factory A given that
it is defective is:

P(A|D)=P(AD)P(D)=P(D|A)P(A)P(D)=(0.015)(0.35)0.01475=0.356

And, the probability that a lamp was manufactured in factory B given that it is defective is:

P(B|D)=P(BD)P(D)=P(D|B)P(B)P(D)=(0.01)(0.35)0.01475=0.237

Note that in each case we effectively turned what we knew upside down on its head to find out
what we really wanted to know! We wanted to find P(A | D), but we knew P(D | A). We
wanted to find P(B | D), but we knew P(D | B). We wanted to find P(C | D), but
we knew P(D | C). It is for this reason that I like to say that we are interested in finding "reverse
conditional probabilities" when we solve such problems.

The probabilities P(A), P(B) and P(C) are often referred to as prior probabilities, because they
are the probabilities of events A, B, and C that we know prior to obtaining any additional
information. The conditional probabilities P(A | D), P(B | D), and P(C | D) are often referred to
as posterior probabilities, because they are the probabilities of the events after we have
obtained additional information.

As a result of our work, we determined:

P(C | D) = 0.407
P(B | D) = 0.237
P(A | D) = 0.356

Calculated posterior probabilities should make intuitive sense, as they do here. For example, the
probability that the randomly selected desk lamp was manufactured in Factory C has increased,
that is, P(C | D) > P(C), because Factory C generates the greatest proportion of defective lamps
(P(D | C) = 0.02). And, the probability that the randomly selected desk lamp was manufactured
in Factory B has decreased, that is, P(B | D) < P(B), because Factory B generates the smallest
proportion of defective lamps (P(D | B) = 0.01). It is, of course, always a good practice to make
sure that your calculated answers make sense.

Let's now go and generalize the kind of calculation we made here in this defective lamp example
.... in doing so, we summarize what is called Bayes' Theorem.