Chapter 4

The role of probability theory in statistics

We collect data so as to provide evidentiary support for
answers we give to our many questions about the world
(and in our particular case, about the business world). As
we have seen, our questions often concern themselves with
very large populations which are nearly impossible to census, so when we collect data, we must restrict ourselves to
rather small samples from these populations.
A natural question that arises is How do we select a particular individual from the population of interest to become
part of the sample we measure? It turns out that random
sampling from a population is the best method to employ
(this fact will be analyzed later in the course; see Chapter
7 of the textbook).
Consequently, the important statistical features of the sample we draw are inherently unpredictable. Nonetheless,
there are general conclusions that can be made, even of
uncertain characteristics like the properties of a randomly
selected sample; these kinds of conclusions are precisely
what probability theory is designed to handle. So we devote some time to studying its basic principles.

Chapter 4

Elements of probability theory

(probabilistic) experiment
situation in which one of a collection of possible outcomes could occur, but precisely which one cannot be
predicted with certainty
sample space (S)
the exhaustive collection of all the possible outcomes of
some probabilistic experiment
event (A, B, . . . )
any result of the experiment described by one or more
possible outcomes from the sample space
probability (P (A))
measure of the likelihood of an event; its long-run relative frequency
subjective make an educated guess
empirical calculate the fraction of attempted trials
in which the event has occurred
a priori use a mathematical model to describe the
likelihood of occurrence
2

Chapter 4

odds
alternative method for describing likelihood of occurrence of an event
If P (A) is the probability that event A occurs, then
the odds in favor of A is given as the ratio P (A) to
P (Ac), while the odds against A is the inverse ratio
P (Ac) to P (A)
Conversely, if the odds in favor of event A is stated as
a
a to b, then P (A) = a+b
, whereas if the odds against
b
event A is stated as a to b, then P (A) = a+b
Venn diagram
diagram of the sample space of an event (represented
by a rectangle) that depicts the relations among various
collections of outcomes (represented by circles which
might overlap); a very useful tool to help in the computation of probabilities

Chapter 4

exhaustive events
events which cover all possibilities from the sample space
of the probabilistic experiment
disjoint/mutually exclusive events
events which have no outcomes in common, that is, can
never occur simultaneously
independent events
events one of whose outcomes has no influence on the
outcomes of the other, that is, the likelihood of the
occurrence of one is unaffected by whether the other
takes place or not

Chapter 4

Formal rules of probability

1. Probability measures likelihood : P (A) lies between 0
and 1 for any event A.
2. Something has to happen: Where S is the event consisting of the entire sample space, P (S) = 1.
3. Equally likely outcomes have equal probabilities: If there
are n equally likely possible outcomes and event A includes exactly k of these outcomes, then P (A) = k/n.
4. Complementary events have complementary probabilities: P (Ac) = 1 P (A).
5. Addition rule for disjoint events: If A and B are disjoint
events, then their total probability is
P (A B) = P (A or B) = P (A) + P (B).
6. Multiplication rule for independent events: If A and B
are independent events, then their joint probability is
P (A B) = P (A and B) = P (A) P (B).

Chapter 4

More probability rules

General Addition Rule
If A and B are any two events, then
P (A or B) = P (A) + P (B) P (A and B).

conditional probability
If A and B are any two events, then the conditional
probability P (B|A) of event B given event A is the
frequency of the outcomes in B conditioned by the
outcomes in A; that is,
(rel.) freq. of outcomes in B also in A
P (B|A) =
(rel.) freq. of outcomes in A
which is equivalent to the definition:
P (A B)
.
P (B|A) =
P (A)
General Multiplication Rule
If A and B are any two events, then
P (A B) = P (A) P (B|A).

Chapter 4

independent events
Events are independent precisely when their conditional
probabilities are the same as their unconditional probabilities; that is, when either one (and thus both) of
these formulas hold:
P (B|A) = P (B),
P (A|B) = P (A).
contingency table
Paired qualitative data is organized in a table whose
columns list the categories of one variable x and whose
rows list the categories of the other variable y; each
cell of the table counts the joint frequency of individuals who simultaneously fall into both that column and
row category
tree diagram
a diagram of the outcomes of pairs of successive events,
in which the first level of branches represent outcomes
of one event and the second layer outcomes of the second; useful for working with conditional probabilities

Chapter 4

total probability rule

To study the influence on event A of event B, it is
useful to separate those outcomes described by A into
those which are common to B, namely the joint event
A B, and those disjoint from B, which is the joint
event A B c; from this it follows that
P (A) = P (A B) + P (A B c)
= P (A|B)P (B) + P (A|B c)P (B c)
prior probability
the probability P (A) of some event A before consideration of new information in the guise of the occurrence
of a second event B; in other words, the unconditional
probability of A relative to B
posterior probability
the conditional probability P (A|B) of event A, evaluated after consideration of new information in the guise
of the occurrence of event B

Chapter 4

The General Multiplication Rule implies that

P (A|B) P (B) = P (A B) = P (B|A) P (A),
but the Total Probability Rule states that
P (B) = P (A B) + P (Ac B)
= P (B|A)P (A) + P (B|Ac)P (Ac),
so we deduce the formula
P (A|B) [P (B|A)P (A) + P (B|Ac)P (Ac)] = P (A B)
= P (B|A) P (A)
from which follows

Bayes Theorem
a formula that describes how to find the posterior probability P (A|B) involving a pair of events A and B when
the probability of the conditional event B is not known:
P (A|B) =

P (B|A)P (A)
P (B|A)P (A) + P (B|Ac)P (Ac)

Chapter 4

Counting Rules
Many probability computations require the enumeration of
outcomes of some probabilistic experiment; consequently,
rules for counting collections of objects are useful to have
available.
n factorial (n!)
the product of all the integers from 1 to n (where by
convention we always define 0! = 1)
permutations (nPx)
arrangements of objects in which the order of selection
matters; if x objects are selected from a total of n
objects, then the number of possible permutations of
these objects is
n!
P
=
n x
(n x)!
combinations (nCx)
arrangements of objects in which the order of selection
does not matter; if x objects are selected from a total
of n objects, then the number of possible combinations
of these objects is
 
n
n!
C
=
=
n x
x
x!(n x)!
10