Probability

Prof. F.P. Kelly

Lent 1996

These notes are maintained by Andrew Rogers.

Comments and corrections to soc-archim-notes@lists.cam.ac.uk.
Revision: 1.1
Date: 1998/06/24 14:38:21

The following people have maintained these notes.

– June 2000 Kate Metcalfe

June 2000 – date Andrew Rogers
Contents

Introduction v

1 Basic Concepts 1
1.1 Sample Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Classical Probability . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.3 Combinatorial Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 Stirling’s Formula . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 The Axiomatic Approach 5

2.1 The Axioms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3 Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.4 Conditional Probability . . . . . . . . . . . . . . . . . . . . . . . . . 9

3 Random Variables 11
3.1 Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2 Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.3 Indicator Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.4 Inclusion - Exclusion Formula . . . . . . . . . . . . . . . . . . . . . 18
3.5 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

4 Inequalities 23
4.1 Jensen’s Inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2 Cauchy-Schwarz Inequality . . . . . . . . . . . . . . . . . . . . . . . 26
4.3 Markov’s Inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.4 Chebyshev’s Inequality . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.5 Law of Large Numbers . . . . . . . . . . . . . . . . . . . . . . . . . 28

5 Generating Functions 31
5.1 Combinatorial Applications . . . . . . . . . . . . . . . . . . . . . . . 34
5.2 Conditional Expectation . . . . . . . . . . . . . . . . . . . . . . . . 34
5.3 Properties of Conditional Expectation . . . . . . . . . . . . . . . . . 36
5.4 Branching Processes . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.5 Random Walks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

6 Continuous Random Variables 47

6.1 Jointly Distributed Random Variables . . . . . . . . . . . . . . . . . 50
6.2 Transformation of Random Variables . . . . . . . . . . . . . . . . . . 57
6.3 Moment Generating Functions . . . . . . . . . . . . . . . . . . . . . 64

iii
iv CONTENTS

6.4 Central Limit Theorem . . . . . . . . . . . . . . . . . . . . . . . . . 67

6.5 Multivariate normal distribution . . . . . . . . . . . . . . . . . . . . 71
Introduction

These notes are based on the course “Probability” given by Prof. F.P. Kelly in Cam-
bridge in the Lent Term 1996. This typed version of the notes is totally unconnected
with Prof. Kelly.
Other sets of notes are available for different courses. At the time of typing these
courses were:
Probability Discrete Mathematics
Analysis Further Analysis
Methods Quantum Mechanics
Fluid Dynamics 1 Quadratic Mathematics
Geometry Dynamics of D.E.’s
Foundations of QM Electrodynamics
Methods of Math. Phys Fluid Dynamics 2
Waves (etc.) Statistical Physics
General Relativity Dynamical Systems
Combinatorics Bifurcations in Nonlinear Convection

They may be downloaded from

http://www.istari.ucam.org/maths/ or
http://www.cam.ac.uk/CambUniv/Societies/archim/notes.htm
or you can email soc-archim-notes@lists.cam.ac.uk to get a copy of the
sets you require.

v
 Copyright (c) The Archimedeans, Cambridge University.
All rights reserved.
Redistribution and use of these notes in electronic or printed form, with or without
modification, are permitted provided that the following conditions are met:

1. Redistributions of the electronic files must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in printed form must reproduce the above copyright notice, this
list of conditions and the following disclaimer.
3. All materials derived from these notes must display the following acknowledge-
ment:

This product includes notes developed by The Archimedeans, Cambridge

University and their contributors.

4. Neither the name of The Archimedeans nor the names of their contributors may
be used to endorse or promote products derived from these notes.
5. Neither these notes nor any derived products may be sold on a for-profit basis,
although a fee may be required for the physical act of copying.
6. You must cause any edited versions to carry prominent notices stating that you
edited them and the date of any change.

THESE NOTES ARE PROVIDED BY THE ARCHIMEDEANS AND CONTRIB-

UTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING,
BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABIL-
ITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
EVENT SHALL THE ARCHIMEDEANS OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSE-
QUENTIAL DAMAGES HOWEVER CAUSED AND ON ANY THEORY OF LI-
ABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUD-
ING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THESE NOTES, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAM-
AGE.
Chapter 1

Basic Concepts

1.1 Sample Space

Suppose we have an experiment with a set of outcomes. Then is called the sample
space. A potential outcome ! 2 is called a sample point.
For instance, if the experiment is tossing coins, then = fH
T g, or if the experi-
ment was tossing two dice, then = f(i
j ) : i
j 2 f1
: : :
6gg.
A subset A of is called an event. An event A occurs is when the experiment is
performed, the outcome ! 2 satisfies ! 2 A. For the coin-tossing experiment, then
the event of a head appearing is A = fH g and for the two dice, the event “rolling a
four” would be A = f(1
3)
(2
2)
(3
1)g.

1.2 Classical Probability

If is finite, = f!1
: : :
!n g, and each of the n sample points is “equally likely”
then the probability of event A occurring is

jAj
P(A) =
j j
Example. Choose r digits from a table of random numbers. Find the probability that
for 0
k
9,

1. no digit exceeds k ,

2. k is the greatest digit drawn.

Solution. The event that no digit exceeds k is

Ak = f(a1
: : :
ar ) : 0
ai
k
i = 1 : : : rg :
Now jAk j = (k + 1)r , so that P(Ak ) = k10
;
r
+1 .

Let Bk be the event that k is the greatest digit drawn. Then Bk = Ak n Ak;1 . Also
Ak;1  Ak , so that jBk j = (k + 1)r ; kr . Thus P(Bk ) = (k+1)10r;k
r r

1
2 CHAPTER 1. BASIC CONCEPTS

The problem of the points

Players A and B play a series of games. The winner of a game wins a point. The two
players are equally skillful and the stake will be won by the first player to reach a target.
They are forced to stop when A is within 2 points and B within 3 points. How should
the stake be divided?
Pascal suggested that the following continuations were equally likely
AAAA AAAB AABB ABBB BBBB
AABA ABBA BABB
ABAA ABAB BBAB
BAAA BABA BBBA
BAAB
BBAA
This makes the ratio 11 : 5. It was previously thought that the ratio should be 6 : 4
on considering termination, but these results are not equally likely.

1.3 Combinatorial Analysis

The fundamental rule is:
Suppose r experiments are such that the first may result in any of n 1 possible out-
comes and such that for each of the possible outcomes of the first i ; 1 experiments
there are ni possible outcomes to experiment i. Let ai be the outcome of experiment i.
Q
r
Then there are a total of i=1 ni distinct r-tuples (a1
: : :
ar ) describing the possible
outcomes of the r experiments.
Proof. Induction.

1.4 Stirling’s Formula

For functions g (n) and h(n), we say that g is asymptotically equivalent to h and write
g(n)  h(n) if hg((nn)) ! 1 as n ! 1.
Theorem 1.1 (Stirling’s Formula). As n ! 1,

log p n! !0
2nnn e;n
p
and thus n!  2nnn e;n.
We first prove the weak form of Stirling’s formula, that log(n!)  n log n.
Proof.
P
log n! = n1 log k. Now
Zn X
n Z n+1
log xdx
log k
log xdx

1 1
R z logx dx = z log z ; z + 1, and so
1

and 1
n log n ; n + 1
log n!
(n + 1) log(n + 1) ; n:
Divide by n log n and let n ! 1 to sandwich nlog n!
log n between terms that tend to 1.
Therefore log n!  n log n.
1.4. STIRLING’S FORMULA 3

Now we prove the strong form.

Proof. For x > 0, we have

1 ; x + x2 ; x3 < 1 +1 x < 1 ; x + x2 :
Now integrate from 0 to y to obtain

y ; y2 =2 + y3 =3 ; y4 =4 < log(1 + y) < y ; y2 =2 + y3 =3:

= log nnn+1
!e n
Let hn 1
=2 . Then we obtain

1 1 1 1
12n2 ; 12n3
hn ; hn+1
12n2 + 6n3 :
For n  2, 0P
hn ; hn+1
P 1
n21. Thus hn is a decreasing sequence, and 0

n
h2 ;hn+1
r=2(hr ;hr+1 )
1 r12 . Therefore hn is bounded below, decreasing
so is convergent. Let the limit be A. We have obtained

n!  eAnn+1=2 e;n :
We need a trick to find A. Let Ir
R
= 0
=2 sinr  d. We obtain the recurrence Ir =
(2n n!)2
r;1 Ir;2 by integrating by parts.
r Therefore I 2n = (2(2nnn)!!)2 =2 and I2n+1 = (2 n+1)! .
Now In is decreasing, so

1
I I2n
II2n;1 = 1 + 21n ! 1:

2n+1 2n+1

But by substituting our formula in, we get that

I2n  2n + 1 2 2
I2n+1  2 n e2A ! e2A :
Therefore e2A = 2 as required.

1 by 1
playing silly buggers with log 1 + n
4 CHAPTER 1. BASIC CONCEPTS
Chapter 2

The Axiomatic Approach

2.1 The Axioms

Let be a sample space. Then probability P is a real valued function defined on subsets
of satisfying :-

1. 0
P(A)
1 for A  ,

2. P( ) = 1,
3. for a finite or infinite sequence A1
A2
  
P  of disjoint events, P( Ai ) =
i P(Au ).
The number P(A) is called the probability of event A.
We can look at some distributions here. Consider an arbitrary finite or countable
= f!1
!2
: : : g and an arbitrary collection fp1
p2
: : : g of non-negative numbers
with sum 1. If we define
X
P(A) = pi

i:!i 2A
it is easy to see that this function satisfies the axioms. The numbers p1
p2
: : : are
called a probability distribution. If is finite with n elements, and if p1 = p2 =    =
pn = n1 we recover the classical definition of probability.
Another example would be to let = f0
1
: : : g and attach to outcome r the
probability pr = e; r! for some  > 0. This is a distribution (as may be easily
r

verified), and is called the Poisson distribution with parameter .

Theorem 2.1 (Properties of P). A probability P satisfies

1. P(Ac ) = 1 ; P(A),

2. P(
) = 0,

3. if A  B then P(A)
P(B ),

4. P(A B ) = P(A) + P(B ) ; P(A \ B ).

5
6 CHAPTER 2. THE AXIOMATIC APPROACH

Proof. Note that = A Ac , and A \ Ac =

. Thus 1 = P( ) = P(A)+ P(Ac ). Now
we can use this to obtain P(
) = 1 ; P(
c ) = 0. If A  B , write B = A (B \ Ac ),
so that P(B ) = P(A) + P(B \ Ac )  P(A). Finally, write A B = A (B \ Ac )
and B = (B \ A) (B \ Ac ). Then P(A B ) = P(A) + P(B \ Ac ) and P(B ) =
P(B \ A) + P(B \ Ac ), which gives the result.
Theorem 2.2 (Boole’s Inequality). For any A1
A2
    ,
n ! X
n
P Ai
P(Ai )

1 ! X
1
1 i
P Ai
P(Ai )
1 i
Proof. Let B1 = A1 and then inductively let Bi
S
= Ai n i1;1 Bk . Thus the Bi ’s are
S S
disjoint and i Bi = i Ai . Therefore
 !  !
P Ai = P Bi
i X i
= P(Bi )
i
X

P(Ai ) as Bi  Ai :
i

Theorem 2.3 (Inclusion-Exclusion Formula).

n ! 0 1
X \
P Ai = (;1)jSj;1 P@ Aj A :
1 S f1:::ng j 2S
S 6=
Proof. We know that P(A1 A2 ) = P(A1 ) + P(A2 ) ; P(A1 \ A2 ). Thus the result
is true for n = 2. We also have that
P(A1    An ) = P(A1    An;1 ) + P(An ) ; P((A1    An;1 ) \ An ) :
But by distributivity, we have
n ! n;1 ! n;1 !
P Ai = P Ai + P(An ) ; P (Ai \ An ) :
i 1 1
Application of the inductive hypothesis yields the result.
Corollary (Bonferroni Inequalities).
0 1
 n !
X \ 
(;1)jSj;1 P@ Aj A or P Ai
S f1:::rg j 2S  1
S 6=
according as r is even or odd. Or in other words, if the inclusion-exclusion formula is
truncated, the error has the sign of the omitted term and is smaller in absolute value.
Note that the case r = 1 is Boole’s inequality.
2.2. INDEPENDENCE 7

Proof. The result is true for n = 2. If true for n ; 1, then it is true for n and 1
r

n ; 1 by the inductive step above, which expresses a n-union in terms of two n ; 1
unions. It is true for r = n by the inclusion-exclusion formula.
Example (Derangements). After a dinner, the n guests take coats at random from a
pile. Find the probability that at least one guest has the right coat.
Solution. Let Ak be the event that guest k has his1 own coat.
S
n
We want P( i=1 Ai ). Now,

P(Ai1 \    \ Air ) =
(n ; r)!

n!
by counting the number of ways of matching guests and coats after i 1
: : :
ir have
taken theirs. Thus
X n
(n ; r)!
P(Ai1 \    \ Air ) =
r n! = r1!

i1 <<ir
and the required probability is
 n ! n;1
P Ai = 1 ; 2!1 + 3!1 +    + (;1)n!

i=1
which tends to 1 ; e;1 as n ! 1.
Furthermore, let Pm(n) be the probability that exactly m guests take the right coat.
Then P0 (n) ! e;1 and n! P0(n) is the number of derangements of n objects. There-
fore
n
1  P (n ; m)  (n ; m)!
Pm (n) =
0
m n!
;
= P0(nm;! m) ! em! as n ! 1:
1

2.2 Independence
Definition 2.1. Two events A and B are said to be independent if

P(A \ B ) = P(A) P(B ) :

More generally, a collection of events Ai , i 2 I are independent if

\ ! Y
P Ai = P(Ai )
i2J i2J
for all finite subsets J  I.
Example. Two fair dice are thrown. Let A1 be the event that the first die shows an odd
number. Let A2 be the event that the second die shows an odd number and finally let
A3 be the event that the sum of the two numbers is odd. Are A1 and A2 independent?
Are A1 and A3 independent? Are A1 , A2 and A3 independent?
1 I’m not being sexist, merely a lazy typist. Sex will be assigned at random...
8 CHAPTER 2. THE AXIOMATIC APPROACH

Solution. We first calculate the probabilities of the events A1 , A2 , A3 , A1 \A2 , A1 \A3

and A1 \ A2 \ A3 .

Event Probability

A1 18
36 = 21
A2 As above, 1
2

A3 63 = 21
36

A1 \ A2 33 = 41
36

33
A1 \ A3 36 = 41
A1 \ A2 \ A3 0
Thus by a series of multiplications, we can see that A1 and A2 are independent, A1
and A3 are independent (also A2 and A3 ), but that A1 , A2 and A3 are not independent.

Now we wish to state what we mean by “2 independent experiments” 2. Consider

1 = f 1
: : : g and 2 = f
1
: : : g with associated probability distributions fp1
: : : g
and fq1
: : : g. Then, by “2 independent experiments”, we mean the sample space
1  2 with probability distribution P(( i

j )) = pi qj .
Now, suppose A  1 and B  2 . The event A can be interpreted as an event in
1  2 , namely A  2 , and similarly for B . Then
X X X
P(A \ B ) = p i qj = pi qj = P(A) P(B )

i 2A i 2A j 2B
j 2B
which is why they are called “independent” experiments. The obvious generalisation
to n experiments can be made, but for an infinite sequence of experiments we mean a
sample space 1  2  : : : satisfying the appropriate formula 8n 2 N .
You might like to find the probability that n independent tosses of a biased coin
with the probability of heads p results in a total of r heads.

2.3 Distributions
The binomial distribution with parameters n and p, 0
p
1 has
;
= f0
: : :
ng and
probabilities pi = ni pi (1 ; p)n;i .

Theorem 2.4 (Poisson approximation to binomial). If n ! 1, p ! 0 with np = 

held fixed, then
n
r
r n;r ; 
r p (1 ; p) ! e r! :
2 or more generally, n.
2.4. CONDITIONAL PROBABILITY 9

Proof.
r n;r n(n ; 1) : : : (n ; r + 1) pr (1 ; p)n;r
r p (1 ; p) = r!
= nn n ; 1 : : : n ; r + 1 (np)r (1 ; p)n;r
n n r!
Yr n ; i + 1
r 
n 
;r
= n r! 1 ; n 1; n
i=1
r
! 1   e;  1
r!
 r
; 
= e r! :

Suppose an infinite sequence of independent trials is to be performed. Each trial

results in a success with probability p 2 (0
1) or a failure with probability 1 ; p. Such
a sequence is called a sequence of Bernoulli trials. The probability that the first success
occurs after exactly r failures is pr = p(1 ; p)r . This is the geometric distribution with
P
parameter p. Since 1 0 pr = 1, the probability that all trials result in failure is zero.

2.4 Conditional Probability

Definition 2.2. Provided P(B ) > 0, we define the conditional probability of AjB 3 to
be
P(A \ B )
P(AjB ) = :
P(B )
Whenever we write P(AjB ), we assume that P(B ) > 0.
Note that if A and B are independent then P(AjB ) = P(A).
Theorem 2.5. 1. P(A \ B ) = P(AjB ) P(B ),
2. P(A \ B \ C ) = P(AjB \ C ) P(B jC ) P(C ),
A\B jC ) ,
3. P(AjB \ C ) = P(P(B jC )

4. the function P(jB ) restricted to subsets of B is a probability function on B .

Proof. Results 1 to 3 are immediate from the definition of conditional probability. For
result 4, note that A \ B  B , so P(A \ B )
P(B ) and thus P(AjB )
1. P(B jB ) =
1 (obviously), so it just remains to show the last axiom. For disjoint Ai ’s,
  ! S
P Ai B = P( iP(A(Bi \) B ))
i P P(A \ B)
= i i
P(B )
X
= P(Ai jB )
as required.
i

3 read “A given B ”.
10 CHAPTER 2. THE AXIOMATIC APPROACH

Theorem 2.6 (Law of total probability). Let B1
B2
: : : be a partition of . Then

X
P(A) = P(AjBi ) P(Bi ) :
i
Proof.
X X
P(AjBi ) P(Bi ) = P(A \ Bi )
 !
=P A \ Bi
i
= P(A)
as required.

Example (Gambler’s Ruin). A fair coin is tossed repeatedly. At each toss a gambler
wins $1 if a head shows and loses $1 if tails. He continues playing until his capital
reaches m or he goes broke. Find px , the probability that he goes broke if his initial
capital is $x.
Solution. Let A be the event that he goes broke before reaching $m, and let H or
T be the outcome of the first toss. We condition on the first toss to get P(A) =
P(AjH ) P(H ) + P(AjT ) P(T ). But P(AjH ) = px+1 and P(AjT ) = px;1 . Thus
we obtain the recurrence

px+1 ; px = px ; px;1 :
Note that px is linear in x, with p0 = 1, pm = 0. Thus px = 1 ; m x.

Theorem 2.7 (Bayes’ Formula). Let B1
B2
: : : be a partition of . Then

P(AjBi ) P(Bi )
P(Bi jA) = P :
j P(AjBj ) P(Bj )
Proof.

P(A \ Bi )
P(Bi jA) =
P(A)
= PP(PA(A
jBi ) P(Bi )
jBj ) P(Bj )


j
by the law of total probability.
Chapter 3

Random Variables

Let be finite or countable, and let p! = P(f!g) for ! 2 .

Definition 3.1. A random variable X is a function X : 7! R.

Note that “random variable” is a somewhat inaccurate term, a random variable is
neither random nor a variable.

Example. If = f(i
j )
1
i
j
tg, then we can define random variables X and

Y by X (i
j ) = i + j and Y (i
j ) = maxfi
j g
Let RX be the image of under X . When the range is finite or countable then the
P
random variable is said to be discrete.
We write P(X = xi ) for !:X (!)=xi p! , and for B  R

X
P(X 2 B) = P(X = x) :
x2B
Then

(P(X = x)
x 2 RX )
is the distribution of the random variable X . Note that it is a probability distribution
over RX .

3.1 Expectation
Definition 3.2. The expectation of a random variable X is the number
X
E X] = pw X (!)
!2
provided that this sum converges absolutely.

11
12 CHAPTER 3. RANDOM VARIABLES

Note that

X
E X] = pw X (!)
! 2
X X
= p! X (!)
x2RX !:X (!)=x
X X
= x p!
x2RX !:X (!)=x
X
= xP(X = x) :
x2RX

Absolute convergence allows the sum to be taken in any order.

If X is a positive random variable and if
P
!2 p! X (! ) = 1 we write E  X ] =
+1. If
X
xP(X = x) = 1 and
x2RX
x 0
X
xP(X = x) = ;1
x2RX
x<0

then E  X ] is undefined.

= r) = e; r! , then E  X ] = .

r
Example. If P(X

Solution.

X
1
E X] = re; rr!
r=0
X
1 r;1
= e; (r ; 1)! = e; e = 
r=1

Example. If P(X
;

= r) = nr pr (1 ; p)n;r then E  X ] = np.
3.1. EXPECTATION 13

Solution.
X
n

E X] = rpr (1 ; p)n;r nr

r=0
Xn
= r r!(nn;! r)! pr (1 ; p)n;r
r=0
Xn
= n (r ;(n1)!( ; 1)!
n ; r )! pr (1 ; p)n;r
r=1
X n
= np (r ;(n1)!( ; 1)!
n ; r )! pr;1 (1 ; p)n;r
r=1
nX;1 n ; 1)!
= np (r()!( n ; r )! pr (1 ; p)n;1;r
r=1
nX;1 n ; 1

= np r n;1;r
r=1 r p (1 ; p)
= np

For any function f : R 7! R the composition of f and X defines a new random

variable f and X defines the new random variable f (X ) given by

f (X )(w) = f (X (w)):
Example. If a, b and c are constants, then a + bX and (X ; c)2 are random variables
defined by

(a + bX )(w) = a + bX (w) and

(X ; c)2 (w) = (X (w) ; c)2 :
Note that E  X ] is a constant.
Theorem 3.1.
1. If X  0 then E  X ]  0.
2. If X  0 and E  X ] = 0 then P(X = 0) = 1.
3. If a and b are constants then E  a + bX ] = a + bE  X ].
4. For any random variables X , Y then E  X + Y ] = E  X ] + E  Y ].
h i
5. E  X ] is the constant which minimises E (X ; c)2 .

Proof. 1. X  0 means Xw  0 8 w 2
X
So E  X ] = p! X (!)  0
! 2
2. If 9! 2 with p! > 0 and X (!) > 0 then E  X ] > 0, therefore P(X = 0) = 1.
14 CHAPTER 3. RANDOM VARIABLES

3.
X
E  a + bX ] = (a + bX (!)) p!
!2
X X
=a p! + b p! X (!)
!2 !2
= a + E X] :
4. Trivial.

5. Now

E
 (X ; c)2 = E  (X ; E  X ] + E  X ] ; c)2

= E (X ; E  X ])2 + 2(X ; E  X ])(E  X ] ; c) + (E  X ] ; c)]2 ]

= E (X ; E  X ])2 + 2(E  X ] ; c)E (X ; E  X ])] + (E  X ] ; c)2

= E (X ; E  X ])2 + (E  X ] ; c)2 :
This is clearly minimised when c = E  X ].

Theorem 3.2. For any random variables X1
X2
::::
Xn

"X
n # X
n
E Xi = E  Xi ]
i=1 i=1
Proof.
"X
n # " nX
;1 #
E Xi = E Xi + Xn
i=1 i=1
" nX #
;1
=E Xi + E  X ]
i=1
Result follows by induction.

3.2 Variance

Var X = E X 2 ; E  X ]2 for Random Variable X
= E  X ; E  X ]]2 = 2
p
Standard Deviation = Var X

Theorem 3.3. Properties of Variance

(i) Var X  0 if Var X = 0, then P(X = E  X ]) = 1

Proof - from property 1 of expectation
(ii) If a
b constants, Var (a + bX ) = b2 Var X
3.2. VARIANCE 15

Proof.
Var a + bX = E  a + bX ; a ; bE  X ]]
= b2 E  X ; E  X ]]
= b2 Var X

(iii) Var X

= E X 2 ; E  X ]2
Proof.
E X

; E  X ]]2 = E X 2 ; 2X E  X ] + (E  X ])2


= E X 2 ; 2E  X ] E  X ] + E  X ]2

= E X 2 ; (E  X ])2

Example. Let X have the geometric distribution P(X = r) = pqr with r = 0
1
2:::
and p + q = 1. Then E  X ] = pq and Var X = pq2 .
Solution.
X
1 X 1
E X] = rpqr = pq rqr;1
r=0 r=0
1
= pq
X1 d (qr ) = pq d  1 
r=0 dq dq 1 ; q
= pq(1 ; q);2 = pq

E
 X 2 = X
1
r2 p2 q2r
r=0
X
1 X
1

= pq r(r + 1)qr;1 ; rqr;1

r=1 r=1
= pq( (1 ;2 q)3 ; (1 ;1 q)2 = 2pq2 ; pq

Var X = E X 2 ; E  X ]2
= 2q ; q ; q
2

p2 p p
= pq2

Definition 3.3. The co-variance of random variables X and Y is:

Cov(X
Y ) = E (X ; E  X ])(Y ; E  Y ])]
The correlation of X and Y is:

Corr(X
Y ) = pCov(X
Y )
Var X Var Y
16 CHAPTER 3. RANDOM VARIABLES

Linear Regression

Theorem 3.4. Var (X + Y ) = Var X + Var Y + 2Cov(X
Y )

Proof.

Var (X + Y ) = E (X + Y )2 ; E  X ] ; E  Y ] 2


= E (X ; E  X ])2 + (Y ; E  Y ])2 + 2(X ; E  X ])(Y ; E  Y ])

= Var X + Var Y + 2Cov(X
Y )

3.3 Indicator Function

Definition 3.4. The Indicator Function I A] of an event A  is the function

(
I A](w) = 1
if ! 2 A
(3.1)
0
if ! 2= A:

NB that I A] is a random variable

1.

E  I A]] = P(A)
X
E  I A]] = p! I A](w)
! 2
= P(A)

2. I Ac ] = 1 ; I A]
3. I A \ B ] = I A]I B ]
4.

I A B ] = I A] + I B ] ; I A]I B ]
I A B ](!) = 1 if ! 2 A or ! 2 B
I A B ](!) = I A](!) + I B ](!) ; I A]I B ](!) WORKS!

Example. n  couples are arranged randomly around a table such that males and fe-
males alternate. Let N = The number of husbands sitting next to their wives. Calculate
3.3. INDICATOR FUNCTION 17

the E  N ] and the Var N .

X
n
N= I Ai ] Ai = event couple i are together
"
i=1
X
n #
E N ] = E I Ai ]
i=1
X
n
= E  I Ai ]]
i=1
Xn 2
= n
i=1
Thus E  N ] = n
2 =2
2 n n
X !2 3

E N2 = E4 I A ] 5 i
2 0i=1n 13
X 2
X
= E 4@ I Ai ] + 2 I Ai ]I Aj ]A5
i=1 i
j

= nE I Ai ] + n(n ; 1)E (I A1 ]I A2 ])]
2

 I A ]2 = E  I A ]] = 2
E i i n
E (I A1 ]I A2 ])] = I E A1 \ B2 ]] = P(A1 \ A2 )
= P(A1 ) P(A2 jA1 )
2
1 1 n;2 2

= n n;1n;1 ; n;1n;1

Var N = E N 2 ; E  N ]2
= n; 2 (1 + 2(n ; 2)) ; 2
1
2(
= n;1n ; 2)
18 CHAPTER 3. RANDOM VARIABLES

3.4 Inclusion - Exclusion Formula


N \
N !c
Ai = Aci
"
N #
1
"
1
N !c #
\
I Ai = I c Ai
1
"\
1
N #
=1;I Aci
1
Y
N
=1; I Aci ]
1
Y
N
=1; (1 ; I Ai ])
1
X
N X
= I Ai ] ; i1
i2 I A1 ]I A2 ]
1
X
+ ::: + (;1)j+1 I A1 ]I A2 ]:::I Aj ] + :::
i1
i2 :::
ij

Take Expectation
"
N # 
N !
E Ai = P Ai
1 1
X
N X
= P(Ai ) ; i1
i2 P(A1 \ A2 )
1
X ;A

+ ::: + (;1)j+1 P i1 \ Ai2 \ :::: \ Aij + :::
i1
i2 :::
ij

3.5 Independence
Definition 3.5. Discrete random variables X1
:::
Xn are independent if and only if
for any x1 :::xn :

Y
N
P(X1 = x1
X2 = x2 :::::::Xn = xn ) = P(Xi = xi )
1

Theorem 3.5 (Preservation of Independence). If

X1
:::
Xn are independent random variables and f1
f2 :::fn are functions R ! R
then f1 (X1 ):::fn (Xn ) are independent random variables
3.5. INDEPENDENCE 19

Proof.
X
P(f1 (X1 ) = y1
: : :
fn (Xn ) = yn ) = P(X1 = x1
: : :
Xn = xn )
x1 :f1 (X1 )=y1 :::
xn :fn (Xn )=yn
YN X
= P(Xi = xi )
1 xi :fi (Xi )=yi
Y
N
= P(fi (Xi ) = yi )
1

Theorem 3.6. If X1 :::::Xn are independent random variables then:

"Y
N # Y
N
E Xi = E  Xi ]
1 1

NOTE that E 
P X ] = P E  X ] without requiring independence.
i i

Proof. Write Ri for RXi the range of Xi

"Y
N # X X
E Xi = :::: x1 ::xn P(X1 = x1
X2 = x2 :::::::
Xn = xn )
x1 2R1 xn 2Rn
N X !
1
Y
= P(Xi = xi )
1 xi 2Ri
Y
N
= E  Xi ]
1

Theorem 3.7. If X1
:::
Xn are independent random variables and f 1 ::::fn are func-
tion R ! R then:
"Y
N # Y
N
E fi (Xi ) = E  fi (Xi )]
1 1

Proof. Obvious from last two theorems!

Theorem 3.8. If X1
:::
Xn are independent random variables then:

X
n ! X
n
Var Xi = Var Xi
i=1 i=1
20 CHAPTER 3. RANDOM VARIABLES

Proof.

X ! 2  n !23 " n #2
n X 5 X
Var Xi = E 4 Xi ; E Xi
i=1 i=1 i=1
2 3 " n #2
X X X
= E 4 Xi2 + Xi Xj 5 ; E Xi
i i6=j i=1
X  X X X
= E X 2 + E  X X ] ; E  X ]2 ; E  X ] E  X ]
i i j i i j
i i6=j i i6=j
X   2 
= E Xi ; E  Xi ]
2

Xi
= Var Xi
i

Theorem 3.9. If X1
:::
Xn are independent identically distributed random variables

then

 X !
1 n
Var n Xi = n1 Var Xi
i=1

Proof.
 X !
1 n
Var n Xi = n12 Var Xi
i=1
Xn
= 1 Var X
n2 i=1 i

= n1 Var Xi

Example. Experimental Design. Two rods of unknown lengths a
b. A rule can

measure the length but with but with error having 0 mean (unbiased) and variance  2 .
Errors independent from measurement to measurement. To estimate a
b we could take
separate measurements A
B of each rod.

E  A] = a Var A = 2
E  B] = b Var B = 2
3.5. INDEPENDENCE 21

Can we do better? YEP! Measure a + b as X and a ; b as Y

E X] = a + b Var X = 2
E Y ] = a ; b Var Y = 2
X +Y 
E
2 =a
Var X + Y = 1 2
X ;Y  2
2
E
2 =b
Var X ; Y 1 2
2 = 2

So this is better.
Example. Non standard dice. You choose 1 then I choose one. Around this cycle

a ! B P(A  B ) = 23 . So the relation ’A better that B’ is not transitive.

22 CHAPTER 3. RANDOM VARIABLES
Chapter 4

Inequalities

4.1 Jensen’s Inequality

A function f  (a
b) ! R is convex if

f (px + qy)
pf (x) + (1 ; p)f (y) - 8x
y 2 (a
b) - 8p 2 (0
1)

Strictly convex if strict inequality holds when x 6= y

f is concave if ;f is convex. f is strictly concave if ;f is strictly convex

23
24 CHAPTER 4. INEQUALITIES

Concave

neither concave or convex.

We know that if f is twice differentiable and f (x)  0 for x 2 (a
b) the if f is
00

convex and strictly convex if f (x)  0 forx 2 (a
b).

00

Example.
f (x) = ; log x
f (x) = ;x1
0

f (x) = x12  0
00

f (x) is strictly convex on (0
1)

Example.
f (x) = ;x log x
f (x) = ;(1 + logx)
0

f (x) = ;x1
0
00

Strictly concave.
Example. f (x = x3 is strictly convex on (0
1) but not on (;1
1)
Theorem 4.1. Let f : (a
b) ! R be a convex function. Then:
X
n X
n !
pi f (xi )  f pi xi
i=1 i=1
x1
: : :
Xn 2 (a
b), p1
: : :
pn 2 (0
1) and
Pp
i = 1. Further more if f is strictly
convex then equality holds if and only if all x’s are equal.
E  f (X )]  f (E  X ])
4.1. JENSEN’S INEQUALITY 25

Proof. By induction on n n = 1 nothing to prove n = 2 definition of convexity.

Assume results holds up to n-1. Consider x1
:::
xn
Pp 2 (a
b), p1
:::
pn 2 (0
1) and
i=1
X
n
For i = 2:::n
set pi = 1 ;pi p
pi = 1
0 0
such that
i i=2
Then by the inductive hypothesis twice, first for n-1, then for 2
X
n X
n
pi fi (xi ) = p1 f (x1 ) + (1 ; p1 ) p i f (x i )
0

1

i=2
X
n !
 p1 f (x1 ) + (1 ; p1 )f pi xi
0

 X
n
i=2 !
 f p1 x1 + (1 ; p1 ) pi xi
0

X
n ! i=2
=f pi xi
i=1
f is strictly convex n  3 and not all the x0i s equal then we assume not all of x2 :::xn
are equal. But then
X
n X
n !
(1 ; pj ) pi f (xi )  (1 ; pj )f pi xi
0 0

i=2 i=2
So the inequality is strict.
Corollary (AM/GM Inequality). Positive real numbers x1
: : :
xn
1Yn ! n
xi

1X
n
x
i=1 n i=1 i
Equality holds if and only if x 1 = x2 =    = xn
Proof. Let

P(X = xi ) = n1
then f (x) = ; log x is a convex function on (0
1).
So
E  f (x)]  f (E  x]) (Jensen’s Inequality)
;E log x]  log E  x] 1]
;
1 X log x
; log 1 X x
n n
Therefore
n i n
Yn ! n1
1 1

xi 1X
n

i=1


n i=1 xi 2]
For strictness since f strictly convex equation holds in [1] and hence [2] if and only if
x1 = x2 =    = xn
26 CHAPTER 4. INEQUALITIES

If f : (a
b) ! R is a convex function then it can be shown that at each point y 2 (a
b)9
a linear function y +
y x such that

f (x)
y +
y x x 2 (a
b)
f (y) = y +
y y
If f is differentiable at y then the linear function is the tangent f (y ) + (x ; y )f (y )
0

Let y = E  X ], = y and
=
y

f (E  X ]) = +
E  X ]
So for any random variable X taking values in (a
b)

E  f (X )]  E  +
X ]
= +
E  X ]
= f (E  X ])

4.2 Cauchy-Schwarz Inequality

Theorem 4.2. For any random variables X
Y ,

E  XY ]
2  E  Y 2

E X2
Proof. For a
b 2 R Let

LetZ = aX ; bY
 
Then0
E Z 2 = E (aX ; bY )2

 
= a2 E X 2 ; 2abE  XY ] + b2 E Y 2
quadratic in a with at most one real root and therefore has discriminant
0.
4.3. MARKOV’S INEQUALITY 27

Take b 6= 0

E  XY ]2
 E Y 2

E X2

Corollary.

jCorr(X
Y )j
1

Proof. Apply Cauchy-Schwarz to the random variables X ; E  X ] and Y ; E  Y ]

4.3 Markov’s Inequality

Theorem 4.3. If X is any random variable with finite mean then,

E  jX j]
P(jX j  a)
for any a  0
a
Proof. Let

A = jX j  a
Then jX j  aI A]
Take expectation

E  jX j]  aP(A)
E  jX j]  aP(jX j  a)

4.4 Chebyshev’s Inequality

Theorem 4.4. Let X be a random variable with E
 X 2
1. Then 8  0

E
 X 2
P(jX j  )

2
Proof.

I jX j  ]
x2 8x
2
28 CHAPTER 4. INEQUALITIES

Then

I jX j  ]
x2
2

Take Expectation
 x2  E
X 2
P(jX j  )
E
2 = 2

Note


1. The result is “distribution free” - no assumption about the distribution of X (other
than E X 2
1).

2. It is the “best possible” inequality, in the following sense

X = + with probability 2c2
= ; with probability 2c2
= 0 with probability 1 ; c2
Then P(jX j  ) = 2
c
 
E X2 = c
c

E X2
P(jX j  ) = 2 =
 2
3. If = E  X ] then applying the inequality to X ; gives

P(X ;  )

Var X
2
Often the most useful form.

4.5 Law of Large Numbers

Theorem 4.5 (Weak law of large numbers). Let X1
X2 ::::: be a sequences of inde-
pendent identically distributed random variables with Variance  2
1 Let

X
n
Sn = Xi
i=1
Then
 Sn 

8  0, P  ;    ! 0 as n ! 1
n
4.5. LAW OF LARGE NUMBERS 29

Proof. By Chebyshev’s Inequality

 S 
E  ( Sn ; )2
P  
n n
n ;   
 2
E (S ; n )2 n
= n2 2 properties of expectation

= Var Sn Since E  S ] = n
n 2
2 n

 S
n = n
 Sn But Var
2

Thus P 
n2 2
;   
2 2 = 2 ! 0
n n  n

Example. A1
A2 ::: are independent events, each with probability p. Let Xi = I Ai ].
Then
Sn = nA = number of times A occurs
n n number of trials

= E  I Ai ]] = P(Ai ) = p
Theorem states that
 Sn 

P  
n ; p   ! 0 as n ! 1
Which recovers the intuitive definition of probability.
Example. A Random Sample of size n is a sequence X1
X2
: : :
Xn of independent
identically distributed random variables (’n observations’)
Pn Xi
X = i=1 n is called the SAMPLE MEAN
Theorem states that provided the variance of Xi is finite, the probability that the sample
mean differs from the mean of the distribution by more than  approaches 0 as n ! 1.
We have shown the weak law of large numbers. Why weak? 9 a strong form of larger
numbers.
S

P
n ! as n ! 1 = 1
n
This is NOT the same as the weak form. What does this mean?
! 2 determines
Sn
n = 1
2
: : :
n
as a sequence of real numbers. Hence it either tends to or it doesn’t.
S (!)

P !:
n ! as n ! 1 = 1
n
30 CHAPTER 4. INEQUALITIES
Chapter 5

Generating Functions

In this chapter, assume that X is a random variable taking values in the range 0
1
2
: : :.
Let pr = P(X = r) r = 0
1
2
: : :
Definition 5.1. The Probability Generating Function (p.g.f) of the random variable
X,or of the distribution pr = 0
1
2
: : : , is

 X 1 X
p(z ) = E z X = z r P(X = r) = pr z r
1
r=0 r=0
This p(z ) is a polynomial or a power series. If a power series then it is convergent for
jz j
1 by comparison with a geometric series.
X X
jp(z )j
pr jz jr
pr = 1
r r
Example.

pr = 16 r = 1
: : :
6
 ;
p(z ) = E z X = 16 1 + z + : : : z 6


= 6z 11;;zz
6

Theorem 5.1. The distribution of X is uniquely determined by the p.g.f p(z ).

Proof. We know that we can differential p(z) term by term for jz j
1

p (z ) = p1 + 2p2z + : : :
0

and so p (0) = p1 (p(0) = p0 )

0

Repeated differentiation gives

X
1 r!
r;i
p(i) (z ) = (r ; i)! pr z
r=i
and has p(i) = 0 = i!pi Thus we can recover p0
p1
: : : from p(z)
31
32 CHAPTER 5. GENERATING FUNCTIONS

Theorem 5.2 (Abel’s Lemma).

E X] = lim p0 (z )
r!1
Proof.
X
1
p0 (z ) = rpr z r;1 jz j
1
r=i
For z 2 (0
1), p0 (z ) is a non decreasing function of z and is bounded above by
X
1
E X] = rpr
r=i
Choose   0, N large enough that
X
N
rpr  E  X ] ; 
r=i
Then
X
1 X
N X
N
lim
r !1
rpr z r;1  rlim
!1 rpr z r;1 = rpr
r=i r=i r=i
True 8  0 and so
E X] = lim p0 (z )
r!1

Usually p0 (z ) is continuous at z=1, then E  X ] = p0 (1).

z 1 ; z6

Recall p(z ) =
6 1;z
Theorem 5.3.
E  X (X ; 1)] = lim p00 (z )
z !1
Proof.
X
1
p00 (z ) = r(r ; 1)pz r;2
r=2
Proof now the same as Abel’s Lemma
Theorem 5.4. Suppose that X1
X2
: : :
Xn are independent random variables with
p.g.f’s p1 (z )
p2 (z )
: : :
pn (z ). Then the p.g.f of
X1 + X2 + : : : Xn
is

p1 (z )p2 (z ) : : : pn (z )
 33

 zX1+X2 +:::Xn = E  zX1 :zX2 : : : :zXn

Proof.
E
  
= E z X1 E z X2 : : : E z Xn
= p1 (z )p2 (z ) : : : pn (z )

Example. Suppose X has Poisson Distribution

r
P(X = r) = e; r! r = 0
1
: : :
Then

E
zX = X
1
z r e; 
r
r=0 r!
= e; e;z
= e;(1;z)
Let’s calculate the variance of X
p = e;(1;z)
0
p = 2 e;(1;z)
00

Then
E X] = lim p (z ) = p (1)( Since p (z ) continuous at z = 1 )E  X ] = 
0 0 0

z !1
E  X (X ; 1)] = p (1) = 2
00


Var X = E X 2 ; E  X ]2
= E  X (X ; 1)] + E  X ] ; E  X ]2
= 2 +  ; 2
=
Example. Suppose that Y has a Poisson Distribution with parameter . If X and Y are

 zX +Y = E  zX E  zY
independent then:
E
= e;(1;z) e; (1;z)
= e;(+ )(1;z)
But this is the p.g.f of a Poisson random variable with parameter  + . By uniqueness
(first theorem of the p.g.f) this must be the distribution for X + Y

Example. X has a binomial Distribution,

P(X = r) = r pr (1 ; p)n;r r = 0
1
: : :
n n

 zX = X
E pr (1 ; p)n;r z r
r
r=0
= (pz + 1 ; p)n
34 CHAPTER 5. GENERATING FUNCTIONS

This shows that X = Y1 + Y2 +    + Yn . Where Y1 + Y2 +    + Yn are independent

random variables each with

P(Yi = 1) = p P(Yi = 0) = 1 ; p
Note if the p.g.f factorizes look to see if the random variable can be written as a sum.

5.1 Combinatorial Applications

Tile a (2  n) bathroom with (2  1) tiles. How many ways can this be done? Say fn

fn = fn;1 + fn;2 f0 = f1 = 1
Let
X
1
F (z ) = fn z n
n=0

fn z n = fn;1z n + fn;2 z n
X
1 X
1 X
1
fn z n = fn;1z n + fn;2 z n
n=2 n=2 n=0
F (z ) ; f0 ; zf1 = z (F (z ) ; f0 ) + z 2 F (z )
F (z )(1 ; z ; z 2 ) = f0 (1 ; z ) + zf1
= 1 ; z + z = 1:
Since f0 = f1 = 1, then F (z ) = 1;z1;z2
Let
p p
1 = 1 +2 5 2 = 1 ;2 5

1
F (z ) = (1 ; z )(1
1 ; 2 z )
= (1 ; z ) ; (1 ; 2 z )
1
1
X
1 X
2
1 !
= ;1 1 n1 z n ; 2 n2 z n
1 2 n=0 n=0
The coefficient of z1n , that is fn , is

fn = ;1 ( n1 +1 ; n2 +1 )
1 2

5.2 Conditional Expectation

Let X and Y be random variables with joint distribution

P(X = x
Y = y)
5.2. CONDITIONAL EXPECTATION 35

Then the distribution of X is

X
P(X = x) = P(X = x
Y = y)
y2Ry
This is often called the Marginal distribution for X . The conditional distribution for X
given by Y = y is

P(X = xjY = y) = P(XP= x
Y = y)

(Y = y)
Definition 5.2. The conditional expectation of X given Y = y is,
X
E X = xjY = y] = xP(X = xjY = y)
x2Rx
The conditional Expectation of X given Y is the random variable E  X jY ] defined by

E  X jY ] (! ) = E  X jY = Y (!)]
Thus E  X jY ] : !R
Example. Let X1
X2
: : :
Xn be independent identically distributed random vari-
ables with P(X1 = 1) = p and P(X1 = 0) = 1 ; p. Let

Y = X1 + X2 +    + Xn
Then

P(X1 = 1jY = r) = P(XP1 (=Y 1=
Yr)= r)

= P(X1 = 1
XP2 (+Y  =
  + Xn = r ; 1)
r)
= P(X1 ) P(X2P+(Y = +r)Xn = r ; 1)
;

p nr;;11 pr;1(1 ; p)n;r
= ;n
r n;r
;n;1
r p (1 ; p)
= r;;n
1
r
= nr
Then

E  X1 jY
= r] = 0  P(X1 = 0jY = r) + 1  P(X1 = 1jY = r)
= nr
1
E  X1 jY = Y (! )] = Y (! )
n
Therefore E  X jY ] = Y
1
1
n
Note a random variable - a function of Y .
36 CHAPTER 5. GENERATING FUNCTIONS

5.3 Properties of Conditional Expectation

Theorem 5.5.
E  E  X jY ]] = E  X ]
Proof.
X
E  E  X jY ]] = P(Y = y) E  X jY = y]
y2Ry
X X
= P(Y = y) P(X = xjY = y)
y x2Rx
XX
= xP(X = xjY = y)
y x
= E X]

Theorem 5.6. If X and Y are independent then

E  X jY ] = E  X ]
Proof. If X and Y are independent then for any y 2 Ry
X X
E  X jY = y] = xP(X = xjY = y) = xP(X = x) = E  X ]
x2Rx x

Example. Let X1
X2
: : : be i.i.d.r.v’s with p.g.f p(z ). Let N be a random variable

independent of X1
X2
: : : with p.g.f h(z ). What is the p.g.f of:
X1 + X2 +    + XN
E
 zX1+:::Xn = E  E  zX1 +:::Xn jN
X1  X1+:::Xn
= P(N = n) E z jN = n
n=0
X
1
= P(N = n) (p(z ))n
n=0
= h(p(z ))
Then for example
d h(p(z ))

E  X1 +
: : :
Xn ] = z=1
dz
= h (1)p (1) = E  N ] E  X1 ]
0 0

2
d 2 h(p(z )) and hence
Exercise Calculate dz
Var X1 +
: : :
Xn
In terms of Var N and Var X1
5.4. BRANCHING PROCESSES 37

5.4 Branching Processes

X0
X1 : : : sequence of random variables. Xn number of individuals in the nth gener-

ation of population. Assume.
1. X0 = 1
2. Each individual lives for unit time then on death produces k offspring, probabil-
P
ity fk . fk = 1
3. All offspring behave independently.
Xn+1 = Y1n + Y2n +    + Ynn
Where Yin are i.i.d.r.v’s. Yin number of offspring of individual i in generation n.
Assume
1. f0  0
2. f0 + f1
1
Let F(z) be the probability generating function ofY in .

F (z ) =
X
1  h i
fk z k = E z Xi = E z Yin
n=0
Let

Fn (z ) = E z Xn
Then F1 (z ) = F (z ) the probability generating function of the offspring distribution.
Theorem 5.7.
Fn+1 (z ) = Fn (F (z )) = F (F (: : : (F (z )) : : : ))
Fn (z ) is an n-fold iterative formula.
38 CHAPTER 5. GENERATING FUNCTIONS

Proof.

Fn+1 (z ) = E z Xn+1

= E E z Xn+1 jXn

X
1 
= P(Xn = k) E z Xn+1 jXn = k
n=0
X
1 h i
= k) E z Y1 +Y2 ++Yn
n n n
= P(Xn
n=0
X
1 h i h i
= k ) E z Y1 : : : E z Yn
n n
= P(Xn
n=0
X
1
= P(Xn = k) (F (z ))k
n=0
= Fn (F (z ))

Theorem 5.8. Mean and Variance of population size

X
1
If m = kfk
1
k=0
X
1
and  2 = (k ; m)2 fk
1
k=0
Mean and Variance of offspring distribution.
Then E  Xn ] = mn
(
2 mn 1 (mn ;1)

m 6= 1
;

Var Xn = m;1 (5.1)

n2
m=1
Proof. Prove by calculating F (z ), F (z ) Alternatively
0 00

E  Xn ] = E  E  Xn jXn;1 ]]
= E  mjXn;1 ]
= mE  Xn;1 ]
= mn by induction
 
E (Xn ; mXn;1 )2 = E E (Xn ; mXn;1 )2 jXn

= E Var (Xn jXn;1 )]

= E 2 Xn;1

= 2 mn;1
Thus
 X 2 ; 2mE  X X ] + m2E  X 2 2 = 2mn;1
E n n n;1 n;1
5.4. BRANCHING PROCESSES 39

Now calculate

E  Xn Xn;1 ] = E  E  Xn Xn;1 jXn;1 ]]

= E  Xn;1 E  Xn jXn;1 ]]
= E  Xn;1 mXn;1]

= mE Xn2;1

Then E Xn2 =  2 mn;1 + m2 E  Xn;1 ]
2

Var Xn = E Xn2 ; E  Xn ]2

= m2 E Xn2;1 + 2 mn;1 ; m2 E  Xn;1 ]2
= m2 Var Xn;1 + 2 mn;1
= m4 Var Xn;2 + 2 (mn;1 + mn )
= m2(n;1) Var X1 + 2 (mn;1 + mn +    + m2n;3 )
= 2 mn;1 (1 + m +    + mn )

To deal with extinction we need to be careful with limits as n ! 1. Let

An = Xn = 0
= Extinction occurs by generation n

1
and let A = An
1
= the event that extinction ever occurs
Can we calculate P(A) from P(An )?
More generally let An be an increasing sequence

A1  A2  : : :
and define


1
A = nlim
!1 An = An
1

Define Bn for n  1

B1 = A1
n;1 !c
Bn = An \ Ai
i=1
c
= An \ An;1
40 CHAPTER 5. GENERATING FUNCTIONS

Bn for n  1 are disjoint events and


1 
1
Ai = Bi
i=1 i=1
n n
Ai = Bi

1 !
i=1 
i=1
1 !

P Ai = P Bi
i=1 i=1
X
1
= P(Bi )
1
X
n
= nlim P(Bi )
!1 1
n
= nlim
!1 Bi
i=1
n
= nlim
!1 Ai
i=1
!1 P(An )
= nlim
Thus
 
P lim A = nlim
n!1 n !1 P(An )
Probability is a continuous set function. Thus

P(extinction ever occurs) = lim P(An )

n!1
!1 P(Xn = 0)
= nlim
= q
Say

Note P(Xn = 0), n = 1
2
3
: : : is an increasing sequence so limit exists. But

P(Xn = 0) = Fn (0) Fn is the p.g.f of Xn
So

q = nlim
!1 Fn (0)
Also
 
F (q) = F nlim!1 Fn (0)
!1 F (Fn (0))
= nlim Since F is continuous

!1 Fn+1 (0)
= nlim
Thus F (q ) = q
5.4. BRANCHING PROCESSES 41

“q” is called the Extinction Probability.

Alternative Derivation
X
q= P(X1 = k) P(extinctionjX1 = k)
k
X
= P(X1 = k) qk
= F (q)
Theorem 5.9. The probability of extinction, q , is the smallest positive root of the equa-
tion F (q ) = q . m is the mean of the offspring distribution.
If m
1 then q = 1
while if m  1thenq
1
Proof.
X
1
F (1) = 1 m= kfk = zlim
0

!1 F (z )
0

X
1
F (z ) =
00
j (j ; 1)z j;2 in 0
z
1 Since f0 + f1
1 Also F (0) = f0  0
j =z
Thus if m
1, there does not exists a q 2 (0
1) with F (q) = q. If m  1 then let

be the smallest positive root of F (z ) = z then
1. Further,

F (0)
F ( ) =
F (F (0))
F ( ) =
Fn (0)
8n  1
q = nlim
!1 Fn (0)
0
q= Since q is a root of F (z ) = z
42 CHAPTER 5. GENERATING FUNCTIONS

5.5 Random Walks

Let X1
X2
: : : be i.i.d.r.vs. Let

Sn = S0 + X1 + X2 +    + Xn Where, usually S0 =0

Then Sn (n = 0
1
2
: : : is a 1 dimensional Random Walk.

We shall assume

(
Xn = 1
with probability p
(5.2)
;1
with probability q

This is a simple random walk. If p = q = 12 then the random walk is called symmetric

Example (Gambler’s Ruin). You have an initial fortune of A and I have an initial
fortune of B . We toss coins repeatedly I win with probability p and you win with
probability q . What is the probability that I bankrupt you before you bankrupt me?
5.5. RANDOM WALKS 43

Set a = A + B and z = B Stop a random walk starting at z when it hits 0 or a.

Let pz be the probability that the random walk hits a before it hits 0, starting from
z . Let qz be the probability that the random walk hits 0 before it hits a, starting from
z . After the first step the gambler’s fortune is either z ; 1 or z + 1 with prob p and q
respectively. From the law of total probability.

pz = qpz;1 + ppz+1 0
z
a
Also p0 = 0 and pa = 1. Must solve pt2 ; t + q = 0.
p p
t = 1  21p; 4pq = 1  21p ; 2p = 1 or qp
General Solution for p 6= q is

z
pz = A + B pq A + B = 0A = 1 q a
1; p
and so
 z
1 ; pq
pz =  a
1 ; pq
If p = q , the general solution is A + Bz

pz = az
To calculate qz , observe that this is the same problem with p
q
z replaced by p
q
a ; z
respectively. Thus
 q a  q z
;
qz = p q a p if p 6= q
p ;1
44 CHAPTER 5. GENERATING FUNCTIONS

or

qz = a ;z z if p = q
Thus qz + pz = 1 and so on, as we expected, the game ends with probability one.
P(hits 0 before a) = qz
 q a q z
;( )
qz = p q a p if p 6= q
p ;1
Or =
a ; z if p = q
z
What happens as a ! 1?

1
P( paths hit 0 ever) = path hits 0 before it hits a
a=z+1
P(hits 0 ever) = lim P(hits 0 before a)
a!1
!1 qz
= alim

= pq pq
=1 p=q
Let G be the ultimate gain or loss.
(
G = a ; z
with probability pz
(5.3)
;z
with probability qz
(
E  G] =
apz ; z
if p 6= q
(5.4)
0
if p = q
Fair game remains fair if the coin is fair then then games based on it have expected
reward 0.
Duration of a Game Let Dz be the expected time until the random walk hits 0
or a, starting from z . Is Dz finite? Dz is bounded above by x the mean of geometric
random variables (number of window’s of size a before a window with all +1 0 s or
;10s). Hence Dz is finite. Consider the first step. Then
Dz = 1 + pDz+1 + qDz;1
E  duration] = E  E  duration j first step]]
= p (E  duration j first step up]) + q (E  duration j first step down])
= p(1 + Dz+1 ) + q(1 + Dz;1 )
Equation holds for 0
z
a with D0 = Da = 0. Let’s try for a particular solution
Dz = Cz
Cz = Cp (z + 1) + Cq (z ; 1) + 1
1
C = q ; p for p 6= q
5.5. RANDOM WALKS 45

Consider the homogeneous relation

pt2 ; t + q = 0 t1 = 1 t2 = qp
General Solution for p 6= q is

z
Dz = A + B pq + q =z p
Substitute z = 0
a to get A and B
 q z
1 ;
Dz = q ;z p ; q ;a p  pq a p 6= q
1; p
If p = q then a particular solution is ;z 2 . General solution

Dz ; z 2 + A + Bz
Substituting the boundary conditions given.,

Dz = z (a ; z ) p=q
Example. Initial Capital.

p q z a P(ruin) E  gain] E  duration]

0.5 0.5 90 100 0.1 0 900
0.45 0.55 9 10 0.21 -1.1 11
0.45 0.55 90 100 0.87 -77 766

Stop the random walk when it hits 0 or a.

We have absorption at 0 or a. Let

Uzn = P(r.w. hits 0 at time n—starts at z)

Uzn+1 = pUz+1n + qUz;1n 0
z
a n0
U0n = Uan = 0 n  0
Ua0 = 1Uz0 = 0 0
z
a
X
1
Let Uz = Uznsn :
n=0
Now multiply by sn+1 and add for n = 0
1
2 : : :

Uz (s) = psUz+1(s) + qsUz;1 (s)

Where U0 (s) = 1 and Ua (s) = 0

Look for a solution

Ux(s) = ((s))z (s)

46 CHAPTER 5. GENERATING FUNCTIONS

Must satisfy

(s) = ps (((s))2 + qs
Two Roots,
p
1 (s)
2 (s) = 1  21ps; 4pqs
2

Every Solution of the form

Uz (s) = A(s) (1 (s))z + B (s) (2 (s))z

Substitute U0 (s) = 1 and Ua (s) = 0.A(s) + B (s) = 1 and

A(s) (1 (s))a + B (s) (2 (s))a = 0

a ))z ; (1 (s))z (2 (s))a
Uz (s) = (1 (s)) ((2((ss)) a ; ( (s))a
1 2
q
But 1 (s)2 (s) = recall quadratic
p q
(1 (s))a;z ; (2 (s))a;z
Uz (s) = p ( (s))a ; ( (s))a
1 2

Same method give generating function for absorption probabilities at the other barrier.
Generating function for the duration of the game is the sum of these two generating
functions.
Chapter 6

Continuous Random Variables

In this chapter we drop the assumption that id finite or countable. Assume we are
given a probability p on some subset of .
For example, spin a pointer, and let ! 2 give the position at which it stops, with
= ! : 0
!
2. Let
P(! 2 0
]) =
 (0

2)
2
Definition 6.1. A continuous random variable X is a function X : ! R for which
Zb
P(a
X (! )
b) = f (x)dx
a
Where f (x) is a function satisfying
1. f (x)  0
R +1 f (x)dx = 1
2. ;1
The function f is called the Probability Density Function.
For example, if X (! ) = ! given position of the pointer then x is a continuous
(1
random variable with p.d.f

f (x) = 2

(0
x
2) (6.1)

0
otherwise
This is an example of a uniformly distributed random variable. On the interval 0
2 ]

47
48 CHAPTER 6. CONTINUOUS RANDOM VARIABLES

in this case. Intuition about probability density functions is based on the approximate
relation.
Z x+xx
P(X 2 x
x + xx]) = f (z )dz
x
Proofs however more often use the distribution function
F (x) = P(X
x)
F (x) is increasing in x.

If X is a continuous random variable then

Zx
F (x) = f (z )dz
;1
and so F is continuous and differentiable.
F (x) = f (x)
0

(At any point x where then fundamental theorem of calculus applies).

The distribution function is also defined for a discrete random variable,
X
F (x) = p!
!:X (!)
x
and so F is a step function.

In either case
P(a
X
b) = P(X
b) ; P(X
a) = F (b) ; F (a)
 49

Example. The exponential distribution. Let

( ;x
F (x) = 1 ; e
0
x
1 (6.2)
0
x
0
The corresponding pdf is

f (x) = e;x 0
x
1

this is known as the exponential distribution with parameter . If X has this distribu-
tion then
P(X
x + z )
P(X
x + z jX
z ) =
P(X
z )
;(x+z)
= e e;z
= e;x
= P(X
x)
This is known as the memoryless property of the exponential distribution.

Theorem 6.1. If X is a continuous random variable with pdf f (x) and h(x) is a con-
tinuous strictly increasing function with h;1 (x) differentiable then h(x) is a continu-
ous random variable with pdf
;
d h;1(x)
fh (x) = f h;1 (x) dx

Proof. The distribution function of h(X ) is

P(h(X )
x) = P
;X
h;1(x)
= F ;h;1(x)

Since h is strictly increasing and F is the distribution function of X Then.

d
dx P(h(X )
x)
is a continuous random variable with pdf as claimed f h . Note usually need to repeat
proof than remember the result.
50 CHAPTER 6. CONTINUOUS RANDOM VARIABLES

Example. Suppose X  U 0
1] that is it is uniformly distributed on 0
1] Consider

Y = ; log x
P(Y
y) = P(; log X
y)
;
= P X  e;Y


Z1
= 1dx
e; Y
= 1 ; e;Y
Thus Y is exponentially distributed.
More generally
Theorem 6.2. Let U  U 0
1]. For any continuous distribution function F, the ran-
dom variable X defined by X = F ;1 (u) has distribution function F .
Proof.

P(X
;

x) = P F ;1 (u)
x


= P(U
F (x))
= F (x)  U 0
1]

Remark

1. a bit more messy for discrete random variables

P(X = Xi ) = pi i = 0
1
: : :
Let

X
j ;1 X
j
X = xj if pi
U
pi U  U 0
1]
i=0 i=0
2. useful for simulations

6.1 Jointly Distributed Random Variables

For two random variables X and Y the joint distribution function is

F (x
y) = P(X
x
Y
y) F : R2 ! 0
1]

Let

FX (x) = P(Xz
x)
= P(X
x
Y
1)
= F (x
1)
!1 F (x
y)
= ylim
6.1. JOINTLY DISTRIBUTED RANDOM VARIABLES 51

This is called the marginal distribution of X. Similarly

FY (x) = F (1
y)
X1
X2
: : :
Xn are jointly distributed continuous random variables if for a set c 2 R b
ZZ Z
P((X1
X2
: : :
Xn ) 2 c) =  f (x1
: : :
xn )dx1 : : : dxn
(x1 :::xn )2c

For some function f called the joint probability density function satisfying the obvious
conditions.
1.

f (x1
: : :
xn )dx1  0

ZZ Z
2.

 f (x1
: : :
xn )dx1 : : : dxn = 1

Rn
Example. (n = 2)
F (x
y) = P(X
x
Y
y)
Zx Zy
= f (u
v)dudv
;1 ;1
and so f (x
y ) =
@ 2 F (x
y)
@x@y
Theorem 6.3. provided defined at (x
y ). If X and y are jointly continuous random
variables then they are individually continuous.
Proof. Since X and Y are jointly continuous random variables

P(X 2 A) = P(X 2 A
Y 2 (;1
+1))

Z Z1
= f (x
y)dxdy
A ;1
= fA fX (x)dx
Z1
where fX (x) = f (x
y)dy
;1
is the pdf of X .
Jointly continuous random variables X and Y are Independent if

f (x
y) = fX (x)fY (y)

Then P(X 2 A
Y 2 B ) = P(X 2 A) P(Y 2 B )

Similarly jointly continuous random variables X 1
: : :
Xn are independent if

Y
n
f (x1
: : :
xn ) = fXi (xi )
i=1
52 CHAPTER 6. CONTINUOUS RANDOM VARIABLES

Where fXi (xi ) are the pdf’s of the individual random variables.

Example. Two points X and Y are tossed at random and independently onto a line
segment of length L. What is the probability that:

jX ; Y j
l?

Suppose that “at random” means uniformly so that

f (x
y) = L12 x
y 2 0
L]2

6.1. JOINTLY DISTRIBUTED RANDOM VARIABLES 53

Desired probability

ZZ
= f (x
y)dxdy
A
= areaL2of A
L2 ; 2 12 (L ; l)2
= L2
= 2Ll ; l
2

L2

Example (Buffon’s Needle Problem). A needle of length l is tossed at random onto a

floor marked with parallel lines a distance L apart l
L. What is the probability that
the needle intersects one of the parallel lines.

Let  2 0
2 ] be the angle between the needle and the parallel lines and let x be
the distance from the bottom of the needle to the line closest to it. It is reasonable to
suppose that X is distributed Uniformly.

X  U 0
L]   U 0
)

and X and  are independent. Thus

f (x
) = l1 0
x
L and 0



54 CHAPTER 6. CONTINUOUS RANDOM VARIABLES

The needle intersects the line if and only if X
sin  The event A
ZZ
= f (x
)dxd
Z
Asin 
=l L d
0
2l
= L
Definition 6.2. The expectation or mean of a continuous random variable X is
Z1
E X] = xf (x)dx
;1
R 1 xf (x)dx and R 0 xf (x)dx are infinite
provided not both of ;1 ;1
Example (Normal Distribution). Let

f (x) = p 1 e
(x;
)2
22 ;1
x
1
;

2
This is non-negative for it to be a pdf we also need to check that
Z1
f (x)dx = 1
;1
Make the substitution z = x;
. Then
Z1
I = p1
(x;
)2
e 22 dx
;

2Z ;1
Z 1 z2
= p1 e 2 dz ;

Z 1 Z 1  2  ;1
Thus I =
2 1 e 2
;x2
dx e 2
;y2
dy
2 ;1 ;1
1 Z1Z1 (y2 +x2 )
= 2 e dxdy
;
2
;1 ;1
Z 2
Z 1
= 21
2
re ;
2 drd
Z 2

0 0

= d = 1
0

Therefore I = 1. A random variable with the pdf f(x) given above has a Normal
distribution with parameters and  2 we write this as

X  N 
2 ]
The Expectation is
1 Z 1 (x
)2
E X] = p xe 22 dx
; ;

2 Z;1 Z1
1 1 (x
)2 1 (x;
)2
=p (x ; )e 2 dx + p e 22 dx:
; ; ;
2 
2 ;1 2 ;1
6.1. JOINTLY DISTRIBUTED RANDOM VARIABLES 55

The first term is convergent and equals zero by symmetry, so that

E X] = 0 +
=
Theorem 6.4. If X is a continuous random variable then,
Z1 Z1
E X] = P(X  x) dx ; P(X
;x) dx
0 0
Proof.
Z1 Z 1 Z 1 
P(X  x) dx = f (y)dy dx
0
Z01 Z 1x
= I y  x]f (y)dydx
Z 0 1 Z0 y
= dxf (y)dy
Z0 1 0

= yf (y)dy
Z1 Z0
0

Similarly P(X
;x) dx = yf (y)dy

0 ;1
result follows.
Note This holds for discrete random variables and is useful as a general way of
finding the expectation whether the random variable is discrete or continuous.
If X takes values in the set 0
1
: : :
] Theorem states
X
1
E X] = P(X  n)
n=0
and a direct proof follows

X
1 X
1 X
1
P(X  n) = I m  n]P(X = m)
n=0 n=0 m=0 !
X
1 X
1
= I m  n] P(X = m)
m=0 n=0
X
1
= mP(X = m)
m=0
Theorem 6.5. Let X be a continuous random variable with pdf f (x) and let h(x) be
a continuous real-valued function. Then provided

Z1
jh(x)j f (x)dx
1
;1 Z1
E  h(x)] = h(x)f (x)dx
;1
56 CHAPTER 6. CONTINUOUS RANDOM VARIABLES

Proof.
Z1
P(h(X )  y ) dy
0
Z 1 "Z #
= f (x)dx dy
x:h(x) 0
Z 1Z
0

= I h(x)  y]f (x)dxdy

x:h(x) 0
Z
0
"Z h(x) 0 #
= dy f (x)dx
x:h(x)
Z 0

= h(x)f (x)dy
x:h(x) 0
Z1 Z
Similarly P(h(X )
;y ) = ; h(x)f (x)dy
0 x:h(x)
0

So the result follows from the last theorem.

Definition 6.3. The variance of a continuous random variable X is


Var X = E (X ; E  X ])2

P R
Note The properties of expectation and variance are the same for discrete and contin-
uous random variables just replace with in the proofs.

Example.

Var X = E X 2 ; E  X ]2
Z1 Z 1
2
= x f (x)dx ;
2
xf (x)dx
;1 ;1

Example. Suppose X  N 
 2 ] Let z = X

; then
X ;

P(Z
z) = P
z

= P(X
+ z )
Z +
z 1 (x
)2
= p e 22 dx
; ;

x ;
Z z 1 2u2
;1
Let u=  = p e 2 du
;

;1 2
= (z ) The distribution function of a N (0
1) random variable
Z  N (0
1)
6.2. TRANSFORMATION OF RANDOM VARIABLES 57

What is the variance of Z ?


Var X = E Z 2 ; E  Z ]2 Last term is zero
1 Z 1 z2
=p z 2e 2 dz ;

2 ;1
 1 z 2 1 Z 1 z 2
= ; p ze 2 + e 2 dz
; ;

2 ;1 ;1
=0+1=1
Var X = 1

Variance of X ?

X = + z
Thus E  X ] = we know that already
Var X = 2 Var Z
Var X = 2
X  (
2 )

6.2 Transformation of Random Variables

Suppose X1
X2
: : :
Xn have joint pdf f (x1
: : :
xn ) let

Y1 = r1 (X1
X2
: : :
Xn )
Y2 = r2 (X1
X2
: : :
Xn )
..
.
Yn = rn (X1
X2
: : :
Xn )

Let R 2 Rn be such that

P((X1
X2
: : :
Xn ) 2 R) = 1

Let S be the image of R under the above transformation suppose the transformation
from R to S is 1-1 (bijective).
58 CHAPTER 6. CONTINUOUS RANDOM VARIABLES

Then 9 inverse functions

x1 = s1 (y1
y2
: : :
yn )
x2 = s2 (y1
y2
: : :
yn ) : : :
xn = sn (y1
y2
: : :
yn )
@si exists and is continuous at every point (y1
y2
: : :
yn ) in S
Assume that @yj

 @s1 : : : @s1 
 @y1 @yn 
J =  ... . . . .. 
@sn 
(6.3)
 @s@yn1 : : :
.
@yn

If A  R

Z Z
P((X1
: : :
Xn ) 2 A) 1] =  f (x1
: : :
xn )dx1 : : : dxn
Z A Z
=  f (s1
: : :
sn ) jJ j dy1 : : : dyn
B

Where B is the image of A

= P((Y1
: : :
Yn ) 2 B ) 2]

Since transformation is 1-1 then [1],[2] are the same

Thus the density for Y1
: : :
Yn is

g((y1
y2
: : :
yn ) = f (s1 (y1
y2
: : :
yn )
: : :
sn (y1
y2
: : :
yn)) jJ j

y1
y2
: : :
yn 2 S
= 0 otherwise.

Example (density of products and quotients). Suppose that (X
Y ) has density

(
f (x
y) = 4xy
for 0
x
1
0
y
1
(6.4)
0
Otherwise.

Let U = XY and V = XY
6.2. TRANSFORMATION OF RANDOM VARIABLES 59

p
r
X = UV Y = VU
p
rv
x = uv y= u
r
@x = 1 v
r
@x = 1 u
@u 2 u @v 2 v
@y = ;1 v 12 @y = p1 :
@u 2 u 32 @v 2 uv
Therefore jJ j = 21u and so

g(u
v) = 21u (4xy)

r
= 21u  4 uv uv
p

= 2 uv if (u
v ) 2 D
= 0 Otherwise:
Note U and V are NOT independent

g(u
v) = 2 uv I (u
v) 2 D]
not product of the two identities.
When the transformations are linear things are simpler still. Let A be the n  n
invertible matrix.

0Y 1 0 X 1
B@ ...1 C
A @.A
= A B .. C :
1

Yn Xn

jJ j = det A;1 = det A;1

Then the pdf of (Y1
: : :
Yn ) is

g(y1
: : :
n ) = det1 A f (A;1 g)

Example. Suppose X1
X2 have the pdf f (x1
x2 ). Calculate the pdf of X1 + X2 .
Let Y = X1 + X2 and Z = X2 . Then X1 = Y ; Z and X2 = Z .

A;1 = 10 ;11 (6.5)

det A;1 = 1 det1 A

60 CHAPTER 6. CONTINUOUS RANDOM VARIABLES

Then

g(y
z ) = f (x1
x2 ) = f (y ; z
z )
joint distributions of Y and X .
Marginal density of Y is
Z1
g(y) = f (y ; z
z )dz ;1
y
1
Z;1
1
or g (y ) = f (z
y ; z )dz By change of variable
;1
If X1 and X2 are independent, with pgf’s f1 and f2 then

f (x1
x2 ) = f (x1 )f (x2 )

Z1
and then g (y ) = f (y ; z )f (z )dz
;1

- the convolution of f1 and f2

For the pdf f(x) x

^ is a mode if f (^x)  f (x)8x
x^ is a median if
Z x^ Z1
f (x)dx ; f (x)dx = 12
;1 x^
For a discrete random variable, x^ is a median if

P(X
x^) 
1 or P(X  x^)  1
2 2
If X1
: : :
Xn is a sample from the distribution then recall that the sample mean is

1X
n
X
n i
1

Let Y1
: : :
Yn (the statistics) be the values of X1
: : :
Xn arranged in increasing

order. Then the sample median is Y n+1 if n is odd or any value in
2
h i
Y n2
Y n+1
2
if n is even
6.2. TRANSFORMATION OF RANDOM VARIABLES 61

If Yn = maxX1
: : :
Xn and X1
: : :
Xn are iidrv’s with distribution F and den-

sity f then,
P(Yn
y) = P(X1
y
: : :
Xn
y)
= (F (y))n
Thus the density of Yn is
d (F (y))n
g(y) = dy
= n (F (y))n;1 f (y)
Similarly Y1 = minX1
: : :
Xn and is
P(Y1
y ) = 1 ; P(X1  y
: : :
Xn  y )
= 1 ; (1 ; F (y))n
Then the density of Y1 is

= n (1 ; F (y))n;1 f (y)
What about the joint density of Y1
Yn ?
G(y
yn ) = P(Y1
y1
Yn
yn )
= P(Yn
yn ) ; P(Yn
yn
Y1 1 )
= P(Yn
yn ) ; P(y1
X1
yn
y1
X2
yn
: : :
y1
Xn
yn)
= (F (yn ))n ; (F (yn ) ; F (y1 ))n
Thus the pdf of Y1
Yn is

g(y1
yn ) = @y@@y G(y1
yn )

2

1 n
= n(n ; 1) (F (yn ) ; F (y1 ))n;2 f (y1 )f (yn ) ; 1
y1
yn
1
=0 otherwise
What happens if the mapping is not 1-1? X = f (x) and jX j = g(x)?
Zb
P(jX j 2 (a
b)) = (f (x) + f (;x)) dx g(x) = f (x) + f (;x)
a
Suppose X1
: : :
Xn are iidrv’s. What is the pdf of Y1
: : :
Yn the order statistics?
(
g(y1
: : :
yn ) = n!f (y1 ) : : : f (yn)
y1
y2
  
yn (6.6)
0
Otherwise
Example. Suppose X1
: : :
Xn are iidrv’s exponentially distributed with parameter
. Let
z1 = Y1
z2 = Y2 ; Y1
..
.
zn = Yn ; Yn;1
62 CHAPTER 6. CONTINUOUS RANDOM VARIABLES

Where Y1
: : :
Yn are the order statistics of X1
: : :
Xn . What is the distribution of

the z 0 s?
Z = AY
01 1
Where
0 0 :::0 0
BB;1 1 0 : : : 0 0C
C
A=BBB 0. ;1 1 : : : 0 0C
C CA (6.7)
@ .. ..
.
..
.
..
.
..
.
0 0 ::: ;1 1
det(A) = 1
h(z1
: : :
zn) = g(y1
: : :
yn )
= n!f (y1 ) : : : f (yn )
= n!n e;y1 : : : e;yn
= n!n e;(y1 ++yn)
= n!n e;(z1 2z2 ++nzn )
Y
n
= ie;izn+1 ; i

i=1
Thus h(z1
: : :
zn ) is expressed as the product of n density functions and
Zn+1;i  exp(i)
exponentially distributed with parameter i, with z1
: : :
zn independent.
Example. Let X and Y be independent N (0:1) random variables. Let
D = R 2 = X 2 + Y2

Y then
then tan  = X
 
d = x2 + y2 and  = arctan xy
 
 2x
jJ j =  x2y 1 =2

2y
2
;

 1+( y )2 1+( y ) 
x (6.8)
x x
6.2. TRANSFORMATION OF RANDOM VARIABLES 63

f (x
y) = p1 e 2x p1 e
2 y2
2
; ;

2 2
1
= 2 e 2
(x2 +y2 )
;

Thus

g(d
) = 41 e 2d 0
d
1 0

2
;

But this is just the product of the densities

gD (d) = 21 e 2d;
0
d
1
g () = 21 0

2
Then D and  are independent. d exponentially mean 2.   U 0
2 ].
Note this is useful for the simulations of the normal random variable.
We know we can simulate N 0
1] random variable by X = f 0 (U ) when U 
U 0
1] but this is difficult for N 0
1] random variable since
Z +x
p1 e
z2
F (x) = (x) = ;
2
;1 2
is difficult.
Let U1 and U2 be independent  U 0
1]. Let R 2 = ;2 log U , so that R2 is
exponential with mean 2.  = 2U2 . Then   U 0
2 ]. Now let
p
X = R cos  = ;2 log U1 cos(2U2 )
p
Y = R sin  = ;2 log U2 sin(2U1 )
Then X and Y are independent N 0
1] random variables.
Example (Bertrand’s Paradox). Calculatepthe probability that a “random chord” of
a circle of radius 1 has length greater that 3. The length of the side of an inscribed
equilateral triangle.
There are at least 3 interpretations of a random chord.
(1) The ends are independently and uniformly distributed over the circumference.

answer = 1
3
64 CHAPTER 6. CONTINUOUS RANDOM VARIABLES

(2)The chord is perpendicular to a given diameter and the point of intersection is

uniformly distributed over the diameter.

 p !2 p 2
a2 + 3 = 3
2
answer = 12
(3) The foot of the perpendicular to the chord from the centre of the circle is uni-
formly distributed over the diameter of the interior circle.

interior circle has radius 12 .


;1

answer = 1222 = 14

6.3 Moment Generating Functions

If X is a continuous random variable then the analogue of the pgf is the moment gen-
erating function defined by

m() = E e x
for those  such that m() is finite
Z1
m() = e xf (x)dx
;1
where f (x) is the pdf of X .
6.3. MOMENT GENERATING FUNCTIONS 65

Theorem 6.6. The moment generating function determines the distribution of X , pro-
vided m() is finite for some interval containing the origin.
Proof. Not proved.
Theorem 6.7. If X and Y are independent random variables with moment generating
function mx () and my () then X + Y has the moment generating function

mx+y () = mx ()  my ()

Proof.
h i 
E e (x+y) = E e xe y
 
= E e x E e y
= mx()my ()

Theorem 6.8. The rth moment of X ie the expected value of X r , E  X r ], is the coeffi-
cient of r! of the series expansion of n().
r

Proof. Sketch of....

e X = 1 + X + 2! X 2 + : : :
2

 2 
E e X = 1 + E  X ] + E X 2 + : : :
2!

Example. Recall X has an exponential distribution, parameter  if it has a density

ex for 0
x
1.
E
 e X = Z 1 e xexdx
Z1
0

= e;(; )xdx
0
 = m() for 

= ;    
E  X ] = m (0) =
0
= 1
( ; ) =0 
2

E
  2 
X 2 = ( ; )2 = 2 2
=0 
Thus

Var X = E X 2 ; E  X ]2
= 2 ; 1 2 2
66 CHAPTER 6. CONTINUOUS RANDOM VARIABLES

Example. Suppose X1
: : :
Xn are iidrvs each exponentially distributed with param-

eter .
Claim : X1
: : :
Xn has a gamma distribution, ;(n
) with parameters n
. With
density

n e;x xn;1 0
x
1

(n ; 1)!
we can check that this is a density by integrating it by parts and show that it equals 1.

h i  
e (X1 ++Xn ) = E e X1 : : : E e Xn
E
 
= E e X1 n

n
= ; 
Suppose that Y  ;(n
).
 Z 1 ne;xxn;1
E e Y = e x (n ; 1)! dx

n Z 1 ( ; )n e;(; )xxn;1
0

= ;  0 (n ; 1)! dx
Hence claim, since moment generating function characterizes distribution.
Example (Normal Distribution). X  N 0
1]
 Z1
e x p 1 e;( 22 ) dx
X x
2
E e =
;

Z;1 2  
1 1 ;1
= p exp (x 2
; 2x + ; 2 x) dx
2 2

Z;1 2  22 
1 1 ;1 ;

= exp 22 (x ; ;  ) ; 2   ;   dx
p 2 2 2 2 4
;1 2
Z  ;1 
2 2 1 1
= e + 2 p exp 22 (x ; ; 2 )2 dx
;1 2
The integral equals 1 are it is the density of N  +  2
 2 ]

2 2
= e + 2

Which is the moment generating function of N 
 2 ] random variable.

Theorem 6.9. Suppose X , Y are independent X  N  1
12 ] and Y  N  2
22 ]
then
1.

X + Y  N  1 + 2
12 + 22 ]
6.4. CENTRAL LIMIT THEOREM 67

2.

aX  N a 1 + a2 2 ]
Proof. 1.
h i  
E e (X +Y ) = E e X E e Y
2 2 2 2
= e( 1 + 21
1 ) e( 2 + 12
2 )
2 2 2
= e( 1 + 2 ) + 21 (
1 +
2 )
which is the moment generating function for

N  1 + 2
12 + 22 ]
2.
h i h i
E e (aX ) = E e( a)X
2 2
= e 1 ( a)+ 12
1 ( a)
2 2 2
= e(a 1 ) + 12 a
1
which is the moment generating function of

N a 1
a2 12 ]

6.4 Central Limit Theorem

X1
: : :
Xn iidrv’s, mean 0 and variance  2 . Xi has density

Var Xi = 2
X1 +    + Xn has Variance
Var X1 +    + Xn = n2
68 CHAPTER 6. CONTINUOUS RANDOM VARIABLES

X1 ++Xn has Variance

n

Var X1 +  n  + Xn = n
2

X1 +p+Xn has Variance

n

Var X1 + p
   + Xn
n = 2

Theorem 6.10. Let X1
: : :
Xn be iidrv’s with E  Xi ] = and Var Xi = 2
1.

X
n
Sn = Xi
1

Then 8(a
b) such that ;1
a
b
1

Zb
lim P a
Sn ;
p n
b = p1 e
z2
 n
;
2 dz
n!1 a 2
Which is the pdf of a N 0
1] random variable.
6.4. CENTRAL LIMIT THEOREM 69

Proof. Sketch of proof.....

WLOG take = 0 and  2 = 1. we can replace Xi by Xi
; . mgf of Xi is

mXi () = E e Xi
 
= 1 + E  Xi ] + 2 E Xi2 + 3! E Xi3 + : : :
2 3

3 
= 1 + 2 + 3! E Xi3 + : : :
2

Sn
The mgf of p n
h i h  (X1 ++Xn)i
e
Sn
E
p
n =E e np

h  i h  i
= E e n X1 : : : E e n Xn
p p

h  in
= E e n X1
p

n
= mX1 p
n
  !n
3 E X 3
= 1 + 2n +
2 2
32 =! e 2 as n ! 1
3!n
Which is the mgf of N 0
1] random variable.
Note if Sn  Binn
p] Xi = 1 with probability p and = 0 with probability (1 ; p).
Then
Sn ; np ' N 0
1]
pnpq
This is called the normal approximation the the binomial distribution. Applies as n !
1 with p constant. Earlier we discussed the Poisson approximation to the binomial.
which applies when n ! 1 and np is constant.
Example. There are two competing airlines. n passengers each select 1 of the 2 plans
at random. Number of passengers in plane one

S  Binn
12 ]
Suppose each plane has s seats and let
f (s) = P(S
s)
S ; np ' n0
1]
pnpq
S ; 1n s ; 1n

f (s) = P 1 p2
1 p2
2 n
2s ; n
2 n
=1; p
n
therefore if n = 1000 and s = 537 then f (s) = 0:01. Planes hold 1074 seats only 74
in excess.
70 CHAPTER 6. CONTINUOUS RANDOM VARIABLES

Example. An unknown fraction of the electorate, p, vote labour. It is desired to find p

within an error no exceeding 0.005. How large should the sample be.
Let the fraction of labour votes in the sample be p . We can never be certain (with-
 
0

 
out complete enumeration), that p ; p
0:005. Instead choose n so that the event
 
0

p ; p 
0:005 have probability  0:95.

0

  
P p ; p 
0:005 = P(jSn ; npj
0:005n)
0

jSn ; npj 0:005pn

=P pnpq
p
n

Choose n such that the probability is  0:95.

Z 1:96
p1 e
x2
;
2 dx = 2(1:96) ; 1
;1:96 2

We must choose n so that

p
0:005
p n  1:96
n

But we don’t know p. But pq
1

4 with the worst case p = q = 12

n  01::005
962 1 ' 40
000
24

If we replace 0.005 by 0.01 the n  10
000 will be sufficient. And is we replace 0.005
by 0.045 then n  475 will suffice.
Note Answer does not depend upon the total population.
6.5. MULTIVARIATE NORMAL DISTRIBUTION 71

6.5 Multivariate normal distribution

Let x1
: : :
Xn be iid N 0
1] random variables with joint density g (x1
: : : xn )
Y
n x2
g(x1
: : : xn ) = p1 e 2 i
;

i=1 2
= 1 n2 e 2 i=1 xi
1 Pn 2
;

(2)
= 1 n2 e 2 ~x^~x
1 ;

(2)
0X 1
Write

BBX12 CC
X~ = B@ ... CA
Xn
and let ~z
;

= ~ + AX~ where A is an invertible matrix ~x = A;1 (~x ; ~ ) . Density of ~z
f (z1
: : :
zn ) = (21) n2 det1 A e 2 (A (~z;~ )) (A (~z;~ ))
; ;1 1 ; T 1

1 1 (~z;~ )T  1 (~z;~ )
= 12 e 2
; ;

(2) 2 jj
n

where AAT . This is the multivariate normal density

~z  MV N ~
]
Cov(zi
zj ) = E (zi ; i )(zj ; j )]
But this is the (i
j ) entry of

E
 (~z ; ~ )(~z ; ~ )T = E h (AX~ )(AX~ )T i

= AE XX T AT
= AIAT = AAT =  Covariance matrix
If the covariance matrix of the MVN distribution is diagonal, then the components of
the random vector ~z are independent since
n Y 1 e ;21 zi;i
i
2
f (z1
: : :
zn) = 1
i=1 (2 ) 2 i

02 0 : : : 1
Where
0
BB 01 22 : : : 0CC
 = B .. .. . . CA
@. . . ..
.
0 0 : : : n2
Not necessarily true if the distribution is no MVN recall sheet 2 question 9.
72 CHAPTER 6. CONTINUOUS RANDOM VARIABLES

Example (bivariate normal).

f (x1
x2 ) = 1 
2(1 ; p2 ) 12 1 2
" "

exp ; 2(1 ;1 p2 ) x1 ; 1 2 ;
1
x1 ; 1
x ;
x ;
2##
2p 1
2
2
2
+ 1 1
1

1
2
0 and ;1
p
+1. Joint distribution of a bivariate normal random

variable.
Example. An example with
;1 ;1

;1 = 1 ;1 p2 p;11;1 p122
2

1 2 2

 = p1 p122
2

1 2 2

E  Xi ] = i and Var Xi = i2 . Cov(X1
X2 ) = 1 2 p.

Correlation(X1
X2 ) = Cov(X1
X2 ) = p
1 2