Good Review Materials

http://www.imageprocessingbook.com/DIP2E/dip2e_downloads/review_material_downloads.htm

Random Variables
and
Random Vectors

(Gonzales & Woods review materials)

Chapt. 1: Linear Algebra Review
Chapt. 2: Probability, Random Variables,
Random Vectors

Tim Marks, Cognitive Science Department

Tim Marks, Cognitive Science Department

Random variables

Discrete random variables

Samples from a random variable are real numbers

Discrete random variables have

a pmf (probability mass function), P
P(X = a) = P(a)
Example: Coin flip
1

A random variable is associated with a probability distribution

over these real values
Two types of random variables
Discrete
Only finitely many possible values for the random variable:
X {a1, a2, , an}
(Could also have a countable infinity of possible values)
e.g., the random variable could take any positive integer value
Each possible value has a finite probability of occurring.

X = 0 if heads
X = 1 if tails
What is the pmf of this
random variable?

Continuous
Infinitely many possible values for the random variable
E.g., X {Real numbers}

P (a )

0.5

0
0

Tim Marks, Cognitive Science Department

Continuous random variables

Discrete random variables have

a pmf (probability mass function), P
1
P(X = a) = P(a)
P (a )
Example: Die roll

Continuous random variables have a

pdf (probability density function), p
Example: Uniform distribution
p(x)

p(1.3) = ?

0
0
1

p(2.4) = ?

What is the probability

that X = 1.3 exactly:
P(X = 1.3) = ?

0.5

0.5

Tim Marks, Cognitive Science Department

Tim Marks, Cognitive Science Department

Discrete random variables

X {1, 2, 3, 4, 5, 6}
What is the
pmf of this
random variable?

Probability corresponds to area

under the pdf.
1.5
P(1 < X < 1.5) = ? p(x) dx = 0.25

Tim Marks, Cognitive Science Department

Continuous random variables

Random variables

What is the total area under any pdf ?

How much change do you have on you?

Asking a person (chosen at random) that question can

be thought of as sampling from a random variable.

p( x)dx = 1?

Example continuous random variable:

Human heights
p(h)

p(h)

Is the random variable

Amount of change people carry
discrete or continuous?

p(h)

Tim Marks, Cognitive Science Department

Tim Marks, Cognitive Science Department

Random variables: Mean & Variance

An important type of random variable

These formulas can be used to find the mean and

variance of a random variable when its true
probability distribution is known.
Definition
Mean

= E(X)

Variance

E ((X ) 2 )

Var(X)

Discrete r.v.

Continuous r.v.

= ai P (ai )

x p(x) dx

(a )

P(ai )

(x )

p(x) dx

Tim Marks, Cognitive Science Department

Tim Marks, Cognitive Science Department

The Gaussian distribution

p(x) =

1
2

Estimating the Mean & Variance

( x )2
2

X ~ N(, 2)

Tim Marks, Cognitive Science Department

After sampling from a random variable n times,

these formulas can be used to estimate the
mean and variance of the random variable.
Samples x1, x2, x3, , xn
Estimated mean:

m=

1 n
xi
n i=1

Estimated variance:

2 =

1 n
(x i m)2
n i=1

2 =

1
(x i m)2 unbiased estimate
n 1 i=1

maximum
likelihood estimate

Tim Marks, Cognitive Science Department

Finding mean, variance in Matlab

Example continuous random variable

Samples x = [x1 x 2 x 3 L x n ]
Mean
>> m = (1/n)*sum(x)

Peoples heights (made up)

Variance
1
= [x1 m
n
2

Method 1:
Method 2:

x1 m

x m
x 2 m L x n m] 2
M

x n m

>> v = (1/n)*(x-m)*(x-m)
>> z = x-m
>> v = (1/n)*z*z

Gaussian
= 67 , 2 = 20
What if you went
to a planet where
heights Gaussian
= 75 , 2 = 5
How would they be
different from us?

Tim Marks, Cognitive Science Department

Tim Marks, Cognitive Science Department

Example continuous random variable

Time people woke up this morning
Gaussian
= 9 , 2 = 1

Random vectors
An n-dimensional random vector consists of n
random variables that are all associated with the same
events.
Example 2-D random vector:
X
V =
Y

where X is random variable of human heights

Y is random variable of wake-up times

Sample n times from V.

v1 v 2 L v n
x1 x 2 L x n

y1 y 2 L y n
Tim Marks, Cognitive Science Department

Random Vectors
What will the graph of V look like?

67
m =
10

Tim Marks, Cognitive Science Department

x (heights)

Mean of a random vector

Estimating the mean
of a random vector

Mean
Mean of X is 67
Mean of Y is 10

y
(wake-up
times)

Tim Marks, Cognitive Science Department

n samples from V

What is mean of V ?

Lets collect some

samples and graph them:

m=

v1 v 2 L v n
x1 x 2 L x n

y1 y 2 L y n

n
n
1
1 x m x
v i = i =

n i=1
n i=1 y i m y

To estimate mean of V in Matlab

>> (1/n)*sum(v,2)
Tim Marks, Cognitive Science Department

Random vector
Example 2-D random vector:
X
V =
Y

where X is random variable of human heights

Y is random variable of human weights
v1 v 2 L v n
x1 x 2 L x n

y1 y 2 L y n

Sample n times from V

What will graph

look like?

Covariance of two random variables

Height and wake-up time are uncorrelated, but
height and weight are correlated.
Covariance
Cov(X, Y) = 0 for X = height, Y = wake-up times
Cov(X, Y) > 0 for X = height, Y = weight
Definition:
Cov(X,Y ) = E ((X x )(Y y ))
If Cov(X, Y) < 0 for two random variables X, Y , what would a
scatterplot of samples from X, Y look like?

Tim Marks, Cognitive Science Department

Tim Marks, Cognitive Science Department

Estimating covariance from samples

Sample n times:
Cov(X,Y ) =
Cov(X,Y ) =

x1

y1

Estimating covariance in Matlab

Samples

x2 L xn

y2 L yn

Means
mx m_x
my m_y

x = [x1 x 2 x 3 L x n ]
y = [y1 y 2 y 3 L y n ]

1 n
(x i mx )(y i my )
n i=1

maximum
likelihood estimate

1 n
(x i mx )(y i my )
n 1 i=1

unbiased estimate

Covariance
1
Cov(X,Y ) = [x1 m x
n

x 2 mx

y1 m y

y 2 my

L x n mx ]
M

y n my

Cov(X, X) = ?Var(X)

Method 1: >> v = (1/n)*(x-m_x)*(y-m_y)

How are Cov(X, Y) and Cov(Y, X) related?

Method 2: >> w = x-m_x

>> z = y-m_y
>> v = (1/n)*w*z

Cov(X, Y) = Cov(Y, X)
Tim Marks, Cognitive Science Department

Covariance matrix of a
D-dimensional random vector

X X
= E
[X X
Y Y

Example covariance matrix

Peoples heights
(made up)

In 2 dimensions
X
V =
Y
T
Cov(V) = E (V )(V )

Tim Marks, Cognitive Science Department

X ~ N(67, 20)

)
Var(X ) Cov(X,Y )
Y Y ] =

Var(Y)
Cov(X,Y )

In D dimensions
T
Cov(V) = E ((V )(V ) )
When is a covariance matrix symmetric?
A. always, B. sometimes, or C. never
Tim Marks, Cognitive Science Department

Time people woke up

this morning
Y ~ N(9, 1)

What is the covariance

matrix of

X
V =
Y

20 0

0 1

Tim Marks, Cognitive Science Department

Estimating the covariance matrix from

samples (including Matlab code)
Sample n times and find mean of samples
v1 v 2 L v n

x
V = 1
y1

x2 L xn

y2 L yn

1
Cov(V ) =
n

x1 m x

y1 m y

1-dimensional Gaussian is completely determined

by its mean, , and variance, 2 :

mx
m=
my

Find the covariance matrix

x1 mx

x 2 mx L x n mx x 2 mx

y 2 my L y n my M

x n mx

>> m = (1/n)*sum(v,2)
>> z = v - repmat(m,1,n)
>> v = (1/n)*z*z
Tim Marks, Cognitive Science Department

Gaussian distribution in D dimensions

X ~ N(, 2)
y1 m y

y 2 my
M

y n my

p(x) =

1
2

(x )2
2

D-dimensional Gaussian is completely determined

by its mean, , and covariance matrix, :
X ~ N(, )

p(x) =

1
(2 )

D/2

1/ 2

12 (x )T 1 (x )

What happens when D = 1 in the Gaussian formula?

Tim Marks, Cognitive Science Department