Expectation and Moments

Expectation and Moments
Farzin Zareian
University of California-Irvine
October 20, 2014
Farzin Zareian (UCIrvine)
October 20, 2014
1 / 20
Partial Descriptors of Random Variables
A random variable is completely defined by its PMF (in the case of

discrete random variables) or its PDF (for continuous random variables).
It is often useful, however, to characterize a random variable by a set of
measures that describe the overall features of its distribution. Such
measures might include its central location, breadth and skewness as well
as other quantities defined by the shape of the distribution function.
October 20, 2014
2 / 20
Partial Descriptors of Random Variables, Mean
Perhaps the most common measure used to describe the central location
of a random variable, X , is its mean, which is usually denoted X or E [X ].
The mean of a random variable is defined as the first moment of its PMF
or PDF, i.e.,
X
For discrete random variables: X =
xpX (x)
allx
For continuous random variables: X =
xfX (x)
October 20, 2014
3 / 20
Partial Descriptors of Random Variables, Median
Another commonly used measure of the central location of a random

variable X is its median, which is denoted x0.5 and x. The median is
defined such that there is a 50% chance that a realization of the random
variable lies below (or above) it. Mathematically:
FX (x0.5 ) = 0.5
October 20, 2014
4 / 20
Partial Descriptors of Random Variables, Mode

The central location of a random variable X can also be described by its
mode, which is denoted x. The mode is defined as that realization of a
random variable that has the highest probability (for discrete random
variables) or probability density (for continuous random variables). The
mode is found by maximizing the PMF or PDF. Note that, unlike the
mean and median, a probability distribution can have more than one mode.
October 20, 2014
5 / 20
Expectation
Let g (X ) denote a function of a random variable X . The expectation of

g (X ) is defined as:
X
For discrete random variables: E [g (X )] =
g (x)pX (x)
allx
For continuous random variables: E [g (X )] =

g (x)fX (x)
Z Z
For vector of random variables: E [g (X)] =
...
g (x)fX (x)dx
|
{z
}
n-fold
October 20, 2014
6 / 20
Expectation
E [g1 (X) + g2 (X)] = E [g1 (X)] + E [g2 (X)]

E [ag1 (X)] = aE [g1 (X)]
E [g1 (X)g2 (X)] 6= E [g1 (X)] + E [g2 (X)]
"
#
m
m
X
X
E a0 +
ai gi (X) = a0 +
E [gi (X)]
i=1
If X = [X1 , X2 ]
i=1
T
Z
then: E [g (X1 )] =
...
| {z
k-fold
g (x1 )fX1 (x1 )dx1
October 20, 2014
7 / 20
Moments of a Random Variable

Consider a function g (X ) = X m of a random variable X , where m is a
constant. The following expectation is called the mth moment of X .
Z
m
E [X ] =
x m fX (x)dx where m = 1, 2, ...
We have already considered the first moment, E [X ] = X , which is known

as the mean of X . As discussed earlier, the mean is one measure
commonly used to quantify the central value of a distribution (i.e., where
the bulk of the probability content is located). The second moment,
E [X 2 ], is called the mean-square of X and it measures the dispersion or
spread of the probability content. The third and fourth moments of a
random variable are related to the skewness and flatness of the probability
distribution respectively.
October 20, 2014
8 / 20
Variance
Now consider the function g (X ) = (X X )m , where x is the mean of

the distribution described above. The mth central moment of X is defined
as:
Z
m
E [(X X ) ] =
(x X )m fX (x)dx where m = 1, 2, ...
Consequently

First moment, m = 1, E (X X )1 = 0

Second moment, m = 2, E (X X )2 = Var(X ) = X2
October 20, 2014
9 / 20
Variance
The variance of a random variable is a measure of the dispersion of it

probability distribution about its mean. The square root of the variance,
x , is known as the standard deviation and the ratio of standard deviation
to mean is called the coeficient of variation (c.o.v.):
X
X =
|X |
Using the linear property of the expectation operator:
X2 = E [X 2 ] E 2 [X ]
October 20, 2014
10 / 20
Coefficient of skewness

Because the third central moment, E (X X )3 , retains the sign of the
deviation from the mean, it provides useful information about the
non-symmetry of a probability distribution about its mean. Coefficient of
skewness, is used to quantify this property. For symmetric distributions,
X = 0. When X > 0, the probability distribution has a longer tail to the
right and is described as right-skewed. Conversely, X < 0 implies a longer
left tail, i.e., left-skewed.

E (X X )3
X =
|X3 |
October 20, 2014
11 / 20
Kurtosis coefficient
The fourth central moment can be used to define the Kurtosis coefficient
which quantifies the flatness of the probability distribution.

E (X X )4
X =
|X4 |
It should be noted that the above moments only provide a partial
description of a probability distribution. It can be shown that all moments
must be known in order to have complete knowledge about a random
variable.
October 20, 2014
12 / 20
Joint Moments of Random Variables
The above definitions can be extended to the case of multiple random

variables. For an n-vector of random variables, X, the joint moment of
order m1 , m2 , ..., mn is defined as:
Z Z
m1 m2
mn
E [X1 X2 ...Xn ] =
...
x1m1 x2m2 ...xnmn fX (x)dx
|
{z
}
n-fold
The joint central moment of the same order is:

mn
m1
m2
E [(X1
Z
Z 1 ) (X2 2 ) ...(Xn n ) ] =
...
(x1 1 )m1 (x2 2 )m2 ...(xn n )mn fX (x)dx
| {z }
n-fold
October 20, 2014
13 / 20

The most useful of these moments are the lowest order joint moments for
two random variables:
Z
Z
E [Xi Xj ] =
xi xj fXi Xj (xi , xj )dxi dxj

Z Z
E [(Xi i )(Xj j )] =
(xi i )(xj j )fXi Xj (xi , xj )dxi dxj
The first-order joint central moment of two random variables is

known as the covariance: ij = Cov (Xi , Xj ) = E [(Xi i )(Xj j )]
The covariance is a measure of linear dependence between two
random variables.
Cov (Xi , Xi ) = Var (Xi ) = i2
Cov (Xi , Xj ) = E [Xi Xj] E [Xi ]E [Xj ]
October 20, 2014
14 / 20
|Cov (Xi , Xj )|
p
Var (Xi )Var (Xj )
The correlation coefficient between random variables Xi and Xj is

Cov (Xi ,Xj )
defined as: ij =
= i ij j
i j
1 ij 1
Independent random variables: E [Xi Xj ] = E [Xi ]E [Xj ] and ij = 0.
Independent random variables are uncorrelated. But the reverse is not
necessarily true.
October 20, 2014
15 / 20
Example 18
Random variable X has a uniform probability distribution fX (x) =

where a x b. Let Y = X 2 and compute XY .
1
ba
October 20, 2014
16 / 20
When dealing with more than two random variables, it is useful, if not
necessary, to introduce a matrix notation for the first and second
moments. For an n-vector of random variables, X = [X1 , X2 , ..., Xn ]T , we
define the n 1 mean vector:
MX = [1 , 2 , ..., n ]T
and the n n covariance matrix,

Var (X1 )
Cov (X1 , X2 ) . . . Cov (X1 , Xn )

..
Cov (X1 , X2 )

Var (X2 )
.

XX =

..
.
.
.
.

.
.
.

Cov (Xn , X1 )
...
...
Var (Xn )
October 20, 2014
17 / 20
XX

Var (X1 )
Cov (X1 , X2 ) . . . Cov (X1 , Xn )

..
Cov (X1 , X2 )
Var (X2 )
.
=
..
..
.
..

.
.

Cov (Xn , X1 )
...
...
Var (Xn )
XX

12
12 1 2 . . . 1n 1 n

..
12 1 2
22
.

=
..
.
.
..
..

.

n1 n 1
...
...
n2
October 20, 2014
18 / 20

XX = E [(X MX )(X MX )T ] = E [XXT ] Mx MT
x
XX =DX RXX DX where
1
DX =
.
.
.
n
1 12 . . . 1n
..
21 1
.
RXX = .
.. = Correlation Coefficient Matrix
..
..
.
.
n1 . . . . . . 1
It can be shown that both of these matrices are, in general, positive
definite. The only exception to this rule occurs when a linear
dependence exists between any subset of random variables, in which
case the covariance matrix and the correlation coefficient matrix are
singular.
October 20, 2014
19 / 20
X1 |X2 = E [X1 |X2 = x2 ]

Var (X1 |X2 = x2 ) = E (X1 X1 |X2 )2 |X2 = x2 ]
MX1 |X2 =x2 = E [X1 |X2 = x2 ]
X1 X1 |X2 =x2 = E [(X1 MX1 |X2 =x2 )(X1 MX1 |X2 =x2 )T |X2 = x2 ]
October 20, 2014
20 / 20

Expectation and Moments

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Expectation and Moments

Încărcat de

Drepturi de autor:

Formate disponibile

Expectation and Moments

October 20, 2014

Farzin Zareian (UCIrvine)