Markov Chain Estimation

Estimation for transition probability

For purpose of inference a Markov chain can be observed in two ways:

One observation of a chain of great length or

Observations on a large number of realizations of same chain.

Observe a finite Markov chain with

transitions from

states

i to j i, j 1, 2,..., m

n
j 1

ij

1, 2,..., m

until

transitions take place. Let

nij

be the number of

. Let

ni

These numbers can be represented as:

1
2
3
M
m

1
n11
n21
n31
M
nm1

3
n13

2
n12
n22
n32
M
nm 2

n23
n33
M
nm 3

Total
Let the stationary transition matrix of the Markov chain is

L
L
L
L
L

m
n1m
n2m
n3m
M
nmm

Total

n1
n2
n3
M
nm
n

P , given by

p11 p12 . . . p1m

p p ... p
2m
21 22

P . .
.

. .
.
pm1 pm2 . . . pmm

We are interested in the estimation of the element

(2)

Pij .

Markov Chain Estimation ~ 1 of 9

For a given initial state

and a number of trials

considered as a sample of size

p
j 1

ij

the sample of transition counts

from a multinomial distribution with probabilities

ni1 , ni 2 ,..., nim

can be

pi1 , pi 2 ,..., pim , such that

1 . The probability of the outcome can be given as:

ni1 !
ni1 !ni 2 !...nim !

Pi1ni1 Pi 2ni 2 ...Pimnim

where

ni

ni ,

nij ni

and

j 1

j 1

ij

..... (3)

1, 2,..., m , when the breakdown of the total number of trials n into

Extending this argument for the m initial states

n1 , n2 ,..., nm
m

is given, the probability of the realization of transition is given by,

n !

ni1 !ni 2i1!...nim !

Pi1ni1 Pi n2i 2 ...Pimnim

i 1

..... (4)

It can be shown that this distribution is independent of the probability elements

its explicit form, we shall denote it by
m

f ( p ) A
i 1

ni1 !
ni1 !ni 2 !...nim !

Pij

and therefore, without going into

A . The likelihood function and natural logarithm L p

Pi1ni1 Pi n2i 2 ...Pimnim

m m

L( p) ln B nij ln Pij

.....

can be expressed as:

..... (5)

(6)

i 1 j 1

where ln B contains all terms independent of

m

under the condition

p
j 1

ij

pij s

To derive maximum likelihood elements, we maximize

1, (i 1,2,.., m) . Incorporating this condition in 6

m m

i 1 j 1

i 1

L( p) ln B nij ln Pij nim ln(1 Pi1 Pi 2 ... Pim 1 )

we can write:

...... (7)

For a specific value I, we have:

m

Li ( p ) ln B nij ln Pij nij ln(1 Pi1 Pi 2 ... Pim 1 )

j 1

Differentiating with respect to pij and setting it equal to zero, we get,

ni1
nim

0
Pi1 1 Pi1 Pi 2 ... Pim 1
ni 2
nim

0
Pi 2 1 Pi1 Pi 2 ... Pim 1
M
nim 1
nim

0
Pim 1 1 Pi1 Pi 2 ... Pim 1
Combining these equations we may write:
Markov Chain Estimation ~ 2 of 9

ni1 ni 2
n
nim

... im 1
Pi1 Pi 2
Pim 1 1 Pi1 Pi 2 ... Pim 1
This leads us to write

ni1
Pi1 Pi1
ni1

ni 2
Pi1 Pi 2
ni1

.....

(8)

nim

Pi1 1 Pi1 Pi 2 ... Pim 1

ni1

Adding both sides we get,

ni1 ni 2 ... nim

Pi1 1
ni1
Which yields the estimate:

n
Pi1 i1
ni
In a similar manner, we can derive estimates of other elements. Thus we get:

Pij

nij
nj

i, j 1, 2,3,..., m

Hypothesis Testing
There are several hypothesis testing problems related to Markov chains. The major ones one of the following:
i.

Tests for a given transition probability matrix.

ii.

Tests for the stationary of the transition probability matrix.

iii.

Tests for the order of the Markov chain.

iv.

Tests for the first order Markov dependence.

Testing Transition Probability Matrix

We consider the null hypothesis

H 0 : P P0
For large

n , it can be shown that nij

the normal distribution with mean

m

ni ( Pij Pij 0 ) 2

j 1

Pij 0

~ 2m 1

are asymptotically normally distributed and that the static

has

pij 1 pij . Thus we can write the test static as:

and variance

ni 2 p ij pij

i 1, 2,..., m ....(1)

2
distributed as with m 1 degrees of freedom.

In obtaining

1 ,

we have assumed that all

pij0

the non zero elements should be considered in

are non-zero. If there are some zero elements in the

i th

row, only

and the degrees of freedom should be decreased by the number

of zeros in it. Thus we can write:

Markov Chain Estimation ~ 3 of 9

0
ni ( Pij Pij ) 2

Pij

i 1 j 1

and the summation in

is

) is the maximized value of the likelihood function. From statistical theory, it is known that when
where f ( p

H0

is

has a

2 with m 1

taken only over

i, j

degrees of freedom, where

for which

is the number of zeros in

p0

pij0 0 .

Likelihood Ratio Test

The likelihood ratio criteria for

f ( P0 )
f ( P )

2 ln ~ 2

true.

H 0 : P P 0 can be obtained as

with

m m 1

degrees of freedom. In this case we have

m m

nij

i 1 j 1

ni Pij0

2 ln 2 L( P ) L( P 0 ) 2 nij ln

Testing the Stationary of Transition Probability

Let

pijt

be the one step transition probability of a time dependent process

Pijt P X (t 1) j | X (t ) i
Let

nijt

be the number of transition

transition counts

nijt ; t 1, 2,..., T

(1)

during the

t th

transition of a process. For a given initial state

2
ni12

3
ni13

L
L

m
1
nim

ni21

ni22

ni23

2
nim

3
M
T

ni31
M
niT1

ni32
M
niT2

ni33
M
niT3

3
nim
M
T
nim

nijt
ni

t 1

where ni

t 1

i , the

can be represented as

1
ni11

The maximum likelihood estimates of

t
ij

i j

.....

X t , such that,

pijt

can be obtained:
m

nijt
j 1

Now we can consider the null hypothesis,

H 0 : pijt pij t 1, 2,..., T

t ) and the likelihood ration criterion
Then the maximum likelihood function is given by f ( p

is given by

f ( Pt )

f ( P)
Here,

2 ln 2 L( P ) L( P)
t

2 n ln
t 1 i 1 j 1

t
ij

nijt
nijt 1 Pij
Markov Chain Estimation ~ 4 of 9

Under

H 0 , 2 ln ~ 2

with

T 1 m m 1

degrees of freedom.

Testing the Order of Markov Chain

Let

pijk P X n k | X n 1 j , X n 2 i
And

nijk

be the corresponding transition count. Also let

pijk

nijt nijk
k

. The estimates of

pijk

is obtained as

nijk
nijt

Now consider the null hypothesis that the Markov chain is a first order Markov chain against the alternative that it is
of order to can be given as

H 0 : pijk p jk i, j , k 1, 2,..., m . To test the Hypothesis we consider the following

test static

ijk P
jk )
nijt ( P

jk
P
i , j,k
which has a 2 distribution with

m m 1

degrees of freedom. The likelihood ratio test static can be written as,

2 ln 2 nijk (ln Pijk ln Pjk ) 2 nijk ln

i , j ,k

i , j ,k

which has a 2 distribution with

Markov chain is of order

r 1

2 ln 2 nijk ln
i , j ,k

with

m r 1 m 1

m m 1

nijk ni
nijt nij

degrees of freedom. Now if we test the null hypothesis that the

against the alternative hypothesis that it is of order

r , the test static is

nijk ni
~ 2
t
nij nij

degrees of freedom.

Testing the First Order Markov Chain Dependence

We consider the null hypothesis that the observations are independent against the alternative that the process is a
first order Markov chain. The test statistics are obtained by using an appropriate

P 0 . The appropriate P 0

should

have identical rows under the hypothesis of independence.

Let

P0

consist of m identical rows

( 1 , 2 ,..., m ) .

When these probabilities are not known, their

maximum likelihood estimates can be determined as follows:

m

Let

nij nij

The log likelihood function can be written as

i 1

L( ) ln B nij ln j
j 1

Hence the log likelihood function leads to the maximum likelihood estimate

n. j
n
Markov Chain Estimation ~ 5 of 9

2
The test statistic can be written as

nn
nij i . j
m m
n

ni n. j
i 1 j 1
n

The likelihood ration statistic can be written as:

nij

2 ln 2 nij ln
ni n. j
i 1 j 1

2
has a distribution with

m 1

degrees of freedom.

Problem
A telephone receptionist at a department store can process only one call at a time. The two states the receptionist
can be in are idle and busy. It is conjectured that a two state Markov chain model would be reasonable for this
individuals walk habits with observations being taken once every 30 seconds. The following data were obtained by
observing the receptionists state of work for a period of 25 minutes (0 = idle; 1= busy)
110111010111001101011110111110100111011010101111010
a)

Derive a estimate for the elements of the transition probability matrix.

b)

Is it fair to assume that the observed process is a realization of a Markov chain with the transition probability
matrix

0.2 0.8
0.5 0.5

Solution
a)

From the given information, we have the following transition probability matrix

0
0 3
1 15

1
14 17
18
33

So the transition probability matrix is

0
0
1
b)

17

15

33

0 0.18 0.82

18
33

1 0.45 0.55

14

17

In this case we consider the null hypothesis

Markov Chain Estimation ~ 6 of 9

 0.2 0.8
H0 : P P
0.5 0.5
0

To test the hypothesis we consider the test as

m

ni Pij Pij 0
Pij

j 1

So that 2
Here

has

0.18 0.82
Here, Pij

0.45 0.55

17(0.18 0.2) 2 17(0.82 0.8) 2 33(0.45) 2 33(0.55 0.5)2

0.3725
0.2
0.8
0.5
0.5

m 1 2 1 1 degrees of freedom. For 0.05 we get

2
tab
3.841
2
2
Since cal
0.3725 tab
3.841, we accept the null hypothesis

So, we may conclude that the observed process is a realization of a Markov chain with the transition probability
matrix

0.2 0.8
0.5 0.5

Estimating Parameters of an M|M|1 Queue
Let us consider an M|M|1 queue with an arrival rate

, we

observe that the arrival process is Poisson. Let

maximum likelihood estimator of

Also since
for

2 S n

is

and a service rate . To estimate the average arrival rate

Sn

be the time required to observe

arrivals, then the

n
.
Sn

has a chi-square distribution with

2n

degrees of freedom, a

100 1 %

confidence interval

is given by

22n ,1 22n ,
2
2

,
2Sn
2 Sn

Again, in order to estimate the average service rate

we assume that the service times are independent

exponentially distributed random variables.

Thus if service times

X 1 , X 2 ,..., X m

have been observed and if we get

Markov Chain Estimation ~ 7 of 9

Ym i
i 1

then the maximum likelihood estimate of

is given by,

m
M
Ym
Since

2 Ym

interval for

has a chi-square distribution with

2m

degrees of freedom, we get a

100 1 %

confidence

is

22n ,1 22n ,
2
2

,
2Ym
2Ym

Now the server utilization

is given by

n
Ym

S
n
R
m
Sn
M m
Ym
n

, we use the ratio

To obtain confidence intervals for

2 Ym
R Ym m / S n n
2m

2 S n

2n
Now, since

Ym

and

Sn

are independent and

R
~ F2 m ,2 n

To obtain a 100 1 %

confidence interval for

and d so that

P( F c) 2

are chi-square distributed it follows that

we write for some constant

and

R
d)

1 P (c F d ) P( c
Select

2 Ym , 2 S n

and

P( F d ) 1 2

Then we get,

c F2 m , 2 n ;1 and
2

d F2 m , 2 n ;

so the required confidence interval

F2 m , 2 n ;

and

is given by ( L,

From the confidence interval of

v), where

F2 m , 2 n ;1

, we can obtain a confidence interval for any monotonically increasing function

H of . Thus H ( L ), H ( u )

is the

100 1 % confidence interval for H .

Problem

Markov Chain Estimation ~ 8 of 9

Assume that a communication channel can be modeled as an M|M|1 queue. Suppose we observe the time until 30
message arrivals to be 59.46 minutes and these 30 messages keep the channel busy for a total of 29 minutes.
Find the estimates of average arrival rate, average service rate and the average channel utilization.

Solution
Here,

m n 30 , S n 59.46

minutes and

ym 29

minutes. So, the average arrival rate is

n
30

0.505
Sn 59.46
message per minute, the average service rate is

m
30

1.03
Ym 29

message per minute, and average channel utilization is

0.488

To obtain 95% confidence intervals for

have
2
60,0.025
83.3

and

we use a 2 with

2m, 2n 60

degrees of freedom. We

2
60,0.975
40.48

and

So the required confidence interval for

is

40.48
83.3

(0.34,0.7)
2 59.46 2 59.46

and for

is

40.48 83.3
,

(0.698,1.436)
2 29 2 29
To obtain a confidence interval for

, we use F

distribution with

60, 60

degrees of freedom. We have,

F60,60, 0.025 1.67

F60,60, 0.975 0.5988
L

F60, 60,0.025

Thus the 95% confidence interval for

is

E(N )

and u

0.292

F60, 60,0.975

0.815

is (0.292, 0.815). The average number of messages queued or in service

0.488
0.953
1 0.488

Thus a 95% confidence interval for

E N

is given by

0.292
0.815
,
(0.412, 0.405)
1 0.292 1 0.815

Markov Chain Estimation ~ 9 of 9