Lecture 9

Continuous-time Markov
chains, part II
9.1 Kolmogorov equations
Please read Ross [2010], Section 6.4.
9.1.1 Instantaneous transition rates
Kolmogorov equations: instantaneous rates
The rate of transition from state i into state j is given by
q
ij
= v
i
P
ij
.
The q
ij
values are called the instantaneous transition rates.
Note that q
ii
= 0 as a consequence of the fact that P
ii
= 0.
Observe that
v
i
= v
i
_

j
P
ij
_
=

j
v
i
P
ij
=

j
q
ij
.
Moreover,
P
ij
=
q
ij
v
i
=
q
ij

j
q
ij
.
13
9.1. Kolmogorov equations 14
Kolmogorov equations: instantaneous rates
Therefore, a continuous-time Markov chain can be dened completely
by the instantaneous transition rates q
ij
.
If i = j, then
lim
h0
P
ij
(h)
h
= q
ij
That is, the instantaneous transition rate q
ij
may be viewed as the
derivative P

ij
(0) of the transition probability P
ij
(t) with respect
to t, evaluated in t = 0.
Kolmogorov equations: instantaneous rates
As

j
P
ij
(h) = 1 P
ii
(h),
it follows that
lim
h0
1 P
ii
(h)
h
=

j
lim
h0
P
ij
(h)
h
=

j
q
ij
= v
i
.
The interchange between the limit and the innite sum is not
always allowed, but can be justied here.
9.1.2 Chapman-Kolmogorov
Kolmogorov equations: Chapman-Kolmogorov
The next result is our point of departure in deriving the Kolmogorov
backward and forward equations.
Chapman-Kolmogorov equations for continuous-time Markov
chains: for all s 0, t 0,
P
ij
(t + s) =

k=0
P
ik
(t)P
kj
(s).
Proof:
In order for a continuous-time Markov chain X to go from state i
to state j in time t + s, it must be at some state k at time t.
9.1. Kolmogorov equations 15
Kolmogorov equations: Chapman-Kolmogorov
Proof (continued):
Thus, by the law of total probability,
P
ij
(t + s) = P (X(t + s) = j | X(0) = i)
=

k=0
P (X(t + s) = j, X(t) = k | X(0) = i)
=

k=0
P (X(t + s) = j | X(t) = k, X(0) = i)
P (X(t) = k | X(0) = i)
=

k=0
P (X(t + s) = j | X(t) = k)
P (X(t) = k | X(0) = i)
=

k=0
P
ik
(t)P
kj
(s).
9.1.3 Backward equations
Kolmogorov equations: backward equations
Kolmogorovs backward equations for continuous-time Markov
chains: for all states i, j and times t 0,
P

ij
(t) =

k=i
q
ik
P
kj
(t) v
i
P
ij
(t).
Proof:
It follows by the Chapman-Kolmogorov equations that
P
ij
(h + t) P
ij
(t) =

k=0
P
ik
(h)P
kj
(t) P
ij
(t)
=

k=i
P
ik
(h)P
kj
(t) + P
ii
(h)P
ij
(t) P
ij
(t)
=

k=i
P
ik
(h)P
kj
(t) {1 P
ii
(h)} P
ij
(t).
9.1. Kolmogorov equations 16
Kolmogorov equations: backward equations
Proof (continued):
Thus,
lim
h0
P
ij
(h + t) P
ij
(t)
h
= lim
h0
_

k=i
P
ik
(h)P
kj
(t) {1 P
ii
(h)} P
ij
(t)
h
_
=

k=i
_
lim
h0
P
ik
(h)
h
_
P
kj
(t)

_
lim
h0
1 P
ii
(h)
h
_
P
ij
(t)
=

k=i
q
ik
P
kj
(t) v
i
P
ij
(t).
The interchange between the limit and the innite sum is not
always allowed, but can be justied here.
Kolmogorov equations: backward equations
For the pure birth process the backward equations are
P

ij
(t) =
i
P
i+1,j
(t)
i
P
ij
(t).
The backward equations for the birth and death process are
P

ij
(t) =
i
P
i+1,j
(t) (
i
+
i
) P
ij
(t).
In particular,
P

0j
(t) =
0
(P
1j
(t) P
0j
(t)) .
9.1. Kolmogorov equations 17
9.1.4 Two states example
Kolmogorov equations: two states example
Consider the machine of Example 6.11 in Ross [2010], p. 386.
The machine works for an exponential time with rate .
Repair takes an exponential time with rate .
Suppose the machine is in working condition at time t = 0. What is
the probability that it is still working at time t = 10?
Kolmogorov equations: two states example
There are two states:
State Description
0 machine is working
1 machine is in repair
The Kolmogorov backward equations are:
P

00
(t) = {P
10
(t) P
00
(t)}
P

10
(t) = {P
00
(t) P
10
(t)}
One may show, see Ross [2010], p. 387, that the Kolmogorov backward
equations together imply
P
10
(t) = (1 P
00
(t))
9.1. Kolmogorov equations 18
Kolmogorov equations: two states example
It follows from the rst Kolmogorov backward equation that
P

00
(t) = (1 P
00
(t)) P
00
(t)
= ( + ) P
00
(t)
= ( + ) h(t)
with
h(t) = P
00
(t)

+
Observe that
h

(t)
h(t)
=
P

00
(t)
h(t)
= ( + ) ,
Integrating both sides yields
ln h(t) = ( + ) t + c,
Kolmogorov equations: two states example
Thus,
P
00
(t) = h(t) +

+
= K exp {( + ) t} +

+
.
Using the fact that P
00
(0) = 1, we obtain
P
00
(t) =

+
exp {( + ) t} +

+
.
Using the fact that P
10
(t) equals (1 P
00
(t)), we obtain
P
10
(t) =

+
(1 exp {( + ) t})
The desired probability is P
00
(10).
9.2. Limiting probabilities 19
9.1.5 Forward equations
Kolmogorov equations: forward equations
Kolmogorovs forward equations for continuous-time Markov
chains: under suitable regularity conditions,
P

ij
(t) =

k=j
q
kj
P
ik
(t) v
j
P
ij
(t).
The regularity conditions do not hold in all models. However these
conditions hold for all birth and death processes and nite state
models.
The forward equations for the birth and death process are
P

ij
(t) =
j1
P
i,j1
(t) +
j+1
P
i,j+1
(t) (
j
+
j
) P
ij
(t).
In particular,
P

i0
(t) =
1
P
i1
(t)
0
P
i0
(t).
9.2 Limiting probabilities
Please read Ross [2010], Section 6.5.
9.2. Limiting probabilities 20
9.2.1 Balance equations
Limiting probabilities: balance equations
Assume that the limiting probability of being in state j
lim
t
P
ij
(t)
exists and is independent of the initial state i.
Denote this limiting probability by P
j
.
Consider now the forward equations, and assume that the regularity
conditions holds.
Then,
lim
t
P

ij
(t) = lim
t
_

k=j
q
kj
P
ik
(t) v
j
P
ij
(t)
_
=

k=j
q
kj
_
lim
t
P
ik
(t)
_
v
j
_
lim
t
P
ij
(t)
_
=

k=j
q
kj
P
k
v
j
P
j
.
Limiting probabilities: balance equations
As P
ij
(t) is a probability, and hence is bounded between 0 and 1, it
follows that if P

ij
(t) converges, then it must converge to 0.
Suppose that P

ij
(t) converges to a non-zero value.
Then P
ij
(t) will exceed one of the boundaries 0 and 1, eventually.
This contradicts the fact that P
ij
(t) is a probability.
9.2. Limiting probabilities 21
Limiting probabilities: balance equations
Thus, to nd the limiting probabilities P
j
, we may solve the balance
equations
v
j
P
j
=

k=j
q
kj
P
k
,
under the condition

j
P
j
= 1.
We may interpret v
j
P
j
as the limiting rate at which the process
leaves state j.
We may interpret

k=j
q
kj
P
k
as the limiting rate at which the
process enters state j.
Thus the balancing equations simply state that both limiting rates
should be equal.
Limiting probabilities: balance equations
The limiting probabilities exist if
all states of the Markov chain communicate, and
the Markov chain is positive recurrent.
Under these two conditions, the limiting probability P
j
can be inter-
preted as the long-run proportion of time that the process is in state
j.
When P
j
s exist, the chain becomes ergodic.
Like in the discrete-time case, P
j
s are also called the stationary prob-
abilities.
9.2.2 Birth and death
Limiting probabilities: birth and death
For a birth and death process, the balance equations are
For i = 0,

0
P
0
=
1
P
1
.
For every i > 0,
(
i
+
i
) P
i
=
i+1
P
i+1
+
i1
P
i1
.
9.2. Limiting probabilities 22
Limiting probabilities: birth and death
It follows by solving the balance equations that the limiting probabili-
ties for a birth and death process should satisfy
P
i
=

0

1

i1

2

i
P
0
for i 1.
By using the fact that

i=1
P
i
= 1, it follows that
P
0
=
1
1 +

i=1

i1

i
.
for i 1.
Limiting probabilities: birth and death
The last equation gives us a necessary and sucient condition for ex-
istence of the limiting probabilities:

n=1

1

n1

2

n
< .
9.2. Limiting probabilities 23
9.2.3 M/M/1 queue
Limiting probabilities: M/M/1 queue
Recall that the M/M/1 queue is a birth and death process with common
arrival rate and common service rate .
As
i
= for every i 0 and
i
= for every i 1, the limiting
probability P
0
further simplies to
P
0
=
1
1 +

n=1
(/)
n
= 1 /
provided that / < 1
Similarly, the limiting probability P
i
simplies to
P
i
=
(/)
i
1 +

n=1
(/)
n
= (/)
i
(1 /)
for i 1, again provided that / < 1.
Limiting probabilities: M/M/1 queue
If / > 1, then the number of customers in the system tends to innity
as time grows.
That is, the system has insucient capacity.
The boundary case / = 1 is very similar to random walk model.
9.2. Limiting probabilities 24
9.2.4 Shoe shine example
Limiting probabilities: shoe shine example
Consider the shoe shine shop of Example 6.15 in Ross [2010], p. 395.
There are two chairs: chair 1 and chair 2.
In chair 1, shoes are cleaned and polished. Service times are in-
dependent exponential random variables with common rate
1
.
In chair 2, the polish is bued. Service times are independent
exponential random variables with common rate
2
.
Potential customers arrive at the shop according to a Poisson pro-
cess with rate . A potential customer will enter the shop only if
both chairs are free.
a What are the states?
b What are the balance equations?
c Determine the proportion of time the process spends in each state
(in the long run).
Limiting probabilities: shoe shine example
a There can be at most one customer in the shop, so the possible states
are
State Description
0 no customers
1 one customer in chair 1
2 one customer in chair 2
b The balance equations are:
Limiting rate at which the process
State leaves state enters state
0 P
0
=
2
P
2
1
1
P
1
= P
0
2
2
P
2
=
1
P
1
9.2. Limiting probabilities 25
Limiting probabilities: shoe shine example
c Recall that the limiting probability P
j
can be interpreted as the long-
run proportion of time that the process is in state j. Solving the balance
equations gives P
1
= (/
1
)P
0
and P
2
= (/
2
)P
0
. The condition
P
0
+ P
1
+ P
2
= 1 now yields
P
0
=
1
1 +

1
+

2
.
It follows that
P
1
=

1
1 +

1
+

2
, P
2
=

2
1 +

1
+

2
.
9.2.5 Gas station example
Limiting probabilities: gas station example
Consider the gas station of Exercise 6.14 in Ross [2010], p. 414.
Potential customers arrive at a one-pump gas station at a Poisson
rate of = 20 cars per hour.
Potential customers will only enter the station if there are no more
than two cars at the pump.
Service times are independent exponential random variables with
mean ve minutes.
That is, with rate = 12 cars per hour.
a. What fraction of the attendants time will be spent on servicing cares?
b. What fraction of potential customers are lost?
9.2. Limiting probabilities 26
Limiting probabilities: gas station example
There can be at most two cars at the gas station, so the possible states
are
State Description
0 no cars at the station
1 one car at the station
2 two cars at the station
The balance equations are:
Limiting rate at which the process
State leaves state enters state
0 P
0
= P
1
1 ( + )P
1
= P
0
+ P
2
2 P
2
= P
1
Limiting probabilities: gas station example
Solving the balance equations gives P
1
= (/)P
0
and P
2
= (/)P
1
=
(/)
2
P
0
.
The condition P
0
+ P
1
+ P
2
= 1 now yields
P
0
=
1
1 +

+
_

_
2
= 0.1837.
It follows that
P
1
=

1 +

+
_

_
2
= 0.3061,
P
2
=
_

_
2
1 +

+
_

_
2
. = 0.5102.
Limiting probabilities: gas station example
a The fraction of time the attendant is busy is P
1
+ P
2
= 0.8163.
b The fraction of time the system is full is P
2
= 0.5102. All customers
arriving during that time are lost. Since customers arrive randomly
over time, also a percentage P
2
is lost.
9.2. Limiting probabilities 27
Limiting probabilities: gas station example
Next consider the addition of a second attendant, while customers still
leave if two cars are being served.
What percentage of customers is lost in that case?
The new balance equations are:
Limiting rate at which the process
State leaves state enters state
0 P
0
= P
1
1 ( + )P
1
= P
0
+ 2P
2
2 2P
2
= P
1
Now proceed as before.
Bibliography
Sheldon M. Ross. Introduction to Probability Models, 10th Edition. Academic
Press, Inc., Orlando, FL, USA, 2010. ISBN 978-0123756862.
28