Chapter 6

Chapter 6:
Continuous-Time
Markov Chains

STAT306, Term I, 2011/12 1

 Chapter 6

Chapter Contents
Introduction
Continuous-Time Markov Chains
Birth and Death Processes
 Pure birth process
 Pure death process
The Transition Probability Function
 Chapman-Kolmogorov Equations
 Instantaneous transition rates
 Kolmogorov’s Backward Equation
 Kolmogorov’s Forward Equation
Limiting Behavior of Birth and Death Processes

STAT306, Term I, 2011/12 2

 Chapter 6
Introduction
• Chapter 4 introduced the Markov chain, which is a Markov
process with discrete-time and discrete-state.
• Chapter 5 introduced the Poisson process, nonhomogeneous
Poisson process, and compound Poisson process.
• In this chapter, we present several important examples of
continuous-time, discrete-state, Markov processes such as
 Pure Birth Process
 Pure Death Process
 Birth and Death Process
which are referred to as continuous-time Markov chain to
reflect the fact that the state space is discrete.
• It turns out that the Poisson process is special pure birth process.

STAT306, Term I, 2011/12 3

 Chapter 6
Continuous-Time Markov Chains

Consider a family of random variables {X(t), t  0} where the

possible values of X(t) are nonnegative integers.

Definition 6.1. A stochastic process {X(t), t  0} is said to be a

continuous-time Markov chain if for all s, t,  0, nonnegative
integers i, j, and x(u) with 0  u < s,
P{X(t + s) = j | X(s) = i, X(u) = x(u), 0  u < s}
= P{X(t + s) = j | X(s) = i}.

We will consider only the case where P{X(t + s) = j | X(s) = i}

is independent of s, i.e., a continuous-time Markov chain
with stationary transition probabilities.
The function Pij(t) = P{X(t + s) = j | X(s) = i}, t > 0, is called
the transition probability function.
STAT306, Term I, 2011/12 4
 Chapter 6
Continuous-Time Markov Chains
Let Ti denote the amount of time that the continuous-time Markov
chain X(t) stays in state i before making a transition into a different
state. We have the following result.

Theorem 6.1. Let Ti, i = 0, 1, 2, …, be defined above. We have

P{Ti > t + s | Ti > s } = P{Ti > t },
and hence the random variable Ti is exponentially distributed.

• The probability statement in Theorem 6.1 follows directly from

the Markovian property: given that the process is in state i at
time s, the probability that the process will not leave state i
during the interval [s, t+s] is just the unconditional probability
that it stays in state i for at least t time units.
• Since Ti is memoryless, it must be exponentially distributed

STAT306, Term I, 2011/12 5

 Chapter 6
Continuous-Time Markov Chains
Thus, a continuous-time Markov chain is a stochastic process
that moves from state to state in accordance with a discrete-
time Markov chain, but is such that the amount of time it
spends in each state is exponentially distributed.
The amount of time the process spends in state i, and next state
visited, must be independent random variables. This is argued
as follows.:
For if the next state visited were dependent on Ti , then
information as to how long the process has already been in
state i would be relevant to the prediction of the next state,
which contradicts the Markovian property.

STAT306, Term I, 2011/12 6 © Zhenlin Yang, SMU

 Chapter 6
Continuous-Time Markov Chains
Definition 6.2. A continuous-time Markov chain can be
equivalently defined as a stochastic process having properties that
each time it enters state i,
(i) The amount of time, Ti , it spends in that state before making a
transition into a different state is exponentially distributed;
(ii) When the process leaves state i, it next enters state j with
probability Pij such that
Pii = 0, all i, and  j Pij  1, all i

There are many quantities related to a continuous-time Markov

chain that are of interest. The foremost ones are
• vi = 1/E(Ti ), the state transition rate, i.e., the rate at which the
process makes a transition when in state i, and
• Pij , the probability that this transition is into state j.
STAT306, Term I, 2011/12 7
 Chapter 6
Continuous-Time Markov Chains
Example 6.1. (A Shoeshine Shop) Consider a shoeshine
establishment consisting of two chairs – chair 1 and chair 2.
• A customer upon arrival goes to initially to chair 1 where his
shoes are cleaned and polish is applied. After this is done the
customer moves on to chair 2 where the polish is buffed.
• The service times at the two chairs are assumed to be
independent random variables that are exponentially distributed
with respective rates 1 and 2.
• Suppose that the potential customers arrive in accordance with a
Poisson process with rate  and that a potential customer will
enter the system only if both chairs are empty.
• Define a continuous-time Markov chain and find vi = 1/E(Ti ),
and Pij .

STAT306, Term I, 2011/12 8

 Chapter 6
Continuous-Time Markov Chains
Solution: The number of customers in the system can only be 0
or 1, but which chair the customer is in matters.
State Interpretation
0 System is empty
1 a customer is in chair 1
2 a customer is in chair 2

v0 = 1/E(T0) = , reciprocal of the mean time to first customer

v1 = 1/E(T1) = 1, reciprocal of the mean time spent in chair 1
v2 = 1/E(T2) = 2, reciprocal of the mean time spent in chair 2
P01 = 1, process once started will surely go to state 1,
P12 = 1, once process is in state 1, it will surely move to state 2,
P20 = 1, given the process in state 2, it will surely go back to the
initial state.
STAT306, Term I, 2011/12 9
 Chapter 6
Pure Birth Processes
We now start to introduce some special continuous-time Markov
chains. Poisson process is in fact a special pure birth process. It is
easier to learn the continuous-time Markov chain by generalizing
the Poisson process.
A natural generalization of the Poisson process is to permit the
chance of an event occurring at a given instant of time to
depend upon the number of events that have already occurred.
An example of this phenomenon is the reproduction of living
organism, in which, if there is sufficient food, no mortality,
and no migration, the infinitesimal probability of a birth at a
given instant is proportional to the population size at that time.
In this case, only ‘births’ occur in the process, and population
is always increasing.

STAT306, Term I, 2011/12 10

 Chapter 6
Pure Birth Processes
Recall the postulates that define the Poisson process given in
Definition 5.2, which can be written more concisely as:
(a)P{X(h+s)  X(s) = 1 | X(s) = i} = h + o(h) as h  0,
(b)P{X(h+s)  X(s) = 0 | X(s) = i} = 1  h + o(h) as h  0
for i = 0, 1, 2, …,
(c) X(0) = 0.

The precise interpretation of (a) is the relationship:

P{ X (h  s )  X ( s )  1 | X ( s )  i}
lim  .
h  0 h
which is independent of s, and independent of i as well.
Based on these postulates one shows that for any s > 0,
 t (t ) n
P{ X (t  s)  X ( s)  n}  e , n  0, 1, . . .
n!
 is called the rate of occurrence of Poisson events.
STAT306, Term I, 2011/12 11
 Chapter 6
Pure Birth Processes
Definition 6.3. A pure birth process is a continuous-time Markov
chain satisfying the following postulates:
(a)P{X(h+s)  X(s) = 1 | X(s) = i} = i h + o(h) as h  0,
(b)P{X(h+s)  X(s) = 0 | X(s) = i} = 1  i h + o(h) as h  0
for i = 0, 1, 2, … .
(c) P{X(h+s)  X(s) < 0 | X(s) = i} = 0 (i  0).
The transition probability function depends on initial state i,
Pij(t) = P{X(t + s) = j | X(s) = i}, t > 0,
The amount of time, Ti , it spends in state i before switching to
a different state is exponentially distributed with rate i ,
A special birth process is the Yule Process where i = i.
It arises in physics and biology and describes the growth of a population in which
each member has a probability h + o(h) of giving birth to a new member during
an interval of time of length h.
STAT306, Term I, 2011/12 12
 Chapter 6
Pure Death Processes
Complementing the increasing pure birth process is the pure
death process, which moves successively through states N, N1, .
. . , 2, 1 and ultimately is absorbed in state 0 (extinction).
• The process is specified by the death rate i > 0, i = 1, 2, . . .,
N, and the exponential (i) sojourn time in state i (time spent
in state i), all sojourn times being independent.

Definition 6.4. A pure death process is a continuous-time Markov

chain satisfying the following postulates:
(a)P{X(h+s) = i1 | X(s) = i} = i h + o(h), i = 1, . . ., N;
(b)P{X(h+s) = i | X(s) = i} = 1  i h + o(h), i = 1, . . ., N;
(c)P{X(h+s) > i | X(s) = i} = 0, i = 0, 1, . . ., N.
• Pure death process arises in failure analyses;
• A special case with i = i gives a linear death process.
STAT306, Term I, 2011/12 13 © Zhenlin Yang, SMU
 Chapter 6
Birth and Death Processes
Consider again the example of the reproduction of living
organism.
If there is sufficient food, no mortality, and no migration, the
infinitesimal probability of a birth at a given instant is
proportional to the population size at that time.
In this case, only ‘births’ occur in the process, and population
is always increasing.
If mortality occurs as well, then birth and death both occur,
and the population size can go up and down.
A birth and death process is more suitable for this
phenomenon.

STAT306, Term I, 2011/12 14

 Chapter 6
Birth and Death Processes
Definition 6.3. A birth and death process is a continuous-time
Markov chain with state space {0, 1, . . .} such that
a)if at time t the process is in state n it may, after a random time,
move to either of the neighboring states n  1 and n + 1,
b)whenever there are n ‘people’ in the system, then new arrivals
enter the system at an exponential rate n, departures from the
system happens at an exponential rate n, and arrivals and
departures are independent.

The birth and death (B-D) process can be viewed as a

continuous-time analogue of the random walk process.
{n} are called the arrival or birth rates and {n} the departure
or death rates
When n = 0, the process becomes the pure birth process;
and when n = 0, it becomes the pure death process.
STAT306, Term I, 2011/12 15
 Chapter 6
Birth and Death Processes
The above definition can equivalently be defined based on some
postulates as we do in defining the Poisson process. Let
Pij(t) = P{X(t + s) = j | X(s) = i}
denote the stationary transition probabilities that a process
presently in state i will be in state j a t time later.

Definition 6.4. A birth and death process is a continuous-time

Markov chain with state space {0, 1, . . .} such that for a small h >
0, Pij(t) satisfies
1)Pi, i+1 (h) = i h + o(h), i  1,
2)Pi, i1 (h) = i h + o(h) , i  1,
3)Pi, i (h) = 1  (i + i)h + o(h) , i  1,
4)Pi, j (0) = i,j (= 1 if i = j, 0 otherwise),
5)0 = 0, 0 > 0, i > 0, i > 0, i = 1, 2, . . . ,

STAT306, Term I, 2011/12 16

 Chapter 6
Birth and Death Processes
Theorem 6.2. For a birth and death process defined above,
the state transition rates are
v0 = 0, vi = i + i , i > 0;
the transition probabilities (into states i+1 or i1 from i) are
i i
Pi ,i 1  , i  0; Pi ,i 1  , i0
i  i i  i
Given at time t the process is in state i,
the time until next birth is exponential with rate i ;
the time until the next death is another independent exponential with rate i ;
the time until the next transition must be an exponential with rate i + i ;
if a birth occurs before a death, then the process moves from state i to state
i+1. The probability of it is the conditional probability of a birth given a
transition has occurred, which is i /(i + i) (see Sec. 5.2.3 of text book);
similarly, the probability of a death before a birth is i /(i + i).

STAT306, Term I, 2011/12 17

 Chapter 6
Birth and Death Processes
Example 6.2. (A Linear Growth Model with Immigration) A
special type of Birth and death process that occurs naturally in the
study of biological reproduction and population growth is called a
linear growth model with immigration, in which
n = n + , n  0
n = n, n1
This is reasoned as follows:
• Each individual in this population is assumed to give birth at an
exponential rate ; in addition, there is an exponential rate of
increase  of the population due to an external source such as
immigration. Hence, the total birth rate where there are n
persons in the system is n + .
• Deaths are assumed to occur at an exponential rate  for each
member of the population, so n = n.
STAT306, Term I, 2011/12 18
 Chapter 6
Birth and Death Processes
Example 6.3. (A Queueing System M/M/1) Suppose that
customers arrive at a single-server service station in accordance
with a Poisson process having rate .
• Upon arrival, each customer goes directly into service if the
server is free; if not, then the customer joins the queue.
• When the server finishing serving a customer, the customer
leaves the system and the next customer in the queue, if there
are any waiting, enters the service.
• The successive service times are assumed to be independent
exponential random variables having mean 1/.
• The preceding is known as the M/M/1, referring to Markovian
arrivals, Markovian services, and 1 server.
If X(t) is the number in the system at time t, then {X(t), t  0} is a
B-D process with n = , n  0; n = , n  1.
STAT306, Term I, 2011/12 19
 Chapter 6
Birth and Death Processes
Example 6.4. (A Multiserver Queueing System M/M/s) Consider
an exponential queueing system in which there are s servers, each
serving at rate . An entering customer first waits in line and then
goes to the first free server. This is a B-D process with parameters
n = , n0
n = n, 1ns
= s, n>s
It is also known as an M/M/s queueing model.

Interesting questions can be asked in a queueing model include:

• what is the fraction of time the system is idle?
• What is the average number of customers in the queue?

STAT306, Term I, 2011/12 20

 Chapter 6
Birth and Death Processes
Time Tij until entering state j (> i), starting from state i.
First note that this time is different from the time until a transition.
Find E(Ti,i+1) and Var(Ti,i+1) (Refer to p. 378-381, textbook).

STAT306, Term I, 2011/12 21

 Chapter 6
Transition Probability Function
Consider the transition probabilities of the continuous-time
Markov chain,
Pij(t) = P{X(t + s) = j | X(s) = i}
i.e., probabilities that a process presently in state i will be in state j
a t time later.

• As these probabilities depend on t, it is referred to as the

transition probability function (TPF).
• Note that Pij(t) differs from Pij defined earlier. Pij is the
probability when process leaves state i it next enters state j.
• Derivation of TPF for a general continuous-time Markov chain
typically involves solving a set of differential equations,
except for certain special cases, e.g., pure birth process.

STAT306, Term I, 2011/12 22

 Chapter 6
Transition Probability Function
Proposition 6.1. For a pure birth process having i  j, i  j,
Pii (t )  e i t
j j
r j 1 j 1
r
Pij (t )   e  k t
  e  k t
 , i j
k i r  k , r i r  k k i r  k , r i r  k

Proof:

STAT306, Term I, 2011/12 23

 Chapter 6
Transition Probability Function
Example 6.5. Consider the Yule process, which is a pure birth
process with linear birth rate i =i. Find P1j(t).

Solution:

STAT306, Term I, 2011/12 24

 Chapter 6
Transition Probability Function
For a general continuous-time Markov chain, the TPF can be
found through Kolmogorov’s Backward Equations, or
Kolmogorov’s Forward Equations, to be introduced later. For
these, we need the following two results.

For any pair of states i and j, let

qij = vi Pij
•where vi is the rate at which the process makes a transition when
in state i and Pij is the probability that this transition is into j, both
being defined at the beginning of the Chapter.
•It follows that qij is the rate, when in state i, at which the process
makes a transition into state j.
•The quantities qij are called the instantaneous transition rates
from state i into state j.
STAT306, Term I, 2011/12 25
 Chapter 6
Transition Probability Function
Lemma 6.1. Alternatively, the state transition rates and the
instantaneous transition rates can be defined as follows,
1  Pii (h) Pij (h)
vi  lim , and qij  lim , i  j.
h 0 h h  0 h
Proof:

STAT306, Term I, 2011/12 26

 Chapter 6
Transition Probability Function
Lemma 6.2. (Chapman-Kolmogorov Equations) For a general
continuous-time Markov chain, we have for all s  0, t  0,

Pij (t  s )   Pik (t ) Pkj ( s )
k 0

Proof:

STAT306, Term I, 2011/12 27

 Chapter 6
Transition Probability Function
Theorem 6.1. (Kolmogorov’s Backward Equations) For a general
continuous-time Markov chain, the TPF Pij(t) satisfies, for all
states i, j, and time t  0, the following differential equation:
Pij(t )   qik Pkj (t )  vi Pij (t )
k i
where ‘  ’ denotes the derivative.

• For the pure birth process, the backward equations become

Pij (t )  i Pi 1, j (t )  i Pij (t )
• For the B-D process, the backward equations become,
P0j (t )  0 P1, j (t )  0 P0 j (t )
Pij(t )  i Pi 1, j (t )  i Pi 1, j (t )  (i  i ) Pij (t ), i  0

STAT306, Term I, 2011/12 28

 Chapter 6
Transition Probability Function
Theorem 6.2. (Kolmogorov’s Forward Equations) For a general
continuous-time Markov chain, the TPF Pij(t) satisfies, for all
states i, j, and time t  0, the following differential equation:
Pij (t )   qkj Pik (t )  v j Pij (t ) .
k j

• For the pure birth process, the backward equations become

Pij (t )   j 1Pi , j 1 (t )   j Pij (t ) .
• For the B-D process, the backward equations become,
Pij (t )   j 1Pi , j 1 (t )   j 1Pi , j 1 (t )  ( j   j ) Pij (t ) .

STAT306, Term I, 2011/12 29

 Chapter 6
Limiting Probabilities
Theorem 6.8. For a general continuous-time Markov chain, if (i)
all states communicate and (ii) the Markov chain is positive
recurrent, then, the limiting probabilities
 j  lim Pij (t )
t 
exist, and are the solutions of the following set of equations:
v j  j   qkj  k , all states j , subject to   j  1
k j j

• The above equations have a nice interpretation: in any interval

(0, t) the number of transitions into state j must equal within 1
the number of transitions out of state j.
• Hence, in the long run, we have
the rate at which transitions into state j occur
= the rate at which transitions out of state j occur.
STAT306, Term I, 2011/12 30
 Chapter 6
Limiting Probabilities
The above is further reasoned as follows:
• When the process is in state j, it leaves at rate vj, and as j is the
proportion of time it is in state j, it thus follows that
vj j = rate at which the process leaves state j
• When process is in state k, it enters j at rate qkj . Hence, as k is
the proportion of time it is in state k, the rate of transitions from
k to j is just qkjk , and
q
k j
kj  k  the rate at which t he process enters state j

Because of their interesting interpretations, the equations defining

the limiting probabilities are referred to as balance equations.

STAT306, Term I, 2011/12 31

 Chapter 6
Limiting Probabilities
Theorem 6.7. For a birth and death process with birth rates {n}
and death rates {n}, define
01 n 1
n  , n 1
12  n
Then, the limiting probabilities are given as

 
1
 0  1  n 1n ,  n  n 0 , n  1




provided that   .
n 1 n

Proof: Recall for B-D process, the state transition rates are
v0 = 0, and vi = i + i , i = 1, 2, . . .
The limiting probabilities must satisfy:
STAT306, Term I, 2011/12 32
 Chapter 6
Limiting Probabilities
State Rate of Leaving = Rate of Entering
0 00 = 11 Taking partial
1 (1+1)1 = 22 + 00 sums of these
2 (2+2)2 = 33 + 11 equations
 gives,
n (n+n)n = n+1n+1 + n1n1

0 0  1 1  1  (0 1 ) 0  1 0 As 0+1 + . . . = 1,
1 1  2 2  2  (1 2 ) 1   2 0 
1   0   0  n
2 2  3 3  3  (2 3 ) 2  3 0 n 1
 1
   
 0  1   n 
n n  n 1 n 1  n  (n 1 n ) n 1   n 0  n 1 

STAT306, Term I, 2011/12 33

 Chapter 6
Limiting Probabilities
Example 6.6. (The M/M/1) In the M/M/1 queue of Example 6.3.
It is a B-D process with n = , n  0; n = , n  1. Then, using
Theorem 6.7, we obtain
n  (01 n 1 ) (12 n )    n
 0  1    ,  n    n  0 , n  1
provided / < 1.

• Thus, the proportion of time that the system is idle = 1  /

• And the average number of customers in the system is

n 0      0   n 1 n  
n 1

 n 
n 0

 
  0   
1

(1    ) 2
(1    )
STAT306, Term I, 2011/12 34
 Chapter 6
Limiting Probabilities
Example 6.7. Reconsider the shoeshine shop of Example 6.1.
Find the limiting probabilities.
As this is not a B-D process, need to apply Theorem 6.6 directly:
State Rate that process leaves = rate that the process enters
0  0 = 2 2
1 1 1 =  0
2 2 2 = 1 1
Solving inters of 0 yields: 1 = (/1) 0 and 2 = (/2) 0, and
using 0 + 1 + 2 =1, gives
12
0  , and
12   ( 1  2 )
2 1
1  , 2 
12   ( 1  2 ) 12   ( 1  2 )
STAT306, Term I, 2011/12 35
 Chapter 6
Limiting Probabilities
Exercise. Each time at machine is repaired it remains for an
exponentially distributed time with rate . It then fails, and its
failure is either of two types. If it is a type 1 failure, then the time
to repair the machine is exponential with rate 1; if it is a type 2
failure, then the repair time is exponential with rate 2. Each
failure is, independent of the time it took the machine to fail, a
type 1 failure with probability p and a type 2 failure with
probability 1 – p.
(a)What proportion of time is the machine down due to a type 1
failure?
(b)What proportion of time is it down due to a type 2 failure?
(c)What proportion of time is it up?

STAT306, Term I, 2011/12 36

 Chapter 6
Limiting Probabilities
Solution:

STAT306, Term I, 2011/12 37