9.

Recurrent and Transient States

9.1 Denitions
9.2 Relations between f
i
and p
(n)
ii
9.3 Limiting Theorems for Generating Functions
9.4 Applications to Markov Chains
9.5 Relations Between f
ij
and p
(n)
ij
9.6 Periodic Processes
9.7 Closed Sets
9.8 Decomposition Theorem
9.9 Remarks on Finite Chains
9.10 Perron-Frobenius Theorem
9.11 Determining Recurrence and Transience when Number of
States is Innite
9.12 Revisiting Statistical Equilibrium
9.13 Appendix. Limit Theorems for Generating Functions
304
9.1 Denitions
Dene f
(n)
ii
= P{X
n
= i, X
1
= i, . . . , X
n1
= i|X
0
= i}
= Probability of rst recurrence to i is at the n
th
step.
f
i
= f
ii
=

n=1
f
(n)
ii
= Prob. of recurrence to i.
Def. A state i is recurrent if f
i
= 1.
Def. A state i is transient if f
i
< 1.
Dene T
i
= Time for rst visit to i given X
0
= 1. This is the same as
Time to rst visit to i given X
k
= i. (Time homogeneous)
m
i
= E(T
i
|X
0
= i) =

n=1
nf
(n)
ii
= mean time for recurrence
Note: f
(n)
ii
= P{T
i
= n|X
0
= i}
305
Similarly we can dene
f
(n)
ij
= P{X
n
= j, X
1
= j, . . . , X
n1
= j|X
0
= i}
= Prob. of reaching state j for rst time in n steps starting from X
0
= i.
f
ij
=

n=1
f
(n)
ij
= Prob. of ever reaching j starting from i.
Consider f
ii
= f
i
= prob. of ever returning to i.
If f
i
< 1, 1 f
i
= prob. of never returning to i.
i.e.
1 f
i
= P{T
i
= |X
0
= i}
f
i
= P{T
i
< |X
0
= i}
306
TH. If N is no. of visits to i|X
0
= i E(N|X
0
= i) = 1/(1 f
i
)
Proof: E(N|X
0
= i) = E[N|T
i
= , X
0
= i]P{T
i
= |X
0
= i}
+E[N|T
i
< , X
0
= i]P{T
i
< |X
0
= i}
E(N|X
o
= i) = 1 (1 f
i
) +f
i
[1 +E(N|X
0
= i)]
If T
i
= except for n = 0 (X
0
= i) , there will never be a visit to i-
i.e. E(N|T
i
= , X
0
= i) = 1. If T
i
< , there is sure to be one visit,
say at X
k
(X
k
= i). But then
E(N|T
i
< , X
k
= i) = E(N|T
i
< , X
0
= i) by Markov property;
i.e.
E[N|T
i
< , X
0
= i] = 1 +E[N|X
0
= i]
.. E[N|X
0
= i] = 1 (1 f
i
) +{1 +E[N|X
0
= i]} (f
i
)
E[N|X
0
= i] = 1/(1 f
i
)
Another expression for E[N|X
0
= i] =

n=0
p
(n)
ii
307
Relation to Geometric Distribution
Suppose i is transient (f
i
< 1) and N
i
=no. of visits to i.
P{N
i
= k + 1|X
0
= i} =f
k
i
(1 f
i
), k = 0, 1, . . .
E(N
i
|X
0
= i) =

k=0
(k + 1)f
k
i
(1 f
i
) =

k=0
kf
k
i
(1 f
i
) + 1
Since
(1 f
i
)
1
=

k=0
f
k
i
(1 f
i
)
2
=
d
df
i
(1 f
i
)
1
=

k=0
kf
k1
i
E(N
i
|X
o
= i) = f
i
(1 f
i
)(1 f
i
)
2
+ 1
= f
i
(1 f
i
)
1
+ 1 = 1/(1 f
i
)
308
TH. E[N|X
0
= i] =

n=0
p
(n)
ii
Proof. Let Y
n
=
_

_
1 if X
n
= i
0 otherwise
N =

n=0
Y
n
Since P{Y
n
= 1|X
0
= i} = P{X
n
= i|X
0
= i} = p
(n)
ii
E(N) =

n=0
E(Y
n
) =

n=0
p
(n)
ii
E(N) may be nite or innite
309
Def: A positive recurrent state is dened by f
i
= 1, m
i
< .
A null recurrent state is dened by f
i
= 1, m
i
=
Ex: f
(n)
ii
=
1
n(n + 1)
=
1
n

1
n + 1
f
i
=

n=1
_
1
n

1
n + 1
_
= 1
But m
i
=

n=1
nf
(n)
ii
=

1
n
(n + 1)n
=

n=1
1
n + 1
= as series does
not converge.
310
Classication of States
f
i
m
i
Positive recurrent state 1 <
Null recurrent state 1
Transient < 1 <
where m
i
= Expected no. of visits to i given X
0
= i.
In addition the recurring and transient states may be characterized by
being periodic or aperiodic.
A state is ergodic if it is aperiodic and positive recurrent.
311
9.2 Relations Between f
i
and p
(n)
ii
Consider p
(n)
ii
. Starting from X
0
= i, the rst recurrence to i may be at
k = 1, 2, . . . , n. Consider the rst visit is at time k and at X
n
, X
n
= i
another visit is made. This probability is f
(k)
ii
p
(nk)
ii
. Summing over all k
results in
() p
(n)
ii
=
n

k=1
f
(k)
ii
p
(nk)
ii
where p
(o)
ii
= 1 = P{X
0
= i|X
0
= i}
Multiplying () by s
n
and summing

n=1
p
(n)
ii
s
n
=

n=1
n

k=1
f
(k)
ii
p
(nk)
ii
s
n
=

k=1
f
(k)
ii
s
k

n=k
p
(nk)
ii
s
nk
312

n=1
p
(n)
ii
s
n
=

n=1
n

k=1
f
(k)
ii
p
(nk)
ii
s
n
=

k=1
f
(k)
ii
s
k

n=k
p
(nk)
ii
s
nk
P
ii
(s) 1 = F
ii
(s)P
ii
(s)
where P
ii
(s) =

n=0
p
(n)
ii
s
n
, F
ii
(s) =

n=1
f
(n)
ii
s
n
P
ii
(s) =
1
1 F
ii
(s)
Note:
lim
s1
F
ii
(s) = F
ii
(1) =

n=1
f
(n)
ii
= f
i
lim
s1
F

ii
(s) = F

ii
(1) =

n=1
nf
(n)
ii
= m
i
313
Theorems
(a) If P
ii
(1) =

0
p
(n)
ii
= f
i
= 1.
Conversely if f
i
= 1, P
ii
(1) =
(b) If P
ii
(1) =

0
p
(n)
ii
< f
i
< 1.
Conversely if f
i
< 1, P
ii
(1) <
314
9.3 Limiting Theorems for Generating Functions
Consider A(s) =

n=0
a
n
s
n
, |s| 1 with a
n
0.
1. lim
n
n

k=0
a
k
= lim
s1
A(s) where s 1 means s 1.
2. Dene
a

(n) =
n

k=0
a
k
/(n + 1)
lim
n
a

(n) = lim
s1
(1 s)A(s)
3. Cesaro Limit
The Cesaro limit is dened by lim
n
a

(n) If the sequence {a

n
} has
a limit = lim
n
a
n
then lim
n
a

(n) = .
The Cesaro limit may exist without the existence of the ordinary limit.
315
Ex. a
n
: 0, 1, 0, 1, 0, 1, . . .
lim
n
a
n
does not exist.
However
a

(n) =
_

_
1
2
if n even
1
2
(1
1
n
) if n is odd
lim
n
a

(n) =
1
2
316
9.4 Application to Markov Chains
Consider p

ii
(n) =
n

k=0
p
(k)
ii
n + 1
n

k=0
p
(k)
ii
is expected no. of visits to i starting from X
0
= i (p
0
ii
= 1).
Dividing by (n + 1), p

ii
(n) is expected no. of visits per unit time.
Ex. n = 29 days, p

ii
(29) = 2/30; i.e. 2 visits per 30 days or 1 visit per
15 days. One would expect mean time between visits = 15 days.
317
TH. lim
n
n

k=0
p
(k)
ii
n + 1
=
1
m
i
, where m
i
= expected no. of visits and
f
i
= 1
Proof: Consider P
ii
(s) =
1
1 F
ii
(s)
lim
s1
(1 s)P
ii
(s) = lim
n
p

ii
(n) = lim
s1
(1 s)
1 F
ii
(s)
.
Since F
ii
(1) = f
i
, if f
i
= 1, the r.h.s. is indeterminate. Using
LHopitals rule
lim
n
p

ii
(n) =
1
F

ii
(1)
=
1
m
i
Recall a positive recurrent state has m
i
< lim
n
p

ii
(n) > 0
A null recurrent state has m
i
=
lim
n
p

ii
(n) = 0 or lim
n
p
(n)
ii
= 0
318
9.5 Relations Between f
ij
and p
(n)
ij
(i = j)
f
(n)
ij
= P{X
n
= j, X
r
= j, r = 1, 2, . . . , n 1|X
0
= i}
= Prob. of starting from i and reaching j for rst time at n
th
step.
f
ij
=

n=1
f
(n)
ij
i = j
Proceeding as before (i = j)
p
(n)
ij
= f
(1)
ij
p
(n1)
jj
+f
(2)
ij
p
(n1)
jj
+. . . +f
(n)
ij
=
n

k=1
f
(k)
ij
p
(nk)
jj
(p
(0)
jj
= 1)
Multiplying by s
n
and summing over n

n=1
p
(n)
ij
s
n
=

n=1
n

k=1
f
(k)
ij
p
(nk)
jj
s
n
319
P
ij
(s) =

k=1
f
(k)
ij
s
k

n=k
p
(nk)
jj
s
nk
P
ij
(s) = F
ij
(s)P
jj
(s) i = j
lim
s1
(1 s)P
jj
(s) = lim
n
p

jj
(n) = lim
s1
(1 s)P
ij
(s)/F
ij
(1)
lim
n
p

jj
(n) = lim
n
p

ij
(n)
F
ij
(1)
=
1
m
i
or lim
n
p

ij
(n) =
F
ij
(1)
m
i
Also P
ij
(1) =

n=1
p
(n)
ij
= F
ij
(1)

n=0
p
(n)
jj
.
Hence if

n=0
p
(n)
jj
=

n=0
p
(n)
ij
= (p
(0)
ij
= 0).
Similarly if

n=0
p
(n)
jj
<

n=0
p
(n)
ij
<
320
Summary
Transient

n=0
p
(n)
ii
< , f
i
< 1, m
i
= 1/(1 f
i
)
lim
n
p
(n)
ii
= 0,

n=1
p
(n)
ij
< , lim
n
p
(n)
ij
= 0
Positive Recurrent

n=0
p
(n)
ii
= , f
i
= 1, m
i
<
lim
n
p

ii
(n) > 0 (= 1/m
i
)
lim
n
p

ij
(n) > 0 (= F
ij
(1)/m
i
)
Negative Recurrent

n=0
p
(n)
ii
= , f
i
= 1, m
i
=
lim
n
p

ii
(n) = 0, lim
n
p
(n)
ii
= 0
321
9.6 Periodic Processes
Suppose transition probabilities have period d.
Then
p
(n)
ij
= 0, p
(n)
ii
= 0 if n = rd r = 1, 2, . . .
p
(n)
ij
0, p
(n)
ii
0 if n = rd
P
ii
(s) =

r=0
p
(rd)
ii
s
rd
=

r=0
p
(rd)
ii
z
r
, z = s
d
F
ii
(s) =

r=1
f
(rd)
ii
s
rd
=

r=1
f
(rd)
ii
z
r
322
We now have a power series in z
lim
z1
P
ii
(Z) =

r=0
p
(rd)
ii
lim
z1
(1 z)P
ii
(z) =

n
n

r=0
p
(rd)
ii
n + 1
Note: E(N|X
0
= i) =

1
nf
(n)
ii
= d

r=1
rf
(rd)
ii
= m
i
However F

ii
(1) =

r=1
f
(rd)
ii
r = m
i
/d
Since P
ii
(Z) = 1/[1 F
ii
(z)]
lim
n
p

ii
(n) = d/m
i
or lim
n
p
(nd)
ii
= d/m
i
323
9.7 Closed Sets
Def. A set of states C is closed if no state outside C can be reached from
any state in C; i.e., p
ij
= 0 if i C and j / C.
Absorbing state: Closed set consisting of a single state.
Irreducible Chain: If only closed set is the set of all states. (Every state
can be reached from any other state).
This means that we can study the behavior of states in C by omitting all
other states.
324
Th. i j, i is recurrent j recurrent
i j, i is transient j transient

r=0
p
(r)
jj

r=0
p
(r+n+m)
jj
=

r=0

kS
p
(m)
jk
p
(r)
kk
p
(n)
kj

r=0
p
(m)
ji
p
(n)
ii
p
(n)
ij
= p
(m)
ji
p
(n)
ij

r=0
p
(r)
ii
Thus if

r=0
p
(r)
ii
= ,

r=0
p
(r)
jj
=
Suppose i is transient, i j and assume j is recurrent. By theorem
just proved i then must be recurrent. However this is a contradiction
j is transient.
325
Summary (Aperiodic, irreducible)
1. If i j and j is positive recurrent i is positive recurrent
lim
n
p
(n)
ij
= lim
n
p
(n)
jj
=
j
= 1/m
j
.
2. If i j and j is null recurrent i is null recurrent
lim
n
p
(n)
jj
= 0, lim
n
p
(n)
ij
= 0
or P
()
= lim
n
P
(n)
= lim
n
P
n
= 0.
3. If i j and j is transient i is transient
lim
n
p
(n)
jj
= 0, lim
n
p
(n)
ij
= 0
326
9.8 Decomposition Theorem
(a) The states of a Markov Chain may be divided into two sets (one of
which may be empty). One set is composed of all the recurring states, the
other of all the transient states.
(b) The recurrent states may be decomposed uniquely into two closed
sets. Within each closed set all states inter-communicate and they are all
of the same type and period. Between any two closed sets no
communication is possible.
327
Ex. Decomposition of a Finite Chain
P =
C
0
C
1
C
2
C
3
C
0
1 0
.
.
. 0 0
.
.
. 0 0
.
.
. 0 0
0 0
.
.
. 0 0
.
.
. 0 0
.
.
. 0 0
C
1
O
.
.
. P
1
.
.
. O
.
.
. O
C
2
O
.
.
. O
.
.
. P
1
.
.
. O
C
3
A
.
.
. B
.
.
. C
.
.
. D
C
0
: Consists of two absorbing states
C
1
: Consists of closed recurrent states
C
2
: Consists of closed recurrent states
C
3
: Consists of transient states
A: Transitions C
3
C
0
B: Transitions C
3
C
1
C: Transitions C
3
C
2
D: Transitions C
3
C
3
328
9.9 Remarks on Finite Chains
1. A nite chain cannot consist only of transient states.
If i, j are transient lim
n
p
(n)
ij
= 0,
However

jS
p
(n)
ij
= 1
leading to a contradiction as n .
2. A nite chain cannot have any null recurrent states.
The one step transition probabilities within a closed set of null
recurrent states form a stochastic matrix P such that
P
n
0 as n . This is impossible as

jS
p
(n)
ij
= 1.
329
9.10 Perron-Frobenius Theorem
Earlier we had seen that if P has a characteristic root (eigenvalue) = 1 of
multiplicity 1, and all other |
i
| < 1, then
P
()
= E
1
.
The conditions under which this is true are proved by the
Perron-Frobenius Theorem. The necessary and sufcient conditions are:
P : aperiodic
P : positive recurrent (m
i
< ).
Then P
()
1
= P

= E
1
= 1y

where y

P = y

and 1 =
_
1 1 1
_

Def. A state is ergodic if it is aperiodic and positive recurrent.

330
9.11 Determining Recurrence and Transience
when Number of States is Innite
Compute f
i
= P{T
i
< |X
0
= i} in a closed communicating class.
All states are recurrent if f
i
= 1; All states are transient if f
i
< 1.
Ex. S = {0, 1, 2, . . . }, p
i,0
= q
i
p
i,i+1
= p
i
,
X
n+1
=
_
_
_
0 with q
i
X
n
+ 1 with p
i
P{T
0
> n|X
0
= 0} = P{X
1
= 1, X
2
= 1, . . . , X
n
= n|X
0
= 0}
=
n1

i=0
p
i
P{T
0
< |X
0
= 0} = 1 lim
n
P{T
0
> n|x
0
= 0} = 1

i=0
p
i
.. State 0 (and all states in closed class) are recurrent iff

i=0
p
i
= 0
331
Ex. Random Walk on Integers
S = {0, 1, 2, . . . }
p
i,i+1
= p, p
i,i1
= q, p +q = 1
p
(2n+1)
00
= 0, n 0 (Go from 0 to 0 in odd number of transitions)
p
(2n)
00
=
_
2n
n
_
p
n
q
n
Consider

n=1
p
(n)
00
=

n=1
p
(2n)
00
=

n=1
(2n)!
n!n!
p
n
q
n
332
Note: Ratio Test: If A =

n=0
a
n
and
a
n+1
a
n
< 1 series converges as n
a
n+1
a
n
> 1 series diverges as n
p
2n+2
00
p
2n
00
=
(2n + 1)(2n + 2)
(n + 1)(n + 1)
pq 4pq as n
If p = q 4pq < 1.
If p = q = 1/2 4pq = 1 and test is inconclusive.
_
2n
s
_
p
s
q
2ns
N(2np, 2npq) = N(n, n/2) if p = 1/2
e
(sn)
2
/n
_
_
2
n
2
333
Since p
(2n)
00

1

n=1
p
(2n)
00

n=1
1

n
series diverges
.. State 0 is recurrent.
Th. An irreducible Markov Chain with S = {0, 1, 2, . . . } and Transition
Prob. {p
ij
} is transient iff
y
i
=

j=1
p
ij
y
j
i = 1, 2, . . .
has a non-zero bounded solution
Proof: P.88
334
Ex. Random walk on S = {0, 1, 2, . . . }
p
i,i+1
= p
i
, p
i,i1
= q
i
p
i,i
= r
i
, q
0
= 0 (p
i
+q
i
+r
i
= 1)
Equations:
y
1
= p
11
y
1
+p
12
y
2
= r
1
y
1
+p
1
y
2
y
2
=
_
1 +
q
1
p
1
_
y
1
y
2
= p
21
y
1
+p
22
y
2
+p
23
y
3
= q
2
y
1
+r
2
y
2
+p
2
y
3
y
3
=
_
1 +
q
1
p
1
+
q
1
q
2
p
1
p
2
_
y
1
335
In general,
y
n
=
_
1 +
n1

k=1
q
1
q
2
. . . q
k
p
1
p
2
. . . p
k
_
y
1
, n 1
y
n
=
_
n1

k=0

k
_
y
1
,
k
=
q
1
q
2
. . . q
k
p
1
p
2
. . . p
k
,
0
= 1
Thus the solution is bounded if

k=0

k
<
336
9.12 Revisiting Statistical Equilibrium
Assume the Markov Chain has all states which are irreducible positive
recurrent aperiodic. (Called Ergodic Chain).
Earlier we had shown
lim
n
p
(n)
ij
= lim
n
p
(n)
jj
= 1/m
j
=
j

j
is given by the solution

j
=

iS

i
p
ij
where

jS

j
= 1
or in matrix notation we can write the linear equations as
= P where : k 1, P : k k.
Proof a
j
(n) = P{X
n
= j} and we will show that lim
n
a
j
(n) =
j
a
j
(n +m) =

iS

i
p
(n)
ij
337
Take m
j
=

iS

i
p
(n)
ij
for any n
If n = 1
j
=

iS

i
p
ij
Allow n
j
=
_

iS

i
_

j
which is true if

is

i
= 1
In the above we made use of
lim
n
p
(n)
ij
= lim
n
p
(n)
jj
=
j
which holds for ergodic chains.
If chain was transient or null recurrent
p
(n)
ij
= p
(n)
jj
0 as n and
j
=

iS

i
p
(n)
ij
0 as n ,
and result does not hold.
338
Th: If a
0
(j) = P{X
0
= j} =
j
then P{a
n
= j} =
j
for all n.
This theorem states that if the initial probabilities correspond to the
limiting probabilities, then for any n p{X
n
= j} =
j
.
Proof:
In general a
j
(n) =

iS
a
0
(i)p
(n)
ij
and in matrix notation we can
write a
n
= P
n
a
0
If a
o
= , a
n
= P
n
. But is dened
by = P. a
n
= P =
This is the reason why is sometimes referred to as the stationary
distribution.
339
9.13 Appendix. Limit Theorems for Generating Functions
Denition: Let A(z) =

n=0
a
n
z
denote a power series. In our application z
will always be real, however all the results also hold if z is a complex
number.
Theorem If {a
n
} are bounded, say |a
n
| B, then A(z) converges for at
least |z| < 1.
Proof : A(z) = |A(z)|

n=0
|a
n
|z
n
| B

n=0
|z|
n
= B/1|z|
Denition: The number R is called the radius of convergence of the
power series A(z) if A(z) converges for |z| > R. Without loss of
generality the radius of convergence can be taken as R = 1. Note if R is
the radius of convergence of A(z) we can write A(z) =

0
b
n
y
n
and the
radius of convergence of y will be unity.
340
Properties of A(z)
(i) If R is the radius of convergence
R
1
= lim
n
sup(a
n
)
1/n
(ii) Within the interval of convergence (R < z < R), A(z) has
derivatives of all orders which may be obtained by term-wise
differentiation. Similarly the integral
_
b
a
A(z)dz is given by term-wise
integration for any (a, b) in (R, R).
(iii) If A(z) and B(z) =

0
b
n
z
n
both converge and are equal for all
|z| < R, then a
n
= b
n
.
(iv) No general statement can be made about the convergence of the
series on the boundary |z| = R; i.e.

n
a
n
R
n
may or may not be nite.
341
Two important theorems on power series are:
Abels Theorem Suppose A(z) has a radius of convergence R = 1 and

0
a
n
is convergent to s. Then
lim
z1
A(z) =

n=0
a
n
If the coefcients {a
n
} are non-negative, the result continues to hold
whether or not the sum on the right is convergent.
342
Note:
A
N
(z) =
N

n=0
a
n
z
n
=
N

n=0
(s
n
s
n1
)z
n
, s
n
=
n

i=0
a
i
=
N

n=0
s
n
z
n
z
N1

n=0
s
n
z
n
= (1 z)
N

n=0
s
n
z
n
+s
N
z
N
lim
z1
A
n
(z) = s
N
and taking the limit as N
lim
N
s
N
= s
343
Theorem: If the sequence {b
n
} converges to a limit b ( lim
n
= b), then
lim
z1
(1 z)

n=0
b
n
z
n
= b
Proof: (1 z)

n=0
b
n
z
n
=

n=0
(b
n
b
n1
)z
n
(b
1
= 0)
We can write
N

n=0
(b
n
b
n1
)z
n
= (1 z)
N

n=0
s
n
z
n
+s)nz
N
where
s
n
=
n

i=0
(b
i
b
i1
) = b
0
+(b
1
b
0
)+(b
2
b
1
)+. . . +(b
n
b
n1
) = b
n
Therefore (1 z)
N

n=0
b
n
a
n
= (1 z)
N

n=0
s
n
z
n
+b
n
z
N
and lim
z1

(1 z)
N

n=0
b
n
z
n
= b
n
, so that as N , lim
N
b
N
= b.
344