Book

4.9
a.
Check that the chain is indeed an ergodic unichain with transition matrix
P =
_

_
1 p p 0 0
1 p 0 p
.
.
.
.
.
.
0 1 p 0
.
.
. 0
.
.
.
.
.
.
.
.
.
.
.
. p
0 0 1 p p
_

_
We want
= P
(I P) = 0
In other words,
p
0
q
1
= 0
p
0
+
1
q
2
= 0
p
1
+
2

3
= 0
.
.
.
.
.
.
.
.
.
p
k3

k2
q
k1
= 0
p
k2
+q
k1
= 0
Notice that if we sum the rows sequentially, we have
p
0
q
1
= 0
p
1
q
2
= 0
.
.
.
p
k3
q
k2
= 0
which implies
i
=
i

0
, thus
_
1 + +
2
+ ... +
k1
_

0
= 1

0
=
_
1
1
k
< 1
1
k
= 1
If p = 0, then no matter where you start, you eventually end up at 0, thus the stable distribution is
= [1, 0, ....., 0], indeed = 0
0
= 1. The same goes for q = 0, then just plug it into the matrix and the
answer is obviously
k1
= 1 (if you dont see it then back solve the system of equations).
b. plug values in.
c.
< 1 :

0
= 1 , and
i
=
i
(1 ), thus the probability of a chain being in k decreases geometrically
with k.
= 1:
Equal probability of being in any of the k states. Thus as k , the chance of being in any one
of them decreases to 0. A very interesting result.
1
> 1:

k
increases geometrically with k, however, we dont know which state has the highest stable
distribution, since it is .
4.10
a.
The trasition matrix for 4.2a is
P
a
=
_

_
0 p 0 q
q 0 p 0
0 q 0 p
p 0 q 0
_

_
By calculation or observation, we can see that the uniform distribution is the steady state distribution.
The transition matrix for 4.2b is
P
b
=
_

_
0 1 0 0
0 0 1
0 1
.
.
.
1
2
.
.
.
1
2
.
.
.
0 1
1
.
.
.
.
.
.
.
.
. 1 0
1
0 0 0 1 0 0
_

_
Thus we have
1
2

4
=
1

1
=
2

2
=
3

3
+
9
=
4
1
2

4
=
5

5
=
6

6
=
7

7
=
8

8
=
9
Implying
1
=
2
=
3
=
5
= ... =
9
, while
4
= 2
1
, that is

1
=
1
10
,
4
=
2
10
b.
P
2
a
=
_

_
2pq 0 p
2
+ q
2
0
0 2pq 0 p
2
+ q
2
p
2
+ q
2
0 2pq 0
0 p
2
+ q
2
0 2pq
_

_
2
P
2
b
=
_

_
0 0 1 0 0 0 0 0 0
0 0 0 1 0 0 0 0 0
1
2
0 0 0
1
2
0 0 0 0
0
1
2
0 0 0
1
2
0 0 0
0 0 0 0 0 0 1 0 0
0 0 0 0 0 0 0 1 0
0 0 0 0 0 0 0 0 1
0 0 0 1 0 0 0 0 0
1
2
0 0 0
1
2
0 0 0 0
_

_
Both chains become reducible, each reduces to two ergodic chains (makes sense because the period is 2).
Rearranging the columns and rows we have
P
2
a
=
_

_
2pq p
2
+ q
2
0 0
p
2
+ q
2
2pq 0 0
0 0 2pq p
2
+ q
2
0 0 p
2
+ q
2
2pq
_

_
P
2
b
=
_

_
0 1 0 0 0 0 0 0 0
1
2
0
1
2
0 0 0 0 0 0
0 0 0 1 0 0 0 0 0
0 0 0 0 1 0 0 0 0
1
2
0
1
2
0 0 0 0 0 0
0 0 0 0 0 0 1 0 0
0 0 0 0 0
1
2
0
1
2
0
0 0 0 0 0 0 0 0 1
0 0 0 0 0 0 1 0 0
_

_
For each chain, we can nd the steady state distribution of each block, then any linear combination of the
distributions would be a steady state distribution of the whole chain, thus the steady state of the whole chain
is no longer unique.
c.
limP
2n
a
=
_

_
1
2
1
2
0 0
1
2
1
2
0 0
0 0
1
2
1
2
0 0
1
2
1
2
_

_
limP
2n
b
=
_

_
1
5

1
5
.
.
.
.
.
.
.
.
.
1
5

1
5
1
5
2
5
1
5
1
5
.
.
.
.
.
.
1
5
2
5
1
5
1
5
_

_
6.9
By theorem 6.5.2, as long as we can construct a set of numbers
1
...
n
such that

i
= 1 and
i
P
ij
=
j
P
ji
,
then weve established reversibility.
3
Let S
i
=

j=i
d
ij
(that is, the sum corresponding to the node i), then we have
P
ij
p
ji
=
dij
Si
dji
Sj
=
S
j
S
i
set
=

j

i
=

j
S
i
S
j
Then

i
=

1
S
i
S
1
However, we want

i
= 1, thus

i
=

1
S
1

S
j
=

1
S
1
S = 1
where S =

i,j
d
ij

1
=
S
1
S
Thus

j
=
S
j
S
Since such set of exists, the chain is reversible.
Worksheets
1.
If the graph is connected, then the chain is irreducible.
2.
a.
Notice the network is undirected, i.e if a piece can move from a to b in one move, then it can move from b
to a in one move. Therefore, as long as we can nd the possible tiles each piece can move to, then we can
nd the # of communicating classes.
Queen: The Queen can move from one place to another in at most 2 steps, thus there is only one
communicating class.
King: The King can move to all states, albeit a bit slower than the Queen. Thus one communicating
class.
Rook: Similar to the Queen. 1.
Knight: 1.
Bishop: The whtie bishop cant meet black bishop, thus there are 2 communicating classes. In fact,
this is a reducible chain.
b.
Queen: Aperiodic. consider the movements a1 b1 a1 and a1 a2 b2 a1. Common divisor
between 2 and 3 is 1, thus aperiodic.
King: Aperiodic. Save movement as the Queen would suce.
4
Rook: Aperodic. a1 a2 a1, a1 a3 a2 a1.
Knight: Periodicity 2. Its easier to draw this out. WLOG start at the center (well, as close as you can
be) of the board, map out all 8 possible moves, call these 1st tier moves, then the however many move
after that, call these second tier moves. Notice that there are no overlaps between second tier and rst
tier moves. This means that one cannot return to starting position with odd number of moves.
Bishop: Aperodic. The movements of the Bishop is similar to the Rook except diagonal.
c.
See code. The steady state distribution of a tile is proportional to the number of moves the piece can make
from that tile.
3.
a.
= P
= (P) P = P
2
=
.
.
.
= P
n
, n
b.
We know

i
=

j
P
ji
Since irreducibility implies all states communicate, i.e P
ji
> 0, i, j and
j
0, j. Then the only way
i
= 0
is if
j
= 0, j, which is a contradiction.
c.
P

(X
n
= i) =

j
P
j
(X
n
= i)
= [P
n
]
i
, i.e i
th
element of [P
n
]
=
i
d.
P

(X
n1
= i, X
n2
= i, ..., X
0
= i) = P

(X
n1
= i|X
n2
= i) P

(X
n2
= i|X
n3
= i)
...P

(X
1
= i|X
0
= i) P (X
0
= i)
= P

(X
n1
= i|X
n2
= i) P

(X
n2
= i|X
n3
= i)
... P

(X
n
= i|X
n1
= i)
. .
a
P

(X
1
= i)
. .
b
a due to Markov property
b because P

(X
1
= i) = P

(X
0
= i) = 1
i
by part c
= P

(X
m
= i, 1 m n)
5
e.
P

(X
0
= i, X
m
= i, 1 m n) = P

(X
m
= i, 1 m n 1)
P

(X
m
= i, 0 m n 1)
= P

(X
m
= i, 1 m n 1)
P

(X
m
= i, 1 m n)
= P

(X
m
= i, 1 m n 1, X
n
= i)
f.
P

(X
0
= i, T
i
n) = P

(X
0
= i, X
m
= i, 1 m n 1)
= P

(X
m
= i, 1 m n 1, X
n
= i)
= P

(T
i
= n)
g.

i
P
i
(T
i
n) = P

(X
0
= i, T
i
n) by denition
= P

(T
i
= n)
h.

i
E
i
[T
i
] =
i

n=0
P
i
(T
i
n) , by tail summation of expectation
=

n=0
P

(T
i
= n) , by g
= P

(T
i
< )
i.
E
i
[T
i
] =
P (T
i
< )

i
< since
i
> 0 by b
j.
Suppose E
j
[T
i
] = . Due to irreducibility a path from i j i, denote this path asT
i,j
+ T
j,i
, where
T
i,j
= # of steps to rst arrive at j when we start at i
Then
E
i
[T
i
] E [T
i,j
] + E [T
j,i
] =
This is a contradiction to part i.
k.
E
j
[T
i
] < P
j
(T
i
< ) = 1
l.
E
i
[T
i
] =
P (T
i
< )

i
=
1

i
6
m.
From part c of problem 2, we see that the steady state distribution is proportional to the number of possible
moves a states has, i.e for a quarter of the chess board
8 8 6 4
8 8 6 4
6 6 4 3
4 4 3 2
That is, if m(i) = # of moves a state has, then

i
=
m(i)
336
Since b1 has 3 moves, then
E
b1
[T
b1
] =
1

b1
=
336
3
= 112
7