MTH6141 Random Structures, Spring 2012 Solutions to Exercise Sheet 3

1. (a) The chain is certainly irreducible. Since there is a path of 3 steps between any pair of states it is also regular. Since it is regular there is a unique equilibrium distribution which satises 1/2 1/2 0 0 0 1/3 2/3 0 = (w1 , w2 , w3 , w4 ) (w1 , w2 , w3 , w4 ) 0 0 1/4 3/4 4/5 0 0 1/5 subject to w1 + w2 + w3 + w4 = 1. Solving the system of linear equations, (w1 , w2 , w3 , w4 ) = (24/73, 18/73, 16/73, 15/73) Since the chain is regular we have that (by a result in lectures) this distribution is also a limiting distribution. (b) The chain is not irreducible (it is impossible to get from state 3 to state 2 for instance). It cannot be regular either (since regularity is a stronger condition). Any probability vector w = (0, , 1 ) satises wP = w so there are many equilibrium distributions. There is no limiting distribution. Indeed (0, 1, 0)P t = (0, 1, 0) and (0, 0, 1)P t = (0, 0, 1) so we cannot have P t tending to a limit with constant rows. (However P t does tend to a limit, can you see what it is?) (c) The chain is not irreducible (it is impossible to get from state 3 to state 2 for instance). It cannot be regular either. The only probability vector which is a solution to wP = w is w = (0, 0, 1) so this is the unique equilibrium distribution. If the chain is run for a long time then it is absorbed in state 3 with probability 1. It follows that there is a limiting distribution and it is also (0, 0, 1). (d) The chain is irreducible. However, it is not regular. (If t is even then (t) (t) p12 = 0; if t is odd then p11 = 0.) Solving the usual equations we see that there is a unique equilibrium distribution w = (1/2, 1/10, 2/5). There is no limiting distribution. We cannot have P t tending to a limit since 1, if k is even; (k) p11 = 0, if k is odd. 1

(e) The chain is clearly irreducible and regular (the matrix P = P 1 has all entries positive). Since it is regular there is a unique equilibrium distribution. By inspection the vector w = (1/4, 1/4, 1/4, 1/4) satises wP = w and so this is the unique equilibrium distribution and also (by a result from lectures) the limiting distribution. Notice that when the chain is not regular there may or may not be a unique equilibirum distribution and there may or may not be a limiting distribution. Also it is possible for there to be no limiting distribution but for P t to tend to a limit. (For a limiting distribution we require that all rows of the limit are equal.) 2. (a) Since the chain is irreducible we know that for any a, b S there is a (k) k with pab > 0. Let m be the maximum of all of these k s (we can do this since there are only nitely many pairs of states). It follows from the denition of m that given any pair of states a, b S (t) (s) we have that pai > 0 for some s m and pib > 0 for some t m. (2m) (s) (2mst) (t) pib > 0. Hence there is Now 2m s t 0 and pab pai pii a positive probability that we will move between any pair of states in 2m steps (or equivalently all the entries of P 2m are positive) and so the chain is regular. (b) The result is no longer true for innite S . Consider the Markov chain with S = Z and pi,i = pi,i+1 = pi,i1 = 1/3 and all for all i Z and all other transition probabilities 0. This is irreducible but given any k we (k ) have that pi,i+k+1 = 0 and so it is not regular. (The part of the proof above which fails here is that we cannot necessarily take the maximum of an innite set of numbers). 3. (a) If p = 0 then the MC is not irreducible. pi3 = 0 for all i, and hence (t1) (1) (t) pi3 = 0 for all t N. (There is no path of any length p13 = 4 i=1 p1i from state 1 to state 3 in the transition graph.)
1 then the MC is irreducible. p21 p24 p41 > 0, p31 If 0 < p 2 p34 p41 > 0 and p41 > 0. (So we can get to state 1 from anywhere.) (2) Also p12 > 0, p13 > 0 and p14 p12 p24 . (So we can get from state 1 to (t) anywhere.) Summarising, for all i, j , pij > 0 for some t 4. (Another way of saying this is that the transition graph is strongly connected. Actually, t 3 is enough.) (2) (2)

(b) If p = 0 the MC is not irreducible and hence not regular. If p = 1 2 then the MC is not regular. Consider the initial distribution (0) = (1, 0, 0, 0). The (3) = (0) P 3 = (1, 0, 0, 0) = (0) . Iterating, (3s) = 2

(1, 0, 0, 0) for all s N. However, , 1 , 0). (3s+1) = (1, 0, 0, 0)P 3s P = (1, 0, 0, 0)P = (0, 1 2 2 Thus (t) does not converge as t . If 0 < p < 1 then the MC is regular. Looking at the pattern of zero 2 entries in powers of P (+ indicates a non-zero entry): 0 + + + + 0 0 + 0 0 0 + + 0 0 0 2 P = 0 0 0 + P = + 0 0 0 + 0 0 0 0 + + + + + + + + + + + + 0 0 + P 5 = + + + + . P4 = + 0 0 + + + + + + + + + + + + + In other words, there is a path of length exactly 5 from every vertex of the transition graph to every vertex. (c) Since the chain is regular it has a limiting distribution w. Moreover w is the unique solution to wP = w. We therefore solve 1 0 2 p 1 p 2 0 0 0 1 = (w1 , w2 , w3 , w4 ) (w1 , w2 , w3 , w4 ) = 0 0 0 1 1 0 0 0 subject to w1 + w2 + w3 + w4 = 1. We get (w1 , w2 , w3 , w4 ) = 1 2p 2 2 , , , . 2p + 5 2p + 5 2p + 5 2p + 5

1 1 (d) This is the case p = 1 . A byproduct of (b) is that w(3s+1) = (0, 2 , 2 , 0) 2 (0) assuming w = (1, 0, 0, 0). Thus Pr(X3s+1 = 1 | X0 = 1) = 0. Now set s = 333 to get Pr(X1000 = 1 | X0 = 1) = 0.

4. This is similar to Question 1(e). If P is an n n matrix then p11 p1n n n 1 1 1 . 1 1 1 . . , n, , n ) . p , p i2 , . . . , (n = i1 . . n i=1 n i=1 n pn1 pnn 1 = n = 3
n

pin
i=1 n

i=1 1 1 (n, n,

1 p1i , n
1 ,n ),

i=1

1 p2i , . . . , n

pni
i=1

where the second equality is by symmetry, and the third follows from the 1 1 ,..., n ) does fact that P is stochastic. Thus the uniform distribution w = ( n satisfy wP = w, and is an equilibrium distribution. 5. Since P is regular, there exists k N such that all entries of Q = P k are strictly positive. According to Lemma 1.7, Q has a limiting distribution, say w. That is to say, Qn W , as n , where W is the matrix with all rows equal to w. Now let r satisfy 0 r < k . Then P nk+r = P r Qn P r W = W , as n . Since any t can be written in the form t = nk + r with 0 r < k , it follows that P t W as t . If you took Convergence and Continuity, you could try making the above argument completely rigorous. What should it mean for the sequence of matrices (P t : t N) to converge to the matrix W ?