LECTURE 17 Markov Processes II Readings: Section 7.

Review Discrete state, discrete time, time-homogeneous Transition probabilities pij Markov property

Lecture outline Review Steady-State behavior Steady-state convergence theorem Balance equations Birth-death processes

rij (n) = P(Xn = j | X0 = i) Key recursion: rij (n) = rik (n 1)pkj

k

Warmup
9 3 4 6 7

Periodic states The states in a recurrent class are periodic if they can be grouped into d > 1 groups so that all transitions from one group lead to the next group

P(X1 = 2, X2 = 6, X3 = 7 | X0 = 1) = P(X4 = 7 | X0 = 2) =
Recurrent and transient states State i is recurrent if: starting from i, and from wherever you can go, there is a way of returning to i If not recurrent, called transient Recurrent class: collection of recurrent states that communicate to each other and to no other state
3 9 5 4 6 8

Expected Frequency of a Particular Transition Consider n transitions of a Markov chain with a single class which is aperiodic, starting from a given initial state. Let qjk (n) be the expected number of such transitions that take the state from j to k . Then, regardless of the Visit frequency interpretation initial state, we have qjk (n) = j pjk . lim n n
j = k pkj
k

Steady-State Probabilities Do the rij (n) converge to some j ? (independent of the initial state i) Yes, if: recurrent states are all in a single class, and single recurrent class is not periodic Assuming yes, start from key recursion rij (n) =
k

Given the frequency interpretation of j and k pkj , the balance equation

(Long run) frequency of being in j : j
m

j =

k pkj

k=1

rik (n 1)pkj

Frequency of transitions k j : the k pexpected kj has an intuitive meaning. It expresses the fact that frequency of visits to j is equal to the sum of the expected frequencies k pkj of transition that lead to j ; see Fig. 7.13. Frequency of transitions into j : p
k kj k

take the limit as n j =

k

1
for all j

1 p1j

j pjj

k pkj ,

2
. . . . . .

Additional equation: j = 1
j

2p2j

m pmj

Figure 7.13: Interpretation of the balance equations in terms of frequencies. In a very large number of transitions, we expect a fraction k pkj that bring the state from k to j . (This also applies to transitions from j to itself, which occur with frequency j pjj .) The sum of the expected frequencies of such transitions is the expected frequency j of being at state j .

Example

0.5 0.5 1 0.2 2

0.8

Birth-death processes In fact, some stronger statements are also true, such as the following. Wheneve 1experiment - p1- q1 1 - q m of the Markov chai 1- p0 we carry out a probabilistic and generate a trajectory over an innite time horizon, the observed long-term frequency with which state j p p0 visited will be exactly equal to j , 1 and the observed long-term frequency of transition 2 3 m 1 0 from j to k will be exactly equal to j p jk . Even though the trajectory is random, thes q q 1 certainty, q2 m equalities hold with essential that is, with probability 1.

.. .

pi i i+1 q i+1

ipi = i+1qi+1

Special case: pi = p and qi = q for all i = p/q =load factor p i+1 = i = i q i = 0 i , i = 0, 1, . . . , m

Assume p < q and m 0 = 1 1

E[Xn] =

(in steady-state)

MIT OpenCourseWare http://ocw.mit.edu

6.041 / 6.431 Probabilistic Systems Analysis and Applied Probability

Fall 2010

For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.