Documente Academic
Documente Profesional
Documente Cultură
5 – Additional Issues
Concerning Discrete-Time
Markov Chains
Topics
• Review of DTMC
• Classification of states
• Economic analysis
• First-time passage
• Absorbing states
Discrete-Time Markov Chain
A stochastic process { Xn } where n N = { 0, 1, 2, . . . } is
called a discrete-time Markov chain if
Pr{ Xn+1 = j | X0 = k0, . . . , Xn-1 = kn-1, Xn = i }
0
a0
1
a1
2
a2
3
a3
4 …
2 1 2 1
(1) (1)
d1 d2 d3
0 1 2 3 4
a1 a2 a3
0 3 6
2 1 5 4
Illustration of Concepts
0
Example 1
State 0 1 2 3
0 0 X X 0
1 X 0 0 0 3 1
2 0 0 0 X
3 X 0 0 X
Example 2 0
State 0 1 2 3 4
0 X X 0 0 0
1 X X 0 0 0 4 1
2 0 0 X 0 0
3 0 0 X X 0
4 X 0 0 0 0
3 2
Example 3 0
State 0 1 2 3
0 0 X X 0
1 0 0 0 X 1
3
2 0 0 0 X
3 X 0 0 0
.6
.7
1 2
.4
3 4
.5
.4 .5 .3 .5 .8
.1
.2
A state j is accessible from state i if pij(n) > 0 for some n > 0.
0 1 2 3 4
0 1 0 0 0 0
1 1-p 0 p 0 0
2 0 1-p 0 p 0
3 0 0 1-p 0 p
4 0 0 0 0 1
0.8 0 0.2
P 0.4 0.3 0.3
1 2
0 0.9 0.1
3
Historical statistics:
1. If a customer had an accident last year then they
have a 10% chance of having one this year;
2. If they had no accident last year then they have a
3% chance of having one this year.
Problem: Find the steady-state probability and the long-
run average annual premium paid by the customer.
Solution:
(N,N) = 0.939, (N,Y) = 0.029, (Y,N) = 0.029, (Y,Y) = 0.003
Transition Matrix
Calculate Regular matrix. Rows sum to 1.
Change 4 Recurrent States
Analyze 1 Recurrent State Class
0 Transient States
State 4 0 1 2 3
Index Names (N, N) (N, Y) (Y, N) (Y, Y) Sum Status
0 (N, N) (N, N) 0.97 0.03 0 0 1 Class-1
1 (N, Y) (N, Y) 0 0 0.9 0.1 1 Class-1
2 (Y, N) (Y, N) 0.97 0.03 0 0 1 Class-1
3 (Y, Y) (Y, Y) 0 0 0.9 0.1 1 Class-1
Sum 1.94 0.06 1.8 0.2
Economic Data and Solution
Economic Data Measure: Cost
Calculate Discount Expected Transition Cost Matrix
Rate State State 0 1 2 3
0 Cost Cost (N, N) (N, Y) (Y, N) (Y, Y)
0 (N, N) 250 250 0 0 0 0
1 (N, Y) 400 400 0 0 0 0
2 (Y, N) 400 400 0 0 0 0
3 (Y, Y) 800 800 0 0 0 0
Steady State The vector shows the long run probabilities of each
Expected
state.
Analysis 0 1 2 3 Cost
(N, N) (N, Y) (Y, N) (Y, Y) per period
Steady State 0.93871 0.029032 0.029032 0.003226 260.483871
Transient Analysis for Insurance Company
Transient
Analysis Average Cost 260.1622 Discounted Cost 5203.243
0 1 2 3 Step Cum. Present
(N, N) (N, Y) (Y, N) (Y, Y) Cost Cost Worth
Start Initial 0 0 0 1 0 0
1 0 0 0.9 0.1 440 440 440
2 0.873 0.027 0.09 0.01 273.05 713.05 713.05
More 3 0.93411 0.02889 0.0333 0.0037 261.3635 974.4135 974.4135
4 0.938388 0.029022 0.029331 0.003259 260.5454 1234.959 1234.959
5 0.938687 0.029032 0.029053 0.003228 260.4882 1495.447 1495.447
Chart 6 0.938708 0.029032 0.029034 0.003226 260.4842 1755.931 1755.931
7 0.93871 0.029032 0.029032 0.003226 260.4839 2016.415 2016.415
8 0.93871 0.029032 0.029032 0.003226 260.4839 2276.899 2276.899
9 0.93871 0.029032 0.029032 0.003226 260.4839 2537.383 2537.383
10 0.93871 0.029032 0.029032 0.003226 260.4839 2797.867 2797.867
First Passage Times
Let ij = expected number of steps to transition
from state i to state j
where 0 1 2 3 4
0 1 0 0 0 0
1 1-p 0 p 0 0
P = 2 0 1-p 0 p 0
3 0 0 1-p 0 p
4 0 0 0 0 1
Solution to Gambler’s Ruin Example
Now we have three equations with three unknowns.
Using p = 0.75 (probability of winning a single bet)
we have
q20 = 0 + 0.25 q10 + 0.75 q30
q10 = 0.25 + 0.75 q20