CH 05 - Markov Model

Ch 04: Markov Processes
ELEXM 621 Random Process and Queuing

Theorem
Assoc. Prof. Dr. Rashid A. Saeed
MSC Comp and Comm
Elex, FoE, SUST,
Agenda
Memoryless Information Processes
Markov Processes and n-gram Models
Stationary Distributions
The Entropy of Markov Processes
Sequences of Symbols
The Adjoint Source of a Markov Source
Extensions of Processes
Infinite Sample Spaces
2
ELEXM 621 Information Theory and Coding Dr. Rashid A. Saeed
Description
Sometimes we are interested in how a random
variable changes over time.
The study of how a random variable evolves
over time includes stochastic processes.
An explanation of stochastic processes in
particular, a type of stochastic process known
as a Markov chain is included.
We begin by defining the concept of a
stochastic process.
A continuous time Markov Chain (CTMC)
A Discrete time Markov Chain (DTMC)
What is a Markov Chain?
One special type of discrete-time is called a Markov
Chain.
Definition: A discrete-time stochastic process is a
Markov chain if, for t = 0,1,2 and all states
P(Xt+1 = it+1|Xt = it, Xt-1=it-1,,X1=i1, X0=i0)
=P(Xt+1=it+1|Xt = it)
Essentially this says that the probability distribution of
the state at time t+1 depends on the state at time t(it)
and does not depend on the states the chain passed
through on the way to it at time t.
In our study of Markov chains, we make further
assumption that for all states i and j and all t,
P(Xt+1 = j|Xt = i) is independent of t.
This assumption allows us to write P(Xt+1 = j|Xt
= i) = pij where pij is the probability that given
the system is in state i at time t, it will be in a
state j at time t+1.
If the system moves from state i during one
period to state j during the next period, we call
that a transition from i to j has occurred.
We call the vector q= [q1, q2,qs] the initial probability
distribution for the Markov chain.
In most applications, the transition probabilities are displayed
as an s x s transition probability matrix P. The transition
probability matrix P may be written as
p11 p12 p1s

p p22 p2 s
P 21

ps1 ps 2 pss
For each I
j s
p
j 1
ij 1
We also know that each entry in the P matrix must be

nonnegative.
Hence, all entries in the transition probability matrix are
nonnegative, and the entries in each row must sum to 1.
Markov processes
ashid A. Saeed
ELEXM 621 Information Theory and
8
Coding
Continuous-time Markov chains (homogeneous
case)
ashid A. Saeed
9
Coding
The Gamblers Ruin Problem
At time 0, I have $2. At times 1, 2, , I play a game
in which I bet $1, with probabilities p, I win the
game, and with probability 1 p, I lose the game.
My goal is to increase my capital to $4, and as soon
as I do, the game is over. The game is also over if my
capital is reduced to 0.
Let Xt represent my capital position after the time
t game (if any) is played
X0, X1, X2, . May be viewed as a discrete-time
stochastic process
The Gamblers Ruin Problem
$0 $1 $2 $3 $4
1 0 0 0 0
1 p 0 p 0 0

P = 0 1 p 0 p 0
0 0 1 p 0 p

0 0 0 0 1
Stationary solution (steady state)
ashid A. Saeed
12
Coding
The Cola Example
Suppose the entire cola industry produces only two colas.
Given that a person last purchased cola 1, there is a 90%
chance that their next purchase will be cola 1.
Given that a person last purchased cola 2, there is an 80%
chance that their next purchase will be cola 2.
1. If a person is currently a cola 2 purchaser, what is the
probability that they will purchase cola 1 two purchases from
now?
2. If a person is currently a cola 1 a purchaser, what is the
probability that they will purchase cola 1 three purchases
from now?
20%
2 1 90%
80%
The Cola Example
90%
We view each persons purchases as a Markov chain with

the state at any given time being the type of cola the
person last purchased.
Hence, each persons cola purchases may be represented
by a two-state Markov chain, where
State 1 = person has last purchased cola 1
State 2 = person has last purchased cola 2
If we define Xn to be the type of cola purchased by a
person on her nth future cola purchase, then X0, X1,
may be described as the Markov chain with the following
transition matrix: X2 X1
X0
90%
The Cola Example X2 X1
X0
90%
Cola1 Cola 2
Cola 1 .90 .10
P
Cola 2 .20 .80

We can now answer questions 1 and 2.

1. We seek P(X2 = 1|X0 = 2) = P21(2) = element 21 of P2:
.90 .10 .90 .10 .83 .17

P
2

.20 .80 .20 .80 .34 .66
The Cola Example
Hence, P21(2) =.34. This means that the probability is .34
that two purchases in the future a cola 2 drinker will
purchase cola 1.
2. We seek P11(3) = element 11 of P3:
.90 .10 .83 .17 .781 .219

P P( P )
3 2

.20 .80 .34 .66 .438 .562
Therefore, P11(3) = .781
Many times we do not know the state of the Markov chain at time 0. Then
we can determine the probability that the system is in state i at time n by
using the reasoning.
Probability of being in state j at time n

is
qi Pij (n)
i 1
where q=[q1, q2, q3].

Hence, qn = qopn = qn-1p
Example, q0 = (.4,.6)
q1= (.4,.6) .90 .10
.20 .80

q1 = (.48,.52)
To illustrate the behavior of the n-step transition
probabilities for large values of n, we have
computed several of the n-step transition
probabilities for the Cola example.
This means that for large n, no matter what the
initial state, there is a .67 chance that a person
will be a cola 1 purchaser.
5.4 Classification of States in a
Markov Chain
To understand the n-step transition in more detail, we need
to study how mathematicians classify the states of a Markov
chain.
The following transition matrix illustrates most of the
following definitions. A graphical representation is shown in
the book (State-Transition diagram)
.4 .6 0 0 0
.5 .5 0 0 0

P 0 0 .3 .7 0

0 0 .5 .4 .1
0 0 0 .8 .2
Information
Processes
Memory Memoryless
Markov Processes
Human
Model
algorithm
Ergodic Markov Stationary

Adjoint Source n-gram Model
Processes Distributions
Probabilities of Memory to Infinite Sample

Entropy
Sequences of Symbols Memoryless Spaces
Extensions of
Processes
20
Dr. Rashid A. Saeed
Memoryless Information Processes
For a memoryless information
source, there are no correlations
between the outputs of the source
at different times.
If all the probability distributions
are the same, the source is said to be
stationary.
21
22
Memoryless information Processes are relatively simple.
More realistic information Processes have memory,
which is the property that the emission of a symbol at any instant
depends on one or more of the symbols that were generated before it.
In this section, we defined it in terms of finite state Markov
chains.
We use the state of a Markov chain to represent the
memory of the source, and
use the transitions between states to represent the next
symbol out of the source.
23
Markov Processes
A Markov information source, is an information
source whose underlying dynamics are given by a
stationary finite Markov chain.
Markov Processes are commonly used in
communication theory, as a model of a
transmitter.
Markov Processes also occur in natural language
processing, where they are used to represent
hidden meaning in a text.
24
Markov Processes
A Markov source consists of:

an alphabet A
a set of states ,
a set of transitions between states,
a set of labels for the transitions and
two sets of probabilities.
25
Diagrammatic representation of a
Markov source
26
27
Markov chain in Network queuing
Markov Chain for M/M/1 system
28
Birth-death chain
The Poisson process

Arrival rate of packets per second
Over a small interval ,
29
Markov model of a scalar passage of
music.
30
n-gram model,
mth-order Markov model

(m+1)-gram model
31
32
Find mth-order Markov model?
33 ELEXM 621 Information Theory and Coding Dr. Rashid A. Saeed

34
A Markov source equivalent to a
3-gram model.
35
ELEXM 621 Information Theory and Coding
n-gram Applications
design kernels that allow machine learning
algorithms
find likely candidates for the correct spelling
of a misspelled word
improve compression in compression
algorithms
assess the probability of a given word
sequence appearing in text of a language of
interest in pattern recognition systems
36
37
38
Stationary Distributions
39
The random process will not change its statistical properties with time
40
Find the stationary distribution matrix
41
42
43
44
Once the source enters either the first state or
the fourth state, it cannot leave it.
A source with two stationary distributions.

45
Information
Processes
Memory Memoryless
Markov
Processes Human
Model algorithm


Entropy
Extensions of
Processes
46
The Adjoint Source of a Markov
Source
47
memory memoryless
48
Dr. Rashid A. Saeed
49
Proof
50
Information
Processes
Memory Memoryless
Markov
Processes Human
Model algorithm


Entropy
Extensions of
Processes
51
Extensions of Processes
52
53
54
55
Extensions of Markov Processes
56
Extensions of Markov
M=1,Processes
n=2, p=1
57
Design the 2nd extension?
Extensions of Markov
M=2,Processes
n=2, p=1
58
59
m=2, n=2, p=1
60
m=4, n=2, p=2
61
m=4, n=2, p=2
62
Infinite Sample Spaces
63
Continuous Sample Space
Differential entropy or
continuous entropy
64
One must take care in trying to apply
properties of discrete entropy to differential
entropy,
Since probability density functions can be
greater than 1.
For example, Uniform(0,0.5) has negative
differential entropy
65
66
67
Exercises
68
Thank You
69 ELEXM 621 Information Theory and Coding

Dr. Rashid A. Saeed

CH 05 - Markov Model

Încărcat de

Informații document

Descriere originală:

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

CH 05 - Markov Model

Încărcat de

Drepturi de autor:

Formate disponibile

Ch 04: Markov Processes

ELEXM 621 Random Process and Queuing

p11 p12 p1s

We also know that each entry in the P matrix must be

We view each persons purchases as a Markov chain with

We can now answer questions 1 and 2.

.90 .10 .90 .10 .83 .17

.90 .10 .83 .17 .781 .219

Probability of being in state j at time n

where q=[q1, q2, q3].

Ergodic Markov Stationary

Probabilities of Memory to Infinite Sample

A Markov source consists of:

The Poisson process

mth-order Markov model

33 ELEXM 621 Information Theory and Coding Dr. Rashid A. Saeed

A source with two stationary distributions.

Ergodic Markov Stationary

Probabilities of Memory to Infinite Sample

Ergodic Markov Stationary

Probabilities of Memory to Infinite Sample

69 ELEXM 621 Information Theory and Coding

S-ar putea să vă placă și