Markov chains

By properties of joint and conditional pdfs: p(xn ) = p(xn , xn1 ) dxn1 = p(xn |xn1 )p(xn1 ) dxn1 = Ln [p(xn1 )]

Markov chains: An example

Transition probabilities between preferred regimes f rom PNA BNAO RNA PNA to BNAO RNA 0.30 0.49 0.36 0.34 0.26 0.30 0.37 0.25 0.34

Evolution of probabilities governed by operator L (which will be independent of time for a stationary process) If space of xn is nite integers, L is a matrix L and pn = Lpn1 = L p0 [where pn = (p(xn = 1), p(xn = 2), ..., p(xn = K))] For processes which conserve probability (e.g. i Lij = 1) the maximum eigenvalue of L is 1 with eigenvector ps (the stationary pdf). As n , pn approaches ps
An Introduction to Probability and Stochastic Processes for Ocean, Atmosphere, and Climate Dynamics2: Stochastic Processes p. 13/64

(from Kondrashov et al., Clim Dyn., 2007.

An Introduction to Probability and Stochastic Processes for Ocean, Atmosphere, and Climate Dynamics2: Stochastic Processes p. 14/64

Chapman-Kolmogorov equation
Again, by properties of joint and conditional pdfs: p(xn |xn2 ) = p(xn |xn1 )p(xn1 |xn2 ) dxn1

Chapman-Kolmogorov equation
For such a process the Chapman-Kolmogorov equation can be transformed into the PDE (with p = p(x(t)|x(t ))) 1 2 t p = x [a(x, t)p] + xx [b2 (x, t)p] 2 1 1 = x a bx b p + x [bx (bp)] 2 2 for the pdf at time t conditioned on the state of the system at time t This equation, known as the Fokker-Planck Equation (FPE), describes probability diffusing conservatively through state space from the original distribution If a, b independent of time pdf will approach stationary pdf ps : 1 2 0 = x [a(x)ps (x)] + xx [b2 (x)ps (x)] 2
An Introduction to Probability and Stochastic Processes for Ocean, Atmosphere, and Climate Dynamics2: Stochastic Processes p. 16/64

an equation for the evolution of the transfer matrix p(xi |xj ); as a nonlinear integral equation its a bit tricky to solve Now focus on smooth processes with continuous paths and with nite mean a(x, t) (the drift) and variance b(x, t) (the diffusion) of the local tendency such that for (for all > 0) lim x(t + ) x(t) [x(t + ) x(t)]2 p(x(t + )|x(t)) dx p(x(t + )|x(t)) dx = = a(x, t) + O( ) b2 (x, t) + O( )

|x(t+)x(t)|<

lim

|x(t+)x(t)|<

An Introduction to Probability and Stochastic Processes for Ocean, Atmosphere, and Climate Dynamics2: Stochastic Processes p. 15/64

The Wiener process W (t)

Consider the case a(W, t) = 0 (no mean tendency) and b(W, t) = 1: 1 t p = ww p 2 This is a classical diffusion equation; with initial condition p(W (t0 )) = (W W0 ) the solution is (W W0 )2 exp p(W (t)) = 2(t t0 ) 2(t t0 ) Gaussian with mean(W ) = 0 and std(W ) = t t0 (sound familiar?) 1 The Wiener process (or Brownian motion) is continuous generalisation of random walk; as std(W ) is time-dependent, this is a non-stationary process
An Introduction to Probability and Stochastic Processes for Ocean, Atmosphere, and Climate Dynamics2: Stochastic Processes p. 17/64

The Wiener process W (t): curious features

1. Paths W (t) are continuous everywhere but differentiable nowhere W (t + ) W (t) as 0 = std 2. Increments tt W = W (t) W (t ) are Gaussian, stationary p(tt W ) = and independent p(t3 t2 W, t2 t1 W ) = exp
(t3 t2 W )2 2(t3 t2 )

1 2(t t )2

exp

(tt W )2 2(t t )
(t2 t1 W )2 2(t2 t1 )

exp

2(t3 t2 )

2(t2 t1 )

in particular, cov(t3 t2 W, t2 t1 W ) = 0
An Introduction to Probability and Stochastic Processes for Ocean, Atmosphere, and Climate Dynamics2: Stochastic Processes p. 18/64

White noise
W (t) not strictly differentiable, but can formally dene white noise W (t + ) W (t) W (t) = lim 0 such that std(W (t)) = and cov(W (t), W (t )) =
, 0

An aside: units of white noise

What are the units of white noise? We have: E W (t)W (t ) = (t t ) so [W ] []1/2 W (t + ) W (t ) What are the units of a delta function? We have (t t ) dt = 1 (dimensionless) so [] [t]1

lim E

W (t + ) W (t)

= (t t ) (because if t = t sufciently small increments dont overlap) White noise innitely fast, innitely strong with E W (t)W (t ) = (t t ) (note: mathematicians usually write dW (t ) = W (t )dt)
An Introduction to Probability and Stochastic Processes for Ocean, Atmosphere, and Climate Dynamics2: Stochastic Processes p. 19/64

We conclude: W (t) has the dimensions of one over the square root of time; has important implications for estimating the coefcient of white noise in stochastic differential equations

An Introduction to Probability and Stochastic Processes for Ocean, Atmosphere, and Climate Dynamics2: Stochastic Processes p. 20/64

Ornstein-Uhlenbeck process (red noise)

Consider FPE with a(x, t) = 1/ and b2 (x, t) = 2 (both constant) and initial condition p(x(t0 )) = (x x0 ); known as an Ornstein-Uhlenbeck (O-U) process, or red noise Can show that solution is Gaussian with mean(x(t)) = x0 et/ var(x(t)) = 2 2 1 e2t/

Red and white noise

With = / , cov(x(t), x(t )) = 2 1 exp(|t t |/ ) 2 2 (tt ) as 0

In this limit, red noise becomes white noise scaled by factor Power spectra are Fourier transforms of autocovariances: 1 2
S()/( )
2 2

As initial transients die out (t ) p(x) approaches stationary pdf with mean zero and stationary autocovariance cov(x(t), x(t )) = 2 exp(|t t |/ ) 2

SW () = E{W ()2 } = Sx () = E{x()2 } =

0.2

2 2 2(1 + 2 2 )

0.15 red noise white noise 0.1

0.05

is the memory of the system

An Introduction to Probability and Stochastic Processes for Ocean, Atmosphere, and Climate Dynamics2: Stochastic Processes p. 21/64

0 5 10

10

10

An Introduction to Probability and Stochastic Processes for Ocean, Atmosphere, and Climate Dynamics2: Stochastic Processes p. 22/64

SDEs: A rst example

If we can (formally) differentiate W (t) to get W (t), can (formally) integrate W (t) to get W (t) back Consider forced linear differential equation with x(0) = x0 : d 1 x= x+ W dt This stochastic differential equation (SDE) has solution
t

SDEs: A rst example

8 6 4 2 0 2 4 0 x(t) W(t)

x(t) = et/ x0 +
0

e(tt )/ W (t ) dt

x(t) is a Gaussian stochastic process with same mean and autocovariance as O-U process: in fact, this is a pathwise description of red noise as a weighted integral of white noise

20

40

60

80

100

dx = x + W dt
An Introduction to Probability and Stochastic Processes for Ocean, Atmosphere, and Climate Dynamics2: Stochastic Processes p. 24/64

An Introduction to Probability and Stochastic Processes for Ocean, Atmosphere, and Climate Dynamics2: Stochastic Processes p. 23/64