agenda: countable markov chainscandes/teaching/stats318/...k=1 n(t+1) k fx tgis a markov chain...

Stats 318: Lecture # 17

Agenda: Countable Markov Chains

I Branching process

I Recurrence and transience

I Classification of states

I Limit theorems

Examples of countable state space

Symmetric random walk

Xt+1 = Xt + Zt+1 Zt iid P(Zt = ±1) = 1/2

Branching processes

Originally considered by Galton & Watson in the 1870’s while seeking for an explanation forthe phenomenon of the disappearance of family names even in a growing population

Xt+1 = Xt + Zt+1 Zt iid P(Zt = ±1) = 1/2

Branching processes

Xt+1 = Xt + Zt+1 Zt iid P(Zt = ±1) = 1/2

Branching processes

Xt+1 = Xt + Zt+1 Zt iid P(Zt = ±1) = 1/2

Branching processes

Problem: Each male in a given family has a prob. pk of having k sons

What is the probability that after n generations, an individual has no male descendants?

I At time t = 0, an individual dies and is replaced at time t = 1 by a random number Nof offsprings: P(N = k) = pk

I Offsprings also die and are replaced at time t = 2, each independently, by a randomnumber of further offsprings with the same distribution, and so on

I Xt : size of population at time t

Xt+1 =

Xt∑k=1

N(t+1)k

{Xt} is a Markov chain

Branching processes

Xt+1 =

Xt∑k=1

N(t+1)k

Branching processes

Xt+1 =

Xt∑k=1

N(t+1)k

Branching processes

Xt+1 =

Xt∑k=1

N(t+1)k

Branching processes

Xt+1 =

Xt∑k=1

N(t+1)k

Question: qt = P(Xt = 0) ?qn : prob. pop dies after n generations

First step analysis: Suppose X1 = k −→ k subpopulationsProb. any of them dies out is qt−1

P(Xt = 0) =∑k≥0

P(Xt = 0|X1 = k)P(X1 = k) =∑k≥1

pkqkt−1

∴ qt = ϕ(qt−1) ϕ(s) =∑k≥0

pksk (MGF of N)

= E(sN )

Remark: ϕ(0) = p0 = P(N = 0) ϕ′(1) = E Nϕ(1) = 1 ϕ′′ ≥ 0 =⇒ ϕ is convex

First step analysis: Suppose X1 = k −→ k subpopulations

Prob. any of them dies out is qt−1

P(Xt = 0) =∑k≥0

P(Xt = 0|X1 = k)P(X1 = k) =∑k≥1

pkqkt−1

∴ qt = ϕ(qt−1) ϕ(s) =∑k≥0

pksk (MGF of N)

= E(sN )

P(Xt = 0) =∑k≥0

P(Xt = 0|X1 = k)P(X1 = k) =∑k≥1

pkqkt−1

∴ qt = ϕ(qt−1) ϕ(s) =∑k≥0

pksk (MGF of N)

= E(sN )

P(Xt = 0) =∑k≥0

P(Xt = 0|X1 = k)P(X1 = k) =∑k≥1

pkqkt−1

∴ qt = ϕ(qt−1) ϕ(s) =∑k≥0

pksk (MGF of N)

= E(sN )

P(Xt = 0) =∑k≥0

P(Xt = 0|X1 = k)P(X1 = k) =∑k≥1

pkqkt−1

∴ qt = ϕ(qt−1) ϕ(s) =∑k≥0

pksk (MGF of N)

= E(sN )

P(Xt = 0) =∑k≥0

P(Xt = 0|X1 = k)P(X1 = k) =∑k≥1

pkqkt−1

∴ qt = ϕ(qt−1) ϕ(s) =∑k≥0

pksk (MGF of N)

= E(sN )

Two possibilities: q0 = 0

ϕ′(1) ≤ 1 ϕ′(1) ≥ 1

qt −→t→∞

1 qt −→t→∞

π0

π0 only fixed point in [0, 1] of ϕ(s) = s

ϕ′(1) ≤ 1 ϕ′(1) ≥ 1

qt −→t→∞

1 qt −→t→∞

π0

ϕ′(1) ≤ 1 ϕ′(1) ≥ 1

qt −→t→∞

1 qt −→t→∞

π0

ϕ′(1) ≤ 1 ϕ′(1) ≥ 1

qt −→t→∞

1 qt −→t→∞

π0

ϕ′(1) ≤ 1 ϕ′(1) ≥ 1

qt −→t→∞

1 qt −→t→∞

π0

More information

ϕXt(s) = E[sXt]= E

[ϕ(s)Xt−1

]= ϕXt−1(ϕ(s))

∴ ϕXt(s) = ϕ ◦ . . . ◦︸︷︷︸t times

ϕ(s)

More information

ϕXt(s) = E[sXt]= E

[ϕ(s)Xt−1

]= ϕXt−1(ϕ(s))

∴ ϕXt(s) = ϕ ◦ . . . ◦︸︷︷︸t times

ϕ(s)

More information

ϕXt(s) = E[sXt]= E

[ϕ(s)Xt−1

]= ϕXt−1(ϕ(s))

∴ ϕXt(s) = ϕ ◦ . . . ◦︸︷︷︸t times

ϕ(s)

Countable Markov chains

Two distinct situations

(1) The chain has an equilibrium distribution, them it essentially behaves like a finite chain

Ergodic theorem holds

Convergence

(2) The chain does not have an equilibrium distribution

(a) It is transient : each state is visited a finite number of times

(b) It is recurrent : infinitely many times but E τx =∞ (average returntime is infinite)

Convergence

Recurrence and transienceReturn times : X0 = x

τx = inf{t ≥ 1 : Xt = x} τ (k+1)x = inf{t > τ (k)x : Xt = x}

Definition

State x is recurrent if q(x) = P(τx

Definition

Why?

(τ(1)x , τ

(2)x , . . . , τ

(k)x

)distributed as

(W

(1)x ,W

(1)x +W

(2)x , . . . ,W

(1)x + . . .+W

(k)x

)W

(k)x (waiting times) iid τx

I If P(τx

Why?(τ(1)x , τ

(2)x , . . . , τ

(k)x

)distributed as

(W

(1)x ,W

(1)x +W

(2)x , . . . ,W

(1)x + . . .+W

(k)x

)

W(k)x (waiting times) iid τx

I If P(τx

Why?(τ(1)x , τ

(2)x , . . . , τ

(k)x

)distributed as

(W

(1)x ,W

(1)x +W

(2)x , . . . ,W

(1)x + . . .+W

(k)x

)W

I If P(τx

Why?(τ(1)x , τ

(2)x , . . . , τ

(k)x

)distributed as

(W

(1)x ,W

(1)x +W

(2)x , . . . ,W

(1)x + . . .+W

(k)x

)W

I If P(τx

Why?(τ(1)x , τ

(2)x , . . . , τ

(k)x

)distributed as

(W

(1)x ,W

(1)x +W

(2)x , . . . ,W

(1)x + . . .+W

(k)x

)W

I If P(τx

Analytical recurrence criterion

x recurrent ⇔ E (Nx|X0 = x) =∞

Nx =∑t≥1

It It =

{1 Xt = x

0 else

E (Nx|X0 = x) =∑t≥0

E(It|X0 = x) =∑t≥1

P(Xt = x|X0 = x) =∑t≥1

P t(x, x)

Proposition

State x is recurrent iff∑t≥1

P t(x, x) =∞

Proposition

{Xt} irreducible chain. Only two cases are possible

I All states are recurrentI All states are transient

Note : If X is finite, then all states are necessarily recurrent

Nx =∑t≥1

It It =

{1 Xt = x

0 else

E (Nx|X0 = x) =∑t≥0

E(It|X0 = x) =∑t≥1

P(Xt = x|X0 = x) =∑t≥1

P t(x, x)

Proposition

P t(x, x) =∞

Proposition

Nx =∑t≥1

It It =

{1 Xt = x

0 else

E (Nx|X0 = x) =∑t≥0

E(It|X0 = x) =∑t≥1

P(Xt = x|X0 = x) =∑t≥1

P t(x, x)

Proposition

P t(x, x) =∞

Proposition

Nx =∑t≥1

It It =

{1 Xt = x

0 else

E (Nx|X0 = x) =∑t≥0

E(It|X0 = x) =∑t≥1

P(Xt = x|X0 = x) =∑t≥1

P t(x, x)

Proposition

P t(x, x) =∞

Proposition

Nx =∑t≥1

It It =

{1 Xt = x

0 else

E (Nx|X0 = x) =∑t≥0

E(It|X0 = x) =∑t≥1

P(Xt = x|X0 = x) =∑t≥1

P t(x, x)

Proposition

P t(x, x) =∞

Proposition

Nx =∑t≥1

It It =

{1 Xt = x

0 else

E (Nx|X0 = x) =∑t≥0

E(It|X0 = x) =∑t≥1

P(Xt = x|X0 = x) =∑t≥1

P t(x, x)

Proposition

P t(x, x) =∞

Proposition

{Xt} irreducible chain. Only two cases are possibleI All states are recurrent

I All states are transient

Nx =∑t≥1

It It =

{1 Xt = x

0 else

E (Nx|X0 = x) =∑t≥0

E(It|X0 = x) =∑t≥1

P(Xt = x|X0 = x) =∑t≥1

P t(x, x)

Proposition

P t(x, x) =∞

Proposition

{Xt} irreducible chain. Only two cases are possibleI All states are recurrentI All states are transient

Nx =∑t≥1

It It =

{1 Xt = x

0 else

E (Nx|X0 = x) =∑t≥0

E(It|X0 = x) =∑t≥1

P(Xt = x|X0 = x) =∑t≥1

P t(x, x)

Proposition

P t(x, x) =∞

Proposition

{Xt} irreducible chain. Only two cases are possibleI All states are recurrentI All states are transient

Proof

Suppose one state x0 is recurrent, then we show that all states are recurrent

Irreducibility : ∃ t1 & t2 s.t. P t1(x, x0) > 0 & P t2(x0, x) > 0

P t(x, x) ≥ P(Xt = x,Xt−t2 = x0, Xt1 = x0|X0 = x) ≥ P t1(x, x0)P t−(t1+t2)(x0, x0)P t2(x0, x)

∴∑t≥1

P t(x, x) ≥∑

t≥t1+t2

P t(x, x) ≥

∑k≥0

P k(x0, x0)

P t1(x, x0)P t2(x0, x) =∞=⇒ x recurrent

Proof

∴∑t≥1

P t(x, x) ≥∑

t≥t1+t2

P t(x, x) ≥

∑k≥0

P k(x0, x0)

Proof

∴∑t≥1

P t(x, x) ≥∑

t≥t1+t2

P t(x, x) ≥

∑k≥0

P k(x0, x0)

Proof

∴∑t≥1

P t(x, x) ≥∑

t≥t1+t2

P t(x, x) ≥

∑k≥0

P k(x0, x0)

P t1(x, x0)P t2(x0, x) =∞

=⇒ x recurrent

Proof

∴∑t≥1

P t(x, x) ≥∑

t≥t1+t2

P t(x, x) ≥

∑k≥0

P k(x0, x0)

If all states are recurrent : 2 possibilities

I All null recurrent

I All positive

Example : SRW

I 1D null recurrence

I 3D+ transience

Why?

Example : SRW

I 3D+ transience

Why?

Example : SRW

I 3D+ transience

Why?

Example : SRW

I 3D+ transience

Why?

Recurrence and transience of SRW’s

I 1D : P(X2n = 0) =(2nn

) (12

)2n ∼ 1√πn

I 2D : P(X2n = 0) =[(

2nn

) (12

)2n]2 ∼ 1πnI 3D+ : P(X2n = 0) ∼ 3

√3

2(πn)3/2

P(τ0

I 1D : P(X2n = 0) =(2nn

) (12

)2n ∼ 1√πn

I 2D : P(X2n = 0) =[(

2nn

) (12

)2n]2 ∼ 1πn

I 3D+ : P(X2n = 0) ∼ 3√3

2(πn)3/2

P(τ0

I 1D : P(X2n = 0) =(2nn

) (12

)2n ∼ 1√πn

I 2D : P(X2n = 0) =[(

2nn

) (12

)2n]2 ∼ 1πnI 3D+ : P(X2n = 0) ∼ 3

√3

2(πn)3/2

P(τ0

Limit theoremsP irreducible : then there is at most one invariant distribution π

If π exists, then π(x) > 0 ∀ x ∈ X

Theorem (Ergodic theorem)

Irreducible chain with invariant dist. π

limT→∞

1

T

T∑t=1

1{Xt = x} = π(x) =1

Eτx

Shows that all states are positive recurrent

Theorem (Convergence theorem)

Irreducible & aperiodic chain with invariant dist. π

π(t) −→ π i.e. limt→∞

P(Xt = x) = π(x)

limT→∞

1

T

T∑t=1

1{Xt = x} = π(x) =1

Eτx

P(Xt = x) = π(x)

limT→∞

1

T

T∑t=1

1{Xt = x} = π(x) =1

Eτx

P(Xt = x) = π(x)

limT→∞

1

T

T∑t=1

1{Xt = x} = π(x) =1

Eτx

P(Xt = x) = π(x)

limT→∞

1

T

T∑t=1

1{Xt = x} = π(x) =1

Eτx

P(Xt = x) = π(x)

Stationary distribution criterion

Theorem

Irreducible (homogeneous) chain is positive recurrent iff there exists a stationary distribution

Moreover, inv. dist. π, when it exists, is unique & π > 0

Example : birth and death process

X = {0, 1, 2, . . .}

p(x) > 0 q(x) > 0 except q(0) = 0

Proposition

{Xt} positive recurrent (exists π) iff∑x≥1

λ(x)

X = {0, 1, 2, . . .}

p(x) > 0 q(x) > 0 except q(0) = 0

Proposition

λ(x)

X = {0, 1, 2, . . .} p(x) > 0 q(x) > 0 except q(0) = 0

Proposition

λ(x)

X = {0, 1, 2, . . .} p(x) > 0 q(x) > 0 except q(0) = 0

Proposition

λ(x)

X = {0, 1, 2, . . .} p(x) > 0 q(x) > 0 except q(0) = 0

Proposition

λ(x)

Proof

Positive recurrence ⇔ ∃ π s.t. πP = π & π(x) > 0 ∀ x ∈ X

∑π(x)P (x, y) = π(y)⇔

{π(0)p(0) = q(1)π(1)

π(x)[p(x) + q(x)] = q(x+ 1)π(x+ 1) + p(x− 1)π(x− 1)

By induction

q(x)π(x) = π(x− 1)p(x− 1) =⇒ π(x) = λ(x)π(0)

i.e. ∃ inv. dist. iff∑λ(x)

Proof

∑π(x)P (x, y) = π(y)⇔

{π(0)p(0) = q(1)π(1)

π(x)[p(x) + q(x)] = q(x+ 1)π(x+ 1) + p(x− 1)π(x− 1)

By induction

q(x)π(x) = π(x− 1)p(x− 1) =⇒ π(x) = λ(x)π(0)

Proof

∑π(x)P (x, y) = π(y)⇔

{π(0)p(0) = q(1)π(1)

π(x)[p(x) + q(x)] = q(x+ 1)π(x+ 1) + p(x− 1)π(x− 1)

By induction

q(x)π(x) = π(x− 1)p(x− 1) =⇒ π(x) = λ(x)π(0)

Proof

∑π(x)P (x, y) = π(y)⇔

{π(0)p(0) = q(1)π(1)

π(x)[p(x) + q(x)] = q(x+ 1)π(x+ 1) + p(x− 1)π(x− 1)

By induction

q(x)π(x) = π(x− 1)p(x− 1) =⇒ π(x) = λ(x)π(0)

Proof

∑π(x)P (x, y) = π(y)⇔

{π(0)p(0) = q(1)π(1)

π(x)[p(x) + q(x)] = q(x+ 1)π(x+ 1) + p(x− 1)π(x− 1)

By induction

q(x)π(x) = π(x− 1)p(x− 1) =⇒ π(x) = λ(x)π(0)

Proof

∑π(x)P (x, y) = π(y)⇔

{π(0)p(0) = q(1)π(1)

π(x)[p(x) + q(x)] = q(x+ 1)π(x+ 1) + p(x− 1)π(x− 1)

By induction

q(x)π(x) = π(x− 1)p(x− 1) =⇒ π(x) = λ(x)π(0)

Proof

∑π(x)P (x, y) = π(y)⇔

{π(0)p(0) = q(1)π(1)

π(x)[p(x) + q(x)] = q(x+ 1)π(x+ 1) + p(x− 1)π(x− 1)

By induction

q(x)π(x) = π(x− 1)p(x− 1) =⇒ π(x) = λ(x)π(0)

agenda: countable markov chainscandes/teaching/stats318/...k=1 n(t+1) k fx tgis a markov chain...

Documents