math 456: mathematical modelingpeople.math.umass.edu/~nguillen/456/week8_countingvisits_etc.pdf ·...

Math 456: Mathematical Modeling

Tuesday, March 6th, 2018

Counting visiting times et cetera


Today

Today we will see:

1. Where are we? A quick review of last few classes.

2. Some comments on Problem Set 4.

3. Proof of the Strong Markov Property.

Next class:

1. Counting how often a state is visited.

2. Statement of the convergence theorem.

ReviewLast couple of classes

1. Transient and recurrent states.

2. Stopping times and exit distributions.

3. Calculus on graphs and the Laplacian.

4. The Strong Markov Property.

5. Probabilities of returns.

ReviewTransient and recurrent states

In a Markov chain, we said first (informally) that a state x isrecurrent if starting from x we are guaranteed to eventuallyreturn to x, and said to be transient if there is some chance ofnever returning to x.

In terms of the first visit time Tx, this is written as

x is said to be transient if Px[Tx <∞] < 1,

x is said to be recurrent if Px[Tx <∞] = 1.

(recall that Tx = min{n ≥ 0 | Xn = x}

)

ReviewStopping times and exit distributions

For a set A ⊂ S, the first arrival time TA is an example of astopping time.We were interested in its exit distribution

Gy(x) = Px[XTA = y],

this defined for y ∈ A and x ∈ S.

ReviewCalculus on graphs and the Laplacian

Going back to the exit distributions, we found that

∆Gy(x) = 0 if x 6∈ A,Gy(x) = 0 if x ∈ A, x 6= y,

Gy(y) = 1.

This system of equations is always sufficient to determine allthe exit probabilities Gy(x)!!

ReviewCalculus on graphs and the Laplacian

Consider a Markov chain with state space S and transitionprobability p(x, y).

Given f : S → R, we defined

∆f(x) =∑y

(f(y)− f(x))p(x, y).

ReviewThe Strong Markov Property

Theorem (Strong Markov Property)

Let Xn denote a Markov Chain. If T is a stopping time andYn := XT+n, then Yn is also a Markov chain. Moreover, thetransition probability for Yn is the same as the one for Xn.

ReviewProbabilities of return

Using the Strong Markov Property, we were able to computethe probability of returning to x at least k times,

Px[T (k)x <∞] = ρkxx

where

ρxy = Px[Ty <∞]

Today

Next

1. Some comments on Problem Set 4.

2. Proof of the Strong Markov Property.

3. Counting how often a state is visited.

The Strong Markov Property






What does this mean? Equivalently, it means that

P[Yn+1 = αn+1, Yn = αn, . . . Y1 = α1]

is equal to p(αn, αn+1)p(αn−1, αn) . . . p(α1, α2)P(Y1 = α1).


Proof

Let us analyze first P[XT+n = xn, . . . , XT+1 = x1, XT = x].

To do this, we first note that

P[XT+n = xn, . . . , XT+1 = x1, XT = x]

=

∞∑t=1

P[Xt+n = xn, . . . , Xt+1 = x1, Xt = x, T = t]


Proof

Fix t ∈ N. Consider the sets

Ht = {α = (α1, . . . , αt) | X1 = α1, . . . , Xt = αt ⇒ T = t}Ht,x = {α ∈ Ht | αt = x}

In words: Ht is the possible trajectories for the chain, up totime t, for which the T = t; and Ht,x are those trajectories upto time t, with T = t, ending in x.


Proof

In particular, this means that

{XT = x, T = t} =⋃

α∈Ht,x

{X1 = α1, X2 = α2, . . . , Xt = αt}


Proof

Therefore, the probability

P[Xt+n = xn, . . . , Xt+1 = x1, Xt = x, T = t]

is given by the sum of the probabilities

P[Xt+n = xn, . . . , Xt+1 = x1, Xt = αt, . . . , X1 = α1]

The sum being over α ∈ Ht,x


Proof

Therefore, the probability

P[Xt+n = xn, . . . , Xt+1 = x1, Xt = x, T = t]

is given by the sum of the probabilities


for all α ∈ Ht,x


Proof

Now, we know that thanks to the Markov property,


is equal to

p(x, x1)p(x1, x2) . . . p(xn−1, xn)P(Xt = αt, . . . X1 = α1)


Proof

Adding up the α’s we see that

P[Xt+n = xn, . . . , Xt+1 = x1, Xt = x, T = t]

is equal to

p(x, x1)p(x1, x2) . . . p(xn−1, xn)P[Xt = x, T = t]


Proof

. . . adding up in t, and dividing by P(XT = x) we see that

P[XT+n = xn, . . . , XT+1 = x1 | XT = x]

is equal to

p(x, x1)p(x1, x2) . . . p(xn−1, xn)

and it follows that XT+n is a Markov Chain with sametransition probability as Xn.

Math 456: Mathematical Modeling


Counting times, periodicity, and convergence

Thursday, March 22th, 2018

Last timeVisiting Times

Given a state y ∈ S, let

N(y) := #{n ≥ 1 | Xn = y}

That is, N(y) is the random variable given as the number oftimes n for which the Markov chain is at the state y. Inparticular, if y is visited infinitely many times, then N(y) =∞.

We are going to prove a few useful facts about N(y).


First, note that

Px(N(y) ≥ 1) = 1− Px(N(y) = 0) = ρxy

Recall that ρxy = Px(Ty <∞), the probability of eventuallyvisiting y starting from x.

Let us computer the probabilities for the other values of N(y)!


We showed that for any k ∈ N

Px[N(y) ≥ k] = Px[T (k)y <∞] = ρxyρ

k−1yy .

From here, one can compute the distribution of N(y)

Px[N(y) = k] = Px[N(y) ≥ k]− Px[N(y) = k + 1]

= ρxyρk−1yy − ρxyρkyy

= ρxyρk−1yy (1− ρyy)

Visiting Times

There is a simple formula for the expected number of visits

Lemma

Ex[N(y)] =ρxy

1− ρyy

Visiting Times

We use the following general (and useful!) fact about theexpectation of a real variable Y

E[Y ] =

∞∑k=1

P(Y ≥ k).

Why is this true?

Let χA denote the random variable which is equal to 1 if theeven happens, and 0 otherwise

χA :=

{1 if A happens0 otherwise

Visiting Times

We use the following general (and useful!) fact about theexpectation of a real variable Y

E[Y ] =

∞∑k=1

P(Y ≥ k).

Why is this true?Let χA denote the random variable which is equal to 1 if theeven happens, and 0 otherwise

χA :=

{1 if A happens0 otherwise

Visiting Times

If a random variable Y takes only the values k = 1, 2, . . ., wemay write it as

Y =

∞∑k=1

χ{Y≥k}

In which case, taking expectation on both sides

E[Y ] =

∞∑k=1

E[χ{Y≥k}]

But, it is clear that

E[χA] = P(A)

and we obtain the formula.

Visiting Times

Proof of the Lemma.

We use the formula for expectation we just obtained, it yields

Ex[N(y)] =

∞∑k=1

Px(N(y) ≥ k)

=

∞∑k=1

ρxyρk−1yy

Therefore, we have

Ex[N(y)] = ρxy

∞∑k=0

ρkyy

Visiting Times

Proof (continued).

Recall that if t ∈ (0, 1), then

1

1− t= 1 + t+ t2 + t3 + . . .

We obtain, using the above with t = ρyy,

Ex[N(y)] =ρxy

1− ρyyif ρyy < 1

Ex[N(y)] =∞ if ρyy = 1

Visiting Times

A few takeaways from this formula: as long as x→ y, Ex[N(y)]gives us information about whether y is recurrent or not.

There is a completely different, formula for Ex[N(y)].

Visiting Times

Lemma

For any states x and y we have

Ex[N(y)] =

∞∑n=1

pn(x, y)

Visiting Times

Proof.

Notice that we may write,

N(y) =

∞∑n=1

χ{Xn=y}

But

Ex[χ{Xn=y}] = pn(x, y)

Therefore,

Ex[N(y)] =

∞∑n=1

pn(x, y)

Visiting Times

Combining these two formulas tell us the following.If y is a transient state, then for any state x

∞∑n=1

pn(x, y) <∞

and in particular, for a transient y the n-step transitionprobability from x to y goes to zero as n goes to infinity!

limn→∞

pn(x, y) = 0

Question:Does pn(x, y) have a limit as n→∞ for recurrent y?

Convergence Theorem

Theorem

Consider an irreducible, aperiodic chain, and let π(y) denote itsstationary distribution.

Then, for any y ∈ S, we have

limn→∞

pn(x, y) = π(y) ∀ x ∈ S.

This is great, but what does this aperiodicity refer to?!.

Period of a state

Definition: Given a Markov Chain, the period of a state x isthe largest common divisor of the numbers in the set

Ix := {n ∈ N | pn(x, x) > 0}

Example: Remember the Ehrenfest chain? For any state inthis chain pn(x, x) > 0 if and only if n is even. Thus, theperiod of every state is 2.

Example: If x is such that p2(x, x) > 0 and p3(x, x) > 0 thenthe period of x is 1.

Period of a stateProperties of the period

• The set Ix is closed under sums, that is, if n ∈ Ix and m ∈ Ix,then n+m ∈ Ix.

• If p(x, x) > 0, then x has period equal to 1.

• If x 7→ y and y 7→ x then x and y have the same period. Inparticular, in an irreducible chain, all states have the sameperiod.

• If x has period 1, then there is a number n0 such thatpn(x, x) > 0 for every number n ≥ n0.

Period of a stateProperties of the period

Definition: If every state in a chain has period equal to 1, thechain is said to be aperiodic.

• If a chain with N states is irreducible and aperiodic, thenpn(x, y) > 0 for all n ≥ N and any states x and y.

math 456: mathematical modelingpeople.math.umass.edu/~nguillen/456/week8_countingvisits_etc.pdf ·...

Documents