ex3s

7/30/2019 ex3s

http://slidepdf.com/reader/full/ex3s 1/4

MTH6141 Random Structures, Spring 2012

Solutions to Exercise Sheet 3

1. (a) The chain is certainly irreducible. Since there is a path of 3 stepsbetween any pair of states it is also regular. Since it is regular there isa unique equilibrium distribution which satisfies

(w1, w2, w3, w4)

1/2 1/2 0 00 1/3 2/3 00 0 1/4 3/4

4/5 0 0 1/5

= (w1, w2, w3, w4)

subject to w1+w2+w3+w4 = 1. Solving the system of linear equations,

(w1, w2, w3, w4) = (24/73, 18/73, 16/73, 15/73)

Since the chain is regular we have that (by a result in lectures) thisdistribution is also a limiting distribution.

(b) The chain is not irreducible (it is impossible to get from state 3 to state2 for instance). It cannot be regular either (since regularity is a strongercondition).

Any probability vector w = (0, α, 1 − α) satisfies wP = w so there aremany equilibrium distributions.

There is no limiting distribution. Indeed (0, 1, 0)P t = (0, 1, 0) and(0, 0, 1)P t = (0, 0, 1) so we cannot have P t tending to a limit with

constant rows. (However P t does tend to a limit, can you see what itis?)

(c) The chain is not irreducible (it is impossible to get from state 3 to state2 for instance). It cannot be regular either.

The only probability vector which is a solution to wP = w is w =(0, 0, 1) so this is the unique equilibrium distribution.

If the chain is run for a long time then it is absorbed in state 3 withprobability 1. It follows that there is a limiting distribution and it isalso (0, 0, 1).

(d) The chain is irreducible. However, it is not regular. (If t is even then p(

t)12 = 0; if t is odd then p(

t)11 = 0.)

Solving the usual equations we see that there is a unique equilibriumdistribution w = (1/2, 1/10, 2/5).

There is no limiting distribution. We cannot have P t tending to a limitsince

p(k)11 =

1, if k is even;

0, if k is odd.

1

7/30/2019 ex3s


(e) The chain is clearly irreducible and regular (the matrix P = P 1 has allentries positive).

Since it is regular there is a unique equilibrium distribution. By inspec-tion the vector w = (1/4, 1/4, 1/4, 1/4) satisfies wP = w and so this isthe unique equilibrium distribution and also (by a result from lectures)the limiting distribution.

Notice that when the chain is not regular there may or may not be a uniqueequilibirum distribution and there may or may not be a limiting distribution.Also it is possible for there to be no limiting distribution but for P t to tendto a limit. (For a limiting distribution we require that all rows of the limitare equal.)

2. (a) Since the chain is irreducible we know that for any a, b ∈ S there is a

k with p(k)ab > 0. Let m be the maximum of all of these ks (we can dothis since there are only finitely many pairs of states).

It follows from the definition of m that given any pair of states a, b ∈ S we have that p

(s)ai > 0 for some s ≤ m and p

(t)ib > 0 for some t ≤ m.

Now 2m − s − t ≥ 0 and p(2m)ab ≥ p

(s)ai p

(2m−s−t)ii p

(t)ib > 0. Hence there is

a positive probability that we will move between any pair of states in2m steps (or equivalently all the entries of P 2m are positive) and so thechain is regular.

(b) The result is no longer true for infinite S . Consider the Markov chain

with S =Z

and pi,i = pi,i+1 = pi,i−1 = 1/3 and all for all i ∈

Zand allother transition probabilities 0. This is irreducible but given any k we

have that p(k)i,i+k+1 = 0 and so it is not regular. (The part of the proof

above which fails here is that we cannot necessarily take the maximumof an infinite set of numbers).

3. (a) If p = 0 then the MC is not irreducible. pi3 = 0 for all i, and hence

p(t)13 =

4i=1 p(

t−1)1i p(1)i3 = 0 for all t ∈ N. (There is no path of any length

from state 1 to state 3 in the transition graph.)

If 0 < p ≤ 12

then the MC is irreducible. p(2)21 ≥ p24 p41 > 0, p(2)31 ≥ p34 p41 > 0 and p41 > 0. (So we can get to state 1 from anywhere.)

Also p12 > 0, p13 > 0 and p(2)14 ≥ p12 p24. (So we can get from state 1 to

anywhere.) Summarising, for all i, j, p(t)ij > 0 for some t ≤ 4. (Another

way of saying this is that the transition graph is strongly connected.Actually, t ≤ 3 is enough.)

(b) If p = 0 the MC is not irreducible and hence not regular. If p = 12

then the MC is not regular. Consider the initial distribution µ(0) =(1, 0, 0, 0). The µ(3) = µ

(0)P 3 = (1, 0, 0, 0) = µ(0). Iterating, µ(3s) =

2

7/30/2019 ex3s


(1, 0, 0, 0) for all s ∈ N. However,

µ(3s+1) = (1, 0, 0, 0)P 3sP = (1, 0, 0, 0)P = (0, 1

2, 12

, 0).

Thus µ(t) does not converge as t → ∞.

If 0 < p < 12

then the MC is regular. Looking at the pattern of zeroentries in powers of P (+ indicates a non-zero entry):

P =

0 + + +0 0 0 +0 0 0 ++ 0 0 0

P 2 =

+ 0 0 ++ 0 0 0+ 0 0 00 + + +

P 4

=

+ + + ++ 0 0 +

+ 0 0 ++ + + +

P 5

=

+ + + ++ + + +

+ + + ++ + + +

.

In other words, there is a path of length exactly 5 from every vertex of the transition graph to every vertex.

(c) Since the chain is regular it has a limiting distribution w. Moreover wis the unique solution to wP = w. We therefore solve

(w1, w2, w3, w4) =

0 1

2p 1

2− p

0 0 0 10 0 0 1

1 0 0 0

= (w1, w2, w3, w4)

subject to w1 + w2 + w3 + w4 = 1. We get

(w1, w2, w3, w4) =

2

2 p + 5,

1

2 p + 5,

2 p

2 p + 5,

2

2 p + 5

.

(d) This is the case p = 12

. A byproduct of (b) is that w(3s+1) = (0, 12

, 12

, 0)assuming w(0) = (1, 0, 0, 0). Thus Pr(X 3s+1 = 1 | X 0 = 1) = 0. Nowset s = 333 to get Pr(X 1000 = 1 | X 0 = 1) = 0.

4. This is similar to Question 1(e). If P is an n × n matrix then

( 1n

, 1n

, · · · , 1n

)

p11 · · · p1n...

... pn1 · · · pnn

=

1

n

ni=1

pi1,1

n

ni=1

pi2, . . . ,1

n

ni=1

pin

=1

n

ni=1

p1i,1

n

ni=1

p2i, . . . ,1

n

ni=1

pni

= ( 1

n, 1n

, · · · , 1n

),

3

7/30/2019 ex3s


where the second equality is by symmetry, and the third follows from thefact that P is stochastic. Thus the uniform distribution w = ( 1

n, . . . , 1

n) does

satisfy wP = w, and is an equilibrium distribution.

5. Since P is regular, there exists k ∈ N such that all entries of Q = P k arestrictly positive. According to Lemma 1.7, Q has a limiting distribution,say w. That is to say, Qn → W , as n → ∞, where W is the matrix withall rows equal to w. Now let r satisfy 0 ≤ r < k. Then P nk+r = P rQn →P rW = W , as n → ∞. Since any t can be written in the form t = nk + rwith 0 ≤ r < k, it follows that P t → W as t → ∞.

If you took Convergence and Continuity , you could try making the aboveargument completely rigorous. What should it mean for the sequence of

matrices (P t

: t ∈ N) to converge to the matrix W ?

4

ex3s

Documents