7/30/2019 ex4s

1/3

MTH6141 Random Processes, Spring 2012

Solutions to Exercise Sheet 4

1. (a) As you showed for Sheet 3 this chain has a limiting distribution. Weshowed in lectures (Theorem 1.7) that if there is a limiting distributionw then the expected proportion of time spent in state k tends to wk forany initial distribution. So the proportion of time spent in state 1 inthe long run is 24/73 for any initial distribution.

(b) If we start in state 2 we never leave it and if we start in state 3 we neverleave it. It follows that the expected proportion of time spent in state2 (or state 3) depends on the initial distribution even after a long time.If we start in state 1 then we must leave it (the state is transient) so inthe long run the expected proportion of time spent in state 1 is 0. If we

start in state 2 or 3 we can never reach state 1. It follows that in thelong run the expected proportion of time spent in state 1 is 0 for anyinitial distribution.

(c) As you showed for Sheet 3 if the chain is run for a long time thenit is certainly absorbed in state 3. It follows that there is a limitingdistribution and it is (0, 0, 1) (note that this doesnt follow directlyfrom the theory since this chain is not regular or even irreducible). Itfollows that the expected proportion of time in each state will in thelong run also be (0, 0, 1). In particular the proportion of time spent instate 1 will be 0 for any initial distribution.

(d) You showed for Sheet 3 that here there is a unique equilibrium distri-bution w = (1/2, 1/10, 2/5). It follows (from Theorem 1.8) that theexpected proportion of time spent in each state tends to this in thelong run. In particular the expected proportion of time spent in state1 in the long run is 1/2 for any initial distribution.

(e) As for part a) there is a limiting distribution w = (1/4, 1/4, 1/4, 1/4).So (by Theorem 1.7) the expected proportion of time spent in each statetends to this in the long run. In particular the expected proportion oftime spent in state 1 in the long run is 1/4 for any initial distribution.

2. Recall that a b means that p(k)ab > 0 for some k and p(l)ba > 0 for some l. To

show that is an equivalence relation we need to show that it is reflexive,symmetric and transitive.

Reflexive: Since p(0)aa = 1 > 0 we have that a a for all a S.

Symmetric: The relation is clearly symmetric since the definition of whatit means for a b is unchanged after exchanging a and b.

1

7/30/2019 ex4s

2/3

Transitive: If p(k)ab > 0 and p

(l)bc > 0 then p

(k+l)ac p

(k)ab p

(l)bc > 0. So if a b

and b c then a c. Similarly, if c b and b a then c a. Itfollows from this that if a b and b c then a c. That is is

transitive.

3. (a) From the transition diagram the communicating classes can be inden-tified by inspection. They are:

{1, 2}, {3, 4, 5}, {6, 7}.

(States 1 and 2 intercommunicate, as do 3, 4 and 5, and finally 6 and 7intercommunicate. On the other hand, states in the second class do notcommunicate with the first and those in the third do not communicatewith the first or second.)

(b) The only way to return to state 1 is directly in one step, or in two steps

via the sequence of states 1 2 1. So f(t)11 = 0 when t > 2, and

f(1)11 = p11 =

13

and f(2)11 = p12p21 =

1245

= 25

. Then f11 =

t=1 f(t)11 =

13

+ 25

= 1115

< 1. State 1 is transient. The expected number of returnsto state 1 is f11/(1 f11) =

114

.

(c) The only way to return to 6 (for the first time) is via a path of the form

6 7 7 7 6. Thus f(t)66 = 1(

34

)t2 14

= 14

(34

)t2, for t 2,

and f(1)66 = 0. Then

f66 =

t=1

f(t)66 =

t=2

14(

34)

t2 = 14(1 +34 + (

34)

2 + ) = 14 4 = 1.

State 6 is recurrent.

(d) A possible sequence of transitions is 3 5 4 6 (or 3 5 4 7), which occurs with probability p35p54p46 =

1213 14

= 124

. State 6 liesoutside the communicating class {3, 4, 5}, and no return to state 3 ispossible from state 6. Thus 1f33 (the probability of never returning tostate 3) is at least 1

24, and hence f33

2324

< 1, and state 3 is transient.

(e) Denote the equilibrium distribution by w = (w1, . . . , w7). It is reason-

able to suppose that w1 = w2 = = w5 = 0, these states being alltransient. (This supposition can be rigorously justified.) Restricting at-tention to states 6 and 7, we have a 2-state MC with = 1 and = 1

4.

(Refer to the analysis of the 2-state MC in the notes.) So the requiredequilibrium distribution would seem to be (0, 0, 0, 0, 0, 1

5, 45

), and thiscan be verified by computing wP and checking it equals w.

4. If there is an equilibrium distribution w = (w0, w1, w2, . . .), it will satisfywj =

iN wipij for all j N. Specifically, w0 = w1p1,0, w1 = w0p0,1 + w2p2,1

2

7/30/2019 ex4s

3/3

and then wj = wj1pj1,j + wj+1pj+1,j for all j 2. Thus the equations weneed to solve are w0 =

23

w1, w1 = w0 +23

w2, and then w2 =13

w1 +23

w3,w3 =

13

w2 +23

w4, w4 =13

w3 +23

w5, etc. We can solve these in sequence for

w1, w2, . . . in terms of w0:

w1 =32

w0,

w2 =32

w1 32

w0 =34

w0,

w3 =32

w2 12

w1 =38

w0,

w4 =32

w3 12

w2 =316

w0.

For w to be a probability distribution we need

w0 + w1 + w2 + = w0 +32w0

1 +12 + (

12)

2

+ (12)

2

+ (12)

3

= 4w0 = 1,

i.e., w0 =14

. Substituting, w1 =3412

, w2 =34

(12

)2, w3 =34

(12

)3, etc. (Ingeneral, wj =

34

(12

)j for all j 1.) So the Markov chain does have anequilibrium distribution. (Compare this with the situation considered in thelectures, where the transition probabilities were all 1

2.)

5. (a) If i and j are in the same equivalence class then there exist a, b > 0

with p(a)ij , p

(b)ji > 0. Suppose that i is 2-periodic, then since

p(a+b)ii p

(a)ij p

(b)ji > 0

we have that a + b must be even.

We want to prove that j is 2-periodic. Suppose that it is not and go fora contradiction.

Ifj is not 2-periodic then there exists an odd number c such that p(c)jj >

0. Hence,p(a+c+b)ii p

(a)ij p

(c)jj p

(b)ji > 0.

Since c is odd and a + b is even a + c + b is odd. This contradicts the2-periodicity of i. We conclude that j is 2-periodic.

(b) There are lots of possibilities. Here is one given by a transition matrix

1/2 1/2 00 0 10 1 0

.

(c) This is impossible. If one state in an irreducible chain is 2-periodic thenevery state in S is 2-periodic (by part (a)). However, a state s with a

loop cannot be 2-periodic because p(1)ss > 0.

3