1

Introduction to Stochastic ModelsIntroduction to Stochastic ModelsGSLM 54100GSLM 54100

2

OutlineOutline

limiting distribution connectivity

types of states and of irreducible DTMCs transient, recurrent, positive recurrent, null

recurrent

periodicity

limiting behavior of irreducible chains

3

ConnectivityConnectivity

4

Connectivity of a Connectivity of a DTMCDTMC

connectivity: one factor that determines the limiting behavior of a DTMC

1 2

1

0.010.99

A

0.99 0.01

0 1

P

1 2

0.99

0.010.99

B

0.01

0.99 0.01

0.01 0.99

P

5

Connectivity of a Connectivity of a DTMCDTMC

1 2

1

0.010.99

A

0.99 0.01

0 1

P

(4) 0.960596 0.03940399

0 1

P

(16) 0.8514576 0.1485422

0 1

P

(64) 0.5255965 0.4744035

0 1

P

(256) 0.07631498 0.923685

0 1

P

(1024) 0.0000339187 0.9999661

0 1

P

( ) 0 1

0 1

P

6

Connectivity of a Connectivity of a DTMCDTMC

1 2

0.99

0.010.99

B

0.01

0.99 0.01

0.01 0.99

P

(4) 0.96118408 0.03881592

0.03881592 0.96118408

P

(16) 0.8618989 0.1381011

0.1381011 0.8618989

P

(64) 0.6372268 0.3627732

0.3627732 0.6372268

P

(256) 0.5028369 0.4971631

0.4971631 0.5028369

P

(1024) 0.5 0.5

0.5 0.5

P

( ) 0.5 0.5

0.5 0.5

P

7

Connectivity of a Connectivity of a DTMCDTMC

1 2

0.99

0.010.99

B

0.01

0.99 0.01

0.01 0.99

P

( ) 0.5 0.5

0.5 0.5

P

1 2

1

0.010.99

A

0.99 0.01

0 1

P

( ) 0 1

0 1

P

8

Connectivity of a Connectivity of a DTMCDTMC

1 2 0.6

0.10.9

C

1

30.4

0.9 0.1 0

0 0.6 0.4

1 0 0

P

( )

0.7407407 0.1851852 0.07407407

0.7407407 0.1851852 0.07407407

0.7407407 0.1851852 0.07407407

P

9

Connectivity of a Connectivity of a DTMCDTMC

1 2

1

0.010.99

A

0.99 0.01

0 1

P

( ) 0 1

0 1

P

1 2

0.8

0.10.9

D

0.20.9 0.1

0.2 0.8

P

( ) 0.666667 0.333333

0.666667 0.333333

P

( )11lim 0n

np

( )

12lim 1n

np

( )21lim 0n

np

( )

22lim 1n

np

( )11lim 2 / 3n

np

( )

12lim 1/ 3n

np

( )21lim 2 / 3n

np

( )

22lim 1/ 3n

np

10

Connectivity of a Connectivity of a DTMCDTMC

rows of P() may not be the same as in the previous examples

1 2

1

0.5

3 4

0.5

0.5

0.5

0.5

5

0.51

1 0 0 0 0

0.5 0 0.5 0 0

0 0.5 0 0.5 0

0 0 0.5 0 0.5

0 0 0 0 1

P

( )

1 0 0 0 0

0.75 0 0 0 025

0.5 0 0 0 0.5

0.25 0 0 0 0.75

0 0 0 0 1

P

11

Connectivity of a Connectivity of a DTMCDTMC

1 2

1

F

1

0 1

1 0

P

(even) 1 0

0 1

P

(odd) 0 1

1 0

P

limit of P(m) may not exist

12

Connectivity of a Connectivity of a DTMCDTMC0 1 0

0.75 0 0.25

0 1 0

P

even0.75 0 0.25

0 1 0

0.75 0 0.25

P 0 1 0

0.75 0 0.25

0 1 0

odd

P

1 2 3

0.25

1

1

0.75

1 2 3

0.25

0.99

1

0.75

0.01

0 1 0

0.75 0 0.25

0.01 0.99 0

P

0.3757803 0.4993758 0.1248439

0.3757803 0.4993758 0.1248439

0.3757803 0.4993758 0.1248439

P

13

Types of States and of Irreducible DTMCs

14

Limiting Results for a Limiting Results for a DTMCDTMC

depending on the type of states and the chain

type: transient, positive recurrent, null recurrent

connectivity and periodicity

15

Transient State Transient State

1 2

1

0.010.99

A

1 2

1

0.5

3 4

0.5

0.5

0.5

0.5

5

0.51

( ) 0 1

0 1

P( )11lim 0

np

( )

1 0 0 0 0

0.75 0 0 0 025

0.5 0 0 0 0.5

0.25 0 0 0 0.75

0 0 0 0 1

P

( )22lim 0

np

( )33lim 0

np

( )44lim 0

np

16

Recurrent State Recurrent State

state i is recurrent if P(return to i|X0 = i) = 1

( )

0

nii

np

17

Recurrent State Recurrent State

two types of recurrent states positive recurrent:

E(# of transitions to return to i|X0 = i) <

null recurrent:

E(# of transitions to return to i|X0 = i) =

( )

0lim

N nii

n

N

p

N

( )

0lim

N nii

n

N

p

N

18

Periodicity

19

PeriodicityPeriodicity

state i is of period d if Xn can return to state i in multiples of d

states 1, 2, 3 are of period 2

state i of period d

1 2 3

0.25

1

1

0.75

( ) 0 for 1nd kiip k d

20

PeriodicityPeriodicity

period of states 1, 2, 3, and 4 = ?

state 4 of period 2 state 4 can return to itself in 2 steps

1 2 3

0.25

0.99

1

0.754

0.01

1

21

Communicating States Communicating States

communicating states are of the same type transient, positive recurrent, null recurrent at the

same time of the same period

states in an irreducible chain are of the same type transient, positive recurrent, null recurrent at the

same time of the same period

22

Limiting Behavior of Irreducible Chains

23

Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain

j = fraction of time at state j

N: a very large positive integer

# of periods at state j j N

balance of flow j N i (i N)pij j = i ipij

[ ]ijpP

24

Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain

j = fraction of time at state j

j = i ipij

1 = 0.91 + 0.22

2 = 0.11 + 0.82

linearly dependent

normalization equation: 1 + 2 = 1

solving: 1 = 2/3, 2 = 1/3

1 2

0.8

0.10.9

C

0.2

( ) 0.666667 0.333333

0.666667 0.333333

P

25

Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain

1 = 0.752 + 0.013

3 = 0.252

1 + 2 + 3 = 1

1 = 301/801, 2 = 400/801, 3 = 100/801

1 2 3

0.25

0.99

1

0.75

0.01

0 1 0

0.75 0 0.25

0.01 0.99 0

P

0.3757803 0.4993758 0.1248439

0.3757803 0.4993758 0.1248439

0.3757803 0.4993758 0.1248439

P

26

Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain

an irreducible DTMC {Xn} is positive there exists a unique nonnegative solution to

 

j: stationary (steady-state) distribution of {Xn}

0

0

1 (normalization eqt)

, for all , (balance eqts)

jj

j i iji

p j

27

Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain

j = fraction of time at state j

j = fraction of expected time at state j

average cost cj for each visit at state j

random i.i.d. Cj for each visit at state j

for aperiodic chain:

1

0lim

k

n

Xk

j jn j

E cc

n

1

0lim ( )

k

n

Xk

j jn j

E CE C

n

0lim ( | )n jn

P X j X

28

Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain

1 = 301/801, 2 = 400/801, 3 = 100/801

profit per state: c1 = 4, c2 = 8, c3 = -2

average profit

1 2 3

0.25

0.99

1

0.75

0.01

0 1 0

0.75 0 0.25

0.01 0.99 0

P 301 400 100801 801 801

4201801

(4) (8) (2)

29

Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain

1 = 301/801, 2 = 400/801, 3 = 100/801

C1 ~ unif[0, 8], C2 ~ Geo(1/8), C3 = -4 w.p. 0.5; and = 0 w.p. 0.5 E(C1) = 4, E(C2) = 8, E(C3) = -2

average profit

1 2 3

0.25

0.99

1

0.75

0.01

0 1 0

0.75 0 0.25

0.01 0.99 0

P

301 400 100801 801 801

4201801

(4) (8) (2)

30

Different Interpretations of Different Interpretations of

balance equations: balance of rates

total rate into a group of states = total rate out of a group of states

{0}: 0 = q1

{0, 1}: p1 = q2

{0, 1, …, n}: pn = qn+1 , n 1

0 1

q

2 3

p

q

p

q

p1

…q

0j i ij

ip

31

Example: Condition for the Example: Condition for the Following Chain to be PositiveFollowing Chain to be Positive

{0}: 0 = q1

{0, 1}: p1 = q2 1

{0, 1, …, n}: pn = qn+1 , n 11

positive {j} exists

0 + 1 + 2 + … = 1 has solution

.

p < q 0 1

q

2 3

p

q

p

q

p1

…q

11 0q

12 1 0

p pq q q

11

0

npn q q

21 1 1

0 1 ... 1p pq q q q q

32

Example 4.24 of RossExample 4.24 of Ross four-state production process states {1, 2, 3, 4} up states {3, 4}, down states {1, 2} find E(up time) & E(down time)

time

1

state2

3

4

time

downstate

up

33

Example 4.24 of RossExample 4.24 of Ross

1 = 3/16, 2 = 1/4, 3 = 7/24, 4 = 13/48

how to find E(up time) E(down time)

0.25 0.25 0.5 0

0 0.25 0.5 0.25

0.25 0.25 0.25 0.25

0.25 0.25 0 0.5

P

34

Example 4.24 of RossExample 4.24 of Ross

fraction of up time = 3 + 4 =

rate of turning from up to down

= (p31+p32)3 + (p41+p42) 4

= rate of turning from down to up

= p131 + (p23+p24) 2

=

0.25 0.25 0.5 0

0 0.25 0.5 0.25

0.25 0.25 0.25 0.25

0.25 0.25 0 0.5

P

(up time)

(up time) (up time)

E

E E

1

(up time) (up time)E E