markov chain

EE 178

Lecture notes 14

Stochastic Processes II: Markov Chains

Lecture Outline

• Discrete State Processes

• Markov Chains

• State Classification

• Steady State Probabilities

• Birth and Death Processes

Reading: Bertsekas & Tsitsiklis 6.1,6.2 ,6.3

Stochastic Processes II: Markov Chains 14–1

EE 178

Discrete-State Process

Markov chains are popular models in many applications areas, includingeconomic systems and networking. Their language is sometimes differentthan the common usage of stochastic processes, but essentially they are aspecial kind of stochastic process.

For simplicity we consider only discrete time models with discrete outputs.

Suppose that {Xn; n = 0, 1, 2, . . .} is a stochastic process

Suppose also that Xn can only take on values in the set {1, 2, . . . ,m}.

Think of these values as “states” of a system. Examples include

• # of unread email messages at the begining of each hour

• # of customers in a bank at each minuteStochastic Processes II: Markov Chains 14–2

EE 178

• # of cars in a parking lot at the beginning of each hour

• # of packets in a queue at each clock period

A Markov chain is a stochastic process where the probability of the nextstate given the current state and the entire past depends only on the currentstate:

P (Xn = xn | Xn−1 = xn−1, Xn−2 = xn−2, · · · ,X0 = x0) = P (Xn = xn |Xn−1 = xn−1)

The Markovian property is a constraint on the memory of the process:knowing the immediate past means the earlier outcomes are no longerrelevant.

Alternatively: The future is conditionally independent of the past given thepresent.


EE 178

Notation: In the Markov chain literature the abbreviation

pij = P (Xn = j | Xn−1 = i)

is often used. Arguably pj|i would be better, but we adopt commonconvention. Be careful when dealing with Markov chains to not confusethis conditional probability with a joint probability.

This can be conveniently expressed using matrix notation as

P =

p11 p12 · · · p1mp21 p22 · · · p2m... ... ... ...pm1 pm2 · · · pmm

Note that here i is the conditioning so that

m∑j=1

pij = 1

i.e., the sums of any row of the matrix is 1.


EE 178

Note that any iid process trivially satisfies the Markov condition, since bothprobabilities are simply P (Xn = xn).

As a simple nontrivial example, consider a sequence of dependent coin flipswhere each coin has probability of 1− p of having the same outcome as theprevious coin flip, regardless of all previous flips.

1 21 1− p p

2 p 1− ppij

Graphs called transition diagrams are often used to depict Markov chains.

Nodes are states, arcs are state transitions (i, j) from state i to state j(only draw transitions with pij > 0)


EE 178

H T1− p1− p

p

p

Graph shows memory structure.

“binary symmetric Markov process”


EE 178

Can construct this process from an iid process: Suppose that {Zn} is aBernoulli process with property p.

Define a new process Xn recursively as follows: Let X0 be a Bernoullirandom variable with some bias q.

DefineXn = Xn−1 ⊕ Zn; n = 1, 2, . . .

⊕ is exclusive-or, addition modulo 2 (0⊕ 0 = 1⊕ 1 = 0, 0⊕ 1 = 1⊕ 0 = 1)You should be able to convince yourself that P (Xn = xn | Xn−1 =xn−1,Xn−2 = xn−2, · · · , X0 = x0)

=

{p xn 6= xn−11− p xn = xn−1

Hence {Xn} is a Markov chain.


EE 178

If you are not convinced, consider:

Note that from binary arithmetic a ⊕ b = c if and only if a = b ⊕ c andhence

P (Xn = xn | Xn−1 = xn−1, Xn−2 = xn−2, · · · ,X0 = x0)= P (Xn−1 ⊕ Zn = xn | Xn−1 = xn−1, Xn−2 = xn−2, · · · , X0 = x0)= P (Zn = Xn−1 ⊕ xn | Xn−1 = xn−1, Xn−2 = xn−2, · · · , X0 = x0)= P (Zn = xn−1 ⊕ xn | Xn−1 = xn−1, Xn−2 = xn−2, · · · ,X0 = x0)= P (Zn = xn−1 ⊕ xn)= pZn(xn−1 ⊕ xn)

The conditioning disappears because the conditioning event involves onlyXk for k ≤ n−1 and hence Zk for k ≤ n−1, but the Zn are by assumptioniid and hence mutually independent.


EE 178

Can also have asymmetric binary Markov process.

State 1 = You are up-to-date in EE178

State 2 = You are behind in EE178

1 21 0.8 0.22 0.6 0.4

pij

1 20.8

0.2

0.6

0.4


EE 178

Another example: Two spiders and a fly

A fly’s possible positions are represented by four states.

States 2, 3 : safely flying in the left or right half of a room

State 1: A spider’s web on the left wall

State 4: A spider’s web on the right wall

1 2 3 41 1.0 0 0 02 0.3 0.4 0.3 03 0 0.3 0.4 0.34 0 0 0 1.0

pij


EE 178

1 2 34 11

0.3

0.4

0.3

0.3

0.3

0.4


EE 178

Some Questions of Interest

• Steady-state occupancy probabilities of each state (long term fraction oftime spent in each state)?

• Probability of hitting certain absorbing states? (e.g., spider web)

For example, in binary symmetric Markov chain, could use known pX0 andpX1|X0 to compute pX1:

pX1(1) =1∑

x0=0

pX1|X0(1 | x0)pX0(x0)

= q(1− p) + (1− q)p

This can then be combined with pX2|X1 = pX1|X0 to compute pX2, and soon.


EE 178

But is there a limit of pXn as n→∞?a steady state probability of the states.


EE 178

In particular, if repeat the computation of pX1 from pX0 to find pX2 frompX1, then repeat again to find pX3 from pX2, and so on to find pXn frompXn−1, then each time we are solving an equation of the form

pXn(1) =1∑x=0

pXn|Xn−1(1 | x)pXn−1(x)

= pXn−1(1)(1− p) + (1− pXn−1(1))p ?

Suppose that indeed pXn(1) has a limit as n→∞, say pX(1) = α, so that

pX(x) ={α x = 11− α x = 0

Substituting this into ?:

α = α(1− p) + (1− α)pStochastic Processes II: Markov Chains 14–14

EE 178

orα(1− 1 + p+ p) = p

so that

α =1

2is the steady state probability that Xn = 1.

shortcut and interpretation: If assume the steady state probabilities exist,can argue that on the average number of transitions from state 1 to state2 must equal the average of transitions in the reverse direction. If furtherassume that the average frequency of events is given by their probability ⇒P( in state 1 and go to state two ) = P( in state 2 and go to state 1) or

αp = (1− α)p

or α = 1/2.

(Later will do this carefully)


EE 178

A similar strategy yields the steady state probability for the asymmetricMarkov chain. Suppose that asymptotically state 1 has probability α andhence state 2 has probability 1− α.In the long run the fraction of transitions from right to left in the graphmust equal that from the left to the right (one cannot enter a state moreoften than one leaves it) which implies that

0.6(1− α) = 0.2α

or α = 3/4.

In both cases write equations for steady state behavior (if it holds) andsolve them.

In general this is a question of writing matrix equations and solving them(but solutions do not always exist).


EE 178

n-step transition probabilities

m states

• rij(n) = P ( state = j after n transitions, starting from state i)

• Recursive formula, starting with rij(1) = pij:

rij(n+ 1) =

m∑k=1

rik(n)pkj; 1 ≤ i, j ≤ m ?

(Special case of a “Chapman-Kolmogorov” equation.)

Note: this is just total probability:


EE 178

rij(n+ 1) = P (Xn+1 = j|X0 = i)

=m∑k=1

P (Xn+1 = j and Xn = k|X0 = i)

=m∑k=1

P (Xn+1 = j | Xn = k,X0 = i)P (Xn = k|X1 = i)

=m∑k=1

P (Xn+1 = j | Xn = k)P (Xn = k|X0 = i)

=m∑k=1

pkjrik(n)


EE 178

• rij(n) can be viewed as the “nth power” of the matrix pij. i.e.,

Define matrix P = {pij; i = 1, . . .m, j = 1, . . .m} and define the matricesR(n) = {rij(n); i = 1, . . .m, j = 1, . . .m}; n = 1, 2, . . . Then therecursion and initial value can be expressed as

R(1) = P

R(2) = R(1)P = P 2

R(3) = R(2)P = P 3

...

R(n) = R(n− 1)P = Pn


EE 178

1 21 0.8 0.22 0.6 0.4

rij(1)

1 21 .76 .242 .72 .28

rij(2)

1 21 .752 .2482 .744 .256

rij(3)

1 21 .7504 .24962 .7488 0.2512

rij(4)


EE 178

1 21 .7501 .24992 .7498 .2502

rij(5)

As n→∞, rij(n) converges to a limit which does not depend on the initialstate i.

Part of the theory of Markov chains is the demonstration of generalconditions under which this behavior occurs, i.e., rij(n) converges to a limitwhich does not depend on the initial state


EE 178

Spider-and-Fly example

1 2 3 41 1.0 0 0 02 0.3 0.4 0.3 03 0 0.3 0.4 0.34 0 0 0 1.0

pij

1 2 3 41 1.0 0 0 02 .42 .25 .24 .093 .09 .24 .25 .424 0 0 0 1.0

rij(2)


EE 178

1 2 3 41 1.0 0 0 02 .50 .17 .17 .163 .16 .17 .17 .504 0 0 0 1.0

rij(3)

1 2 3 41 1.0 0 0 02 .55 .12 .12 .213 .21 .12 .12 .554 0 0 0 1.0

rij(4)

1 2 3 41 1.0 0 0 02 2/3 0 0 1/33 1/3 0 0 2/34 0 0 0 1.0


EE 178

rij(∞)

Note that

• States 2 and 3 have steady-state probability 0

• The steady-state probabilities of states 1 and 4 depend on the initialstate.


EE 178

Recurrent and Transient States

• Let A(i) be the set of states that are accessible from state i.

• State i is recurrent if starting from i, than any accessible state j mustbe such that i is accessible from j, i.e., j is in A(i) iff i is in A(j).

• Note: if i is recurrent, then necessarily i ∈ A(i).

• A state is transient if it is not recurrent.

• Example:

1 2 34


EE 178

• Recurrence/transcience is determined by the arcs (transitions withnonzero probability), not be actual values of probabilities.

• A recurrent class = set of accessible states A(i) from some recurrentstate i.

Note: Suppose that i is recurrent and hence A(i) is a recurrent class. Thenif k ∈ A(i), k is also recurrent and A(k) = A(i).Proof:

Suppose i is recurrent and k ∈ A(i). Then k is accessible from i and hence,since i is recurrent, k can access i and hence, through i, any state in A(i).Thus A(i) ⊂ A(k). Since i can access k and hence any state in A(k),A(k) ⊂ A(i). Thus A(i) = A(k).k must be recurrent since it can access any state in A(k) and hence anystate in A(i), which can access i (since i is recurrent), which in turn canaccess k.

Note: Recurrent classes contain no transient states.


EE 178

Note: Two recurrent classes must be identical or disjoint.

proof: Suppose have A(i) and A(j) where i and j are distinct and each isrecurrent.

If j ∈ A(i), then have seen that j is recurrent and A(j) = A(i).If j 6∈ A(i), then no state in A(j) can be also be in A(i) so thatA(i) ∩ A(j) = ∅. To see this, assume the contrary. Suppose that l ∈ A(j)is also in A(i). Then j can access l which in turn can be accessed by i.Since i is recurrent, l can access i and hence j ∈ A(i), a contradiction.

Note: A Markov chain can be decomposed into one or more recurrentclasses plus possibly some transient states.

A recurrent state is accessible from all states in its class, but it is notaccessible from states in other recurrent classes.


EE 178

Periodic Classes

A recurrent class is called periodic if the states can be grouped in d > 1subsets such that all transitions from one subset lead to the next subset.

1 2 31 0 1.0 02 1.0 0 03 0.5 0.5 0

pij

1 2

3

0.50.5

1

1

rii(n) ={1 if n is even0 if n is odd


EE 178

1

2

3

4

5

6

S1

S2

S3

If the class is periodic, rii(n) never converges to a steady-state value as nincreases.


EE 178

Steady-State Probabilities

Steady-State Convergence Theorem

If there is only one recurrent class and it is not periodic, then rij(n) tendsto a steady-state value πj that is independent of i:

limn→∞ rij(n) = πj, for all i

Steady-state equations

Take limit as n→∞ of

rij(n+ 1) =m∑k=1

rik(n)pkj; 1 ≤ i, j ≤ m

get


EE 178

πj =m∑k=1

πkpkj, j = 1, 2, . . . ,m

m∑j=1

πj = 1


EE 178

Computation of Steady State Probabilities

Solve the steady state probabilities

πj =m∑k=1

πkpkj, j = 1, 2, . . . ,m

m∑j=1

πj = 1

by elimination or by computer.


EE 178

Example Model of class mobility

1 2 3

Upper Middle Lower

.45

0.1

0.45

0.05

0.7

0.25

0.5

0.45

0.05

π1 = 0.45π1 + 0.05π2 + 0.05π3

π2 = 0.45π1 + 0.7π2 + 0.5π3

π3 = 0.1π1 + 0.25π2 + 0.45π3

1 = π1 + π2 + π3


EE 178

Expected Relative Frequency Interpretation

Interpretation: Under suitable conditions should have the long-termexpected frequency of visits to state j

πj = limn→∞

E[# of visits to j in n transitions]

n

Reminder: “suitable conditions” above means that the steady stateconvergence theorem holds.

This is an interpretation of the steady state “balance” equations

πj =m∑k=1

πkpkj, j = 1, 2, . . . ,m

which followed from combining the steady-state convergence theorem withthe Chapman-Kolmogorov equation.


EE 178

We proved the Chapman-Kolmogorov equation, we did not prove the steady-state convergence theorem. (Just stated it with some handwaving, B&Tsketch a proof. The usual careful proofs involve the limiting properties ofpowers of stochastic matrices.)

This “interpretation” is actually a theorem, a variation on the law of largenumbers relating probabilities to relative frequencies (to be proved in thenext lecture notes)

Another “stability” result follows for transitions into and out of a fixedstate:

Rewrite the steady state equations as

m∑k=1,k 6=j

πkpkj =m∑

k=1,k 6=jπjpjk, j = 1, 2, . . . ,m

i.e., original form: πj =∑mk=1 πkpkj, j = 1, 2, . . . ,m


EE 178

The left hand side can be written as

πj = πj

m∑k=1

pjk

=m∑k=1

pjkπj

= pjjπj +

m∑k=1,k 6=j

pjkπj

The right hand side can be written as

m∑k=1

πkpkj = pjjπj +m∑

k=1,k 6=jπkpkj

and subtracting the pjjπj from both sides yields the desired form.


EE 178

1

2

m

1

2

m

π1p1j

πmpmj

πjpj2π2p2j

πjpjm

......

j

πjpj1

E[ frequency of trans. into j] = E[ frequency of trans. out of j]

As when looking at frequency of visits to states, the frequency of transitionsbetween states is an interpretation of the steady state equations in termsof relative frequencies. Here the interpretation is that πipij is the expectedfrequency of transitions from state i into state j. As before, this is actuallya theorem, another variation on the law of large numbers.


EE 178

Although we have not proved the connection between probabilities andrelative frequencies for Markov chains with one recurrent class that is notperiodic, the result is very useful for quick derivations and will be used.


EE 178

Birth-death Markov Chains

• State space= {1, 2, . . . ,m}

• Only three transitions possible: go left, go right, stay where you are

1 2 i i+ 1· · · · · ·

• Cut the chain between states i and i+ 1

• Apply frequency intepretation:

πipi,i+1 = πi+1pi+1,i


EE 178

Here by “frequency interpretation” we mean specifically that the expectedfrequency (under “suitable conditions”) of making a transition from statei into state j is πipij, i.e., one small piece of the transition frequencyinterpretation.

The “balance equations” or limiting probabilities simplify greatly for birth-death Markov chains because each state has only limited transitions possible.

If we interpret the probabilities of transitions, we must have as manytranstitions going left at cut as we have going right, e.g.,

If we go left through a cut, we will not be able to go left again through ituntil we have gone right through it, there is no other route possible. Henceat any time we can have at most a difference of 1 between the number ofleft passages and right passages (and 1/n→ 0)This implies that πipi,i+1 = πi+1pi+1,i

In fact, you do not need the expected frequency interpretation to come tothis conclusion, it is just the easiest way to do so.


EE 178

To do this, recall the steady state (or balance) equation.

πj =m∑k=1

πkpkj, j = 1, 2, . . . ,m

For state i=1 (the leftmost), this reduces to

π1 = π1p11 + π2p21

and hence since p11 = 1− p12, this is

π1p12 = π2p21

Next use mathematical induction: Assume the result is true for i − 1, i.e.,that πi−1pi−1,i = πipi,i−1, and prove that it holds for i. (We know it holdsfor i = 1, so this will complete the proof.)


EE 178

πi =m∑k=1

πkpki

= πi−1pi−1,i + πipii + πi+1pi+1,i= πi−1pi−1,i + πi(1− pi,i−1 − pi,i+1) + πi+1pi+1,i= πipi,i−1 + πi(1− pi,i−1 − pi,i+1) + πi+1p(i+1)i= πi(1− pi,i+1) + πi+1pi+1,i

so thatπipi,i+1 = πi+1pi+1,i

as claimed.


EE 178

Queueing Example

Packets arrive at a communication node with storage capacity m. Time isdiscretized in very small periods. At each period:

• Prob. of 1 arrival = p

• Prob. of 1 transmission = q

• Prob. of 0 arrivals and 0 transmissions = 1− p− q

0 1 m· · ·

p

q

1− p− q 1− q

q q q

p p

1− p− q1− p

m− 1


EE 178

π0p = π1q ⇒ π1 =(p

q

)π0

π1p = π2q ⇒ π2 =(p

q

)π1 =

(p

q

)2π0

π2p = π3q ⇒ π3 =(p

q

)π2 =

(p

q

)3π0

...

πm−1p = πmq ⇒ πm =(p

q

)πm−1 =

(p

q

)mπ0

or defining ρ = p/qπi = ρ

iπ0

Since π0 + π1 + · · ·+ πm = 1,

π0(1 + ρ+ ρ2 + · · ·+ ρm) = 1


EE 178

hence for i = 1, . . . ,m

πi =ρi

1 + ρ+ ρ2 + · · ·+ ρm

Using geometric progression formula,

πi = ρi 1− ρ1− ρm+1 →m→∞

ρi

1− ρif ρ < 1


markov chain

Documents