1

Probabilistic Operations Research (POR)

Topic # 10 – Limiting Behavior in Discrete-Time Markov Chains

(Chapter 6.2 and 6.4 of Cassady and Nachlas textbook)

Finding the n-step Transition Probability

The one-step transition probability matrix (tpm) can be used to examine the dynamic behavior of

the system.

Consider the discrete-time Markov chain { ( ) } with state space and tpm . Then,

the n-step transition probability from state to state for all and

is expressed as

( )

( ( ) ( ) )

The collection of all the n-step transition probabilities is the n-step transition probability

matrix, ( ).

Chapman-Kolmogorov Equations

There can be several feasible ways to make an n-step transition and we can consider the

transition to an intermediate state from state when transitioning to state in n steps using the

Chapman-Kolmogorov equations shown below.

∑

for all .

( | ) ∑ ( | )

∑ ( | ) ( )

∑

In general, the summation of products on the right hand side is a matrix multiplication, then

( ) ( ) ( )

for all .

A useful special case is: ( ) ( ) for all .

2

Limiting behavior

Consider the “Weather Forecast” example in Topic 8:

Suppose that whether or not it rains tomorrow depends only on today’s weather condition. If it

rains today, it will rain tomorrow with probability . If it does not rain today, it will rain

tomorrow with probability .

The tpm is

[

]

When we find the n-step tpm for n=2, 4, 8, and 16:

( ) [

] ( ) [

]

( ) [

] ( ) [

]

Note that the transition probabilities converge to the same value as n and is the same

for all .

There seems to be a limiting probability that the process will be in state after a large number of

transitions and this probability is independent of the initial state .

In order to determine the step-dependent probability distribution on the identity of the state

occupied at any point in time, we need to know the distribution on state occupancy at the

beginning of the analysis.

Initial occupancy probability: ( )

The vector of initial occupancy probabilities:

n-step occupancy probability: ( )

( )

The vector of n-step occupancy probabilities: ( ) ( )

( )

Note that

( )

∑ ( )

for all n = 1, 2, … and

( ) ( ) for all n = 1, 2, …

3

If a DTMC is irreducible and has a finite state space, the initial state becomes irrelevant as the

number of transitions increases. This steady-state behavior is represented by the limiting

(stationary) probabilities, .

For an irreducible and ergodic (positive recurrent and aperiodic) Markov chain,

( )

for all .

Equivalently,

( )

for all .

The vector of limiting probabilities:

The vector is the unique non-negative solution to the set of linear equations

∑ for all (or equivalently, ) and

∑