8/13/2019 DTMC Classification Review

1/3

1

Discrete-Time Markov Chains Review

Chapman-Kolmogorov Equations:

, for all

Initial occupancy probability:

n -step occupancy probability:

Classification of States

- Accessibilityo If the n-step transition probability from state to state is non-negative for some ,

state is accessible from state .

- Communicationo States that are accessible from each other communicate.

- Classo States that communicate with each other are in the same class.o Classes are disjoint, that is, a state cannot be in more than one class.

- Irreducibilityo A MC with only one class is irreducible.

- Recurrenceo If the probability that the process will reenter state , starting from state , is equal to 1,

( ) state is recurrent. Also, state is recurrent if .

o If a state is recurrent, the process will reenter that state infinitely often.o If the expected time to reenter the recurrent state is finite, state is positive recurrent.

All recurrent states in finite-state MCs are positive recurrent!

- Transienceo If the probability that the process will reenter state , starting from state , is less than 1,

( ) state is transient. Also, state is transient if .

8/13/2019 DTMC Classification Review

2/3

2

Since a transient state is only visited a finite number of times, in a finite-stateMC, not all states can be transient! If all states were transient, after a finite timeno states would be visited, which contradicts the fact that the process should bein some state after any finite number of time intervals.Therefore, in a finite-state MC, at least one state should be recurrent!

o If a state is transient, each time the process enters that state there will be a positiveprobability ( ) that it will never reenter that state.

Starting in state , the probability that the process will be in state for exactly time periods is . Note that this is the pmf of the geometricdistribution with parameter . Therefore the average number of timeperiods that the process will be in state is .

o In the long-run, the fraction of time spent in a transient state would be 0.

- Periodicityo State has period if for values of that are NOT divisible by , and is the

largest such integer with this property. is the greatest common divisor of all possible number of transitions to reenter

a state.o The fact that a state has period does not necessarily mean that you can reenter that

state in transitions!Example: If it is possible to reenter a state in transitions, the period is

, but 2 is not in the set of possible number of transitions to reenter that state.o If for state , state is aperiodic.

- Ergodicityo A positive recurrent and aperiodic MC is ergodic.o The tpm for ergodic MCs converge to the limiting distribution ( ) as the number of

transitions converge to infinity ( ). The limiting distribution provides the probability of observing the stochastic

process in each state in the long-run. The limiting (stationary) probability value for state ( ) is independent of the

initial state.

8/13/2019 DTMC Classification Review

3/3

3

Identify the classes of the following Markov Chains and specify which ones are transient, recurrent, etc.

[ ]

[ ]

[ ]

[ ]