chapter 4. markov chains - nchu.edu.t ch4 part i1.pdf · only on multiples of some positive integer...
TRANSCRIPT
Chapter 4. Markov Chains
1
2
Example. (Weather Forecasting 2)
Other Examples
3
Queuing Systems ─ number of customers waiting in line at a bank at the beginning of each
hour ─ number of machines waiting for repair at the end of each day ─ number of patients waiting for a lung transplant at beginning of each
week
Inventory Management ─ number of units in stock at beginning of each week ─ number of backordered units at end of each day
Airline Overbooking ─ number of coach seats reserved at beginning of each day
Finance ─ price of a given security when market closes each day
Other Examples (con’t)
4
Epidemiology ─ number of foot-and-mouth infected cows at the beginning of each day
Population Growth ─ size of US population at end of each year
Genetics ─ genetic makeup of your descendants in each subsequent generation
Workforce Planning ─ number of employees in a firm with each level of experience at the end
of each year
5
Independent of n Independent of past states
Introduction
Introduction
6
Introduction
7
Transforming a Process into a Markov Chain
8
No!
Transforming a Process into a Markov Chain (con’t)
9
The transition matrix for this Markov chain is: Today & Tomorrow
Yesterday & Today
… rain sun rain sun sun rain sun sun sun rain …
day n
state 2 state 3
day n+1
Example 1. The M/G/1 Queue
10
Example 1. The M/G/1 Queue
11
X(t) arrival
departure
Example 1. The M/G/1 Queue
12
﹏ ﹏ ﹏
Example 1. The M/G/1 Queue
13
Example 1. The M/G/1 Queue
14
Example 2. The G/M/1 Queue
15
﹏ ﹏ ﹏
Example 2. The G/M/1 Queue
16
Example 3. The General Random Walk
17
Example 4. The Simple Random Walk
18
Example 4. The Simple Random Walk (con’t)
19
Example 4. The Simple Random Walk (con’t)
20
Example 4. The Simple Random Walk (con’t)
21
Other Examples
22
Queuing Systems ─ number of customers waiting in line at a bank at the beginning of each
hour ─ number of machines waiting for repair at the end of each day ─ number of patients waiting for a lung transplant at beginning of each
week
Inventory Management ─ number of units in stock at beginning of each week ─ number of backordered units at end of each day
Airline Overbooking ─ number of coach seats reserved at beginning of each day
Finance ─ price of a given security when market closes each day
Other Examples (con’t)
23
Epidemiology ─ number of foot-and-mouth infected cows at the beginning of each day
Population Growth ─ size of US population at end of each year
Genetics ─ genetic makeup of your descendants in each subsequent generation
Workforce Planning ─ number of employees in a firm with each level of experience at the end
of each year
Chapman-Kolmogorov Equations
24
Chapman-Kolmogorov Equations (con’t)
25
Chapman-Kolmogorov Equations (con’t)
26
27
Example. (Weather Forecasting 1)
P { rain-rain-rain + rain-no rain-rain } = 0.7 x 0.7 + 0.3 x 0.4 = 0.61
28
Example. (Weather Forecasting 2)
Example. (Weather Forecasting 2) (con’t)
29
Mon. Tue. Wed. Thur.
Yesterday & Today (state 0)
Today & Tomorrow P
Yesterday & Today
Today & Tomorrow P
rain rain ? rain
The desired probability is given by
30
Classification of States
(reflexive)
(symmetric)
(transient)
31
Classification of States (con’t)
31
0 1 2 3 Transition Diagram => Three classes {0, 1}, {2}, {3}
Classification of States (con’t)
32
0 1 2
• Transient – A state is transient if it is possible to leave the state and never return.
• Periodic – A state is periodic if it is not transient, and if that state is returned to
only on multiples of some positive integer greater than 1. This integer is known as the period of the state.
• Ergodic – A state is ergodic if it is neither transient nor periodic.
• An absorbing state is a state that returns to itself immediately with probability 1
33
Classification of States (con’t)
Tip1: Always find the communicating classes before classifying the states. Tip2: Identify states in the order of the definition: transient states -> periodic states second -> ergodic states.
Tip3: A state that has a loop cannot be periodic
Tip4: If a communicating class is periodic, each state has the same period
Classification of States (con’t)
34
g.c.d: Greatest Common Denominator
35
•No. of communicating classes: 5 { a, b, c } :
{ d} : { e} : { f, g, h, i, j } :
1. f, g, h, i, j form a communicating class. 2. Once you enter, you never leave
3. look at state h
transient transient absorbing, ergodic
periodic w/ d=3
-> not transient.
Once we leave it we will always return in 3 transitions
-> period=3 -> f, g, h, i, j all have period=3 -> f, g, h, i, j are periodic
{ k, l, m} : ergodic
36
A Graph of Period Two. A Non-periodic Graph.
37
Examples of Periodicity
Example: Gambler’s Ruin
Example: Weather Prediction
0 1 2 3
Classification of States (con’t)
38
(Periodicity is a class property)
Classification of States (con’t)
39
Classification of States (con’t)
40
Example:
41
The probability of eventually returning to state 0 is
State 0 is recurrent nonnull or recurrent null ?
the state must be recurrent null.
Thus, state 0 is recurrent.
Classification of States (con’t)
42
Proof (con’t)
43
Classification of States (con’t)
44
=> A finite-state Markov must have at least one recurrent state.
45
Examples of Recurrent and Transient States Example:
Example:
1. The chain is finite
2. All states communicate
Communicating classes: {0,1} {2,3} {4}
1 0 4
1/2 1/2
1/2 1/2
3 2
1/2 1/2
1/2
1/4 1/4
1/2
=> States 0 and 1 are recurrent! Similarly, States 2 and 3 are recurrent!
=> State 4 is transient!
=> At least one recurrent state. => All states are recurrent!
Classification of States (con’t)
46
(Recurrent is a class property)
Example. The Simple Random Walk
47
1 2 0 -1 -2 … …
p p p p
1-p 1-p 1-p 1-p
Example. The Simple Random Walk (con’t)
48
Example. The Simple Random Walk (con’t)
49
Example. The Simple Random Walk (con’t)
50
Classification of States (con’t)
51
Classification of States (con’t)
52
Limit Theorems
53
Limit Theorems
54
Strong Law of Renewal Process
Elementary Renewal Theorem
Blackwell Theorem
Limit Theorems
55
stationary probability, limiting probabilities, equilibrium probabilities, or balance probabilities
Limit Theorems
56
Limit Theorems
57
Limit Theorems
58
Note that for a finite Markov chain, there must be at least one recurrent state
only one class
limiting probabilities equilibrium probabilities
=> There are infinite transient states!
Only one stationary distribution
Proof.
59
(1)
Proof. (con’t)
60
Contradiction!!
balance equations