1

Markov Analysis

Jørn Vatn

NTNU

2

Introduction

Markov analysis is used to model systems which have many different states

These states range from “perfect function” to a total fault state

The migration between the different states may often be described by a so-called Markov-model

The possible transitions between the states may further be described by a Markov diagram

3

Purpose

Markov analysis is well suited for deciding reliability characteristics of a system

Especially the method is well suited for small systems with complicated maintenance strategies

In a Markov analysis the following topics will be of interest Estimating the average time the system is in each state. These

numbers might further form a basis for economic considerations. Estimating how frequent the system in average “visits” the various

states. This information might further be used to estimate the need for spare parts, and maintenance personnel.

Estimate the mean time until the system enters one specific state, for example a critical state.

4

Markov Analysis procedure

1. Make a sketch of the system

2. Define the system states

3. Draw the Markov diagram with the transition rates

4. Quantitative assessment

5. Compilation and presentation of the result from the analysis

5

Make a sketch of the system

Pump system wit active pump and a spare pump in standby

Active pump

Standby pump

6

Definition of system states

x1 = state of active pump

x2 = state of standby pump

1 if component is functioning

0 if component is in a fault statei

ix

i

System state xS

Component state Comments

x1 x2

2 1 1 Both pumps functioning

1 0 1 The active pump is in a fault state, the standby pump is functioning

0 0 0 Both pumps in a fault state

7

State transitions

For this system we have assumed that if the active pump fails, the standby pump could always be started

Further we assume that if both pumps have failed, they will both be repaired before the system is put into service again

The following transition rates are defined1 = failure rate of the active pump2 = failure rate of the standby pump (while running, 2 = 0 in

standby position)1 = repair rate of the active pump (1/1 = Mean Down Time

when the active pump has failed)B = repair rate when both pumps are in a fault state. I.e. we

assume that if the active pump has failed, and a repair with repair rate 1 is started, one will ”start over again” with repair rate B, if the standby pump also fails, independent of “how much” have been repaired on the active pump.

8

Markov state space diagram

The circles represent the system states, and the arrows represent the transition rates between the different system states

The Markov diagram and the description of states represent the total qualitative description of the system

2 1 0

1 2

B1

9

Quantitative assessment

We want to assess the following quantities Average time the system remain in the various

system states The visiting frequencies to each system state

2 1 0

1 2

B1

10

Transition matrix

The indexing starts on 0, and moves to r, e.g. there are r +1 system states

Each cell in the matrix has two indexes,where the first (row index) represent the ”from” state, whereas the second (column index) represent the “to” state.

The cells represent transition rates from one state to another

aij is thus the transition rate from state i to state j The diagonal elements are a kind of ”dummy”-elements,

which are filled in at the end, and shall fulfil the condition that all cells in a row adds up to zero

00 01 0

10 11 1

0 1

r

r

ij

r r rr

a a a

a a a

a

a a a

A

11

Example transition matrix: (From , To )

2 2 1 1

1 1

0 1 2

0 0

1

2 0

B B

A

2 1 0

1 2

B1

12

State probabilities

Let Pi(t) represent the probability that the system is in state i at time t

Now introduce vector notation, i.e. P(t) = [P0(t), P1(t),…,Pr(t)] From the definition of the matrix diagram it might be

shown that the Markov state equations are given by:

P(t) A = d P(t)/d t

These equations may be used to establish both the steady state probabilities, and the time dependent solution

13

Steady state probabilities

Let the vector P = [P0, P1,…,Pr] represent the average time the system is in the various system states in the long time run

For example, P0 is average fraction of the time the system is in state 0, P1 is average fraction of the time the system is in state 1

The elements P = [P0, P1,…,Pr] are also denoted steady state probabilities to indicate that in the stationary situation Pi represents the probability that the system is in state i.

14

The steady state solution

In the long run when the system has stabilized we must have that d P(t)/d t = 0, hence

P A = 0 This system of equations is over-determined, hence we

may delete one column, and replace it with the fact thatP0+ P1+…+Pr = 1

Hence, we have

15

The steady state solution

P A1 = b

where

and

b = [0,0, …,0,1]

00 01

10 111

0 1

1

1

1r r

a a

a a

a a

A

16

Example

which gives

0 1 2 2 2 1

1

0 1

1 0 0 1

0 1

B

P P P

BB

P

)()( 1212

210

BB

BP

)()( 1212

11

BB

BP

)()(

)(

1212

122

17

Numerical solution

To solve the steady state equations P A1 = b is a tedious task

Often we therefore solve these equations by numerical methods

The Markov.xls program does this, where we have to: Define the transition rates Assign numerical values to the transition rates Specify the Markov state space matrix

18

Program for simple Markov analysis

Transition matrixParameter Value Steady state pr. Visit frequencies To 0 To 1 To 2Dim 3 P0 0.000915 v0 3.81317E-05 From 0 -0.04167 0 0.041667Init 2 P1 0.007627 v1 0.000991458 From 1 0.005 -0.13 0.125SystFail 0 P2 0.991458 v2 0.000991458 From 2 0 0.001 -0.001

1 1.00E-03 -0

2 5.00E-03

1 0.125

B 0.0416667

MTTFS 26200.02

Parameter Numeric valuesnames of the parameters

(Give the cells names)

2 2 1 1

1 1

0 1 2

0 0

1

2 0

B B

A

2 1 0

1 2

B1

19

Visiting frequencies

Often we are interested in evaluating how many times the system enters the various states, i.e. the visiting frequencies

The visiting frequency for state j is denoted j, and could be obtained by:

j = -Pjajj

From our example we obtain the “system failure rate”

1 20 0 00

2 1 2 1( ) ( )B

B B

Pa

20

Time dependent solution

Up to now we have investigated the steady state situation In some situations we also want to investigate the time

dependent solution, i.e. the probability that the system is in e.g. state 0 at time t

We now let Pi(t) be the probability that the system is in state i at time t

The time dependent solution may be found by: P(t) A = d P(t)/d t Which could be solved by Laplace methods, or numerical

methods For numerical methods we apply Markov.xls

21

Spare parts – Simple model

Assume a spare part regime where there is only one spare in the stock

Upon a demand (with demand rate ) a new spare is ordered

The intensity of arrival of a new spare is = 1/ MTA (Mean Time to Arrival)

1 0

A

22

Spare parts 2 spares in stock

Assume a spare part regime where there are two spares in the stock

Upon a demand (with demand rate ) a new spare is ordered

The intensity of arrival of a new spares is = 1/ MTA (Mean Time to Arrival) independent of how many in order

2 1 0

0 1 2

0 0

1

2 0

A