SJEM2252 : STOCHASTIC PROCESSES

:

What is Markov Chain?

A stochastic model that describes the probabilities of

transition among the states of a system

Change of state depends probabilistically only on

the current state of the system

Independent of the past given that the present state

is known.

The behaviour depends on the structure of the

transition matrix P

Markov chain modelling is one of the most powerful

tools for analysing complex stochastic systems (Kim

and Smith, 1989)

Markov chain models have become popular in

manpower system planning. Several researchers

have adopted Markov chain models to clarify

manpower policy issues.

Method of Data Collection

• Based on the journal, the staff movement, in

steps, in this system is considered to be an

absorbing chain. A forty year data of staff

movements, as specified by state space

described above, was obtained from the

organization studied.

• Study Area: University of Benin, Nigeria

Table 1 : Original state space

Table 2 : Lumped state space

States 1970-2010 States 1970-2010

Contract 282 Special leave 5

Dismissal 2 Study leave 2

Late 109 Suspension 2

Leave of absence 65 Temporary 617

Left 108 Termination

appointment

3

Permanent 3051 Training leave 63

Resign 12 Visiting appointment 12

Retired 446 Withdrawal 7

Sabatical 49

States Manpower

Retirement 458

Wastage 227

Disciplinary case 463

Leave 191

Staff Stock 3109

Recruitment 911

Total 5359

State space was reduced to six from seventeen as follows:

(i) Recruitment (Rc)

(ii) Staff Stock (Ss)

(iii) Training leave (T)

(iv) Suspension (S)

(v) Wastage (W)

(vi) Retirement (R).

We developed the theory of canonical forms of transition probability

matrix of a finite aperiodic recurrent set decomposable Markov chain

of order n with steady state probability row vector.

By states, it is specifically implied the condition, the status, the

position or situation a Markov Chain (object or staff) undergoes in the

transition process towards the final position of retirement, if the staff

can ever get there.

• Retirement, (R) : is the point where a person stops

employment completely.

• Wastage(W): reduction in size of a workforce by

voluntary resignation.

• Suspension (S): To bar for a period from a privilege,

office, or position, usually as a punishment.

• Training Leave (T): company provide opportunity to

training or outside study.

• Staff stock(Ss) : official employee.

• Recruitment(Rc) : temporary worker.

Markov Transition Matrix:

A square matrix describing the probabilities of moving from one state to

another in a dynamic system.

In each row are the probabilities of moving from the state represented

by that row, to the other states. Thus the rows of a Markov transition

matrix each add to one.

Probabilities were organized in canonical form.

The data obtained appear pure/original although we cannot

guarantee for perfection in the way the records were kept.

The data obtained were used to compute the transition probability

matrix (TPM) which we depict in Table 1. The estimates of the transition

probabilities were based on frequency distributions or tabulations of the

number of transitions from one state to the other in the system

considered. The frequency were converted to TPM by dividing each row

by its total.

It is assumed that after this number of movements, the matrix T would have attained stationarity.

Transition matrix, P is

P =

18607.063501.003901.0004636.009355.0

17751.060581.003722.009022.0008924.0

18588.063436.003897.009447.004632.00

000100

000010

000001

Rc

Ss

T

S

W

R

R W S T Ss Rc

DECOMPOSITION OF A STATE SPACE INTO EQUIVALENCE CLASSES

• We decompose state space into equivalence classes

• States in the same class will have the same characterizing attributes.

• For state i, we let:

C(i) denote the class containing state i.

T(i) be the set of all states that are accessible from i.

F(i) be the set of states from which state I is accessible.

Algorithm:

• Let T(i)={i} and F(i)={Φ}

• For each state k in T(i), add to T(i) all states j such that pkj>0 (if k is not already there). Reiterate.

• Add state j to F(i) if state i is in T(j). Reiterate.

• C(i)= F(i) ∩ T(i).

Transient and Recurrent Classes

• State i is recurrent if i is accessible from every state that is accessible from i.

• If i communicate with j, but j does not communicate with I, then I is transient.

Canonical Form

• By grouping the states together in their classes and list the recurrent classes first, the transition matrix P can be written the following canonical form:

R 0Q T

Absorbing State:

pii=1

Stays forever

Recurrent

Forms a class by itself

Absorbing Chain:

All of its states are either transient or absorbing.

It implies that it is reducible chain

The transition matrix P can be organized into the following canonical form:

𝐈 𝐎𝐐 𝐓

PERIODIC STATES AND PERIODIC CLASS

• A state i has period d if the chain can only revisit it a multiple of d steps later.

• d is the greatest common divisor of the set of integers

• If d=1, then the state has period 1 and is said to be aperiodic. Otherwise, if d>1,the state has period more than 1 and is said to be periodic.

EXPECTED NUMBER OF VISITS TO A TRANSIENT STATE

• Let vij be the total number of visits the chain makes to state j throughout its life, starting from state i.

• Matrix U is called the fundamental matrix.

EXPECTED ABSORPTION TIME

• Let ui be the absorption time or the number of steps before the chain is absorbed into one of the absorbing states, starting from a transient state i.

• The expected absorption :

• In matrix form:

where e is a column vector with all elements equal 1.

ABSORBING PROBABILITIES

• Q - transient to absorbing portion

• T - transient to transient portion

• [UQ]i,j - probability that the chain starts from transient state i, and then moves to the absorbing state j rather than any other absorbing state regardless of the number of steps taken.

LIMITING DISTRIBUTIONS

• Dependent on initial conditions

• After infinite steps, probability of visiting transient state equals to 0

• State space, S = { Retirement,(R) , Wastage,(W) , Suspension(S) ,Training Leave,(T), Staff Stock,(Ss) , Recruitment,(Rc) }

• Transition matrix, P is

P =

18607.063501.003901.0004636.009355.0

17751.060581.003722.009022.0008924.0

18588.063436.003897.009447.004632.00

000100

000010

000001

Rc

Ss

T

S

W

R

R W S T Ss Rc

DECOMPOSITION OF STATE SPACE

State, i T ( i ) F ( i ) C ( i )

R R R, T, Ss, Rc R

W W W, T, Ss, Rc W

S S S T, Ss, Rc S

T T, W, S, Ss, Rc, R T, Ss, Rc T, Ss, Rc

Ss Ss, R, S, T, Rc, W T, Ss, Rc T, Ss, Rc

Rc Rc, R, W, T, Ss, S T, Ss, Rc T, Ss, Rc

• Closed class

I. E1 = { R }

II. E2 = { W }

III. E3 = { S }

• Non-closed class

T = { T, Ss, Rc }

R,W ,S are recurrent states

T, Ss, Rc are transient states

Absorbing state

{ R, W, S }

Transient state

{ T, Ss, Rc }

Absorbing chain

P =

P2 =

18607.063501.003901.0004636.009355.0

17751.060581.003722.009022.0008924.0

18588.063436.003897.009447.004632.00

000100

000010

000001

Rc

Ss

T

S

W

R

15459.015459.052760.006098.005679.016763.0

14749.050334.003092.014839.000995.015991.0

15444.052706.003238.015538.005674.007400.0

000100

000010

000001

R W S T Ss Rc

P3 =

P4 =

12844.043835.002693.011164.006546.022917.0

12254.041820.002569.019672.001822.021862.0

12831.043791.002690.020599.006540.013548.0

000100

000010

000001

10672.036421.002238.015373.007266.028031.0

10181.034746.002135.023688.002509.026741.0

10661.036383.002235.024804.007260.018656.0

000100

000010

000001

P5 =

P6 =

08867.030260.001859.018870.007865.032279.0

08459.028869.001774.027025.003080.030794.0

08858.030229.001857.028298.007857.022901.0

000100

000010

000001

1 0 0 0 0 0

0 1 0 0 0 0

0 0 1 0 0 0

0.26427 0.08354 0.31201 0.01543 0.25116 0.07359

0.34161 0.03555 0.29797 0.01474 0.23986 0.07028

0.35809 0.08362 0.21776 0.01545 0.25142 0.07367

1 0 0 0 0 0

0 1 0 0 0 0

0 0 1 0 0 0

0.29357 0.08767 0.33612 0.01282 0.20868 0.06114

0.36959 0.03949 0.32100 0.01224 0.19928 0.05839

0.38742 0.08775 0.24190 0.01283 0.20889 0.06121

P7 =

P8 =

1 0 0 0 0 0

0 1 0 0 0 0

0 0 1 0 0 0

0.31791 0.09110 0.35616 0.01065 0.17338 0.05080

0.39284 0.04276 0.34013 0.01017 0.16558 0.04852

0.41178 0.09118 0.26196 0.01066 0.17356 0.05085

P9 =

P10 =

1 0 0 0 0 0

0 1 0 0 0 0

0 0 1 0 0 0

0.33813 0.09394 0.37281 0.00885 0.14405 0.04221

0.41216 0.04548 0.35603 0.00845 0.13757 0.04031

0.43203 0.09403 0.27863 0.00886 0.14420 0.04225

1 0 0 0 0 0

0 1 0 0 0 0

0 0 1 0 0 0

0.35494 0.09631 0.38664 0.00735 0.11969 0.03507

0.42820 0.04774 0.36924 0.00702 0.11430 0.03349

0.44885 0.09640 0.29247 0.00736 0.11981 0.03511

For i = 1,2,3,4,5,and 6

Pii(n) > 0 ,

when n = 1,2,…,10,…

Greatest common divisor d is 1. Hence d=1.

That is, all the states are aperiodic state.

Hence, the chain is aperiodic.

EXPECTED NUMBER OF VISIT TO TRANSIENT STATES

U = ( I – T )-1

= 1 0 00 1 00 0 1

−0.03897 0.63436 0.185880.03722 0.60581 0.177510.03901 0.63501 0.18607

-1

𝑇 𝑆𝑠 𝑅𝑐

= 𝑇𝑆𝑠𝑅𝑐

1.23040 3.75028 1.098890.22004 4.58150 1.049430.23064 3.75413 2.10002

If we are currently on training leave,T : We are expected to go for the next training leave 1.23040 steps

later.

We are expected to be official employee 3.75028 steps later

We are expected to be recruited as a temporary worker 1.09889

steps later

If we are now an official employee , Ss : We are expected to go for next training leave 0.22004 steps later.

We are expected to stay as an official employee 4.58150 steps

later

We are expected to be recruited to be as a temporary worker

1.04943 steps later

If we are now recruited, Rc : We are expected to go for next training leave 0.23064 steps later.

We are expected to be an official employee 3.75413 steps later

We are expected to be recruited again 2.10002 steps later

EXPECTED ABSORPTION TIME

ui = Ue

= 1.23040 3.75028 1.098890.22004 4.58150 1.049430.23064 3.75413 2.10002

111

= 𝑇𝑆𝑠𝑅𝑐

6.0795775.8509626.084779

If we are now on training leave (T), we will move

6.079577 number of steps before getting

absorbed into retirement (R),wastage (W) or

suspension (S)

If we are now an official employee (Ss), we will

move 5.850962 number of steps before getting

absorbed into retirement (R),wastage (W) or

suspension (S)

If we are now recruited (Rc), we will move

6.084779 number of steps before getting

absorbed into retirement (R),wastage (W) or

suspension (S)

ABSORBING PROBABILITIES

A= UQ

= 1.23040 3.75028 1.098890.22004 4.58150 1.049430.23064 3.75413 2.10002

0 0.04632 0.094470.08924 0 0.090220.09355 0.04636 0

𝑅 𝑊 𝑆

= 𝑇𝑆𝑠𝑅𝑐

0.43748 0.10794 0.454590.50703 0.05884 0.434130.53148 0.10804 0.36049

If we are currently on training leave,T : We will end up in retirement with probability of 0.43748

We will end up in wastage with probability of 0.10794

We will end up being suspended with probability of 0.45459

If we are now an official employee , Ss : We will end up in retirement with probability of 0.50703

We will end up in wastage with probability of 0.05884

We will end up being suspended with probability of 0.43413

If we are now recruited, Rc : We will end up in retirement with probability of 0.53148

We will end up in wastage with probability of 0.10804

We will end up being suspended with probability of 0.36049

LIMITING

DISTRIBUTIONS

𝑃(𝑘) = 𝐼 0𝐴 𝑇𝑘

𝑃(∞)= 𝐼 0𝐴 0

The limiting probabilities, 𝑃(∞)

𝑃(∞) =

After infinite years, employees must go to Retirement(R ) ,Wastage(W), and Suspension(S)

00036049.010804.053148.0

00043413.005884.050703.0

00045459.010794.043748.0

000100

000010

000001

Rc

Ss

T

S

W

R

R W S T Ss Rc

Arising from the foregoing analysis and discussion, it is evident that the Markov Chain model applied has been able to provide useful insight into the behaviour pattern for a long run manpower policy for the organization studied.

The transition probability matrix, after several sets of transitions, shows the properties of stochastic regularities and aperiodic.The fundamental matrix, U when multiplied with the Q, do, gave us a dependable probability matrix that provides reliable manpower forecast outcomes in line with the theory of Markov Chain.

The results of this study appear useful for understanding the long run manpower policy for a university system.

• A.C. Igboanugo & M.K. Onifade(2011). Markov Chain Analysis of Manpower Data of a Nigerian University. Journal of Innovative Research in Engineering and Science 2, 2(2), 107-123.

• Adriana Irawati Nur Ibrahim. (2013). SJEM2252: Stochastic Processes, week 5-11 notes [pdf].

• Retrieved from http://www.thefreedictionary.com