nonparametric estimation of reliability and survival function for continuous-time finite markov...

Journal of Statistical Planning andInference 133 (2005) 1 – 21

www.elsevier.com/locate/jspi

Nonparametric estimation of reliability and survivalfunction for continuous-time finite Markov

processes

Amr Sadek∗, Nikolaos LimniosUniversité de Technologie de Compiègne, Centre de Recherche de Royallieu, LMAC, BP 20529, 60205

Compiègne cedex, France

Received 3 April 2003; received in revised form 9 May 2003; accepted 9 March 2004

Abstract

In this paper, problems of nonparametric statistical inference for reliability/survival, availabilityand failure rate functions of continuous-time Markov processes are discussed. We assume the statespace to be finite. We shall discuss some results on the maximum likelihood estimator of generators forcontinuous-time Markov processes. The asymptotic properties of this estimator have been discussedand we present the extension of these results by obtaining the asymptotic properties for the estimatorsof transition matrix and of reliability/survival, availability and failure rate functions for a continuous-time Markov process. The confidence intervals for the proposed estimators are given. Finally we givea numerical example to illustrate the results.© 2004 Elsevier B.V. All rights reserved.

MSC:46N30; 47N30; 62G05; 62G20

Keywords:Markov process; Infinitesimal generator; Phase distribution; Reliability; Survival function;Availability; Failure rate; Estimators; Asymptotic properties

1. Introduction

Continuous-time finite state space Markov processes have been used extensively to con-struct stochastic models in a variety of disciplines such as biology, chemistry, electricalengineering, medicine, and physics. The most important problem here is the estimationof reliability and its measurements for these models. Many investigations involve the

∗ Corresponding author. Tel.: +33-344-234499; fax: +33-344-234477E-mail addresses:[email protected](A. Sadek),[email protected](N. Limnios).

0378-3758/$ - see front matter © 2004 Elsevier B.V. All rights reserved.doi:10.1016/j.jspi.2004.03.010

http://www.elsevier.com/locate/jspi

mailto:[email protected]

mailto:[email protected]

2 A. Sadek, N. Limnios / Journal of Statistical Planning and Inference 133 (2005) 1–21

collection of data in the form of lifetimes, for example, of patients suffering from a partic-ular disease, or of electrical components in a piece of machinery. In the analysis of suchdata, we want to find out how the lifetime is affected by various explanatory variables; forexample, age and treatment in the case of disease, materials and operating environment inthe case of machinery. In order to use Markov processes in stochastic modelling of realsystems, we have to estimate first their characteristics.

In the literature concerning statistical inference of stochastic processes, two types ofobservational procedures are discussed. One either observes a single realization over thefixed time interval[0, T ],or observesK >1 independent and identical copies of the process,each over the fixed duration[0, T ].

Corresponding to the two procedures of observation mentioned above, there are twomethods of studying the asymptotic properties of procedures of statistical inference. In thecase when a single realization of the process is observed, the asymptotic properties areobtained as the periodT of observation becomes large. In the second case,T is kept fixedand the numberK of observed realizations is allowed to grow to infinity.

The problems of statistical inference for a Markov process have been discussed by sev-eral authors, cf.Billingsley (1960, 1961), Basawa and Prakasa Rao (1980), Albert (1962),Grenander (1981), andAdke and Manjunath (1984). All previous works have been con-cerned with the statistical inference about the matrix of transition probabilities.Albert(1962)derived the maximum likelihood estimator for the infinitesimal generator of a con-tinuous time, finite state space Markov process and investigated its asymptotic properties.Grenander (1981)discussed also the inference problems for the infinitesimal generator of aMarkov process. Grenander used the transformation of the generator by the Jordan decom-position into diagonal form. So his results are a particular case of Albert’s results when thegenerator matrix has simple eigenvalues.

In Sadek and Limnios (2002), we discussed the asymptotic properties for the maximumlikelihood estimators for reliability, availability and failure rates for a discrete-time Markovprocess. Recently, there have been several works in the field of survival analysis usingMarkov model. For example,Chen and Sen (2001)used a multistate model to estimatethe mean quality-adjusted survival based on the assumption that patient’s health status isMarkovian, which allows the estimator to accommodate a health status process with periodicobservations.Andersen et al. (1993)studied a point process approach for Markov processes.Ouhbi and Limnios (1999)studied the empirical estimators of reliability and failure ratefunctions of a semi-Markov system and investigated their asymptotic properties. It shouldbe mentioned here that a Markov process is a particular case of a semi-Markov process.So we can apply the results given inOuhbi and Limnios (1999)to estimate the reliabilityand the failure rate functions for Markov systems. Nevertheless, we present here a moreadequate and simpler approach for Markov processes. The principal reason is that there is noneed here to estimate the Markov renewal function. We get a direct estimation of transitionprobabilities by the exponential formula. On the other hand, when we use the generatormatrix, the transition rates are constant and it is easier to estimate them. In this paper, wepresent the maximum likelihood estimators for reliability and its measurements. Startingfrom Albert’s results, the asymptotic properties for these estimators are investigated.

All the results given here for the reliability estimator concern equallyphase-typedis-tributions. Thephase-typedistribution is the distribution of the time until absorption in a

A. Sadek, N. Limnios / Journal of Statistical Planning and Inference 133 (2005) 1–21 3

Markov process. For more details about these distributions, see, e.g.,Neuts (1981). Thephase-typedistributions are widely used in the study of stochastic models, (seeCox, 1955;Limnios and Oprisan, 2001). Significant studies of the estimation problem forphase-typedistribution are given byBaum et al. (1970), Bobbio and Telek (1992), Asmussen et al.(1996)andOlsson (1996).

The paper is organized as follows: in the next section, we present the maximum likelihoodestimator of the infinitesimal generator of a continuous-time, finite state space Markov pro-cess. Asymptotic properties for this estimator are discussed. The estimator of the transitionmatrix and its asymptotic properties are discussed in Section 3. In Section 4, we present theprincipal results of this paper which are the asymptotic properties for availability, reliabilityand failure rate estimators of Markov processes. Using the asymptotic results we constructthe confidence intervals for availability, reliability and failure rate. Finally, in Section 5,wegive a numerical example.

2. Preliminaries

Let us give here some auxiliary results which are used in the following sections. Somedefinitions and basic concepts, such as availability, reliability/survival and failure rate func-tions and their maximum likelihood estimators for this model, are given.

2.1. The model

Let {X(t), t�0} be a time-homogeneous continuous-time Markov process with finitestate spaceE = {1,2, . . . , s}, and infinitesimal generator matrixA = (aij )i,j∈E, where

aij�0, i �= j and aii = −ai = −∑k∈Ei �=k

aik. (1)

This process is a regular one, i.e., almost surely the paths are step functions and in any finiteinterval, the number of jumps is finite. We will suppose further thatX is a càdlàg irreducibleprocess.

Let � = (�(i))i∈E be the initial distribution ofX, i.e.,�(i)= P[X(0)= i], i ∈ E and letP(t)=(pij (t)), t�0, be the transition function ofX, i.e., for alli, j ∈ E, t�0 andh�0,

pij (t)= P[X(t)= j |X(0)= i] = P[X(t + h)= j |X(h)= i].In this case, we have, as the solution of the Kolmogorov equation,

P(t)= etA . (2)

Let us denote by 0=T0<T1< · · ·<Tn <Tn+1< · · · the jump times of the process, and letWn := Tn+1 − Tn, n�0 be the sojourn times in successively visited states. The discrete-time process{Xn, n�0}, is the so-called embedded Markov chain (EMC) ofX(t), t�0.That is,

Xn =X(Tn), n�0 and X(t)=XN(t), t�0,

whereN(t)= sup{n�0 : Tn� t} is the number of jumps in the time interval(0, t].


The transition probabilities matrixQ of the EMC{Xn} is

Q(i, j)={ aijai

if i �= j,0 if i = j. (3)

A sample path of the Process{X(t), t�0} in [0, T ] can be represented as an orderedsequence

((X0,W0), (X1,W1), . . . , (XN(T )−1,WN(T )−1), (XN(T ), UT )), UT = T − TN(T ).The path starts at stateX0 at time zero, remains inX0 forW0 units of time, makes a jump toX1, remains inX1 forW1 units of time,. . ., jumps toXN(T )−1, remains there forWN(T )−1units of time and then makes the final jump toXN(T ) and remains there at least untiltimeT .

2.2. Some known results

In this subsection, we present some known results that we need in the rest of this paper.Albert (1962)derived the maximum likelihood estimator ofA and investigated its asymptoticproperties. These results are the continuous-time versions of the results stated byAndersonand Goodman (1957)and byBillingsley (1961)who study the asymptotic behavior of themaximum likelihood estimator of the transition probability matrix of a discrete-time Markovchain.

• Maximum likelihood estimation of the generator.

SupposeK independent sample paths of{X(t), 0� t�T } are observed. LetNij (T ,K) bethe total number of transitions from statei to statej observed duringK trials in the timeinterval [0, T ] and letVi(T ,K) be the total length of time that statei is occupied duringK trials. Then we denote the maximum likelihood estimator for the generator matrixA byA(t; T ,K)= (aij (T ,K))i,j∈E, where

aij (T ,K)=

Nij (T ,K)

Vi(T ,K)if i �= j, Vi(T ,K) �= 0,

−∑l∈E\{i}Nil(T ,K)Vi(T ,K)

if i = j, Vi(T ,K) �= 0,

0 if Vi(T ,K)= 0.

(4)

Remark. If Vi(T ,K)= 0, the maximum likelihood estimator does not exist and we adoptthe convention that

aij (T ,K)= 0 if i �= j andVi(T ,K)= 0. (5)

It is worth noting here thatAlbert (1962)does not mention the estimation of the diagonalcomponents ofA.

• Asymptotic properties.Albert (1962)investigated the asymptotic properties of the vector{aij (T ,K), i, j ∈ E,j �= i} in two ways. A single realization of the processes can be observed over a long period


of time. This corresponds to the investigation ofaij (T ,K), asT → ∞,whileK is fixed. Weshall suppose thatK=1 to simplify calculations. In this case we denoteaij (T ,1) by aij (T ).On the other hand, many independent realizations of the process{X(t),0� t < T } couldbe observed. This corresponds to an investigation of the behavioraij (T ,K), asK → ∞,with T fixed. In the second case, we denoteaij (T ,K) by aij (K). Albert obtained resultspertaining to the consistency and asymptotic normality of these estimators in both cases.We will present these results in the following theorems.

Theorem A (Albert, 1962). For all i, j ∈ E, we havelimT→∞ aij (T )= aij a.s. (6)

and the distribution of the random vector

{T 1/2(aij (T )− aij )}si,j=1,j �=i

is asymptotically normal and independent components with zero mean and covariancematrixH=(H(i, j ; k, l))s(s−1)×s(s−1),where theelementH(i, j ; k, l) is definedas follows:

H(i, j ; k, l)= �(i, j ; k, l) aij �/A(i,i), (7)

where

�(i, j ; k, l)={

1 if i = k, j = l,0 otherwise,

and� is the product of the non-zero eigenvalues ofA andA(i,i) is the(i, i)th cofactor ofA.

Theorem B (Albert, 1962). If there is a positive probability that the ith visited state willbe occupied at some timet�0, then

limK→∞ aij (K)= aij a.s. (8)

and if every state has positive probability of being occupied, then the random vector

{K1/2(aij (K)− aij )}si,j=1,j �=i

is asymptotically normal, as K → ∞, with mean zero and covariance matrixC =(C(i, j ; k, l))s(s−1)×s(s−1), where the elementC(i, j ; k, l) is defined as follows

C(i, j ; k, l)= �(i, j ; k, l)aij /�i (T ), (9)

where�i (T )=∫ T

0 P[X(t)= i] dt.

Remark. The assumption of positive probability occupation is naturally fulfilled when theMarkov process is irreducible (ergodic).


2.3. Reliability estimation and its measurements

In reliability and survival studies the state spaceE is partitioned into two sets:U = {1, . . . , r} is the set of working states andD= {r + 1, . . . , s} is the set of down states(i.e.,E = U ∪ D, U ∩ D = ∅ andU �= ∅, D �= ∅). According to this partition, we willpartition the generator matrixA and the initial distribution� as follows:

A =U D(A11 A12A21 A22

)U

D

, � = [U

�1

D

�2].

Let us denote by1s, r=(1, . . . ,1,0, . . . ,0)� thes-dimensional column vector whoser firstelements are equal to 1 and the others are equal to zero. We note that1r = 1r, r .

The definitions of availability, reliability/survival and failure rate functions for continuous-time Markov processes (see, e.g.,Balakrishnan et al., 2001), can be expressed as follows:

• Availability function: In this work, we discuss the instantaneous availability at timet, A(t). It is the probability that the system is up at momentt, i.e.,

A(t)= P(X(t) ∈ U), t�0.

The instantaneous availability for a Markov process is given by

A(t)= � etA 1s, r , t�0. (10)

We propose the following estimator forA(t):

A(t; T ,K)= � etA(T ,K) 1s, r , t�0. (11)

• Reliability and survival function: The reliabilityR(t) is the probability of no failure in(0, t], givenX(0)= i ∈ U, i.e.,

R(t)= P(∀u ∈ [0, t], X(u) ∈ U), t�0. (12)

The reliability for a Markov process is given by

R(t)= �1 etA11 1r . (13)

The reliability estimator of this system is defined by

R(t; T ,K)= �1 etA11(T ,K) 1r . (14)

Remark. In the case of survival analysis the survival function is usually denoted byS(t)andis defined by (12) and is also expressed by formula (13) as reliability. Thus we will consider


the same estimator (14) for the survival functionS(t). The distribution 1− �1 etA11 1r onR+ is known as a phase type distribution.

• Failure rate function: In the continuous time case, the failure rate function is defined by

�(t)= limh↓0

1

hP(t < T � t + h | T > t),

whereT is the hitting time ofD, i.e.,

T = inf {t�0 : X(t) ∈ D}.This is the so-called lifetime of the system.

The failure rate function for a Markov process is

�(t)=

−

dR

dt (t)

R(t)= −�1 etA11A111r

�1 etA111r, if R(t)>0,

0 if R(t)= 0.

(15)

Let us denote byr (t)= �1 etA11A111r .If we replaceA11 by its estimator we obtain an estimator of the failure rate for a continuous

Markov system which is given by

�(t; T ,K)=− r (t; T ,K)

R(t; T ,K)= − �1 etA11(T ,K)A11(T ,K)1r

�1 etA11(T ,K)1r, if R(t; T ,K)>0,

0 if R(t; T ,K)=0.(16)

Remark. In the proposed estimators, we suppose that the initial distribution is known. Ifthe initial distribution is unknown we can estimate it as follows:

�(i)= Ni(0)

K, Ni(0)=

K∑l=1

1(xl0=i), K�1.

3. Asymptotic properties for transition probabilities matrix estimator

In this section, we propose an estimator of the transition probabilities matrix for acontinuous-time finite state Markov process and we investigate the asymptotic properties(consistency and asymptotic normality) of this estimator in both frameworks, asT → ∞,whileK is fixed, and asK → ∞, with T fixed.

If we replaceA in (2) by its estimator we obtain an estimator of the transition matrix fora continuous Markov process which is given by

P (t; T ,K)= exp{tA(T ,K)}. (17)


We will now investigate the asymptotic properties for the transition matrix estimatorP (t; T ,K). For one trajectory, i.e.K = 1, we write P (t; T ) instead ofP (t; T ,K) and whenT isfixed, we denoteP (t; T ,K) by P (t;K).

Theorem 1. (a)For K = 1, the estimatorpij (t; T ), for all i, j ∈ E, is uniformly stronglyconsistent, asT → ∞, on compact[0, L], for L ∈ R+, in the sense that

sup0<t�L

| pij (t; T )− pij (t) | a.s.→T→∞ 0, (18)

and the random vector

(T 1/2(pij (t; T )− pij (t)))i,j∈Eis asymptotically normal,asT → ∞,with zeromeanandcovariancematrix�(t)=(�(i, j ;k, l; t))s2×s2, where

�(i, j ; k, l; t)=

s∑u,v=1u �=v

([ ∞∑n=1

tn

n!n∑h=1

(−a(h−1)iu a

(n−h)uj + a(h−1)

iu a(n−h)vj )

]

×[ ∞∑n=1

tn

n!n∑h=1

(−a(h−1)ku a

(n−h)ul + a(h−1)

ku a(n−h)vl )

] [ auv�A(u,u)

]), (19)

anda(n)ij is the(i, j) entry of the matrixAn.(b)For fixedT , the estimatorpij (t;K), for all i, j ∈ E, is uniformly strongly consistent

as,K → ∞, on compact[0, L], for L ∈ R+, in the sense that

sup0<t�L

| pij (t;K)− pij (t) | a.s.→K→∞ 0, (20)

and the random vector

(K1/2(pij (t;K)− pij (t)))i,j∈Eis asymptotically normal,asK → ∞,withmean zero and covariancematrix�(t)=(�(i, j ;k, l; t))s2×s2, where

�(i, j ; k, l; t)=s∑

u,v=1u �=v

([ ∞∑n=1

tn

n!n∑h=1

(−a(h−1)iu a

(n−h)uj + a(h−1)

iu a(n−h)vj )

]

×[ ∞∑n=1

tn

n!n∑h=1

(−a(h−1)ku a

(n−h)ul + a(h−1)

ku a(n−h)vl )

][auv/�i (T )]

).

(21)


Proof. Consistency.From the consistency of the generator estimatorsaij (T ,K)whenT →∞ (see relation (6)) and whenK → ∞ (see relation (8)), we obtain the consistency of therandom vector(aij (T ,K))i,j∈E.We can write every transitionpij (t; T ,K) as a continuousfunction of this vector. Then the strong consistency forpij (t; T ) andpij (t;K), are valid.To prove that these estimators are strongly consistent uniformly on compact[0, L], forL ∈ R+, observe thatpij (t; T ,K) is monotone and continuous, so, relations (18) and (20)are valid.Normality. To prove the normality for the transition probability matrix estimator, we

observe that the transition matrix is the exponential function of the generator matrixA. Let

us define the continuous mapping� : Rs(s−1) → Rs2, where�((auv)u,v∈E,u�=v) = etA .

From Theorem A and Theorem B, applying the delta method (see, e.g.,VanDerVaart, 2000,p. 25) we obtain that the random vector(T 1/2(pij (t; T ) − pij (t)))i,j∈E is asymptotically

normal, asT → ∞, with zero mean and covariance matrix� = �′ · H · �′�. In thesame way, the random vector(K1/2(pij (t;K)−pij (t)))i,j∈E is asymptotically normal, as

K → ∞, with zero mean and covariance matrix� = �′ · C · �′�, where

�′ = (�′12, . . . ,�

′1s , . . . ,�

′s1, . . . ,�

′s(s−1))

is ans2 × s(s − 1) matrix.�′uv is as2-column vector whose elements are taken row-wise

reshaped from matrix�′uv, (i.e. rewriting the matrix�

′uv as a column vector), where

�′uv = �etA

�auv=

∞∑n=1

tn

n!n∑h=1

Ah−1 �A�auv

An−h, u �= v. (22)

Denoting bya(n)ij the(i, j) entry of the matrixAn,we can write the(i, j) entry of the matrix

�′uv as follows

�′uv(i, j)=

∞∑n=1

tn

n!n∑h=1

(−a(h−1)iu a

(n−h)uj + a(h−1)

iu a(n−h)vj ).

Then, from relations (7) and (9), we can write the covariance matrix� as in relation (19)and the covariance matrix� as in relation (21).

Remark. Matrix (22) is well defined since‖ ∑nh=1A

h−1 �A�auv

An−h ‖ �2n ‖ A‖n−1,

where‖ A ‖ =maxi∑j |aij |, and so�

′uv is bounded in norm by 2t et‖A‖.

4. Asymptotic properties for reliability and its measurements

Here we present the asymptotic properties of availability, reliability and failure rateestimators, when either we observe a single realization of the process over a long period oftime or we observeK realizations over the same fixed period of time and we letK tend toinfinity.


Theorem 2. The maximum likelihood estimators of the availability, reliability and failurerate of a Markov process are strongly consistent uniformly on compact[0, L], for L ∈ R+,as T tends to infinity. That is, for all t >0, we have:

(i) sup0<t�L

| A(t; T )− A(t) | a.s.→T→∞ 0,

(ii ) sup0<t�L

| R(t; T )− R(t) | a.s.→T→∞ 0,

and,

(iii ) sup0<t�L

| �(t; T )− �(t) | a.s.→T→∞ 0.

Proof. (i) From the definition of availability and its estimator (10) and (11), we get

sup0<t�L

| A(t, T )− A(t) | = sup0<t�L

‖ �P (t, T ) 1s,r − �P(t)1s, r ‖

� ‖ � ‖ sup0<t�L

‖ P (t, T )− P(t) ‖ ‖ 1s, r ‖ .

From (18), we see thatA(t, T ) converges strongly and uniformly toA(t), asT → ∞.(ii) In the same way, on the basis of the definition of reliability and its estimator, we get

the uniform strong consistency for reliability.(iii) To prove the consistency for failure rate, from the definition of failure rate (15) and

its estimator (16) we obtain that

sup0� t�L

| �(t, T )− �(t) |

= sup0� t�L

∣∣∣∣− r (t, T )

R(t, T )+ r (t)R(t)

∣∣∣∣= sup

0� t�L

∣∣∣∣∣ r (t, T )− r (t)

R(t, T )− r (t)R(t, T )− r (t)R(t)

R(t, T )R(t)

∣∣∣∣∣= sup

0� t�L

∣∣∣∣∣ r (t, T )− r (t)

R(t, T )− r (t)R(t)

R(t, T )− R(t)R(t, T )

∣∣∣∣∣� sup

0� t�L

∣∣∣∣ r (t, T )− r (t)

R(t, T )

∣∣∣∣+ sup0� t�L

∣∣∣∣∣ r (t)R(t)

R(t, T )− R(t)R(t, T )

∣∣∣∣∣ . (23)


Now, we get

sup0� t�L

|r (t, T )− r (t)|= sup0� t�L

|�1 etA11(T )A11(T )1r − �1 etA11A111r |

� ||�1|| sup0� t�L

|| etA11(T )A11(T )− etA11A11|| ||1r ||

� ||�1|| sup0� t�L

||(etA11(T ) − etA11)A11

+ etA11(T )(A11(T )− A11)|| ||1r ||� ||�1|| sup

0� t�L||(etA11(T ) − etA11)A11|| ||1r ||

+ ||�1|| sup0� t�L

||etA11(T )(A11(T )− A11)|| ||1r ||.

From the strong consistency of the generator estimatorA(T ) and the uniform strong con-

sistency of etA(T ),we conclude that the right-hand side term of the last equation above con-verges almost surely to zero asT tends to infinity. Then the estimatorr (t, T ) is uniformlystrongly consistent and from the uniform strong consistency for the reliability estimatorR(t, T ) we obtain that both terms on the right-hand side of (23) converge almost surely tozero asT tends to infinity.

In the sequel of this section, we will give asymptotic normality results for the estimatorsA, R, and�.

Theorem 3. For any fixedt�0,√T (A(t, T )−A(t)) converges in distribution, as T tends

to infinity, to a centered at its mean normal random variable with variance2A(t), where

2A(t)=

s∑i,k=1

r∑j,l=1

�(i) �(k)

×

s∑u,v=1u �=v

([ ∞∑n=1

tn

n!n∑h=1

(−a(h−1)iu a

(n−h)uj + a(h−1)

iu a(n−h)vj )

]

×[ ∞∑n=1

tn

n!n∑h=1

(−a(h−1)ku a

(n−h)ul + a(h−1)

ku a(n−h)vl )

] [ auv�A(u,u)

]) . (24)

Proof. From the definition of availability and its estimator in (10) and (11), respectively,we get

√T (A(t, T )− A(t))= √

T

s∑i=1

r∑j=1

�(i)(etA(T )(i, j)− etA(i, j)),

where etA(i, j) is the(i, j) entry of the matrix etA and etA(T )(i, j) is the entry(i, j) of the

matrix etA(T ).


We observe thatA(t) can be written as a continuous function of the vector(pij (t))i,j∈E .

Let us define the continuous mapping : Rs2 → R, by

((pij (t))i,j∈E)=s∑i=1

r∑j=1

�(i)pij (t).

From Theorem 1and the delta method, we obtain that the random variable√T (A(t, T ) −

A(t)) converges in distribution, asT tends to infinity, to a zero mean normal random variablewith variance

2A(t)= ′ · �(t) · ′�, (25)

where′ is the row vector of first order derivatives of with respect topij (t), for alli, j ∈ E, i.e.,

′ = �(1) · · · �(1)︸︷︷︸

r

0 · · · 0︸︷︷︸s−r

, �(2) · · · �(2)︸︷︷︸r

0 · · · 0︸︷︷︸s−r

, . . . , �(s) · · · �(s)︸︷︷︸r

0 · · · 0︸︷︷︸s−r

.

Thus, from relations (25) and (19), we obtain the desired result (24).

Theorem 4. For any fixedt�0,√T (R(t, T )−R(t)) converges in distribution, as T tends

to infinity, to a centered at its expectation normal random variable with variance2R(t),

where

2R(t)=

r∑i,k=1

r∑j,l=1

�(i) �(k)

×

∑

u∈U,v∈Eu �=v

([ ∞∑n=1

tn

n!n∑h=1

(−a(h−1)iu a

(n−h)uj + a(h−1)

iu a(n−h)vj )

]

×[ ∞∑n=1

tn

n!n∑h=1

(−a(h−1)ku a

(n−h)ul + a(h−1)

ku a(n−h)vl )

] [ auv �A(u,u)

] ) . (26)

Proof. We follow the same steps as in the preceding proof, but with the definition ofreliability and its estimator as is given in (13) and (14). We get that the random variable√T (R(t, T )−R(t)) converges in distribution, asT tends to infinity, to a zero mean normal

random variable with variance

2R(t)= ′

1 · �11(t) · ′1�, (27)

where′1 is a row vector of first order derivatives of1 =

r∑i,j=1

�(i)pij (t) with respect to

pij (t) for all i, j ∈ U, and�11(t) is the limit of covariance matrix for the random vector


(T 1/2(pij (t; T ) − pij (t)))i,j∈U , asT tends to infinity. From the proof of Theorem 1, wehave

�11(t)=∑

u∈U,v∈Eu �=v

([ ∞∑n=1

tn

n!n∑h=1

(−a(h−1)iu a

(n−h)uj + a(h−1)

iu a(n−h)vj )

]

×[ ∞∑n=1

tn

n!n∑h=1

(−a(h−1)ku a

(n−h)ul + a(h−1)

ku a(n−h)vl )

] [ auv �A(u,u)

]). (28)

From relations (27) and (28), we get the desired result.

Theorem 5. For any fixedt�0,√T (�(t, T )− �(t)) converges in distribution, as T tends

to infinity, to a centered normal random variable with variance2�(t), defined as follows

2�(t)=

r∑u,v=1u �=v

(Z′uv(t))

2 auv �/A(u,u),

where

Z′uv(t)=

1

R(t)2

×r (t)

r∑i,j=1

�(i)∞∑n=0

tn

n!n∑h=1

(−a(h−1)iu a

(n−h)uj + a(h−1)

iu a(n−h)vj )

−R(t) r∑i,j=1

�(i)r∑k=1

( ∞∑n=0

tn

n!n∑h=1

(−a(h−1)iu a

(n−h)uj + a(h−1)

iu a(n−h)vk )

)akj

.(29)

Proof. From the definition of the failure rate and its estimator in (15) and (16), we have

√T (�(t, T )− �(t))= √

T

(− r (t, T )

R(t, T )+ r (t)R(t)

).

We write− r (t)R(t)

as a function of the row random vector((auv)u,v∈U,u�=v), as follows

Z((auv)u,v∈U)= − r (t)R(t)

= −�1 etA11A11 1r�1 etA11 1r

.

By computing the first order derivatives of this function with respect to the vector((auv)v,v∈U,u�=v), we obtain

Z′(t)= r (t)R′(t)− R(t)r ′(t)R(t)2

,


whereR′(t)= ((R′uv(t))u,v∈U,u �=v) andR′

uv(t) denotes the first derivative ofRwith respectto auv,

R′uv(t)=

�R(t)�auv

=r∑

i,j=1

�(i)(

�etA11

�auv

)ij

=r∑

i,j=1

�(i)∞∑n=0

tn

n!n∑h=1

(−a(h−1)iu a

(n−h)uj + a(h−1)

iu a(n−h)vj ),

and

r ′uv(t)=�r (t)�auv

=r∑

i,j=1

�(i)(

� etA11A11

�auv

)ij

=r∑

i,j=1

�(i)

[r∑k=1

(�etA11

�auv

)ik

akj +r∑k=1

(etA11)ik

(�A11

�auv

)kj

],

where

(�A11

�auv

)ij

={1 if i = u, v = j

−1 if i = j = u= v0 otherwise.

We can easily see that∑ri,j=1�(i) (

∑rk=1(e

tA11)ik(�A11�auv

)kj )= 0. Then we obtainZ′uv(t) as

in relation (29).All the first order derivatives ofZwith respect toauv for all u, v ∈ U, u �= v exist and are

continuous. ThusZ is differentiable and we note from Theorem A that the vector function{T 1/2(auv(T ) − auv)}ru,v=1,u �=v converges, asT tends to infinity, to the centered normaldistribution with covariance matrixH1 = (H(u, v; k, l))r(r−1)×r(r−1), whereH(u, v; k, l)is defined in relation (7). Then, using the delta method, we obtain, asT → ∞,

√T (Z((auv(T ))u,v∈U

u �=v)− Z((aij )u,v∈U

u �=v))

d→N (0, Z′H1Z′�).

Using the same technique, we obtain the consistency and the asymptotic normality for theseestimators when we observe many independent realizations of the process, asK (the numberof realizations) tends to infinity. We present these results in the next theorem. The proofs ofthese results are the same as above, when we use Theorem B instead of Theorem A.

Theorem 6. (i) The maximum likelihood estimators of availability, reliability and failurerate functions of a Markov process are uniformly strongly consistent, as K tends to infinity.


(ii) For any fixed T andt�0,√K(A(t,K)−A(t)) converges in distribution, as K tends

to infinity, to a centered at expectation normal randomvariable with variance2A1(t),where

2A1(t)=

s∑i,k=1

r∑j,l=1

�(i) �(k)

×

s∑u,v=1u �=v

([ ∞∑n=1

tn

n!n∑h=1

(−a(h−1)iu a

(n−h)uj + a(h−1)

iu a(n−h)vj )

]

×[ ∞∑n=1

tn

n!n∑h=1

(−a(h−1)ku a

(n−h)ul + a(h−1)

ku a(n−h)vl )

][auv

�u(T )

]) . (30)

(iii) For any fixed T andt�0,√K(R(t,K)−R(t)) converges in distribution, as K tends to

infinity, to a centered at expectation normal random variable with variance2R1(t), where

2R1(t)=

r∑i,k=1

r∑j,l=1

�(i)�(k)

×

∑

u∈U,v∈Eu �=v

([ ∞∑n=1

tn

n!n∑h=1

(−a(h−1)iu a

(n−h)uj + a(h−1)

iu a(n−h)vj )

]

×[ ∞∑n=1

tn

n!n∑h=1

(−a(h−1)ku a

(n−h)ul + a(h−1)

ku a(n−h)vl )

][auv

�u(T )

]) . (31)

(iv) For any fixed T andt�0,√K(�(t,K)− �(t)) converges in distribution, as K tends to

infinity, to a centered at expectation normal random variable with variance2�1(t), where

2�1(t)=

r∑u,v=1u �=v

(Z′uv(t))

2 auv/�u(T ).

Using the asymptotic results for availability, reliability and failure rate estimators of aMarkov process we can construct confidence intervals for these estimators. In order to obtainthe confidence intervals of availability, we estimate the variance,A(t), by replacing all theelements of the generatorA with their estimators in relation (24). The resulting confidenceintervals at level 100(1 − �)%, � ∈]0,1[ is given by

AT (t)− z�/2 A(t)√T

�A(t)�AT (t)+ z�/2 A(t)√T,

wherez�/2 is given by 1√2�

∫ z�/2−z�/2 e− t2

2 dt = 1 − �.


1 2

3

0.9

1

0.1

1

Fig. 1. Graph of states.

Using the same technique we can obtain confidence intervals of reliability and failurerate.

5. Numerical example

Let us consider a Markov system whose graph of states is given byFig. 1. This system hasthree states: states 1 and 2 are working states and state 3 is a failure state. ThusU = {1,2}andD = {3}. The generator matrixA for this system is the following

A =[−0.02 0.02 0

0.027 −0.03 0.0030.01 0 −0.01

].

The sojourn times in the different states are independent and follow exponential distributionswith parameters 0.02,0.03, and 0.01, and we suppose that the initial distribution is known.The numerical values of transition rates between states are similar to those encountered ina reliability study of production systems.

We generate one sample trajectory of this system in the time interval[0, T ].We use theembedded Markov chain method for the simulation of this system. This method consists


0 500 1000 1500 2000 2500 3000 3500 40000.7

0.75

0.8

0.85

0.9

0.95

1

t

ava

ilabi

lity

true valueML estimator (T=20000)ML estimator (T=15000)ML estimator (T=10000)

Fig. 2. Maximum likelihood estimator of availability.

0 50 100 150 200 250 300 350 400 450 5000.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

t

ava

ilabi

lity

true valueML estimatorC.Interval(90%)C.Interval(80%)

Fig. 3. Confidence intervals for availability at levels 90% and 80%.

simply in using the transition probabilities of the embedded Markov chain in order to findthe next state of the system and in generating the corresponding holding times to the visitedstates by using the exponential distribution functions.

Fig. 2 presents the estimation of availability of this system for differentT = 10000,15000,20000.We remark here the consistency of the availability estimator when the totaltime of observationTbecomes large. It should be mentioned here that the substantial constantbias fort >500 is due to the ergodicity of the system, i.e., limt→∞pij (t) = �j . In Fig. 3,


0 500 1000 1500 2000 2500 3000 3500 40000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

t

rel

iabi

lity


Fig. 4. Maximum likelihood estimator of reliability.

0 500 1000 1500 2000 2500 3000 3500 40000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

t

rel

iabi

lity


Fig. 5. Confidence intervals for reliability at levels 90% and 80%.

we present the confidence intervals at levels 80% and 90%, for the availability when thetotal timeT = 20,000. Figs. 4and6 present the consistency for the reliability estimatorand the failure rate estimator, respectively. InFigs. 5 and 7 we present the confidenceintervals, at levels 80% and 90%, for the reliability and the failure rate when the total timeT = 20,000. We remark that the confidence intervals for these estimators contain all thetrue value.


0 500 1000 1500 2000 2500 3000 3500 40000

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2x 10

3

t

failu

re r

ate


Fig. 6. Maximum likelihood estimator of failure rate.

0 500 1000 1500 2000 2500 3000 3500 40000

0.5

1

1.5

2

2.5

3x 10

3

t

failu

re r

ate


Fig. 7. Confidence intervals for failure rate at levels 90% and 80%.

Let us compare the maximum likelihood estimator for reliability with the usual empiricalestimator. We use the same numerical example for comparing these estimators, but in thiscase we use the stationary distribution to calculate the initial distribution. From the relation,�A= 0 ,we can calculate the stationary distribution�. Then we generate a trajectory using


0 500 1000 1500 2000 2500 3000 3500 40000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

t

rel

iabi

lity

true valueML estimatorempirical estimator (T=20000)

Fig. 8. Reliability estimators comparison.

the following initial distribution,

�(1)= �(1)�(1)+ �(2)

and �(2)= �(2)�(1)+ �(2)

.

From this trajectory, we calculate the maximum likelihood estimator as above and theempirical estimator for reliability.

LetL1, . . . , Ln be the continuous random variables of successive sojourn times in work-ing statesU when transitions are performed from the working state to the failure state.

Then the empirical reliability in this case is defined by

Rn(t)= 1

n

n∑i=1

1{Li>t}, t ∈ R+.

In Fig. 8 we present the comparison between the empirical and maximum likelihood esti-mators of reliability. We notice that the maximum likelihood estimator is closer to the truevalue than the empirical estimator.

References

Adke, S.R., Manjunath, S.M., 1984. An Introduction to Finite Markov Processes. Wiley Eastern Limited, NewYork.

Albert, A., 1962. Estimating the infinitesimal generator of a continuous time finite state Markov process. Ann.Math. Statist. 38, 727–753.

Anderson, A.W., Goodman, L.A., 1957. Statistical inference about Markov chains. Ann. Math. Statist. 28,89–110.


Andersen, P.K., Borgan,]., Gill, R.D., Keiding, N., 1993. Statistical Models Based On Counting Processes.Springer, Berlin.

Asmussen, S., Nerman, O., Olsson, M., 1996. Fitting phase-type distribution via the EM algorithm. Scand. J.Statist. 23, 419–441.

Balakrishnan, N., Limnios, N., Papadopoulos, C., 2001. Basic probabilistic models in reliability. in: Balakrishnan,N., Rao, C.R. (Eds.), Advances in Reliability. Elsevier, Amsterdam, pp. 1–42.

Basawa, I.V., Prakasa Rao, B.L.S., 1980. Statistical inference for Stochastic Processes. Academic Press, London.Baum, L.E., Petrif, T., Soules, G., Weiss, N., 1970. A maximization technique occurring in the statistical analysis

of probabilistic functions of Markov chains. Ann. Math. Staist. 41, 164–171.Billingsley, P., 1960. Statistical Methods in Markov Chains, The Stanford meetings of the Institute of Mathematical

Statistics.Billingsley, P., 1961. Statistical Inference for Markov Processes. The University of Chicago, Chicago Press.Bobbio, A., Telek, M., 1992. Parameter estimation of phase type distributions. Technical Report. Instituto

Elettrotecnico Nazionale Galileo Ferraris, Torino, ItalyChen, P.L., Sen, P.K., 2001. Quality adjusted survival estimation with periodic observation. Biometrics 57,

868–874.Cox, D.R., 1955. A use of complex probabilities in the theory of stochastic processes. Proc. Cambridge Philo.

Soc. 51, 313–319.Grenander, U., 1981. Abstract Inference. Wiley, New York.Limnios, N., Oprisan, G., 2001. Semi-Markov Processes and Reliability. Birkhäuser, Boston.Neuts, M.F., 1981. Matrix-Geometric Solutions in Stochastic Models. The Johns Hopkins University Press,

London.Olsson, M., 1996. Estimation of phase type distributions from censored data. Scand. J. Statist. 23, 443–460.Ouhbi, B., Limnios, N., 1999. Failure rate estimation of Semi-Markov systems. in: Janssen, J., Limnios, N. (Eds.),

Semi-Markov Models and Applications. Kluwer, Dordrecht, pp. 207–218.Sadek, A., Limnios, N., 2002. Asymptotic properties for maximum likelihood estimators for reliability and failure

rates of Markov chains. Commun. Statistics, Theory Methods 31 (10), 1837–1861.Van Der Vaart, A.W., 2000. Asymptotic Statistics. Cambridge Series in Statistical and Probabilistic Mathematics,

Vol. 3. Cambridge University Press, Cambridge.

nonparametric estimation of reliability and survival function for continuous-time finite markov...

Documents