RWTH Aachen University

Chair for Computer Science IISoftware Modeling and VerificationProf. Dr. Ir. Joost-Pieter Katoen

Quantitative Analysis ofMarkov Automata

by

Dennis GuckMatr.Nr: 273138

Master Thesisin Computer Science

1. Reviewer: Prof. Dr. Ir. Joost-Pieter Katoen

2. Reviewer: Prof. Dr. Gerhard Lakemeyer

Statutory Declaration

I hereby declare, that I have created this work independently and used noother sources or tools than the ones listed.

Eidesstattliche Erklarung

Hiermit versichere ich, dass ich die vorliegende Arbeit selbststandig verfasstund keine anderen als die angegebenen Hilfsmittel verwendet habe.

Aachen, June 24, 2012

Dennis Guck

Abstract

Markov automata (MA) are stochastic models in which non-deterministic,probabilistic and timed events co-exists. Hence, they are a combination ofinteractive Markov chains (IMCs) and probabilistic automata (PAs). SinceMAs were introduced recently, the analysis of them is mostly unexplored. Inthis thesis, we elaborate techniques which allows us to check the reliabilityand availability of components in MAs. Therefore, we provide on the onehand an algorithm to compute the expected time of reaching a set of goalstates. On the other hand we provide an algorithm to compute the long-runaverage fraction of time spent in a set of goal states. Further, we prove thecorrectness of our results and show that a stationary deterministic schedulerwill be sufficient to solve the non-determinism, with respect to obtain theminimum and maximum values. Moreover, we have a prototypical tool IMCAwhich supports these algorithms. Moreover, three case studies show thefeasibility and scalability of the algorithms.

Zusammenfassung

Markov Automaten sind stochastische Modelle in denen Nichtdeterminis-mus, probabilistische Distributionen und kontinuierliche Zeit nebeneinanderexistieren. Sie sind somit eine Kombination aus interaktiven Markov Ket-ten (IMCs) und probabilistischen Automaten (PAs). Dadurch dass diesesModel relativ neu ist, sind Analysen dafur noch großtenteils unerforscht.In dieser Arbeit erarbeiten wir Techniken um MAs auf Reliabilitat sowieVerfugbarkeit zu uberprufen. Dazu entwickeln wir Algorithmen zur Berech-nung der expected-time, welche die erwartete Zeit um eine Menge von Ziel-zustanden zu erreichen angibt, sowie der Berechnung der long-run average,welche das Langzeitverhalten auf einer Menge von Zustanden beschreibt. DesWeiteren beweisen wir die Korrektheit und zeigen, dass ein stationarer deter-ministischer Scheduler ausreicht, um den Nichtdeterminismus zu losen. Weit-erhin haben wir die Algorithmen in dem prototypischen Werkzeug IMCA im-plementiert. Dieses nutzen wir in drei Fallstudien, um die Durchfuhrbarkeitund Skalierbarkeit unserer Algorithmen zu demonstrieren.

Contents

1 Introduction 11.1 Contribution of the thesis . . . . . . . . . . . . . . . . . . . . 11.2 Outline of the thesis . . . . . . . . . . . . . . . . . . . . . . . 2

2 Markov Automata 52.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1.1 Markov Automata . . . . . . . . . . . . . . . . . . . . 52.1.2 Semantics . . . . . . . . . . . . . . . . . . . . . . . . . 72.1.3 Properties . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.2 Paths in Markov automata . . . . . . . . . . . . . . . . . . . . 112.2.1 Zenoness and Deadlocks . . . . . . . . . . . . . . . . . 122.2.2 Non-determinism and schedulers . . . . . . . . . . . . . 13

2.3 Uniformization . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3 Expected time 193.1 Expected time analysis . . . . . . . . . . . . . . . . . . . . . . 21

3.1.1 Expected Time . . . . . . . . . . . . . . . . . . . . . . 213.1.2 Minimum expected time . . . . . . . . . . . . . . . . . 223.1.3 Maximum expected time . . . . . . . . . . . . . . . . . 243.1.4 Uniformization and expected time . . . . . . . . . . . . 26

3.2 Expected time computation . . . . . . . . . . . . . . . . . . . 283.2.1 Expected time probabilities . . . . . . . . . . . . . . . 283.2.2 Linear programming . . . . . . . . . . . . . . . . . . . 31

4 Long-run average 354.1 Long-run average . . . . . . . . . . . . . . . . . . . . . . . . . 36

4.1.1 Long-run ratio in MDPs . . . . . . . . . . . . . . . . . 374.2 Long-run average analysis . . . . . . . . . . . . . . . . . . . . 38

4.2.1 Long-run average for unichain MAs . . . . . . . . . . . 394.2.2 Reduction to a stochastic shortest path problem . . . . 424.2.3 Uniformization and long-run average . . . . . . . . . . 47

v

5 IMCA 495.1 What is IMCA? . . . . . . . . . . . . . . . . . . . . . . . . . . 495.2 Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

5.2.1 Storage management . . . . . . . . . . . . . . . . . . . 505.2.2 Input . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515.2.3 Computation . . . . . . . . . . . . . . . . . . . . . . . 535.2.4 Scoop-IMCA tool chain . . . . . . . . . . . . . . . . . . 54

6 Case studies 556.1 Workstation cluster . . . . . . . . . . . . . . . . . . . . . . . . 55

6.1.1 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 576.1.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

6.2 Google file system . . . . . . . . . . . . . . . . . . . . . . . . . 626.2.1 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 646.2.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

6.3 Polling system . . . . . . . . . . . . . . . . . . . . . . . . . . . 676.3.1 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 696.3.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

7 Conclusion 73

vi

Chapter 1

Introduction

System behaviour can be modeled by a variety of probabilistic and stochasticmodels. Therefore, a plethora of models has been proposed which incorpo-rates probability distributions into the classical transition system formalism.We distinguish them along two dimensions. On the one hand between dis-crete and continuous time. Hence, we can measure time either in discreteentities or continuously. Other than discrete-time models, stochastic modelsapply a continuous time notion. That means, transitions are delayed by arandom amount of time, such that the time is governed by a continuous prob-ability distribution. On the other hand we distinguish between determinismand non-determinism. For a deterministic model, the behaviour is fully spec-ified by its deterministic or continuous probability distributions. By contrastnon-deterministic models have to deal with different choices. Thus, theirdeterministic or stochastic behaviour is not completely determined. Thisbehaviour can occur, if for example the modeler can not specify the proba-bility distribution for a specific state, because of insufficient information, andtherefore leaves it unspecified. Another example is the scheduling freedomwhich occurs in randomized distribution systems.

1.1 Contribution of the thesis

In probabilistic automata (PAs) [37, 39] there exists no global notion of time.Therefor concurrent processes may perform random experiments inside atransition. Hence, we have non-deterministic choices between actions whichactivates a transition leading into a probability distribution. Remark thatlabeled transition systems (LTSs) are a subclass of PAs where the transitionsare restricted to Dirac distributions, thus assigning a full probability to onestate.

1

Interactive Markov chains (IMCs) [25, 26] are a natural mixture of con-tinuous time Markov chains (CTMCs) and LTSs. Besides, the CADP toolbox [14] supports compositional generation and minimization of IMCs. Theoccurrence of IMCs have led to several academic and industrial applications[7, 8, 9, 11, 21, 44].

In recent development, a new type of model was introduced: Markov au-tomata (MA) [17, 19]. This new type of model captures a richer behaviourthan IMCs. MAs support non-deterministic probabilistic branching and ex-ponentially distributed delays in continuous time. Hence, they can be seenas a unification of PAs and IMCs. Moreover, they provide a natural se-mantics for a variety of specifications for concurrent systems, as for exampledynamic fault trees (DFTs) [8] and the standardized Architectural Analy-sis and Design Language (AADL) [10]. Moreover, they are able to expressthe complete semantics [19] of generalized stochastic Petri nets (GSPNs) [1]and of stochastic activity networks (SANs) [32]. Thus, MAs are capable ofdescribing most behaviors that are modeled today. Moreover, lately a data-rich specification language for MAs has been introduced [41]. Nevertheless,model checking MAs is an almost unexplored topic thus far.

Our focus is on the quantitative timed analysis of MAs, in particular withfocus on the non-deterministic probability branching. The goal is to pro-vide techniques, which enables reliability and availability analysis on MAs.Therefore, we provide algorithms for the expected time analysis and long-run average fraction of time analysis on MAs. Moreover, we show that theseproblems can be reduced to a non-negative stochastic shortest path (SSP)problem [6]. Besides, we provide a prototypical tool that supports our al-gorithms and is used to illustrate the feasibility of our results. Further, wepresent a case study which requires the full expressiveness of MAs, and hencecould not be faithfully analyzed prior to our work.

1.2 Outline of the thesis

• In Chapter 2 we introduce the preliminaries of MAs and define thesemantics used in this thesis. Further, we define paths in MAs andgive a short overview on schedulers.

• In Chapter 3 we define the expected time reachability on MAs. More-over, we present an algorithm to compute the minimum and maximumexpected time by reducing it to a SSP problem.

• In Chapter 4 we continue our research on the long-run average fractionof time on MAs. This results in splitting the algorithm into three parts.

2

Further, we present that parts of the long-run average computation canbe reduced to a SSP problem.

• In Chapter 5 we present a prototypical tool IMCA that implements thealgorithms presented in Chapter 3 and 4.

• In Chapter 6 we provide three different case studies. The goal is toshow the feasibility and scalability of our algorithms.

• In Chapter 7 we mention our results and some directions for furtherresearch and conclude.

3

4

Chapter 2

Markov Automata

2.1 Preliminaries

2.1.1 Markov Automata

MAs are finite transition systems with on the one hand action-labeled prob-abilistic transitions which lead to a probability distribution over states andon the other hand Markovian transitions which are labeled with a positivereal number identifying the rate of an exponential distribution. MAs strictlyseparate probabilistic from Markovian transitions. Thus, MAs can be seen asan extension of probabilistic automata with exponentially distributed delays.

Definition 1 (Markov automaton) A Markov automaton (MA) is a tu-pleM = (S,Act, −→ ,=⇒, s0) where S is a nonempty finite set of states, withinitial state s0 ∈ S, Act is a finite set of actions,

• −→ ⊆ S × Act×Dist(S) is a set of probabilistic transitions, and

• =⇒ ⊆ S × R>0 × S is a set of Markovian transitions.

Let MT (s) be the set of Markovian transitions of a state s ∈ S and PT (s) itsset of probabilistic transitions, respectively. We define the set of Markovianstates as MS = s ∈ S|MT (s) 6= ∅∧PT (s) = ∅ and the set of probabilisticstates as PS = s ∈ S|PT (s) 6= ∅ ∧MS(s) = ∅. Due to the definition ofMAs, it is also possible that a state has outgoing Markovian and probabilistictransitions. Such states are called hybrid and are contained in the set HS =s ∈ S|MT (s) 6= ∅ ∧ PT (s) 6= ∅. Further, we abbreviate (s, α, µ) ∈ −→by s α−→µ and (s, λ, s′) ∈ =⇒ by s

λ=⇒ s′. Note, that the actions can be

distinguished in external actions Acte and internal actions Acti, where the set

5

s0s1

s2

s3

s5s4 s6

s7 s8 s9

α1, 1α2, 1

2 3

1

α 3, 1

α4

0.6 0.4

1

5

α5, 1

α6 , 1

1

3

1

Figure 2.1: Example Markov automata.

Act = Acte ∪ Acti. This distinction is made, because we can compose MAsvia synchronization over a set of external actions Acte, where the internalactions Acti are not observable from outside. For a detailed discussion aboutparallel composition and bisimulation of MAs we refer to [19]. As our maininterest is the analysis of MAs, we consider closed MAs, i.e. we focus onMAs that are not subject to any further synchronization.

Definition 2 (Closed MA) MA M = (S,Act, −→ ,=⇒, s0) is closed iffActe = ∅.

Thus, we focus on MAs that are obtained as the final outcome of the com-position, where all remaining external actions can safely be hidden. Forsimplicity in developing the theory, we assume w.l.o.g. that each internalaction α ∈ Acti has a unique distribution, denoted µα. Note that this isno restriction. If there exists transitions s α−→µ and s α−→ ν with µ 6= ν, wemay replace them by new transitions s

αµ−−→µ and s αν−−→ ν with fresh internalactions αµ and αν .

Implied models

Markov automata subsume many different concurrent systems [18]. In thefollowing we will shortly describe in which case we obtain which system. Ifwe restrict the probabilistic transitions to Dirac distributions we obtain in-teractive Markov chains (IMCs). We call a distribution Dirac, if thereexists only one successor for each action. If we go further, and restrict thatPT (s) = ∅ for all s ∈ S we obtain continuous-time Markov chains(CTMCs). Alternatively, if MT (s) = ∅ for all s ∈ S we obtain proba-bilistic automata (PAs). In addition, if the distributions are deterministic,

6

which means that for all s ∈ S with s α−→µ1 and s α−→µ2 implies that µ1 = µ2,we obtain Markov decision processes (MDPs). If now additively for alls ∈ S, |Act(s)| = 1 we obtain discrete-time Markov chains (DTMCs)and if also all transitions are Dirac we obtain a labeled transition system(LTS). Hence, the results in this thesis can be applied to all these models.

2.1.2 Semantics

Since we consider closed MAs where internal actions are not subject to inter-action, they can happen immediately, whereas the probability for Markoviantransitions to happen immediately is zero.

Definition 3 (Maximal progress) In any MA, probabilistic transitions la-beled with internal actions take precedence over Markovian transitions.

Therefore, we assume that for each state s ∈ S, s either has only outgoingprobabilistic transitions or only outgoing Markovian transitions. Such statesare called probabilistic and Markovian, respectively. Thus, if we apply themaximal progress assumption on a MA, it holds that HS = ∅. Let ⊥ bea special action, that indicates a Markovian transition. We will denote theset of actions extended with the bottom action ⊥ with Act⊥ = Act ∪ ⊥.We assume at this point, that ⊥ ∈ Act and thus Act = Act⊥. Let Act(s) =α ∈ Act|∃s′ ∈ S.µα(s′) > 0 be the set of enabled actions in s, if s ∈ PSand Act(s) = ⊥ if s ∈ MS. For a Markovian state s ∈ MS we define

R(s, s′) =∑λ|s λ

=⇒ s′ as the rate to move from state s to state s′.Further we define E(s) =

∑s′∈S R(s, s′) as the exit rate of state s. The

discrete branching probability to move from a Markovian state s to s′ isP(s, s′) = R(s,s′)

E(s).

Example 1 Let M be the MA depicted in Figure 2.1. On the one hand,the semantics of a Markovian state equals that of a CTMC state. Consider

the Markovian state s2 and the transition s23

=⇒ s1 that leads from states2 to state s1 with rate λ = 3. Thus, the transition’s delay is exponentiallydistributed with rate λ, which means it expires in the next t ∈ R≥0 time units

with probability∫ t0

3e−3edt = (1−e−3t). Since s2 has two outgoing Markoviantransitions, they compete for execution, and the MA will move along thetransition, whose delay expires first. The sojourn time of a state s ∈ MS isdetermined by the exit rate E(s). In general, the probability to move from astate s ∈ MS to a successor state s′ ∈ S is equal to the probability that the

Markovian transition from s to s′ wins the race. For transition s23

=⇒ s1we have R(s2, s1) = 3, E(s2) = 3 + 1 = 4, and P(s2, s1) = 3

4. On the other

7

hand, the semantics of the probabilistic states equals that of a PA. Considerthe probabilistic state s3 and the transition s3

α4−−→µ4 that leads from state s3with action α4 to distribution µ4 with µ4(s5) = 0.6 and µ4(s6) = 0.4. Thus,if we choose action α4 in state s4 we will reach s5 as successor state withprobability 3

5and s6 as successor with probability 2

5. Note, that we have two

choices in s3, such that we can choose action α3 instead of α4.

2.1.3 Properties

We will define the set of successors according to an action α ∈ Act as:

Post(s, α) =

s′|s α−→µ with µ(s′) > 0 if α ∈ Act \ ⊥s′|s λ

=⇒ s′ if α = ⊥

Note that we don’t always consider the whole MA but rather a fraction ofit. Therefore, we will define sub-MAs.

Definition 4 (Sub-MA) Let M = (S,Act, −→ ,=⇒, s0) be a MA. A sub-MA is a pair (S ′, A) where S ′ ⊆ S with S ′ 6= ∅ and A : S ′ → 2Act is afunction such that:

• ∅ 6= A(s) ⊆ Act(s) for all states s ∈ S ′, and

• s ∈ S ′ and α ∈ A(s) implies Post(s, α) ⊆ S ′.

Hence, a sub-MA is a fraction of the original MA, where the behaviour ofprobabilistic states is equivalent for the actions in A(s). Further, the equiva-lent behaviour for Markovian states is given by the condition, that if a states ∈ MS is in the sub-MA, all of his successors are also contained. Thus,MS ′ ⊆ MS and for all s ∈ MS ′ it holds that E ′(s) = E(s). To simplifysome general description of behaviour we will define the underlying graph-structure of an MA.

Definition 5 (Digraph) Let M = (S,Act⊥, −→ ,=⇒, s0) be a MA. Thecorresponding digraph G induced by M is a pair (V,E) where:

• V = S ∪ 〈s, α〉|s ∈ S ∧ α ∈ Act(s) is the set of vertices, and

• E ⊆ V × V , where si−→〈si, α〉 if α ∈ Act(si) and 〈si, α〉−→ sj if sj ∈Post(si, α) for all si ∈ S.

This reduces the MA to a directed graph, on which we can perform efficientgraph algorithms.

8

Definition 6 (End component) An end component of MA M is a sub-MA (S ′, A) such that the digraph G(S′,A) induced by (S ′, A) is strongly con-nected. We denote the set of end components of M with EC(M).

Example 2 Consider the MA M of Figure 2.1. The sub-MA (S ′, A) withS ′ = s3, s4, s7, A(s3) = α3 and A(s4) = A(s7) = ⊥ is an end compo-nent of M. The corresponding digraph is given by G(S′,A) = (V,E) where

• V = s3, 〈s3, α3〉, s4, 〈s4,⊥〉, s7, 〈s7,⊥〉 and

• E = s3 → 〈s3, α3〉, 〈s3, α3〉 → s4, s4 → 〈s4,⊥〉, 〈s4,⊥〉 → s7, s7 →〈s7,⊥〉, 〈s7,⊥〉 → s3.

Moreover, we can subsume end components to a maximal end component.

Definition 7 (Maximal end components) An end component (S ′, A) ofMA M is called maximal if there is no end component (S ′′, A′) of M suchthat (S ′, A) 6= (S ′′, A′) and S ′ ⊆ S ′′ and A(s) ⊆ A′(s) for all s ∈ S ′. Wedenote the set of maximal end components of M with MEC(M).

Any end component of M is contained in exactly one maximal end com-ponent. This follows from the fact that also the union of end components(S1, A1) and (S2, A2) with (S1, A1) 6= (S2, A2) and S1 ∩ S2 6= ∅ is an endcomponent. Further, the number of maximal end components is boundedabove by the number of states in M.

Example 3 Let M be the MA depicted in Figure 2.1. M has two maximalend components. M1 ∈ MEC(M) is given by (S1, A1) with S1 = s1, s2and A1(s1) = A1(s2) = ⊥. M2 ∈ MEC(M) is given by (S2, A2) with S2 =s3, s4, s5, s6, s7, s8, s9 and A2(s3) = α3, α4, A2(s5) = α5, α6, and A2(s4)= A2(s6) = A2(s7) = A2(s8) = A2(s9) = ⊥. We observe that the end com-ponent from example 2 is contained in M2.

Definition 8 (Bottom strongly connected component)Let M = (S,Act, −→ ,=⇒, s0) and (S ′, A) ∈ MEC(M). (S ′, A) is a bottomstrongly connected component (BSCC) if and only if for all states s ∈ S ′ itapplies that A(s) = Act(s).

Note that the definition of a BSCC is stricter than the definition of a MEC.Thus, it is not possible to leave a BSCC in the overlying MA, whereas for aMEC it is possible that there exists a state s ∈ S ′ with A(s) ⊂ Act(s). Hence,there exists a transition in the overlying MA of MEC that leads out of themaximal end component.

9

s0 s1

s2

α

0.4

0.6

3

1

β, 1

γ, 1

(a) Example MAM.

s0 s1

s2

α

0.4

0.6

4

34

14

β, 1

γ, 1

(b) Induced mlts(M).

Figure 2.2: Example Markov automata and its induced MLTS.

Example 4 Let M be the MA depicted in Figure 2.1. Both maximal endcomponents in example 3 are also BSCCs, since A(s) = Act⊥ for all s ∈ S1

and s ∈ S2.

To study the behaviour of an MA, it can be more sufficient to consider itsderived structure. Therefore, we define Markov labeled transition systems[17].

Definition 9 (Markov labeled transition system) A Markov labeledtransition system (MLTS) is a tuple (S,Act, →) where S is a set of states,Act is a set of actions, and →⊆ S × (Act ∪ R>0)×Dist(S).

Consider a given MA M = (S,Act, −→ ,=⇒, s0), then the correspondingMLTS mlts(M) is given by (S,Act, →) where:

• Act = Act \ ⊥

• for α ∈ Act the transition sα→ µ iff s α−→µ

• for λ ∈ R>0, sλ→ µ iff E(s) = λ and µ(si) =

∑λi · pi|s

λi=⇒ si wherepi = λi

E(s).

Example 5 Let M be the MA depicted in Figure 2.2(a). For states s0, s2 ∈PS the transition relation in the mlts(M) depicted in Figure 2.2(b) is equal toM, whereas for s1 ∈ MS the transition relation has changed. There are two

Markovian transitions, s23

=⇒ s1 and s21

=⇒ s2. Thus, we have E(s1) = 4

and obtain s14→ µ with µ(s0) = 3

4and µ(s2) = 1

4.

10

2.2 Paths in Markov automata

To unify the notion of probabilistic and Markovian transitions we use thebottom action and let σ range over Act. Thereby, we denote a finite path assequence

π = s0σ0,µ0,t0−−−−−→ s1

σ1,µ1,t1−−−−−→ s2σ2,µ2,t2−−−−−→ . . . σn−1,µn−1,tn−1−−−−−−−−−−→ sn

with si ∈ S, σi ∈ Act⊥, µi ∈ Dist(S), and ti ∈ R≥0. The time instant tiis the amount of time spent in state si. The occurrence of a probabilistictransition in state si in π of action α is denoted si

α,µi,0−−−−→ si+1 iff siα−→µi with

µi(si+1) > 0. The occurrence of a Markovian transition after t time units in

state si is denoted si⊥,1,t−−−→ si+1 iff si

λ=⇒ si+1. Let Paths∗ denote the set of

finite paths in MA and Paths the set of infinite paths respectively. Furtherlet π[i] := si denote the (i+1)-st state along the path π. Let π〈i〉 := ti denotethe time spent in state si. Note that for all si ∈ PS we have π〈i〉 = 0. Let|π| = n be the length of a finite path π, where π↓ = sn denotes the last stateof π.

Definition 10 (Elapsed time) For a finite path π of an MA M, the timeelapsed on path π is given by |π|n =

∑n−1i=0 π〈i〉.

Hence, |π|i :=∑i−1

j=0 π〈j〉 denotes the time spent on path π until enteringstate si. Moreover, for t ∈ R≥0, let π@t denote the set of states that πoccupies at time t. Note that π@t is not a single state, but rather a sequenceof states, as an MA can execute probabilistic transitions immediately, andtherefore occupy various states at the same time.

Definition 11 (π@t) Let π@t ∈ (S∗∪Sω) and denote the sequence of statesthat are visited on π during time point t ∈ R≥0. Let i be the smallest index,such that t ≤ |π|i. Then π[i] will be the first state on π that is visited at orafter time t. If such a state don’t exists then let π@t = 〈〉. Otherwise wedistinguish two cases. First, if t < |π|i, then we define π@t = 〈si−1〉. Second,if t = |π|i, then there exists a largest index j such that t = |π|j and we defineπ@t = 〈si . . . sj〉.

Note that we write s ∈ 〈si . . . sj〉 iff s ∈ si, . . . , sj.

Example 6 Consider the path

π = s0 α0,µ0,0−−−−−−−→ s1 α1,µ1,0−−−−−−−→ s2 α2,µ2,0−−−−−−−→ s3 ⊥,1,t3−−−−−−→ s4 α4,µ4,0−−−−−−−→ s5 ⊥,1,t5−−−−−−→ s6 α6,µ6,0−−−−−−−→ s7.

Let 0 < ε < mint3, t5. The derivations for the sequence π@0, π@(t3 −ε), π@(t3), π@(t3 + ε) and π@t5 are depicted in Table 2.1, where X indicateswhere t ≤ |π|i, and × denotes the states where t > |π|i.

11

t ≤ |π|i 0 1 2 3 4 5 6 7 min i max j π@t0 X X X X X X X X 0 3 〈s0s1s2s3〉

t3 − ε × × × × X X X X 4 - 〈s3〉t3 × × × × X X X X 4 5 〈s4s5〉

t3 + ε × × × × × × X X 6 - 〈s5〉t5 × × × × × × X X 6 7 〈s6s7〉

Table 2.1: An example derivation of π@t for Markov automata.

Time-abstract path

The definition of time-abstract paths differs from the previous one such thatwe don’t keep track of time. Thus a time-abstract path is a sequence

π = s0σ0,µ0−−−−→ s1

σ1,µ1−−−−→ s2σ2,µ2−−−−→ . . .

where the definitions are similar to the previous ones. Let Paths∗abs denotethe set of finite time-abstract paths and Pathsabs the set of infinite time-abstract paths respectively. We denote with abs(π) the time-abstract pathof π ∈ Paths∗ and similar for π ∈ Paths.

2.2.1 Zenoness and Deadlocks

Through the combination of probabilistic and Markovian transitions in MAs,there can occur different behaviors regarding time elapsing along a path.

Definition 12 (Zenoness) An infinite path π of an MA M is Zeno ifflimn→∞

∑ni=0 ti 6=∞.

Hence, an infinite path π is non-Zeno whenever the time along its path di-verges to infinity.

Example 7 Let M be the MA depicted in Figure 2.3. We have two infinitepaths

• π1 = s0α0,µ0(s1),0−−−−−−−→ (s1

⊥,1,t1−−−−→ s2α2,µ2(s1),0−−−−−−−→ s1)

ω, and

• π2 = s0α0,µ0(s3),0−−−−−−−→ (s3

α3,µ3(s4),0−−−−−−−→ s4α4,µ4(s3),0−−−−−−−→ s3)

ω.

The time elapsed on path π1 will be ∞, since every time we pass state s1some time will elapse. Thus, π1 is not a Zeno path. On the contrary π2 is aZeno path, since no time elapses at all.

12

s0

s1

s2

s3

s4

2 α2, 1 α3, 1 α4, 1

α0

0.5 0.5

Figure 2.3: MA with Zeno behaviour.

Note that in a finite MA where PS = ∅ and MS 6= ∅, thus only containingMarkovian transitions, the probability that a Zeno path occurs is zero [4].A finite MA M is non-Zeno if and only if no strongly connected componentwith only probabilistic states is reachable from initial state s0. Besides, if astate s ∈ S has no outgoing transitions, i.e. MT (s) = ∅ and PT (s) = ∅,it is a deadlock state. We assume that entering such a state results in atime lock. Thus, we will equip all deadlock states s ∈ S with an internalself-loop s α−→µ such that µ(s) = 1. Note that it is also possible to add aMarkovian transition where time continuous, instead of an internal self loop.In this case, we would have a continuous time flow towards ∞ instead of atime lock. This will not change our result. In the rest of the thesis, we willassume MAs to be deadlock free.

2.2.2 Non-determinism and schedulers

An MA M is non-deterministic iff there exists (s, α, µ), (s, β, ν) ∈ PT (s)with α 6= β and µ 6= ν. Thus, if both internal actions will execute instanta-neously, the distribution to use is not uniquely determined. To resolve thisnon-determinism, we need schedulers. If we reach a non-deterministic statein M along a path π ∈ Paths∗, a scheduler yields a probability distributionover the set of actions enabled in that state.A path π is a concatenation of a state and a sequence of combined transitionsfrom the set Ω = Act×Dist(S)×R≥0×S. Hence, a path π can be displayed asπ = s0m0m1 . . .mn−1 with mi = (σi, µi, ti, si+1) ∈ Ω. For time-abstractpaths πabs we have Ωabs = Act×Dist(S)× S and mi = (σi, µi, si+1) ∈ Ωabs,respectively.

13

Measurable space

We will shortly describe the concepts of measure-theory. A detailed descrip-tion of probability and measure theory can be found in [2]. Furthermore,for an in-depth discussion about probability measures, measurable space andschedulers we refer to [4, 27, 33, 35, 42]. In a MAM, events are measurablesets of paths. These paths are Cartesian products of combined transitions[27]. Therefore, we will define the σ-field as

F = σ(B(R≥0)× FAct × FS)

on subsets of Ω, where B describes the Borel σ-field [23], FAct = 2Act andFS = 2S. We will derive the product σ-field FPathsn = (S0 ×Mo × . . . ×Mn−1|S0 ∈ FS,Mi ∈ F). The cylinder set construction [2] extends thisproperty to infinite paths.

Definition 13 (Generic measurable scheduler) A generic scheduler onan MA M = (S,Act, −→ ,=⇒, s0) is a partial mapping D : Paths∗× FAct →[0, 1] where FAct = 2Act, D(π, ·) ∈ Distr(Act(π↓)) for all π ∈ Paths∗ andπ↓ ∈ PS. D is a generic measurable scheduler (GM-scheduler) iff for allA ∈ FAct , D

−1(A) : Paths∗ → [0, 1] is measurable.

Thus, the measurability condition states that for all A ∈ FAct and B ∈B([0, 1]) we have π|D(π,A) ∈ B ∈ FPaths∗ . This means, that schedulerswhich will resolve the non-determinism in a way that induces non-measurablesets are excluded. Note, that for states s ∈MS no non-determinism occurs.Nevertheless, we assume that if π↓ ∈ MS, that D(π, ·) = ⊥ 7→ 1 so thatD yields a distribution over Act.

Example 8 If state s1 in Figure 2.4 is reached along path π = s0⊥,1,1.5−−−−−→ s1,

then D(π) might yield the distribution α4 7→ 1, whereas for

π′ = s0⊥,1,2.4−−−−−→ s1 the scheduler D(π′) might yield the distribution

α 7→ 0.2, β 7→ 0.8.

Hence, a generic measurable scheduler (GM scheduler)D depends on the timeelapsed on path π as well as on the history of π. We call a GM schedulerdeterministic (GMD) if and only if D(π, ·) is degenerated for all π ∈ Paths∗.Moreover, a GM scheduler is time-abstract if and only if abs(π) = abs(π′)implies D(π, ·) = D(π′, ·). Further, we can distinguish schedulers in twomore classes, in history-dependent and stationary schedulers. The schedulerspreviously described can be summarized in the class of history-dependentschedulers. Thus, schedulers whose decision is based on the path π. However,a stationary scheduler, also called memoryless, only depends on the last state

14

s0 s1

s2

s3

s4

3

1

α

β

0.5

0.5

0.50.5

1

1

1

Figure 2.4: An example MA.

of π. Hence, we will have D : S × FAct → [0, 1]. In this thesis we will usestationary deterministic schedulers. Thus, we will define D : S → Act.

Example 9 LetM be the MA depicted in Figure 2.4. For s0, s2, s3, s4 ∈MSwe will have D(si) = ⊥, since for Markovian states we choose always ⊥. Fors1 ∈ PS there exist two stationary deterministic schedulers: D1(s1) = α andD2(s1) = β.

Stochastic shortest path problem

The stochastic shortest path (SSP) problem, is the problem of finding theminimum cost along a path π, for reaching a set of goal states. We willconsider the SSP problem on Markov decision processes (MDPs).

Definition 14 (MDP) A Markov decision process (MDP) is a tuple P =(S,Act,P, s0), where S,Act and s0 are as before and P : S × Act × S 7→[0, 1] is a transition probability function such that for all s ∈ S and α ∈Act,

∑s′∈S P(s, α, s′) ∈ 0, 1.

A path in an MDP is a sequence

π = s0α0−−→ s1

α1−−→ s2α2−−→ . . .

if and only if si ∈ S and P(si, αi, si+1) > 0 for all i ≥ 0.

Definition 15 (SSP problem) A non-negative stochastic shortest path prob-lem (SSP problem) is a tuple S = (S,Act,P, s0, G, c, g) where (S,Act,P, s0)is an MDP, G ⊆ S is a set of goal states, c : S \ G × Act → R≥0 is a costfunction and g : G→ R≥0 is a terminal cost function.

Hence, for the smallest index k of a path π with sk ∈ G, the accumulatedcosts along π of reaching G is given by

∑k−1j=0 c(sj) + g(sk), denoted CG(π).

Therefore, the minimum expected cost reachability of G starting from s ∈ Sfor the SSP S is given by

cRmin(s,♦G) = infD

∑π∈Pathsabs

CG(π) · Prabss,D(π),

15

where Prabss,D stands for the probability measure on sets of MDP paths thatis induced by scheduler D and initial state s. In [6, 15] it is shown, thatcRmin(s,♦G) can be obtained by solving the following linear program withvariables xss∈S\G:

maximize∑

s∈S\G xs subject to

xs ≤ c(s, α) +∑

s′∈S\G

P(s, α, s′) · xs′ +∑s′∈G

P(s, α, s′) · g(s′)

for each s ∈ S \G and α ∈ Act.

2.3 Uniformization

The exit rate of a Markovian state depends on the rates of its outgoingtransitions. Hence, for MA M and s ∈MS the exit rate is given by E(s) =∑

s′∈S R(s, s′) and can be vary from state to state. For example, during thecomputation of transient probabilities for CTMCs with numerical algorithms,the presence of different exit rates can lead to a numerical instability [3, 4].In the following, we will shortly define uniformity in a Markov automata.

Definition 16 (Uniform Markov automaton) Let MA M = (S,Act,−→ ,=⇒, s0) where S and Act are finite. M is uniform, iff E(s) = e forall s ∈ MS for some e ∈ R>0.

Hence, an MA where the exit rate of all Markovian states are equal is uniform.If the MA is not uniform, it is possible to perform a uniformization.

Definition 17 (Uniformization) Let MA M = (S,Act, −→ ,=⇒, s0) bedefined as before. Let e ∈ R≥0 such that e ≥ maxs∈MS E(s). Then, unif(e,M)is the tuple (S,Act, −→ ,=⇒e, s0) such that for all s ∈ MS where E(s) < ewe add a Markovian self-loop (s, λ, s) with λ = e− E(s).

Hence, we choose the uniformization rate e as the shortest mean residencetime in MAM. Thus, all rates are normalized according to e. Uniformizationchanges the probabilities of the MA for Markovian states in the following way:

P(s, s′) = E(s)e·P(s, s′) if s 6= s′

P(s, s) = E(s)e·P(s, s) + 1− E(s)

eotherwise

16

s0 s1

s2

α

0.4

0.6

2

1

5

(a) Example MAM.

s0 s1

s2

α

0.4

0.6

2

1

2

5

(b) Uniformed MA unif(M, e).

Figure 2.5: Example Markov automata and it’s uniformed MA.

Example 10 Consider the MA in Figure 2.5(a). There are two Markovianstates, s1 and s2 with E(s1) = 3 and E(s2) = 5. Thus, we choose e = 5 toobtain the uniformed MA depicted in Figure 2.5(b). As the uniformizationrate e = E(s2), the transition relation, and thus although the induced proba-bilities stay the same for s2. Besides, for state s1 we have to add a Markovianself-loop with λ = e − E(s1) = 2. The induced probabilities P(s1, s0) = 2

3,

P(s1, s2) = 13

and P(s1, s1) = 0 will change to P(s1, s0) = 25, P(s1, s2) = 1

5

and P(s1, s1) = 25.

17

18

Chapter 3

Expected time

For critical systems the reliability of components is mandatory for correctfunctionality. Further, if the system relies on continuous time, it is not onlyvital to reach the component but rather what time passes until we reach it.A popular way to model i.e. malfunctions of a complex technical system isto use dynamic fault trees (DFTs) [22]. In [9] it is shown that DFTs areindeed IMCs. Thus, it is possible to transform a DFT into an IMC andcompute the minimum and maximum expected time until the componentwill fail. Moreover, with Markov automata we can specify Markovian tran-sitions alongside with probabilistic transitions, instead of only Dirac actiontransitions. Therefore, the expected time computation on MAs depends onthe Markovian transitions, thus on the elapsed time of Markovian states, thenon-deterministic choices in probabilistic states and the probability distribu-tions induced by them.

Zeno behaviour

Note that for a MAM it is possible that there exist Zeno paths, as describedin Section 2.2.1. Thus, MAM has Zeno behaviour if there exists at least onereachable strongly connected component containing only probabilistic states.Therefore, a Zeno path will consists of a cycle that only contains probabilistictransitions. If we reach such a cycle, we can stay in it with no time elapsing.Regardless of that, we define that the amount of time to reach a goal state is∞, if we never leave such a cycle. This decision is reasonable, since althoughno time will elapse, we never reach a goal state. Besides cycles with Zenobehaviour, the MA can also have cycles where time elapses, but no goal stateis included. Thus, if we stay in such a cycle the amount of time to reacha goal state will naturally increases towards ∞. Hence, we will summarizestates in such cycles in a set, denoted Lock . Therefore, we have to distinguish

19

the behaviour of a MA for the minimum and maximum expected time. Incase of the minimum expected time, we will only stay in a such cycles if andonly if there exists no action which will lead to a transition outside the cycle.

Definition 18 (Lockmin(G)) Let MA M = (S,Act⊥, −→ ,=⇒, s0) and G ⊆S a set of goal states. We define the set

Lockmin(G) =⋃

B∈BSCC(M)

B∩G=∅

B .

Hence, all states in a BSCC without a goal state will never reach one. Thus,all states s ∈ Lockmin(G) will have an expected time of ∞ to eventuallyreach a goal state.In case of the maximum expected time, we can define a coarser set of states,since a scheduler D will choose the action, such that we will stay in a cycle,also if there exists an action which will lead to a transition outside the cycle.

Definition 19 (Lockmax (G)) Let MA M = (S,Act⊥, −→ ,=⇒, s0) and G ⊆S a set of goal states. We define the set

Lockmax (G) =⋃

E∈EC(M)

E∩G=∅

E .

Hence, all states in an end component without a goal state will never reachone. Note that Lockmin(G) ⊆ Lockmax (G). Furthermore, for the remain-der of this section we will write Lockmin for Lockmin(G) and Lockmax forLockmax (G).

Example 11 LetM be the MA depicted in Figure 3.1 with G = s3. There

exists a zeno cycle s0α1,1,0−−−−→ s1

α4,1,0−−−−→ s4α5,1,0−−−−→ s0. First off all we consider

the minimum expected time. Therefore Lockmin = ∅. Hence, scheduler Dwill avoid the Zeno cycle and thus D(s0) = α2 and D(s2) = α4. For themaximum expected time we obtain Lockmax = s0 , s1 , s4. Thus, we will stayin the Zeno cycle with D(s0) = α1.

20

s0

s2

s1 s3

s4 s5 s6

α1, 1 α2

0.6

0.4

α4 α5

α 3, 1

α4 , 1

1

3

1

Figure 3.1: Example Markov automata with Zeno paths.

3.1 Expected time analysis

3.1.1 Expected Time

Let M be a MA with state space S and G ⊆ S a set of goal states. Wedefine the extended random variable VG : Paths → R∞≥0 as the elapsedtime until the first visit of some state in G, i.e. for the infinite path π =s0

σ0,µ0,t0−−−−−→ s1σ1,µ1,t1−−−−−→ s2

σ2,µ2,t2−−−−−→ . . ., let VG(π) = mint ∈ R≥0 | G ∩ π@t 6=∅ where min(∅) = +∞.

Definition 20 (Expected time) The expected time to reach G from s ∈ Sfor scheduler D is given by

eTD(s,♦G) = Es,D(VG) =

∫Paths

VG(π) Prs,D

(dπ).

Note that by definition of VG, only the time before entering the first statein G is relevant. We can use this fact, such that we turn all s ∈ G intoabsorbing states, without affecting the expected time reachability. Thus, wewill transform a given MA M in the following way: for all s ∈ G the only

outgoing transition is sλ

=⇒ s for some λ > 0. For the remainder of thischapter, we will assume that MA M has absorbing goal states. Note thatdue to non-determinism we can not compute the expected time to reach aset of goal states. Instead we have to define the minimum and the maximumexpected time.

21

3.1.2 Minimum expected time

Regarding to Definition 20 we can define the minimum expected time asfollowing:

Definition 21 (Minimum expected time) The minimum expected timeto reach G from s ∈ S is given by

eTmin(s,♦G) = infD

Es,D(VG) = infD

∫Paths

VG(π) Prs,D

(dπ).

Hence, we have to find a scheduler D such that we obtain the smallest fractionof time spent on states until reaching G. For simplicity, we define the newsets MS′ = MS \ Lockmin and PS′ = PS \ Lockmin .

Theorem 1 The function eTmin is a fixpoint of the Bellman operator:

[L(v)] (s) =

1

E(s)+∑s′∈S

P(s, s′) · v(s′) if s ∈ MS′ \G

minα∈Act(s)

∑s′∈S

µα(s) · v(s′) if s ∈ PS′ \G

0 if s ∈ G∞ if s ∈ Lockmin .

Theorem 1 implies that we add the sojourn times in all Markovian states be-fore visiting a goal sate. Besides, for all states in PS ′, the non-determinismis resolved by taken the action α ∈ Act that minimizes the expected reacha-bility time from the reachable states induced by distribution µα. Moreover,for all goal states, the expected reachability time is 0, whereas for all statesin Lockmin the expected reachability time is ∞.

Proof. We show that L(eTmin(s,♦G)) = eTmin(s,♦G), for all s ∈ S. There-fore, we will distinguish four cases: s ∈ MS ′ \ G, s ∈ PS ′ \ G, s ∈ G ands ∈ Lockmin .

22

(i) if s ∈MS ′ \G, we derive

eTmin(s,♦G) =

∫D

Es,D(VD) = infD

∫Paths

VG(π) Prs,D

(dπ)

= infD

∫ ∞0

t · E(s)e−E(s)t +∑s∈S

P(s, s′) · Es′,D(s

⊥,1,t−−−−→ ·)(VG)dt

=

∫ ∞0

t · E(s)e−E(s)t +∑s∈S

P(s, s′) · infD

Es′,D(s

⊥,1,t−−−−→ ·)(VG)dt

=

∫ ∞0

t · E(s)e−E(s)t +∑s∈S

P(s, s′) · infD

Es′,D(VG)dt

=

∫ ∞0

t · E(s)e−E(s)tdt+∑s∈S

P(s, s′) · eTmin(s′,♦G)

=1

E(s)+∑s∈S

P(s, s′) · eTmin(s′,♦G)

= L(eTmin(s,♦G)).

(ii) if s ∈ PS ′ \G, we derive

eTmin(s,♦G) = infD

Es,D(VG) = infD

∫Paths

VG(π)(dπ)

= infD

∑s

α,µ,0−−−→ s′

D(s)(α) · Es′,D(s

α,µ,0−−−→ ·)(VG).

Each action α uniquely determines a distribution µα, such that thesuccessor state s′, with s α,µα,0−−−−→ s′, satisfies µα(s′) > 0.

α = arg mins

α−→µα

infD

∑s′∈S

µα(s′) · Es′,D(VG)

Hence, all optimal schedulers must choose α with probability 1, i.e.D(s)(α) = 1 and D(s)(σ) = 0 for all σ 6= α. Thus, we obtain

eTmin(s,♦G) = infD

mins

α−→µα

∑s′∈S

µα(s′) · Es′,D(s

α,µα,0−−−−→ ·)(VG)

= mins

α−→µα

infD

∑s′∈S

µα(s′) · Es′,D(s

α,µα,0−−−−→ ·)(VG)

= mins

α−→µα

infD

∑s′∈S

µα(s′) · Es′,D(VG)

= mins

α−→µα

∑s′∈S

µα(s′) · eTmin(s′,♦G)

= minα∈Act(s)

∑s′∈S

µα(s′) · eTmin(s′,♦G)

= L(eTmin(s,♦G)).

23

(iii) if s ∈ G, we derive

eTmin(s,♦G) = infD

∫Paths

VG(π) Prs,D

(dπ) = 0 = L(eTmin(s,♦G)).

(iv) if s ∈ Lockmin , we derive

eTmin(s,♦G) = infD

∫Paths

VG(π) Prs,D

(dπ) =∞ = L(eTmin(s,♦G)).

Thus, the minimum expected time as defined in Definition 21 is indeed afixpoint of the Bellmann equation in Theorem 1.

3.1.3 Maximum expected time

For the maximum expected time the definitions and proof will be analogous.However, for completeness we will shortly give the proper definitions.

Definition 22 (Maximum expected time) The maximum expected timeto reach G from s ∈ S is given by

eTmax(s,♦G) = supD

Es,D(VG) = supD

∫Paths

VG(π) Prs,D

(dπ).

Thus, in comparison with Definition 21, we search now for the supremumover all schedulers D instead of the infimum. As described in Section 3 thisinfluences the set of Lock states. Thus, we define MS ′′ = MS \Lockmax andPS ′′ = PS \ Lockmax .

Theorem 2 The function eTmax is a fixpoint of the Bellman operator:

[L(v)] (s) =

1

E(s)+∑s′∈S

P(s, s′) · v(s′) if s ∈ MS′′ \G

maxα∈Act(s)

∑s′∈S

µα(s) · v(s′) if s ∈ PS′′ \G

0 if s ∈ G∞ if s ∈ Lockmax .

24

The Bellman operator shows the same behaviour as in Theorem 1, exeptfor states in PS ′′ \ G. For those probabilistic states, the non-determinismis now resolved by taking the action α ∈ Act that maximizes the expectedreachability time from the reachable states induced by distribution µα.

Proof. We show that L(eTmax(s,♦G)) = eTmax(s,♦G), for all s ∈ S. Forthe three cases: s ∈MS ′′ \G, s ∈ G and s ∈ Lockmax . The proof is the sameas for eTmin. We show the case for s ∈ PS ′′ \G.If s ∈ PS ′′ \G, we derive

eTmax(s,♦G) = supD

Es,D(VG) = supD

∫Paths

VG(π)(dπ)

= supD

∑s

α,µ,0−−−→ s′

D(s)(α) · Es′,D(s

α,µ,0−−−→ ·)(VG).

Each action α uniquely determines a distribution µα, such that the successorstate s′, with s α,µα,0−−−−→ s′, satisfies µα(s′) > 0.

α = arg maxs

α−→µα

infD

∑s′∈S

µα(s′) · Es′,D(VG)

Hence, all optimal schedulers must choose α with probability 1, i.e. D(s)(α) =1 and D(s)(σ) = 0 for all σ 6= α. Thus, we obtain

eTmin(s,♦G) = supD

maxs

α−→µα

∑s′∈S

µα(s′) · Es′,D(s

α,µα,0−−−−→ ·)(VG)

= maxs

α−→µα

infD

∑s′∈S

µα(s′) · Es′,D(s

α,µα,0−−−−→ ·)(VG)

= maxs

α−→µα

infD

∑s′∈S

µα(s′) · Es′,D(VG)

= maxs

α−→µα

∑s′∈S

µα(s′) · eTmax(s′,♦G)

= maxα∈Act(s)

∑s′∈S

µα(s′) · eTmax(s′,♦G)

= L(eTmax(s,♦G)).

25

Summarizing, eTmin and eTmax are the unique fixpoint of the Bellmanoperator of Theorem 1 and Theorem 2, respectively. Further, the main differ-ence between the minimum and maximum expected time is the choice of thescheduler D in the probabilistic states. Although, for the maximum expectedtime it is possible to define a coarser set of Lock states.

Example 12 Let M be the MA depicted in Figure 3.1 with G = s3. LetLockmin = ∅ and Lockmax = s0 , s1 , s4 as described in Example 11. First ofall we can make s3 absorbing and remove all outgoing transitions and replace

them with one transition s31

=⇒ s3. Lets consider the maximum expectedtime. Observe that initial state s0 ∈ Lockmax . Thus, there exists a schedulerD such that eTmax(s0, G) =∞. Hence, for the optimal scheduler D it holdsthat D(s0) = α1. If we consider now the minimum expected time, we canobserve that there exists a scheduler D such that we reach s3 from initialstate s0. For the optimal scheduler D with D(s0) = α2 and D(s2) = α4 weobtain eTmin(s0, G) = 0.2.

3.1.4 Uniformization and expected time

As we have introduced uniformization in Section 2.3, we also want to researchthe behaviour of the expected time in an uniformized MA. For this purpose,we first consider the minimum expected time. The uniformization of an MAM has an impact on the sojourn time of states in MS. Nevertheless, thisdoes not contradict that the expected time will be equal.

Theorem 3 For any MA M and its uniformized MA unif(M, e) it holds:

eTminM (s,♦G) = eTminunif(e,M)(s,♦G).

Hence, Theorem 3 states, that despite of the change of the sojourn times forMarkovian states, the minimum expected time will stay the same. Observethat besides of the sojourn time, also the induced probabilities in the MA forMarkovian states will change. Intuitively, those changes will be in balancesuch that the expected time stays the same.

Proof. We show that eTminunif(e,M)(s,♦G) = eTminM (s,♦G). Therefore, we

show for the four cases s ∈ MS ′ \ G, s ∈ PS ′ \ G, s ∈ G and s ∈ Lockmin ,that we obtain the same expected time.

26

(i) if s ∈MS ′ \G, we derive

eTminunif(e,M)(s,♦G) =1

e+∑s′∈S

P(s, s′) · eTminunif(e,M)(s′,♦G)

=1

e+∑

s′∈S\s

P(s, s′) · eTminunif(e,M)(s′,♦G)

+P(s, s) · eTminunif(e,M)(s,♦G)

=1

e+∑

s′∈S\s

E(s)

e·P(s, s′) · eTminunif(e,M)(s

′,♦G)

+

(E(s)

e·P(s, s) + 1− E(s)

e

)· eTminunif(e,M)(s,♦G)

=1

e+∑

s′∈S\s

E(s)

e·P(s, s′) · eTminunif(e,M)(s

′,♦G)

+E(s)

e·P(s, s) · eTminunif(e,M)(s,♦G)

+eTminunif(e,M)(s,♦G)−E(s)

e· eTminunif(e,M)(s,♦G)

⇔ E(s)

e· eTminunif(e,M)(s,♦G) =

1

e+∑

s′∈S\s

E(s)

e·P(s, s′) · eTminunif(e,M)(s

′,♦G)

+E(s)

e·P(s, s) · eTminunif(e,M)(s,♦G)

⇔ eTminunif(e,M)(s,♦G) =1

E(s)+∑

s′∈S\s

P(s, s′) · eTminunif(e,M)(s′,♦G)

+P(s, s) · eTminunif(e,M)(s,♦G)

=1

E(s)+∑s′∈S

P(s, s′) · eTminunif(e,M)(s′,♦G)

=1

E(s)+∑s′∈S

P(s, s′) · eTminM (s′,♦G)

= eTminM (s,♦G)

(ii) if s ∈ PS ′ \G, we derive

eTminunif(e,M)(s,♦G) = minα∈Act(s)

∑s′∈S

µα(s′) · eTminunif(e,M)(s

′,♦G)

= minα∈Act(s)

∑s′∈S

µα(s′) · eTmin(s′,♦G)

= eTmin(s,♦G)

since the uniformization only effect Markovian states and the proba-bilistic states remain equivalent.

27

(iii) if s ∈ G we derive

eTminunif(e,M)(s,♦G) = 0 = eTmin(s,♦G)

since the uniformization has no influence on the expected time, if weare already in a goal state.

(iv) if s ∈ Lockmin we derive

eTminunif(e,M)(s,♦G) =∞ = eTmin(s,♦G)

since the uniformization has no influence on the expected time, if wewill never reach a goal state.

Thus, the uniformization has no influence on the minimum expected time.This is the case, because the uniformization don’t only affects the sojourntime, it also affects the probabilities of the MA for Markovian states.

The results we have obtained for the minimum expected time with respect touniformization can also be applied to the case of maximum expected time.

Theorem 4 For any MA M and its uniformized MA unif(M, e) it holds:

eTmaxM (s,♦G) = eTmaxunif(e,M)(s,♦G).

Note that the proof will be analogous to the previous one and therefore weomit it.

3.2 Expected time computation

3.2.1 Expected time probabilities

Minimum expected time

The characterization of eTmin(s,♦G) in Theorem 1 allows us to reduce theproblem of computing the minimum expected time reachability in an MA toa non-negative SSP problem [6, 15].

28

s0 s1

s2

s3 s4

α 0.4

0.6

1

13

β 0.5

0.5

2

(a) Example MAM.

s0 s1

s2

s3 s4

α 0.4

0.6

⊥, 1

⊥

0.75 0.25

β 0.5

0.5

⊥, 1

(b) SSP SeTmin(M).

Figure 3.2: Example MA and its corresponding SSP.

Definition 23 (SSP for minimum expected time reachability) TheSSP of MAM = (S,Act, −→ ,=⇒, s0) for the minimum expected time reacha-bility of G ⊆ S is given by SeTmin(M) = (S,Act,P, s0, G, c, g) where g(s) = 0for all s ∈ G and

P(s, σ, s′) =

R(s,s′)E(s)

if s ∈MS ∧ σ = ⊥µσ(s′) if s ∈ PS ∧ s σ−→µσ

0 otherwise, and

c(s, σ) =

1

E(s)if s ∈MS ′ \G ∧ σ = ⊥

∞ if s ∈ Lockmin

0 otherwise.

Hence, the probabilistic behaviour of the Markovian states is as explainedin Section 2.1.2. For the probabilistic states, the probabilities are gainedfrom the distribution µα induced by action α ∈ Act. Further, the reward of aMarkovian state is given by its mean residence time, whereas for probabilisticstates will be zero, since no time elapses in those states. Remark, that ifs ∈ Lockmin the reward will be ∞, since for such a state, no shortest path toa goal state exists. Besides, the terminal costs - so the cost for goal states -are set to zero.

Example 13 Let M be the MA depicted in Figure 3.2(a). The probabilitiesfor SSP SeTmin(M) are depicted in Figure 3.2(b). Further, g(s1) = 0 andc(s0, α) = 0, c(s1,⊥) = 0, c(s2,⊥) = 0.25, c(s3, β) = 0 and c(s4,⊥) = 0.5.

29

Theorem 5 (Correctness) For MA M and its induced SSP SeTmin(M) itholds that

eTmin(s,♦G) = cRmin(s,♦G)

where cRmin(s,♦G) denotes the minimal cost reachability of G in SSP SeTmin(M).

Thus, Theorem 5 implies, that we can derive the minimum expected timedefined in Definition 21 by reducing the problem to a SSP problem.

Proof. As specified in Section 2.2.2 and according to [6, 15], cRmin(s,♦G)is the unique fixpoint of the Bellman operator L′ defined as

[L′(v)](s) = minα∈Act(s)

c(s, α) +∑

s′∈S\G

P(s, α, s′) · v(s′) +∑s′∈G

P(s, α, s′) · g(s′).

We show that the Bellman operator L for MA M defined in Theorem 1equals L′ for the induced SSP SeTmin(M). Note that by definition g(s) = 0for all s ∈ G. Thus

[L′(v)](s) = minα∈Act(s)

c(s, α) +∑

s′∈S\G

P(s, α, s′) · v(s′).

Now we distinguish between Markovian, probabilistic, goal and Lockmin states.If s ∈ MS with Act(s) = ⊥ and s ∈ G then c(s,⊥) = 0, and if s ∈ Lockmin

then c(s,⊥) =∞. Moreover if s ∈ PS and s ∈ Lockmin then also c(s, α) =∞for all α ∈ Act(s). Otherwise, there exists a unique distribution µα such thatfor all s′ ∈ S with µα(s′) > 0,P(s, α, s′) = µα(s′). Thus we can rewrite L′ asfollows:

[L′(v)](s) =

c(s,⊥) +∑

s′∈S\G

P(s,⊥, s′) · v(s) if s ∈MS ′ \G

minα∈Act(s)

c(s, α) +∑

s′∈S\G

P(s, α, s′) · v(s′) if s ∈ PS ′ \G

0 if s ∈ G∞ if s ∈ Lockmin

By observing that C(s,⊥) = 1E(s)

if s ∈MS ′\G and c(s, α) = 0 if s ∈ PS ′\Gfor all α ∈ Act(s), we can rewrite L′ to yield the Bellman operator L definedin Theorem 1.

30

Note that only probabilistic states may exhibit non-determinism. We canobserve that from the fixpoint characterization in Theorem 1 it suffices forprobabilistic states to choose the distribution µ which minimizes

∑s′∈S µ(s′)·

v(s′). Further, the Bellman operator L from Theorem 1 yields the minimalcost reachability in SSP SeTmin(M). With the observations we have madeand the fact from [6, 15] that stationary deterministic schedulers suffice toattain the minimum expected cost of an SSP, it yields:

Corollary 1 There exists a stationary deterministic scheduler that yieldseTmin(s,♦G).

Maximum expected time

Note that the previous gained results also can be used for the maximumexpected time. Thus we will build the same SSP as in Definition 23, ex-cept that we use the sets MS ′′ and Lockmax . Observe that we have to findcRmax(s,♦G), which denotes the maximal cost reachability of G in SSPSeTmax(M). Thus, cRmax(s,♦G) is the unique fixpoint of the Bellman op-erator L′′ defined as:

[L′′(v)](s) = maxα∈Act(s)

c(s, α)∑

s′∈S\G

P(s, α, s′) · v(s′) +∑s′∈G

P(s, α, s′) · g(s′).

The correctness proof that eTmax(s,♦G) = cRmax(s,♦G) is analogous toTheorem 5. Remark that the observation from [6, 15] that stationary deter-ministic schedulers suffice to attain the minimum expected cost of an SSP canbe adopted to the maximum expected cost. Thus for the maximum expectedtime it yields:

Corollary 2 There exists a stationary deterministic scheduler that yieldseTmax(s,♦G).

3.2.2 Linear programming

Minimum expected time computation.

In Section 3.2.1 we have shown that we can reduce the minimum expectedtime computation to an SSP problem. The uniqueness of the minimumexpected cost of an SSP [6, 15] now yields:

Corollary 3 eTmin is the unique fixpoint of L defined in Theorem 1.

31

Hence, we will transform the function L from Theorem 1 in the followinginequality system:

max∑s∈S

xs

xsi ≤1

E(si)+∑s′∈S

P(si,⊥, s′) · xs′ if si ∈MS ′ \G

xsi ≤ minα∈Act(si)

∑s′∈S

P(si, α, s′)(s′) · xs′ if si ∈ PS ′ \G

xsi = ∞ if si ∈ Lockmin

xsi = 0 if si ∈ G

Hence, all states s ∈ Lockmin are bounded by∞ and s ∈ G by 0. Further, forstates s ∈ PS ′ \G we choose the minimum expected time over all α ∈ Act(s)and for states in s ∈MS ′ \G we add up the sojourn time in s.

Maximum expected time computation

The previous results can also be applied to the maximum expected time.Hence it yield:

Corollary 4 eTmax is the unique fixpoint of L defined in Theorem 2.

Hence, we will transform the function L from Theorem 2 in the followinginequality system:

min∑s∈S

xs

xsi ≥1

E(si)+∑s′∈S

P(s,⊥, s′) · xs′ if si ∈MS ′′ \G

xsi ≥ maxα∈Act(si)

∑s′∈S

P(si, α, s′) · xs′ if si ∈ PS ′′ \G

xsi = ∞ if si ∈ Lockmax

xsi = 0 if si ∈ G

Example 14 Consider the MA in Figure 3.3. Let G = s4 be the set of goalstates. Thus, we can make s4 absorbing by removing all existing transitions,

in this case s41

=⇒ s3, and replacing them with one self-loop s41

=⇒ s4.In the next step we will compute the Lock sets and obtain Lockmin = ∅and Lockmax = ∅. Thus, for all states in S we eventually always reachG. For the underlying MDP P(M), we have for the Markovian states s0

32

s0

s1

s2

s3

s4

6

44

2

α

β

0.5

0.5

0.8

0.2

α, 1

1

Figure 3.3: Example MA for expected time reachability.

and s1: P(s0,⊥, s1) = 35,P(s0,⊥, s2) = 2

5, E(s0) = 10 and P(s1,⊥, s2) =

23,P(s1,⊥, s3) = 1

3, E(s1) = 6. For the probabilistic states s1 and s3 we

have: P(s2, α, s1) = 12,P(s2, α, s4) = 1

2,P(s2, β, s0) = 4

5,P(s2, α, s4) = 1

5

and P(s3, α, s0) = 1. Now we can build the inequality systems as describedbefore, for eTmin and eTmax. Solving those linear programming problemsyields for initial state s0: eT

min(s0,♦G) = 0.5 and eTmax(s0,♦G) = 1.25.The difference between the minimum and maximum expected time is inducedby the decision which scheduler D makes in state s2. Hence, for the minimumexpected time D(s2) = α and for the maximum expected time D(s2) = β.

33

34

Chapter 4

Long-run average

Alongside with the reliability, the availability of components is an importantquality. Hence, in cyclic systems certain states have to be available at mostof the time. Remark, that the goal of most systems is a high availability. Theproblem we address in this section is to analyze this property for complexsystems where probabilistic branching as well as continuous time are manda-tory. Thus, we provide algorithms such that we can compute the long-runaverage on MAs. Note, that for acyclic MAs the long-run average will be0. For example, if we consider a DFT there exists no cycle, and thus eachstate is only visited at most once. Hence, in an acyclic model we stay almostsurely in one of its absorbing states on the long-run.

Zeno and goal state behaviour

As in Section 3 we have to reconsider the case of Zeno behaviour. Note thatthe long-run average time spent in a probabilistic state is always 0. Thisfollows from the fact, that the choice which is made in a probabilistic state isinstantaneously and therefore the sojourn time is 0 by definition. Hence, wemay assume w.l.o.g. that G ⊆MS. Further, we can not make the goal statesabsorbing as for expected time reachability. In the following we will shortlyrecap Zeno behaviour. In a finite MA M we have Zeno behaviour if thereexists a strongly connected component only containing probabilistic states.By definition, the sojourn time in such a component will be 0. Therefore,we can exclude maximal end components only containing probabilistic statesfrom the Lra computation and directly assign 0 to them. Note that for aMEC with state space S ′ ⊆ S and S ′ ∩ G = ∅ it holds that the long-runaverage time spent on states in G will be 0.

35

4.1 Long-run average

Let M be a MA with state space S and G ⊆ S a set of goal states. We useG as an indicator with G = 1 if s ∈ G and 0 otherwise. The function of timespent in G on an infinite path π inM up to time bound t ∈ R≥0 is given bythe random variable:

AG,t(π) =1

t

∫ t

0

G(π@t)du.

Now we consider the random variable where t→∞:

AG(π) = limt→∞AG,t(π) = lim

t→∞

1

t

∫ t

0

G(π@t)du.

Definition 24 (Long-run average) The long-run average time spent in Gstarting from s ∈ S under scheduler D is given by

LraD(s,G) = Es,D(AG) =

∫Paths

AG(π) Prs,D

(dπ).

Note that we can not just compute the long-run average. Due to non-determinism we have to distinguish between the minimum and maximum.In the remainder of this section we will discuss in detail how to obtain theminimum and maximum long-run average fraction on time spent on statesin G in a MA M with initial state s0. The general idea to determine thelong-run average is divided into three steps:

1. Determine the maximal end components M1, . . . ,Mk of MA M.

2. Determine Lra(G) in maximal end componentsMi for all i ∈ 1, . . . , k.

3. Reduce the computation of Lra(s,G) in MA M to an SSP problem.

The first point can be performed by a graph-based algorithm as for exampleproposed in [15] and recently improved in [12]. The second and third pointinstead will be reduced to solving a linear programming problem.

36

4.1.1 Long-run ratio in MDPs

In this section we will recap long-run ratio objectives in MDPs. The reasonfor this step is, that we later reduce the long-run average computation downto long-run ratio objectives in MDPs.

Definition 25 (Long-run ratio) Let P = (S,Act,P, s0) be a MDP. Fur-ther, let c1, c2 : S×Act→ R≥0 be cost functions. The long-run ratio betweenthe accumulated costs c1 and c2 along the infinite path π in P is defined bythe random variable:

R(π) = limn→∞

∑n−1i=0 c1(s, σ)∑n−1i=0 c2(s, σ)

,

where the cost c1(s, α) is incurred for state s ∈ S when selecting α ∈ Act,and similar for c2.

Thus, our interest in the long-run ratio is the ratio between the accumulatedcosts c1 and c2 along an infinite path π.

Definition 26 (Minimum long-run objective) Let P = (S,Act,P, s0)be a MDP. The minimum long-run ratio objective for state s ∈ S is definedas:

Rmin(s) = infD

Es,D(R) = infD

∑π∈PathsD(s)

abs

R(π) · Prs,D

abs(π).

Thus, we search for a schedulerD such that we obtain the combined minimumlong-run ratio over all induced time abstract paths π starting from s underscheduler D. It follows from [15] that Rmin(s) can be obtained by solvingthe following linear program.

Definition 27 (Linear programming solution) Let P = (S,Act,P, s0)be a MDP and k and xss∈S be real variables.Maximize k subject to:

xs ≤ c1(s, α)− k · c2(s, α) +∑s′∈S

P(s, α, s′) · xs′

for all s ∈ S and α ∈ Act.

Hence, by solving the linear program we will obtain for k the minimum long-run. This is also applicable for the maximum long-run ratio objective.

37

Definition 28 (Maximum long-run objective) Let P = (S,Act,P, s0)be a MDP. The maximum long-run ratio objective for state s ∈ S is definedas:

Rmax(s) = infD

Es,D(R) = supD

∑π∈PathsD(s)

abs

R(π) · Prs,D

abs(π).

Thus, we will adjust the linear programming solution in the following way:Minimize k subject to:

xs ≥ c1(s, α)− k · c2(s, α) +∑s′∈S

P(s, α, s′) · xs′

for all s ∈ S and α ∈ Act.

4.2 Long-run average analysis

We will divide the computation of the long-run average in several steps. Firstof all we will consider the difference between minimum and maximum long-run average. Regarding to Definition 24 we can define the minimum long-runaverage time as following:

Definition 29 (Minimum long-run average) The minimum long-run av-erage time spent in G starting from s ∈ S is given by

Lramin(s,G) = infD

LraD(s,G) = infD

Es,D(AG).

Hence, we have to find a scheduler D such that we obtain the smallest fractionof time spent on states in G. This will result in the following theorem.

Theorem 6 There exists a stationary deterministic scheduler that yieldsLRAmin(s,G).

In comparison, for the maximum long-run average we have to find a schedulerD that yields the longest fraction of time spent on states in G.

Definition 30 (Maximum long-run average) The maximum long-run av-erage time spent in G starting from s ∈ S is given by

Lramax(s,G) = supD

LraD(s,G) = supD

Es,D(AG).

Hence, we have to take the supremum over all schedulers instead of theinfimum.

Theorem 7 There exists a stationary deterministic scheduler that yieldsLRAmax(s,G).

38

s0

s2

s1 s3

s4 s5 s6

α1, 1 α2

0.6

0.4

1 5

α 3, 1

α4 , 1

1

3

1

Figure 4.1: Example unichain Markov automata.

4.2.1 Long-run average for unichain MAs

The first step in computing the long-run average is to consider unichainMAs. Remark that we describe with unichain, that under any stationarydeterministic scheduler D the MAM yields a strongly connected component.

Definition 31 (Unichain MA) An MA M = (S,Act, −→ ,=⇒, s0) isunichain iff every stationary deterministic scheduler D on M induces a sin-gle bottom strongly connected component.

Example 15 Let consider the MA M depicted in Figure 4.1. We haves0, s2 ∈ PS with non-deterministic choices where |Act(s0)| = |Act(s2)| =2. Thus, we have four different combinations for stationary deterministicscheduler D. For D(s0) = α1 and an arbitrary choice for D(s2) we obtainSCC = s0, s1, s4. For D(s0) = α2 and D(s2) = α3 we obtain SCC =s0, s1, s2, s3, s4, s5 and with D(s2) = α4 we obtain SCC = s0, s2, s3, s6.Hence, for all stationary deterministic schedulers D we obtain a stronglyconnected component in M. Therefore, M is unichain.

The reason we consider unichain MAs is based on the fact, that we firstdetermine the long-run average in maximal end components. We will nowreduce the long-run average objective in unichain MAs to the long-run ratioobjective in MDPs defined in Definition 25.

Definition 32 Let M = (S,Act, −→ ,=⇒, s0) be an unichain MA and G ⊆S. The induced MDP is P (M) = (S,Act,P, s0) with cost functions c1, c2 :S × Act→ R≥0, where

P(s, σ, s′) =

R(s,s′)E(s) if s ∈MS ∧ σ = ⊥µσ(s

′) if s ∈ PS ∧ s σ−−→µσ

0 otherwise

c1(s, σ) =

1

E(s)if s ∈MS ∩G ∧ σ = ⊥

0 otherwise,c2(s, σ) =

1

E(s)if s ∈MS ∧ σ = ⊥

0 otherwise.

39

The cost function c2 provides us with the information of the average residencetime in state s ∈ S, whereas cost function c1 only keeps track of states s ∈ G.The long-run ratio between between those two costs c1 and c2 along an infinitepath π is defined in Definition 25.

Theorem 8 For unichain MAM, Lramin(s,G) = Rmin(s) in MDP P (M).

Hence, the computation of the long-run average in an unichain MA can bereduced to the long-run ratio objective for its induced MDP. We will nowadapt the proof from [21] for unichain IMCs to our case.

Proof. Let M be an unichain MA with state space S and G ⊆ S. Weconsider a stationary deterministic scheduler D onM. AsM is unichain, Dwill induce an ergodic CTMC with

R(s, s′) =

∑λ|s λ

=⇒ s′ if s ∈MS

∞ if s ∈ PS ∧ s D(s)−−−→µD(s) ∧ µD(s)(s′) > 0

Hence, for Markovian states the rates remain obtained. In contrary we equipprobabilistic transitions with a rate of∞. First off all,∞ is not characterizedas a negative exponential distribution but rather to model the instantaneousexecution of the probabilistic transition. Note that this will not contradictthe applied results for CTMCs.The long-run average for state s ∈ S and a set of goal states G ⊆ S is givenby

LRAD(s,G) = Es,D(AG(π)) = Es,D(

limt→∞

1

t

∫ t

0

G(Xu)du)

where Xu is the random variable which denotes the state s at time point u.Lets recap the ergodic theorem from [36]:

P(

1

t

∫ t

0

1xs=ids→1

miqias t→∞

)= 1

where mi = Ei(Ti) is the expected return time to state si. Therefore, in ourinduced ergodic CTMC, almost surely

Esi(

limt→∞

1

t

∫ t

0

Gsi(Xu)du)

=1

mi · E(si). (4.1)

Hence, in the long-run almost surely all paths will stay in si for 1mi·E(si)

frac-

tion of time, where we assume that 1∞ = 0.

40

Now we consider the probability µi to stay in si in the long-run in the em-bedded DTMC of our ergodic CTMC, where

P(s, s′) =R(s, s′)

E(s).

Thus µ · P = µ where µ is the vector containing µi for all states si ∈ S.Therefore the expected return time to si is given by

mi =

∑sj∈S µj · E(sj)

−1

µi. (4.2)

Now we can gather those results and obtain:

LRAD(s,G) = Es,D(

limt→∞

1

t

∫ t

0

G(Xu)du)

= Es,D

(limt→∞

1

t

∫ t

0

∑si∈G

Gsi(Xu)du

)

=∑si∈G

Es,D(

limt→∞

1

t

∫ t

0

Gsi(Xu)du)

(4.1)=

∑si∈G

1

mi · E(si)

(4.2)=

∑si∈G

µi∑sj∈S µj · E(sj)−1

· 1

E(si)

=

∑si∈G µi · E(si)

−1∑sj∈S µj · E(sj)−1

=

∑si∈S Gsiµi · E(si)

−1∑sj∈S µj · E(sj)−1

=

∑si∈S µi · (Gsi · E(si)

−1)∑sj∈S µj · E(sj)−1

=

∑si∈S µi · c1(si, D(si))∑sj∈S µj · c2(sj, D(sj))

[16]= Es,D(R)

41

Further, we have a one to one correspondence between the scheduler Dof M and its corresponding MDP P(M). This yields for the minimumlong-run average that LRAmin(s,G) = infD LRA

D(s,G) in MA M equalsRmin(s) = infD Es,D(R) in MDP P(M).

Hence, the minimum long-run average fraction of time spent in a set of goalstates G ⊆ S in unichain MAs can be reduced to the minimum long-runration in MDPs. This result can also be adapted to the maximum long-runaverage.

Theorem 9 For unichain MAM, Lramax(s,G) = Rmax(s) in MDP P (M).

Remark that the proof for Theorem 8 can directly applied to the maximumlong-run average. Thus we obtain that Lramax(s,G) = supDLra

D(s,G) inMA M equals Rmax(s) = supDEs,D(R) in MDP P(M).

Example 16 Let M be the unichain MA depicted in Figure 4.1 with G =s3, s4. For the underlying MDP P(M) we have for s ∈MS:

P(s1,⊥, s4) = 1, c1(s1,⊥) = 0, c2(s1,⊥) = 1,P(s3,⊥, s0) = 1, c1(s3,⊥) = 1, c2(s3,⊥) = 1,P(s4,⊥, s0) = 1, c1(s4,⊥) = 1

5, c2(s4,⊥) = 1

5,

P(s5,⊥, s4) = 1, c1(s5,⊥) = 0, c2(s5,⊥) = 1,P(s6,⊥, s3) = 1, c1(s6,⊥) = 0, c2(s6,⊥) = 1

3.

Therefore, we can solve the linear programming problem for minimum andmaximum long-run ratio objectives and obtain for MA M Lramin(G) = 1

2

and Lramax(G) = 56. Observe, that scheduler D chooses for minimum long-

run average D(s0) = α1 such that we visit only goal state s4. For maximumlong-run average D(s0) = α2 and D(s2) = α4, such that we only visit goalstate s3. In case D(s2) we will visit both goal states, but the fraction of timewe will stay in G would be less than the maximum, but more as the minimum.

4.2.2 Reduction to a stochastic shortest path problem

The last step in the long-run average analysis is to reduce our computationto a stochastic shortest path problem. Note that we divide this section in twoparts. First of all we will present how to obtain Lramin(s0, G) by reducingthe computation to an SSP problem and in the second part we present theadjustment for Lramax(s0, G).

42

s0 s1

s3

s2

s5 s4

2 α

0.6

0.4

α, 1

β, 1

1

3

1

Figure 4.2: Example Markov automata.

Minimum long-run average

Lemma 1 Let M = (S,Act, −→ ,=⇒, s0) be a MA, G ⊆ S a set of goalstates and M1, . . . ,Mk the set of maximal end components in M withstate spaces S1, . . . , Sk ⊆ S. Then

Lramin(s0, G) = infD

k∑i=1

Lramini (G) · PrD(s0 |= ♦Si)

where PrD(s0 |= ♦Si) is the probability to eventually reach and stay instates in Si from initial state s0 under scheduler D and Lramini (G) is thelong-run average fraction of time spent in G ∩ Si in unichain MA Mi.

Proof. We provide a sketch proof for Lemma 1. Let M be a finite MA withmaximal end components M1, . . . ,Mk, G ⊆ S a set of goal states andπ ∈ Paths(M) an infinite path in M. We consider D as a deterministicstationary scheduler. Therefore, we can partition π into two fragments:

πs0s = s0α0,µ0,t0−−−−−→ s1

α1,µ1,t1−−−−−→ . . . αn,µn,tn−−−−−−→ s, and

πωs = s αs,µs,ts−−−−−→ . . . αi,µi,ti−−−−−→ s . . . .

Hence, πs0s is the path starting from s0, which ends in a state s ∈Mi. Fur-ther, all states on path πωs belong to maximal end componentMi. Note thatit is possible that a state on πs0s is part of another maximal end componentMj. Hence, the fact that we eventually reach a maximal end component isnot sufficient, and we have to consider the probability to eventually reach amaximal end component and stay in it. That means, we obtain the minimalLra when the Lra for each MEC Mi is minimal and the combined Lra ofall MECs is minimal according to their persistence under scheduler D.

Example 17 Let M be the MA depicted in Figure 4.2. There are two max-imal end componentsM1 andM2, whereM2 only consists of state s5 where

43

s5 6∈ G. Thus, Lra2(G) = 0. For the other maximal end component itholds: M1 = (S1, A) with S1 = s1, s2, s3, s4, where S1 ∩ G 6= ∅ andA(s2) = A(s4) = ⊥, A(s1) = α and A(s3) = β. We can observe thatAct(s3) = α, β and thus A(s3) 6= Act(s3). Therefore, a stationary deter-ministic scheduler D will choose α in s3 to obtain the minimum long-runaverage. Hence, PrD(s0 |= ♦S1) = 0 and Lramin(s0, G) = 0.

We will now reduce the problem of computing the minimal long-run averageto a non-negative SSP problem [6, 15]. Therefore, we introduce for eachmaximal end component Mi a new state ui and qi. Formally, let U =u1, . . . , uk and Q = q1, . . . , qk be sets of fresh states such that U∩Q∩S =∅. Note that we use the state qi to represent the maximal end componentMi, whereas ui represents a decision state. In more detail, ui contains onetransition which lead into qi and all probabilistic transitions which lead outof Si.

Definition 33 (SSP for minimum long-run average) LetM,P(M),S,Mi and Si be as before. The SSP induced by M for the long-run averagefraction of time spent in G is the tuple SLramin(M) = (S ′, Act′,P′, Q, c, g),

where S ′ = S \⋃ki=1 Si ∪ U ∪Q, Act′ = Act ∪ τ and

P′(s, σ, s′) =

P(s, σ, s′) if s, s′ ∈ S \⋃ki=1 Si∑

s′∈SiP(s, σ, s′) if s ∈ S \

⋃ki=1 Si ∧ s′ = ui ∈ U∑

s∈Si∧s′ 6∈S\SiP(s, σ, s′) if s = ui ∈ U ∧ s′ ∈

⋃ki=1 Si ∪ U ∧ σ = τ

1 if s = ui ∈ U ∧ s′ = qi ∈ Q ∧ σ = τ

1 if s = s′ = qi ∈ Q ∧ σ = ⊥0 otherwise

Further, g(qi) = Lramini (G) for qi ∈ Q and c(s, σ) = 0 for all s and σ ∈ Act′.

Thus, the SSP consists of all states in MA M where each maximal endcomponent Mi is replaced with two states ui and qi. The state qi representthe maximal end componentMi and is equipped with a ⊥-labeled self-loop.Besides, the state ui is equipped with a ⊥-labeled transition which leads intoqi. Further, ui is equipped with all probabilistic transitions which will leadout of the maximal end component Mi in MA M.

Example 18 Let M be the MA depicted in Figure 4.2. As described inExample 17 we have two maximal end componentsM1 andM2. Therefore weintroduce the new sets U = u1, u2 and Q = q1, q2. If we apply Definition33, we obtain the SSP depicted in Figure 4.3 with g(q1) = LRAmin1 (G) andg(q2) = LRAmin2 (G).

44

s0 u1 q1

u2 q2

⊥, 1 ⊥, 1

α, 1

⊥, 1

⊥, 1

⊥, 1

Figure 4.3: Induced SSP for MA in Figure 4.2

The following theorem states the correctness of the reduction.

Theorem 10 (Correctness of the reduction.) For MA M and its in-duced SSP SLramin(M) it holds:

Lramin(s,G) = cRmin(s,♦Q).

Proof. We show that the reduction to the induced SSP problem is correct.

Rmin(s,♦Q) = infD

Es,Dg(XTQ) = infD

k∑i=1

g(XTqi) · PrD(s |= ♦ qi)

= infD

k∑i=1

Lramini (G) · PrD(s |= ♦ qi) (4.3)

= infD

k∑i=1

Lramini (G) · PrD(s |= ♦Si) (4.4)

= Lramin(s,G).

Observe that we change from PrD(s |= ♦ qi) in (4.3) to PrD(s |= ♦Si) in(4.4). This step is legitimate due to Definition 33. Hence, if PrD(s |= ♦ qi) >0, we eventually reach the maximal end component Mi and stay forever init. Otherwise, if PrD(s |= ♦ qi) = 0, scheduler D will choose an action suchthat we leave the maximal end component Mi.

Hence, the minimum long-run average can indeed be reduced to the minimumcost reachability for a SSP.

Maximum long-run average

The reduction to an SSP problem for the maximum long-run average fractionof time in a set of goal states G ⊆ S is analogous to Definition 33.

45

s0

s1

s2

s3s4s5

s6 s7 s8

s9

s10

α1, 1

α2, 1

4 4

α3

0.5

0.5

6

3

γ1

γ220γ3

5

5 44

β1

β2

0.50.5

0.5

0.5

2

Figure 4.4: Example Markov automata with two MECs.

Definition 34 (SSP for maximum long-run average) LetM,P(M),S,Mi and Si be as before. The SSP induced byM for the long-run average frac-tion of time spent in G is the tuple SLramax(M) = (S ′, Act′ ∪ τ,P′, Q, c, g),where S ′, Act′ and P′ are defined as in Definition 33 and g(qi) = Lramaxi (G)for qi ∈ Q and c(s, σ) = 0 for all s and σ ∈ Act.

Hence, the difference between SLramin(M) and SLramax(M) is in the terminalcost function.

Theorem 11 (Correctness of the reduction.) For MA M and its in-duced SSP SLramax(M) it holds:

Lramax(s,G) = cRmax(s,♦Q).

The proof is analogous to the previous one, with the difference that we searchfor the supremum over all schedulers D instead of the infimum.

Example 19 Let M be the MA depicted in Figure 4.4 and let G = s7, s8be the set of goal states. In the first step we determine the maximal endcomponents. M contains two MECs M1 and M2 with state space S1 =s4, s5, s6, s7 and s2 = s3, s8, s9, s10. The second step includes the com-putation of the maximum and minimum long-run average for M1 and M2.Therefore we construct the underlying MDPs P(M1) and P(M2) and solvetheir linear programming problems. Therefore, we obtain Lramin1 (G) = 2

3,

Lramax1 (G) = 45

and Lramin2 (G) = 925,Lramax2 (G) = 5

9. The last step is now

to construct the SSP SLramin(M) and SLramax(M) and compute the long-run average for M. For both SSP we substitute M1 and M2 with freshstates u1, q1 and u2, q2 where g(q1) = 2

3, g(q2) = 9

25for SLramin(M) and

g(q1) = 45, g(q2) = 5

9for SLramax(M). Finally, by solving the linear pro-

gramming problem for both SSPs we obtain Lramin(s0, G) = 925

by choosingα2 in state s0 and β1 in state s9 and Lramin(s0, G) = 61

90by choosing α1 in

state s0, β2 in state s9 and γ2 in state s4.

46

4.2.3 Uniformization and long-run average

As in section 3.1.4, we want to research the behaviour of the long-run averagein an uniformized MA. For this purpose we first show that the result holdsfor unichain MAs and then for the SSP reduction.

Theorem 12 For any unichain MAM and its uniformized MA unif(e,M)it holds:

LraminM (G) = Lraminunif(e,M)(G).

Proof. We will divide the proof in three cases, where we consider states inMS \G, PS and in G. Observe that we have shown in Theorem 8 that thecomputation of the long-run average for unichain MAs can be reduced tothe long-run objective for its induced MDP. Thus, we will show through thelinear programming solution defined in Definition 27 that the uniformizationwill not influence the long-run average.Case 1: s ∈ G

xs ≤ c1(s, α)− k · c2(s, α) +∑s′∈S

P(s, α, s′) · xs′

≤ 1

e− k · 1

e+∑s′∈S\s

E(s)

e·P(s, α, s′) · xs′

+E(s)

e·P(s, α, s) · xs + xs −

E(s)

e· xs

E(s)

exs ≤

1

e− k · 1

e+E(s)

e·∑s′∈S\s

P(s, α, s′) · xs′ + E(s)

e·P(s, α, s) · xs

≤ 1

e− k · 1

e+E(s)

e·∑s′∈S

P(s, α, s′) · xs′

xs ≤1

E(s)− k · 1

E(s)+∑s′∈S

P(s, α, s′) · xs′

≤ c1(s, α)− k · c2(s, α) +∑s′∈S

P(s, α, s′) · xs′

Case 2: s ∈ MS \GThe proof is analogue to case 1, regarding that c1(s, α) = 0 = c1(s, α).Case 3: s ∈ PSThe proof is analogue to case 1, regarding that c1(s, α) = 0 = c1(s, α) andc2(s, α) = 0 = c2(s, α).

Therefore, the minimum long-run average for unichain MAs is equal in case

47

of uniformization. Remark, that this holds also for the maximum long-runaverage.

Theorem 13 For any MA M and its uniformized MA unif(e,M) it holds:

LraminM (s0, G) = Lraminunif(e,M)(s0, G).

Proof. The computation of the minimum long-run average for MA M andunif(e,M) can be reduced to a stochastic shortest path problem as describedin the previous section. Remark that the SSP problem can also be solvedby a linear programming problem. Therefore, we show that the linear pro-gramming solution for the SSP problem of M is identical to the solution ofunif(e,M). We only show the case for s ∈MS.

xs ≤∑

s′∈S\G

P(s, α, s′) · xs′ +∑s′∈G

P(s, α, s′) · g(s′)

xs ≤∑

s′∈S\G\s

P(s, α, s′) · xs′ +P(s, α, s) · xs +∑s′∈G

P(s, α, s′) · g(s′)

Remark that if s ∈ G, this implies s is a probabilistic state with a self-loopregarding to Definition 33.

xs ≤∑

s′∈S\G\s

E(s)

eP(s, α, s′) · xs′ +

(E(s)

eP(s, α, s) + 1− E(s)

e

)· xs

+∑s′∈G

E(s)

eP(s, α, s′) · g(s′)

E(s)

exs ≤

E(s)

e

∑s′∈S\G\s

P(s, α, s′) · xs′ +E(s)

eP(s, α, s) · xs

+E(s)

e

∑s′∈G

P(s, α, s′) · g(s′)

xs ≤∑

s′∈S\G\s

P(s, α, s′) · xs′ +P(s, α, s) · xs +∑s′∈G

P(s, α, s′) · g(s′)

xs ≤∑

s′∈S\G

P(s, α, s′) · xs′ +∑s′∈G

P(s, α, s′) · g(s′)

For all other cases the proof is trivial and will be omitted. Hence, the SSPsolution for Lraminunif(e,M)(s0, G) is equal to LraminM (s0, G).

Remark, that this holds also for the maximum long-run average.

48

Chapter 5

IMCA

Since we have elaborated novel algorithms for Markov automata, we are alsointerested in implement them into a tool. Therefore, we will extend theexisting prototypical tool IMCA (Interactive Markov Chain Analyzer). Inthe following section we give a small overview what IMCA provides and howit works.

5.1 What is IMCA?

IMCA is developed as a prototypical tool for the quantitative analysis ofIMCs. The functionality of the previous IMCA version is depicted in Fig-ure 5.1. Summarized, it is possible to compute time- and interval-boundedreachability probabilities, expected-time and steps and the long-run average.Further, it is possible to generate a scheduler, with the drawback of a decel-erated computation. It is also possible to minimize the input model with asimple bisimulation minimization, based on the Kanellakis-Smolka algorithm[28].

Our goal was to extend this tool with support of Markov automata andthe presented algorithms. Since IMCs are basically MAs with the restrictionthat there only exists Dirac transitions for states in PS, it is possible to cre-ate one basic model structure for MAs and IMCs. Thus, IMCA was not justextended with a new model, but rather build new. This means, that some ofthe previous functionalities are not present in the current version of IMCA,depicted in Figure 5.5. Thus, this version currently does not support thegeneration of schedulers, a minimization algorithm and time- and interval-bounded reachability probabilities.

49

Figure 5.1: Functionality of IMCA for IMCs

5.2 Functionality

The architecture of IMCA has slightly changed, and thereby also aspectsregarding the computation. In the following, we will summarize the mainaspects of our IMCA version.

5.2.1 Storage management

Since a Markov automata can be very complex, we need a sophisticated stor-age system for our models. The most complex part is to store and distinguishthe Markovian and probabilistic transitions. Note, that a Markovian tran-

sition sλ

=⇒ s′ can be represented, such that we distinguish between theprobability to reach the successor state s′, and the residence time in state s,which is determined by exit rate E(s). Thus, λ = P(s, s′) · E(s). Therefore,we can store the probabilities for the Markovian transitions alongside withthe probabilities of the probabilistic transitions. Besides, we will store theexit rates for all Markovian states separately.

In Figure 5.2 we have depicted the structure used in our tool IMCA tostore a Markov automata. Besides of basic informations as the number ofstates or the initial and goal states, the most complex part is the transi-tion storage. Therefore, we will shortly describe how this works. The en-tries rate counts, row counts and choice counts can be seen as pointerto our entries for the exitrates in exit rates, the probabilities in non zero

and the successor states in cols. In more detail, to access the informa-tion given by our pointers for si ∈ S we have to proceed as described inthe following: To obtain the exitrate we access rate counts, such thatrate start=rate counts[i] and rate end=rate counts[i+1]. Further, ifrate start and rate end are distinct, we can access exit rates[rate start]

and obtain E(si). To obtain the probabilities and the successors, we first haveto access row counts, such that row start = row counts[i] and row end

50

1 struct SparseMatrix

2

3 unsigned long n; /∗ # of states ∗/4 unsigned long ms_n; /∗ # of Markovian states ∗/5 map <string ,unsigned long> states; /∗ state names mapped to # ∗/6 map <unsigned long,string > states_nr; /∗ state # mapped to names ∗/7 bool *initials; /∗ i n i t i a l s tates = true , otherwise fa l se ∗/8 bool *goals; /∗ goal s tates = true , otherwise fa l se ∗/9 bool *isPS; /∗ probab i l i s t i c state = true , otherwise fa l se ∗/

1011 Real *non_zeros; /∗ transit ion probab i l i t i e s ∗/12 Real *exit_rates; /∗ ex i t rates for Markovian states ∗/13 unsigned long *cols; /∗ successors ∗/14 unsigned char *row_counts; /∗ pointer to state ∗/15 unsigned char *rate_counts; /∗ pointer to ex i t rates ∗/16 unsigned char *choice_counts; /∗ pointer to dis tr ibut ion ∗/17 ;

Figure 5.2: Sparse matrix for MA in C++

= row counts[i+1]. Then we can access choice counts, which containsthe nondeterministic choices for probabilistic states as well as the transi-tions for Markovian states. That means, we access a probabilistic distribu-tion, such that we use choice start = choice counts [row starts] andchoice end = choice counts[row starts+1]-1. We increment row start

until row start = row end-1 and access choice counts as described to ob-tain all distributions of one state. With the access of non zero[choice start]

until non zero[choice end] as well as for cols, we obtain the probabilitiesand the corresponding successor states.

To get a better intuition for the storage system, we have displayed thetransitions of the example depicted in Figure 5.3(a) in the sparse matrixrepresentation depicted in Figure 5.3(b). We show now how to obtain thetransition information for s1. For the exitrate we get rate start=1 andrate end=1. Thus, since rate start = rate end there exists no exit ratefor s1, and therefore it is a probabilistic state. From row counts we getrow start = 1 and row end = 3. With this information, we can see that s1 isalso a non-deterministic state, since we have two distributions, which resultsfrom row start - row end > 1. For the first distribution we get choice start

= 2 and choice end = 3 and for the second distribution choice start = 4 andchoice end = 5. Thus, we obtain s1

τ1−−→µ1 with µ1(s0) = 14

and µ1(s2) = 34,

and s1τ2−−→µ2 with µ2(s1) = 1

2and µ2(s2) = 1

2.

5.2.2 Input

The input file extension for IMCA is .ma. This file contains the transitionrelations for Markovian and probabilistic states and a labeling to distinguishbetween them. Thus a probabilistic transition can have an arbitrary labelexcept for ”!” which is the indicator for a Markovian state.

51

s0 s1

s2

4

2

α β0.25

0.75

0.5

0.5

3

(a) Example Markov automata.

0 1 3 4

0 2 4 6 7

2/3 1/3 1/4 3/4 1/2 1/2 1

s1 s2 s0 s2 s1 s2 s2

0 1 1 2

6 3

row_counts:

choice_counts:

non_zeros:

cols:

exit_rates:

rate_counts:

s0 s1 s2states:

0 1 2 3 4 5 6

(b) Sparse matrix representation of (a).

Figure 5.3: Example Markov automata and sparse matrix transition relation.

File structure:

MA_File = Header Body

Header = #INITIALS \n

<initial state > \n

#GOALS \n

<goal state > \n

#TRANSITIONS \n

Body = <from state > <label > \n

* <to state > <rate/probability > \n

* <to state > <rate/probability > \n

Body

| <from state > <label > \n

* <to state > <rate/probability > \n

* <to state > <rate/probability > \n

The header contains the information of the initial and goal states of themodel as well as the indication where the transition relations start. The bodycontains the transitions and transition labels, where ”from state” containsthe state from where the transitions start, and ”label” is the choice that ismade. Note that for Markovian states the label ”!” represents the bottomaction ⊥. After this line, a number of lines follow which depict the successor

52

#INITIALS

s0

#GOALS

s2

s6

#TRANSITIONS

s0 a1

* s3 1

s0 a2

* s1 1

s1 !

* s2 2

s2 !

* s1 3

* s2 1

s3 a3

* s4 1

s3 a4

* s5 0.6

* s6 0.4

s4 !

* s7 1

s5 a1

* s8 1

s5 a2

* s9 1

s6 !

* s3 1

s7 !

* s3 5

s8 !

* s7 1

s9 !

* s6 3

Figure 5.4: Input file for the example in Figure 2.1

states ”to state” as well as the probability to reach them, if the sourcestate is a probabilistic state, otherwise the rate. Note that all distributionsfor one state should be given consecutively. Besides, the states can be namedarbitrary. In Figure 5.4 we have given an example .ma file, which specificationresults in the MA depicted in Figure 2.1.

5.2.3 Computation

The current version of IMCA provides three different types of computation:

• unbounded reachability,

• expected-time, and

• long-run average.

Since IMCA have to deal with non-determinism, we provide the minimumand the maximum values for each computation. Further, we solve all prob-lems with the help of the SoPlex library [43], which is a linear programmingsolver based on the revised simplex algorithm. A simplified schematic depic-tion of IMCA is given in Figure 5.5.

53

.maData Storage

expected-time evaluation

unbounded reachability

long-run average

.imc

min

max

Figure 5.5: Functionality of IMCA for MAs

5.2.4 Scoop-IMCA tool chain

Scoop [40] is a tool which provides among other things a framework forefficient modeling and generation of Markov automata. Therefore, it existsa Markov Automata Process Algebra (MAPA) introduced in [41]. Thus, itis possible to specify a Markov automata with the MAPA specification andgenerate the corresponding state space, transition rates and distributionswith Scoop and obtain a .ma file as described before. Some advantages ofScoop are, that it can apply the maximal-progress assumption as well asdead variable reduction and transition merging during the generation. Formore information about Scoop and MAPA we refer to [40, 41]. We use thisfunctionality later on in Section 6.3.

54

Chapter 6

Case studies

In this section we will provide several case studies. Our goal is, to showscalability, the influence of non-determinism on the computation, as wellas the impact between the minimum and maximum. All experiments wereconducted on a single core of a 2.8 GHz Intel Core i7 processor with 4GBRAM running on Linux.

6.1 Workstation cluster

In this section we will focus on the scalability of our approach. The results areobtained with the use of the dependable workstation cluster which is modeledby a GSPN in [24]. The setting of the workstation cluster is depicted inFigure 6.1. Two identical sub-clusters are connected via a backbone. Eachof the sub-clusters consists of N ∈ N>0 workstations connected in a star-topology by a switch. Moreover, the switches provide the interface to thebackbone and enables the workstations of each sub-cluster to communicateamong each other. All components of the system can break down. Remarkthat it exists a single repair unit (not shown in Figure 6.1) which takescare of repairing the components. We use the failure and repair rates for

event duration event durationBackboneFail 5000h BackboneRepair 8hLeftSwitchFail 4000h LeftSwitchRepair 4hRightSwitchFail 4000h RightSwitchRepair 4hLeftWorkstationFail 500h LeftWorkstationRepair 0.5hRightWorkstationFail 500h RightWorkstationRepair 0.5h

Table 6.1: Average duration for component failures and repairs.

55

left sub-cluster right sub-cluster

......

½

¾

Æ

½

¾

Æ

1

2

N N

2

1

backbone

left switch

right switch

Figure 6.1: Workstation cluster with 2N workstations.

the components given in [24] and shown in Table 6.1. This system can bemodeled as GSPN depicted in Figure 6.2, where timed transitions are givenas white rectangles and immediate transitions as black ones. We describethe GSPN top-down. The first two rows represent the N workstations of theleft and right sub-cluster, respectively. Using the average duration that aworkstation fails after 500h, we can associate a failure rate of 1

500. Hence, the

timed transitions LeftWorkstationFail as well as RightWorkstationFailare marking dependent and if n tokens are in place LeftWorkstationUp andRightWorkstationUp respectively, each of them fails with rate 1

500. Thus, the

timed transition LeftWorkstationFail and RightWorkstationFail has raten500

. The next two rows modeling the failure of the left and right switch andthe fifth the failure of the backbone. The failure rates for the correspondingevents are constructed as described for the workstations. At the bottom ofthe GSPN the repair unit is depicted. Once a component in the cluster fails,and the repair unit is available, it can repair the component. Hence, therepair unit only takes care of one failed component at a time. Dependenton the component, the repair operation takes a different amount of time,compare Table 6.1. The rates for the given repair events are constructed asdescribed before.

Note that the repair unit depicted in Figure 6.2 can have a decisionproblem. If at least two different components have to be repaired, andthe repair unit is available, the choice which is next is non-deterministic.This non-determinism is depicted in the GSPN model by the transitionsLeftWorkstationInspect, RightWorkstationInspect, LeftSwitchInspectand so on. Hence, the GSPN contains timed transitions as well as non-deterministic choices. We use the transformation of a GSPN model into anIMC from [35]. Remark that an IMC is a MA where for all s ∈ PS thereexists a unique successor for α ∈ Act. Hence, if s′ ∈ Post(s, α) it followsthat µα(s′) = 1 for s α−→µα. Note, that the workstation cluster consists of a

56

Æ

Æ

LeftWorkstationUp

RightWorkstationUp

LeftSwitchUp

RightSwitchUp

BackboneUp

LeftWorkstationInRepair

RightWorkstationInRepair

LeftSwitchInRepair

RightSwitchInRepair

BackboneInRepair

LeftWorkstationDown

RightWorkstationDown

LeftSwitchDown

RightSwitchDown

BackboneDown

RepairUnitAvailable

BackboneFail

LeftSwitchFail

RightWorkstationFail

LeftWorkstationFail

RightSwitchFail

BackboneInspect BackboneRepair

RightSwitchInspect RightSwitchRepair

LeftSwitchInspect LeftSwitchRepair

RightWorkstationInspect RightWorkstationRepair

LeftWorkstationInspect LeftWorkstationRepair

Figure 6.2: Workstation cluster as described in [24].

single maximal end component.

6.1.1 Analysis

In the following we will analyze the reliability and availability regarding to theminimum quality of service (QoS). The QoS criterion of a workstation clusterwith 2N workstations is given by k ∈ 2, . . . , 2N numbers of workstationsthat are required to be operational and mutually connected. For example,given a workstation cluster with N = 8 and k = 12, at least 12 out of 16workstations must be operational, which means that also the backbone aswell as both switches are operational. We will define some basic formulas forthe workstation cluster regarding to a marking m ∈M which corresponds toa state s ∈ S:

LeftOperationali(m) = m(LeftWorkstationUp) ≥ i ∧m(LeftSwitchUp) > 0

RightOperationali(m) = m(RightWorkstationUp) ≥ i ∧m(RightSwitch) > 0

Conn(m) = m(LeftSwitchUp) > 0 ∧m(BackboneUp) > 0 ∧m(RightSwitchUp) > 0

Operationali(m) = m(LeftWorkstationUp) +m(RightWorkstationUp) ≥ i ∧ Conn(m)

Hence, if k < N it can be sufficient if k workstations are available eitherin the left or right cluster, and therefore no communication between the twosub-clusters over the backbone is required. In the case of i < k and j < kworkstations are operational in the left and right cluster, the connectivityvia the switches over the backbone must be available and i + j ≥ k. We

57

will use a function mink : S → true, false to identify all states s ∈ Swhich corresponds to the markings for the QoS requirement in the underlyingGSPN:

mink(s) =

true if LeftOperationalk(s) ∨RightOperationalk(s) ∨Operationalk(s)false otherwise.

Our interest is to analyze the reliability and availability regarding to theQoS requirement for different parameter N and k. In our experiments wewill use the following QoS requirements:

• k = 34·N

• k = N

• k = 34· 2N

• k = 2N

Further, we assume that for the initial state all workstations, switches andbackbones are up, as well as the repair unit is available. Hence, initial states0 is the corresponding state to the marking in Figure 6.2. We will providethe following case studies:

Case 1: Reliability of the dependable workstation cluster. Therefore, wewant to know how long it takes that the QoS requirement is violated. Fur-ther, we will check how much time we spent on states that violates the QoSrequirement. Hence, we add all states s ∈ S that not fulfill mink(s) into theset of goal states G and compute the expected-time as well as the long-runaverage starting from state s0.

Case 2: Availability of the dependable workstation cluster. Therefore, wewant to know how much time we spent on states that fulfill the QoS require-ment. Hence, we will add all states s ∈ S that fulfill mink(s) into the set ofgoal states G and compute the long-run average starting from sate s0.

6.1.2 Results

The results for the first case are depicted in Table 6.2 and 6.3 and the resultfor the second case in Table 6.4. For all experiments we have used workstationclusters with N = 4, 8, 16, 32 and the corresponding QoS requirementsstated before. In the following we describe the results for reliability and

58

availability of the dependable workstation cluster. As pointed out before, inthe dependable workstation cluster occurs non-determinism whenever morethan one components have failed and the repair unit is available. Hence, weexpect that there is a difference between scheduling strategies. Therefore, wehave always computed the minimum and maximum expected time as well aslong-run average.

First of all we consider the expected time to violate the QoS requirement.Observe that the expected time to violate the QoS requirement increases withdecreasing amount of required workstations k. Further, the more stringentthe QoS requirement, and with increasing amount of workstations N , theexpected time to violate the QoS requirement decreases. Note that if k > Nthe QoS constraint is violated as soon as one switch or the backbone fails.Further, all the more workstations are in one sub-cluster, the probabilitythat one will fail increases. Hence, the expected time to violate the QoSconstraint for N = 32 and k = 64 is very short compared to N = 4 andk = 8. This observation can be made for all QoS constraints with k ≥ N .Moreover, we can observe that the scheduling strategies have no real influenceon the expected time for k > N , as the results for eTmax and eTmin areequal. If k ≤ N , the QoS requirement is violated at a later time for aworkstation cluster with more workstations per sub-cluster. In this casealso the scheduling strategy matters. For example, if several workstations inthe left sub-cluster have failed as well as the left switch, a good schedulingstrategy is to repair the switch first and afterwards the workstations, sincewithout the switch also all working workstations of the left sub-cluster arenot operational. Those observations also apply to the long-run. Besides, if wecompare the availability of states fulfilling the QoS constraint against stateswhere the constraint is violated, we can observe that the long-run average onstates fulfilling the QoS requirement is significant higher than on those whodoes not. This can be explained by use of Table 6.1. Observe that the repairoperations are very fast compared to the failure time of a component andtherefore we obtain the results depicted in Table 6.3 and 6.4 for the long-runaverage.

Now we focus on the scalability of our algorithms. The last two columnsin Table 6.2, 6.3 and 6.4 referring to the computation time in seconds. Ob-serve that if our set of goal states increases the computation time for theexpected time will decrease. Further, it seems that there is a connectionbetween the computed expected time and the time the program needs tocompute it. Hence, the longer it takes to violate the QoS requirement, thelonger the program needs to reach the fixpoint for the expected time. How-ever, this assumption is not applicable to the difference between the timeto compute eTmax and eTmin. Due to our experiments, it takes more time

59

to reach the fixpoint for eTmin as for eTmax, where the difference betweenthe maximum and minimum is marginal. Note that this observation cannot made for the long-run average. Hence, the computation time betweenLramin and Lramax have not such a clear indication. For example, observethe computation times for N = 32 in Table 6.3 and 6.4. Nevertheless, thelong-run average computation is more expensive than the expected time.This corresponds to the fact, that for the expected time goal states are madeabsorbing. This reduces the number of transitions in the model as well asthe constraints in the linear programming problem. Furthermore, we assumethat the computation time depends on the chosen set of goal states. Hence,in general the long-run average computation on states that does not fulfillthe QoS requirement is faster than the computation on states that fulfill it.Besides, with an increasing set of goal states also the long-run average com-putation time increases. Remark that this observation does not hold in everycase.

Summarizing, the computation time for the expected time as well as forthe long-run average depends on several factors despite the state space. Onthe one hand the transition rates have influence on how fast we reach thefixpoint by solving the linear programming problem. On the other handthe amount of non-determinism increases the amount of constraints to solve.Furthermore, the chosen set of goal states has influence on the computa-tion. Moreover, we assume that the resulting expected time has influence onthe computation time. In the next section we review those observations onanother model.

60

time (seconds)N k |S| |G| eTmax(s0,♦G) eTmin(s0,♦G) max min

4 3 819 566 1125179.46 1122465.40 0.014 0.0234 4 819 692 51704.89 51699.58 0.002 0.0024 6 819 807 1427.22 1427.22 0.007 0.0114 8 819 818 59.88 59.88 0.002 0.002

8 6 2771 2009 1724343.30 1719447.05 0.119 0.2678 8 2771 2482 25610.45 25604.95 0.039 0.0448 12 2771 2736 1428.57 1428.57 0.017 0.0168 16 2771 2770 30.58 30.58 0.008 0.007

16 12 10131 7544 1966511.37 1963868.17 1.243 3.50516 16 10131 9374 12751.19 12745.39 0.241 0.31116 24 10131 10014 1428.57 1428.57 0.071 0.07616 32 10131 10130 15.46 15.46 0.053 0.052

32 24 38675 29210 1982468.90 1978880.69 20.716 114.11132 32 38675 36406 6642.52 6636.17 2.141 3.25632 48 38675 38250 1428.57 1428.57 0.791 0.90932 64 38675 38674 7.77 7.77 0.643 0.671

Table 6.2: Workstation cluster expected time !(QoS).

time (seconds)

N k |S| |G| Lramax(s0, G) Lramin(s0, G) max min

4 3 819 566 3.6294e-06 3.5817e-06 0.105 0.0544 4 819 692 7.7037e-05 7.6463e-05 0.126 0.0714 6 819 807 3.5989e-03 3.5987e-03 0.124 0.0444 8 819 818 1.1587e-02 1.1587e-02 0.108 0.045

8 6 2771 2009 2.3781e-06 2.3477e-06 1.352 0.5398 8 2771 2482 1.6352e-04 1.6212e-04 1.356 0.2858 12 2771 2736 3.6015e-03 3.6010e-03 1.314 0.2788 16 2771 2770 1.9579e-02 1.9579e-02 1.272 0.278

16 12 10131 7544 2.0637e-06 2.0466e-06 22.419 8.12816 16 10131 9374 3.4831e-04 3.4457e-04 19.661 2.58616 24 10131 10014 3.6080e-03 3.6066e-03 18.751 2.10316 32 10131 10130 3.5561e-02 3.5561e-02 22.941 2.091

32 24 38675 29210 2.0440e-06 2.0316e-06 522.334 600.20032 32 38675 36406 7.0502e-04 6.9403e-04 463.274 35.27932 48 38675 38250 3.6223e-03 3.6177e-03 433.429 25.17432 64 38675 38674 6.7524e-02 6.7524e-02 634.656 24.168

Table 6.3: Workstation cluster long-run average !(QoS).

61

time (seconds)

N k |S| |G| Lramax(s0, G) Lramin(s0, G) max min

4 3 819 253 0.999996 0.999996 0.323 0.2844 4 819 127 0.999924 0.999923 0.239 0.2994 6 819 12 0.996401 0.996401 0.321 0.3194 8 819 1 0.988413 0.988413 0.321 0.269

8 6 2771 762 0.999998 0.999998 3.838 3.2648 8 2771 289 0.999838 0.999836 2.607 3.4188 12 2771 35 0.996399 0.996398 3.948 3.5398 16 2771 1 0.980421 0.980421 3.360 3.227

16 12 10131 2587 0.999998 0.999998 86.148 49.54716 16 10131 757 0.999655 0.999652 50.691 302.06416 24 10131 117 0.996393 0.996392 47.571 260.78216 32 10131 1 0.964439 0.964439 45.891 337.817

32 24 38675 9465 0.999998 0.999998 1913.229 24342.27532 32 38675 2269 0.999306 0.999271 1345.557 7267.62732 48 38675 425 0.996382 0.996378 1440.494 4964.06832 64 38675 1 0.932476 0.932476 1104.782 13232.694

Table 6.4: Workstation cluster long-run average QoS.

6.2 Google file system

In this section we focus on the scalability of our approach and compare theprevious observations with those we obtain in the following. We considera model which is addressing a replicated file system as used of the Googleengine [20]. In this file system model, files are divided into chunks of equalsize where several copies of each chunk reside at several chunk servers. Ifa user requests access to a certain chunk of file, the file system asks themaster for the location and afterwards the data transfer takes place directlybetween the chunk server and the user. To obtain a MA we use the GSPNmodel developed in [13] and depicted in Figure 6.3. The GSPN describes thelife cycle of a single chunk but with consideration of the load by the otherchunks. Observe that the GSPN is divided into three parts: master, copiesand servers. In the first part, the master can be up running (M up), failedbut without knowledge of the failure (M1), or affected by a hardware failure(M hard d) or software failure (M soft d). The middle part considers thenumber of copies which are available (R present) as well as lost (R lost).The last part considers M chunk servers which can be up running (C up), aswell as failing (C 1) where a copy of a chunk can get lost or the server onlystores chunks we do not consider and thus no copies get lost. Further, asfor the master, chunk servers can fail but without knowledge of the failure

62

(C 2), or be affected by a hardware failure (C hard d) or software failure(C soft d). The corresponding rates and probabilities for the transitions aredepicted in Table 6.5. Note that we consider a model where the probabilityof a hardware or a software failure in the chunk server is unknown. Hence,we have a non-deterministic choice in our analysis every time a chunk serverfails.

To obtain a MA from the GSPN we use a slightly modified transforma-tion of [13]. Consider the GSPN depicted in Figure 6.3 and the transitionrates and probabilities in Table 6.5. Hence, the GSPN can be transformed asshown in [13], such that all immediate transitions are resolved. For example,consider the immediate transitions m soft and m hard as well as the timedtransition m fail. Using the rates and probabilities given in Table 6.5 wecan replace those transitions with two new timed transitions m soft fail andm hard fail with rate 0.0005 · 0.95 and 0.0005 · 0.05, respectively. Note thatthe place M1 get obsolete. The resulting GSPN can easily transformed into aCTMC using the PRISM [31] modeling language. Note that for our examplethe transitions c soft and c hard can not be resolved. Thus, we introducetwo rates λsoft and λhard which are not occurring in our model, and add themto the corresponding transitions in the transformed GSPN. This enables usto identify those transitions later on as our non-deterministic choices. More-over, we export the model from PRISM as MRMC [30] .tra and .lab file.Therefore, we interpret the transitions with the previous attached rates asour non-deterministic choices and transform the corresponding CTMDP intoan IMC [34]. Note, that we will obtain an IMC, since our non-deterministicchoices have a direct successor.

transition rate probabilitym fail 0.0005m soft re #M soft d·12m hard re 6.0c fail 0.05c soft re #C soft d·12c hard re 1.0

replicate20.0 if #R present> 02.0 if #R present= 0

m soft 0.95m hard 0.05

destroy#R present

#C up

keep 1− #R present

#C up

Table 6.5: Transition rates and probabilities.

63

M

Cup

c hard re c sof t re

c f ai l

C 1

R present

C sof t d C hard d

c sof t c hard

C 2

R l ost

Mup

mhard re msof t re

Msof t d Mhard d

msof t mhard

M1 mfai l

repl i cate

dest roy

keep

master

copies

servers

Figure 6.3: The Google file system as described in [13].

6.2.1 Analysis

The model features three parameters: The number of chunk servers M , thenumber of chunks S a chunk server can store and the total number of chunksN . In our analysis we will fix the parameters S = 5000 and N = 100000,whereas M ∈ 20, 30, 40, 50. Further we consider the following three condi-tions:

severe hardware disaster = (M hard d = 1) ∧ (C hard d > 0.75 ·M)

severe software disaster = (M soft d = 1) ∧ (C soft d > 0.75 ·M)

service level 1 = (M up = 1) ∧ (R present ≥ 1)

For the reliability analysis of the Google file system, we provide two casestudies. In both we add all states that fulfill the service level 1 conditioninto our set of goal states G. For the first case, we use all states fulfilling thesevere hardware disaster condition as initial state and for the second thesevere software disaster condition, respectively. Note that we have onlydefined MAs with one initial state. This can be achieved by introducing anew state which non-deterministically chooses one of the states fulfilling theinitial condition. For the availability analysis we add all states fulfilling theservice level 1 condition into our set of goal states G. Further, as initial

64

state we choose non-deterministically one state that fulfills service level 1.Note that we obtain a single maximal end component with a set of goal statesfulfilling service level 1.

6.2.2 Results

The results of the experiments are depicted in Table 6.6, 6.7 and 6.8. Inthe following we describe the results obtained for reliability and availabilityof the Google file system. As explained before, the only non-determinismwhich occurs is the choice if a hardware or software failure of a chunk serveroccurs. In the first part of our analysis we shortly describe the meaning ofthe obtained results and afterwards compare the scalability results with theresults obtained in the previous section.

First of all we compare the expected time analysis starting from stateswith severe hardware disaster against the analysis starting from states withsevere software disaster. Observe that the maximum as well as minimumexpected time to reach a state fulfilling service level 1 needs more time if weare in a state of severe hardware disaster as from severe software disaster.The difference between those to starting conditions is that a hardware repaircosts more time than to repair the software. More interesting is the differ-ence between eTmax and eTmin. This will be induced by the non-deterministicchoice which is made if a chunk server fails. Since we have only the choicebetween a hardware and a software failure, it is obvious that the schedulerwill choose the transition which leads to the hardware failure to increase themaximum expected time. Hence, the maximum expected time describes ascenario where only hardware failures of chunk servers occur, whereas theminimum expected time describes a scenario where only software failures ofchunk servers occur. This observation can also be made on the availabilityanalysis.

The more interesting part of this case study is the comparison of the scala-bility with the dependable workstation cluster. The last two columns in Table6.6, 6.7 and 6.8 referring to the computation time in seconds. Recall that forthe workstation cluster the results obtained for eTmax and eTmin were nearlyidentical, where the computation for the minimum were more expensive.However, for the Google file system the difference between eTmax and eTmin

is significant. This is also reflected by the computation time. Hence, despitethat the minimum expected time computation is more expensive than themaximum, the computation time is inferior in all cases except for M = 20.This shows there exists a connection between the resulting expected timeand the duration of the computation. Nevertheless, comparing the results of

65

time (seconds)M |S| |G| eTmax(s0,♦G) eTmin(s0,♦G) max min

20 7176 1713 21.523572 0.166668 2.298 2.73830 16156 3918 21.514897 0.166667 15.413 11.92540 28736 7023 21.514897 0.166667 57.631 32.09250 44916 11028 21.514897 0.166667 166.198 73.408

Table 6.6: Google file system expected time from severe hardware disaster.

Table 6.6 and 6.7 indicates, that the result obtained for eTmax or eTmin haveno direct influence on the computation time. For example, consider the casefor M = 40. The result for eTmax from a severe hardware disaster is clearlyhigher than from a severe software failure, but the computation time isequal in both cases. Hence, the influence of the result is not directly appli-cable on the computation time. However, the difference between eTmax andeTmin have influence on the computation time. Hence, as described before,the minimum expected time computation is more expensive, but neverthelessif there exists a significant difference between eTmax and eTmin, the compu-tation time for the minimum is faster. Besides, if we consider Table 6.8 wecan observe that the computation of Lramin is more expensive than Lramax

for the Google file system. Regarding to the results for the workstation clus-ter, we can not made a general assumption if the minimum or maximumcomputation is more expensive. Nevertheless, we assume that the set of goalstates influence this outcome.

Summarizing, we can make the following observations for scalability. Thedegree of non-determinism in a Markov automata have influence on the com-putation time. Hence, more non-deterministic choices results in more con-straints which have to be solved. Further, the transition rates have alsoinfluence on the computation time. Thus, if the transitions in a MA areequipped with low rates, as for the workstation cluster, this can result ina slower convergence of the algorithms. Moreover, the state space and alsothe set of goal states will influence our computation time. Hence, with agrowing state space we have more variables to solve which takes more time.This also means for the long run average, that if we have several small MECsinstead of one big, we have to solve a linear programming problem for each,but with less constraints and variables. Hence, the amount and size of MECsalso influences the computation time of the long-run average. Further, if wecan exclude states regarding to the Lock set defined in Chapter 3 for theexpected time, this also reduces the amount of constraints and variables.

66

time (seconds)M |S| |G| eTmax(s0,♦G) eTmin(s0,♦G) max min

20 7176 1713 0.843552 0.083333 2.266 2.87530 16156 3918 0.618453 0.083333 16.229 11.76340 28736 7023 0.599445 0.083333 57.988 32.08850 44916 11028 0.592696 0.083333 164.944 72.979

Table 6.7: Google file system expected time from severe software disaster.

time (seconds)M |S| |G| Lramax(s0, G) Lramin(s0, G) max min

20 7176 1713 0.998493 0.149956 2.925 3.96930 16156 3918 0.999105 0.100247 14.009 25.36640 28736 7023 0.999950 0.075271 91.175 111.09950 44916 11028 0.999939 0.060224 136.333 316.998

Table 6.8: Google file system long-run average on service level 1.

6.3 Polling system

In this section we will consider a polling system as for example described in[29] and depicted in Figure 6.4 as GSPN. This simple polling system consistsof a single server who cyclically visits the queues, providing service to thewaiting customers and depart them afterwards from the system. This is onlyone possibility to model a polling system. However, we will not use thisGSPN and transform it into a MA as in the previous sections. For this casestudy, we use the Markov automata process algebra (MAPA) introduced in[41] to specify a MA polling system. Moreover, the main goal of this sectionis to show the influence of non-determinism and continuous time which makesa Markov automata mandatory.

In Figure 6.5 we have depicted a MA for a polling system consisting of oneserver and two stations which is inspired by [38]. Each state is representedas a tuple (s1, s2, j), where s1 represents the number of jobs in station one,s2 the number of jobs in station two, and j the number of jobs in the server.In this example, the maximal number of jobs for each station and server isbounded by 1. Further, the incoming requests for station one and two haverate λ1 and λ2 respectively, and are stored until the server will fetch them.Moreover, when polling a station there is a 1

10probability that the job is acci-

dentally kept in the station. If it is the case that both stations contain a job,the server will choose non-deterministically which station will be served nextwhere jobs are processed with rate µ. Observe that the probabilistic transi-

67

p(1)sp(1)q

p(1)p p(1)wp(0)p p(0)w

p(0)sp(0)q

p(2)a

p(2)qp(2)s

p(2)w p(2)p

p(0)a p(1)a

p(3)a

p(3)s p(3)q

p(3)pp(3)w

T(3)s

T (3)a

T(2)s

T (2)a

T(0)a

T(0)s

T (0)w

T(2)wT(3)

w

T(1)s

T (1)a

T(1)w

t(3)w

t(3)s

t(2)w

t(2)s

t(0)s

t(0)w t(1)w

t(1)s

Figure 6.4: GSPN representation of a cyclic single-server polling system [29].

tions (1, 0, 0) τ1−−→µτ1 and (0, 1, 0) τ2−−→µτ2 can be resolved by replacing them

with (0, 0, 0)0.1·λ1=⇒ (1, 0, 1), (0, 0, 0)

0.9·λ1=⇒ (0, 0, 1) and (0, 0, 0)0.1·λ2=⇒ (0, 1, 1),

(0, 0, 0)0.9·λ2=⇒ (0, 0, 1), respectively. However, the probabilistic transitions

(1, 1, 0) α−→µα and (1, 1, 0) β−→µβ can not be encoded in this way, because ofthe non-determinism. Further, those two transitions can also not be repre-sented in an IMC, due to the probability distributions. Hence, for this casestudy it is mandatory to use Markov automata. We use MAPA to specify aslightly complex version of the just described example polling system. Thespecification is depicted in Figure 6.6. Observe, that the number of differentkind of jobs as well as the size of buffers for each station is not bounded byone as in the example. Besides, we kept the construction of the polling sys-tem with two stations and one server. The specification uses two constantsto define the number of job types N and the queue size Q. Further, it usesthree data types: a set Jobs which includes the possible incoming jobs, a setStations to identify the two different stations and a built-in type Queue tobuffer the requests for each station. Note that the arrival rate for station i isset to 2i+ 1. Hence, if we interpret those as Markovian rates, we obtain forstation one λ1 = 3 and for station two λ2 = 5. Moreover, the rate that jobj is served is given by 2j. A station will receive jobs, as long as its queue isnot full and can deliver jobs if the queue is not empty. As in the example,with probability 1

10the job will be kept accidentally in the station. Further,

the server will continuously poll the stations and work on their jobs. Fur-thermore, the system will communicate over the poll and deliver actions.Note that we hide all actions, since the system is considered as closed MA.

68

0, 0, 0

1, 0, 0

0, 1, 0

0, 0, 1

1, 0, 1

0, 1, 1

1, 1, 1 1, 1, 0

λ1

λ2

µ

λ1

λ2

λ2

µ

λ1

µ

µ

τ1

τ2

α

β

110

910

910

110

910

110

910

110

Figure 6.5: Example MA for a polling system.

Besides, our initial state is (0, 0, 0), such that all queues are empty as well asno job is picked by the server.

6.3.1 Analysis

The model features two parameters: The number of different job types Nand the size of the queue Q. In our analysis we will restrain those parametersby N = 1, 2 and Q = 1, 2, 4. Hence, we will analyze six different settingsfor the described polling system with two stations and one server. Note thatthe polling system is a cyclic model. Moreover, our model is an unichainMA. We will make two case studies, one with respect to reliability and onewith respect to availability.

Case 1: For the reliability analysis, we will compute the expected time ittakes until the first job is completed, starting from an empty polling system.Therefore, we add all states that are reached after the server completes a jobinto the set of goal states. Recall that in case N = 1 only one job type exists,whereas for N = 2 two job types exists.

Case 2: In this case we focus on the long-run average. Therefore, we definethree different sets of goal states:

station 1 = (s1, s2, i)|(s1, s2, i)λ1=⇒ (s′1, s

′2, i′)

station 2 = (s1, s2, i)|(s1, s2, i)λ2=⇒ (s′1, s

′2, i′)

station 1 or 2 = station 1 ∪ station 2

Hence, station 1 and station 2 contains states where jobs can arrive for sta-tion 1 and station 2, respectively. The third set contains all states where

69

1 constant NrOfJobTypes = N, QueueSizeInStation = Q

2 type Jobs = 1.. NrOfJobTypes

3 type Stations = 1..2

45 Station(i:Stations , q:Queue , size :0.. QueueSizeInStation ) =

6 size < QueueSizeInStation => <2 * i + 1> . sum(j:Jobs , arrive(j). Station[i, add(q,j), size + 1])

7 ++ size > 0 => deliver(i,head(q)). psum (0.9 -> Station[i,tail(q),size -1] ++ 0.1

8 -> Station[i, q, size])

910 Server = sum(n:1..2 , sum(j:Jobs , poll(n,j).<2 * j>. complete(j). Server []))

1112 init Station[1,empty ,0] || Station[2,empty ,0] || Server []

1314 comm (poll , deliver , copy)

1516 encap poll , deliver

1718 hide copy , arrive , complete

Figure 6.6: Input MAPA file for a polling system in Scoop.

jobs can arrive. Hence, we will compute the long-run average time we spenton states where stations can receive jobs.

6.3.2 Results

The result for the first case is depicted in Table 6.9 and for the second case inTable 6.10, 6.11 and 6.12. In the following we describe the results obtainedfor both cases for the polling system. Our focus is on the non-determinism.Hence, we will check how much influence the non-deterministic choice willhave in this example on the expected time as well as the long-run average.

First consider the results in Table 6.9. Observe that the results for eTmax

and eTmin don’t change for a different queue size. Hence, the first job whichwill be handled by the server, starting in an empty polling system, is notinfluenced by the number of queues for a station. Furthermore, if thereexists only one type of job, the maximum and minimum expected time isequal. However, for N = 2 we obtain different results for eTmax and eTmin.Remark, that job type one and two have different process rates. Therefore,one job is processed faster as the other. Hence, the decision that is madefor the non-deterministic choices in the polling system have influence on theexpected time.

For the long-run average we consider the results in Table 6.10, 6.11 and6.12. We can observe, that for all cases the difference between Lramax andLramin is significant. Hence, the non-deterministic choice has a clear in-fluence on the long-run average. Moreover, with increasing queue size foreach station also the gap between Lramax and Lramin increases. Remarkthat not at all probabilistic states non-determinism occurs. The column #nd

70

lists the number of states in which non-determinism occurs. Observe thatwith increasing queue size as well as number of job types, the number ofnon-deterministic states increases. Hence, the number of buffers in a queuecan increase the long-run fraction on time on states where jobs can arrive,whereas the thereby increasing amount of non-determinism can lead to ashort long-run fraction on time on those states. For example consider thelong-run average on states where jobs can arrive for station 1. If all buffersin the queue of station 1 are occupied, no more jobs can arrive for station1. Further, at least one job is buffered for station 2. The server has tochoose non-deterministically from which station he pulls a job. In the casefor Lramin, the scheduler chooses always the action such that the server pullsfrom station 2 and thus the queue for station 1 remains full. Remark thatthe results in Table 6.12 for Lramin are significant higher than in Table 6.10and 6.11. This is the case, because the server has to pull a job from onestation, and therefore the polling system visits again a state where a job canarrive for either of the stations. Moreover, the maximum long-run averagenaturally increases with a bigger queue size.

Summarizing, in this section we have shown that the non-deterministicchoices can have a significant influence on the difference between the maxi-mum and minimum expected time as well as the long-run average. Moreover,we have shown that Markov automata have the ability to model behaviorswhich can not represented with another stochastic model.

N Q |S| |MS| |PS| #nd eTmax(s0,♦G) eTmin(s0,♦G)

1 1 18 5 13 1 0.625 0.6251 2 41 10 31 4 0.625 0.6251 4 117 26 91 16 0.625 0.6252 1 50 19 31 18 0.625 0.3752 2 282 99 183 122 0.625 0.3752 4 5706 1923 3783 2762 0.625 0.375

Table 6.9: Polling system expected time until first job is completed.

71

N Q |S| |MS| |PS| #nd Lramax(s0, G) Lramin(s0, G)

1 1 18 5 13 1 0.385116 0.1341761 2 41 10 31 4 0.490683 0.0505921 4 117 26 91 16 0.566285 0.0064792 1 50 19 31 18 0.579652 0.1341762 2 282 99 183 122 0.731727 0.0505922 4 5706 1923 3783 2762 0.866058 0.006479

Table 6.10: Polling system long-run average on station 1.

N Q |S| |MS| |PS| #nd Lramax(s0, G) Lramin(s0, G)

1 1 18 5 13 1 0.274283 0.1224551 2 41 10 31 4 0.329283 0.0648941 4 117 26 91 16 0.356110 0.0202182 1 50 19 31 18 0.458053 0.1224552 2 282 99 183 122 0.560563 0.0648942 4 5706 1923 3783 2762 0.653264 0.020218

Table 6.11: Polling system long-run average on station 2.

N Q |S| |MS| |PS| #nd Lramax(s0, G) Lramin(s0, G)

1 1 18 5 13 1 0.417647 0.3360951 2 41 10 31 4 0.498213 0.3404011 4 117 26 91 16 0.566643 0.3399392 1 50 19 31 18 0.640889 0.3360952 2 282 99 183 122 0.754282 0.3404012 4 5706 1923 3783 2762 0.868817 0.339939

Table 6.12: Polling system long-run average on station 1 or 2.

72

Chapter 7

Conclusion

In this thesis we have focused on a relatively new model, Markov automata.Our interest was, if it is possible to analyze it and how efficient this wouldbe. Before the research on MAs started we have worked on IMCs. Therefore,we were researching if it is possible to check reliability as well as availability.During this research we developed algorithms which were capable of handlingthe occurring non-determinism. That means, we were able to check reliabilityas well as availability on IMCs with respect to computing the maximum andminimum expected time as well as long-run average..

The successful results on IMCs encouraged further research. Hence, thequestions occurred, if the results are applicable to MAs. As described inChapter 2, IMCs are a subclass of MAs. Thus, both models share the com-bination of Markovian transitions and non-deterministic choices. This ledus to the assumption, that the theory and resulting algorithms for IMCsmight hold for MAs. However, the results for IMCs could not just directlytransfered to MAs. The fact, that we have to deal with probability distribu-tions instead of immediate transitions forced us to rethink our results fromscratch. Hence, all theories and algorithms for IMCs are not only adjustedand instead developed new with respect on the results gained from IMCs.

To conclude the thesis, we summarize our achievements as well as proposedirections for future research:

• We developed in Chapter 3 the theory and algorithms to compute theexpected time to reach a given set of goal states in MAs. Moreover,we were able to reduce the computation to a non-negative stochasticshortest path problem in MDPs. Besides, we have proven that a uni-formization will not affect the results for the maximum and minimumexpected time. Currently we only support the expected time for simple

73

reachability. For future research it is an open question if the resultsalso hold for different instances of until formulas.

• Further, we developed in Chapter 4 the theory and algorithms to com-pute the long-run average fraction of time on a set of goal states inMAs. As well as for the expected time, we were able to reduce thecomputation to a non-negative SSP problem. Moreover, the resultsof the minimum and maximum long-run average are not affected byuniformization. For future work, the long-run average with respect torewards in MAs remain unsolved and have to be seized.

• In Chapter 6 we provided three different case studies. The first twowere directed towards the scalability of our approach. A huge impactis the state space and the degree of non-determinism. The third casestudy focused on the impact of non-determinism for the minimum andmaximum. For future work, it is worthwhile to consider reductiontechniques, as for example bisimulation minimization, to reduce thestate space. Further, with the fact that uniformization will not affectour results, it is worthwhile to determine if this can be used to reducethe computation time.

74

List of Figures

2.1 Example Markov automata. . . . . . . . . . . . . . . . . . . . 62.2 Example Markov automata and its induced MLTS. . . . . . . 102.3 MA with Zeno behaviour. . . . . . . . . . . . . . . . . . . . . 132.4 An example MA. . . . . . . . . . . . . . . . . . . . . . . . . . 152.5 Example Markov automata and it’s uniformed MA. . . . . . . 17

3.1 Example Markov automata with Zeno paths. . . . . . . . . . . 213.2 Example MA and its corresponding SSP. . . . . . . . . . . . . 293.3 Example MA for expected time reachability. . . . . . . . . . . 33

4.1 Example unichain Markov automata. . . . . . . . . . . . . . . 394.2 Example Markov automata. . . . . . . . . . . . . . . . . . . . 434.3 Induced SSP for MA in Figure 4.2 . . . . . . . . . . . . . . . . 454.4 Example Markov automata with two MECs. . . . . . . . . . . 46

5.1 Functionality of IMCA for IMCs . . . . . . . . . . . . . . . . . 505.2 Sparse matrix for MA in C++ . . . . . . . . . . . . . . . . . . . 515.3 Example Markov automata and sparse matrix transition rela-

tion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525.4 Input file for the example in Figure 2.1 . . . . . . . . . . . . . 535.5 Functionality of IMCA for MAs . . . . . . . . . . . . . . . . . 54

6.1 Workstation cluster with 2N workstations. . . . . . . . . . . . 566.2 Workstation cluster as described in [24]. . . . . . . . . . . . . 576.3 The Google file system as described in [13]. . . . . . . . . . . . 646.4 GSPN representation of a cyclic single-server polling system

[29]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 686.5 Example MA for a polling system. . . . . . . . . . . . . . . . . 696.6 Input MAPA file for a polling system in Scoop. . . . . . . . . 70

75

76

List of Tables

2.1 An example derivation of π@t for Markov automata. . . . . . 12

6.1 Average duration for component failures and repairs. . . . . . 556.2 Workstation cluster expected time !(QoS). . . . . . . . . . . . 616.3 Workstation cluster long-run average !(QoS). . . . . . . . . . . 616.4 Workstation cluster long-run average QoS. . . . . . . . . . . . 626.5 Transition rates and probabilities. . . . . . . . . . . . . . . . . 636.6 Google file system expected time from severe hardware disaster. 666.7 Google file system expected time from severe software disaster. 676.8 Google file system long-run average on service level 1. . . . . . 676.9 Polling system expected time until first job is completed. . . . 716.10 Polling system long-run average on station 1. . . . . . . . . . . 726.11 Polling system long-run average on station 2. . . . . . . . . . . 726.12 Polling system long-run average on station 1 or 2. . . . . . . . 72

77

78

Bibliography

[1] M. Ajmone Marsan, G. Conte, and G. Balbo. A class of generalizedstochastic petri nets for the performance evaluation of multiprocessorsystems. ACM Trans. Comput. Syst., 2(2):93–122, May 1984.

[2] R. B. Ash and C. Dolans-Dade. Probability and Measure Theory. Aca-demic Press, 2nd edition, 2000.

[3] C. Baier, B. Haverkort, H. Hermanns, and J.-P. Katoen. Model checkingcontinuous-time markov chains by transient analysis. In E. Emersonand A. Sistla, editors, Computer Aided Verification, volume 1855 ofLecture Notes in Computer Science, pages 358–372. Springer Berlin /Heidelberg, 2000.

[4] C. Baier, B. Haverkort, H. Hermanns, and J.-P. Katoen. Model-checkingalgorithms for continuous-time markov chains. Software Engineering,IEEE Transactions on, 29(6):524 – 541, june 2003.

[5] C. Baier, H. Hermanns, J.-P. Katoen, and B. R. Haverkort. Effi-cient computation of time-bounded reachability probabilities in uni-form continuous-time markov decision processes. Theor. Comput. Sci.,345(1):2–26, Nov. 2005.

[6] D. P. Bertsekas and J. N. Tsitsiklis. An analysis of stochastic shortestpath problems. Mathematics of Operations Research, 16(3):pp. 580–595,1991.

[7] H. Boudali, P. Crouzen, B. Haverkort, M. Kuntz, and Stoelinga. Archi-tectural dependability evaluation with arcade. In Dependable Systemsand Networks With FTCS and DCC, 2008. DSN 2008. IEEE Interna-tional Conference on, pages 512 –521, june 2008.

[8] H. Boudali, P. Crouzen, and M. Stoelinga. Dynamic fault tree analysisusing input/output interactive markov chains. In Dependable Systems

79

and Networks, 2007. DSN ’07. 37th Annual IEEE/IFIP InternationalConference on, pages 708 –717, june 2007.

[9] H. Boudali, P. Crouzen, and M. Stoelinga. A rigorous, compositional,and extensible framework for dynamic fault tree analysis. Dependableand Secure Computing, IEEE Transactions on, 7(2):128 –143, april-june2010.

[10] M. Bozzano, A. Cimatti, J.-P. Katoen, V. Nguyen, T. Noll, andM. Roveri. Safety, dependability and performance analysis of extendedAADL models. The Computer Journal, 54(5):754–775, 2011.

[11] M. Bozzano, A. Cimatti, M. Roveri, J.-P. Katoen, V. Y. Nguyen, andT. Noll. Verification and performance evaluation of AADL models. InProceedings of the the 7th joint meeting of the European software en-gineering conference and the ACM SIGSOFT symposium on The foun-dations of software engineering, ESEC/FSE ’09, pages 285–286, NewYork, NY, USA, 2009. ACM.

[12] K. Chatterjee and M. Henzinger. Faster and dynamic algorithms formaximal end-component decomposition and related graph problems inprobabilistic verification. In Symp. on Discrete Algorithms (SODA),pages 1318–1336. SIAM, 2011.

[13] L. Cloth and B. Haverkort. Model checking for survivability! In Quan-titative Evaluation of Systems, 2005. Second International Conferenceon the, pages 145 – 154, sept. 2005.

[14] N. Coste, H. Garavel, H. Hermanns, F. Lang, R. Mateescu, andW. Serwe. Ten years of performance evaluation for concurrent systemsusing CADP. In ISoLA, volume 6416 of LNCS, pages 128–142. Springer,2010.

[15] L. de Alfaro. Formal Verification of Probabilistic Systems. PhD thesis,Stanford University, 1997.

[16] L. de Alfaro. How to specify and verify the long-run average behaviorof probabilistic systems. In LICS, pages 454–465. IEEE CS Press, 1998.

[17] Y. Deng and M. Hennessy. On the semantics of markov automata. InL. Aceto, M. Henzinger, and J. Sgall, editors, Automata, Languages andProgramming, volume 6756 of Lecture Notes in Computer Science, pages307–318. Springer Berlin / Heidelberg, 2011.

80

[18] C. Eisentraut, H. Hermanns, and L. Zhang. Concurrency and compo-sition in a stochastic world. In P. Gastin and F. Laroussinie, editors,CONCUR 2010 - Concurrency Theory, volume 6269 of Lecture Notes inComputer Science, pages 21–39. Springer Berlin / Heidelberg, 2010.

[19] C. Eisentraut, H. Hermanns, and L. Zhang. On probabilistic automatain continuous time. Logic in Computer Science, Symposium on, 0:342–351, 2010.

[20] S. Ghemawat, H. Gobioff, and S.-T. Leung. The google file system.SIGOPS Oper. Syst. Rev., 37(5):29–43, Oct. 2003.

[21] D. Guck, T. Han, J.-P. Katoen, and M. R. Neuhaußer. Quantitativetimed analysis of interactive Markov chains. In NASA Formal Methods,LNCS 7226, pages 8–23. Springer, 2012.

[22] D. Haasl, N. Roberts, W. Vesely, and F. Goldberg. Fault tree hand-book. Technical Report NUREG-0492, Nuclear Regulatory Commis-sion, Washington, DC (USA). Office of Nuclear Regulatory Research,Jan 1981.

[23] P. R. Halmos. Measure theory. Princeton, New Jersey : D. Van NostrandCompany, 1950.

[24] B. Haverkort, H. Hermanns, and J.-P. Katoen. On the use of modelchecking techniques for dependability evaluation. In Reliable DistributedSystems, 2000. SRDS-2000. Proceedings The 19th IEEE Symposium on,pages 228 –237, 2000.

[25] H. Hermanns. Interactive Markov Chains and the Quest for QuantifiedQuality, volume 2428 of LNCS. Springer, 2002.

[26] H. Hermanns and J.-P. Katoen. The how and why of interactive Markovchains. In FMCO, volume 6286 of LNCS, pages 311–337. Springer, 2009.

[27] S. Johr. PModel Checking Compostitional Markov Systems. PhD thesis,Universitat des Saarlandes, 2007.

[28] P. C. Kanellakis and S. A. Smolka. CCS expressions, finite state pro-cesses, and three problems of equivalence. In Proceedings of the secondannual ACM symposium on Principles of distributed computing, PODC’83, pages 228–240, New York, NY, USA, 1983. ACM.

81

[29] D. Kartson, G. Balbo, S. Donatelli, G. Franceschinis, and G. Conte.Modelling with Generalized Stochastic Petri Nets. John Wiley & Sons,Inc., New York, NY, USA, 1st edition, 1994.

[30] J.-P. Katoen, I. S. Zapreev, E. M. Hahn, H. Hermanns, and D. N. Jansen.The ins and outs of the probabilistic model checker mrmc. PerformanceEvaluation, 68(2):90 – 104, 2011.

[31] M. Kwiatkowska, G. Norman, and D. Parker. PRISM 4.0: Verificationof probabilistic real-time systems. In G. Gopalakrishnan and S. Qadeer,editors, Proc. 23rd International Conference on Computer Aided Verifi-cation (CAV’11), volume 6806 of LNCS, pages 585–591. Springer, 2011.

[32] J. F. Meyer, A. Movaghar, and W. H. Sanders. Stochastic activity net-works: Structure, behavior, and application. In International Workshopon Timed Petri Nets, pages 106–115, Washington, DC, USA, 1985. IEEEComputer Society.

[33] M. Neuhaußer, M. Stoelinga, and J.-P. Katoen. Delayed nondeterminismin continuous-time markov decision processes. In L. de Alfaro, editor,Foundations of Software Science and Computational Structures, volume5504 of Lecture Notes in Computer Science, pages 364–379. SpringerBerlin / Heidelberg, 2009.

[34] M. Neuhaußer and L. Zhang. Time-bounded reachability probabilitiesin continuous-time markov decision processes. In Quantitative Evalua-tion of Systems (QEST), 2010 Seventh International Conference on the,pages 209 –218, sept. 2010.

[35] M. R. Neuhaußer. Model checking nondeterministic and randomly timedsystems. PhD thesis, RWTH Aachen, Enschede, January 2010. CTITPh.D.-Thesis Series No. 09-165, ISSN 1381-3617.

[36] J. Norris. Markov Chains. Cambridge University Press, 1997.

[37] R. Segala. Modeling and Verification of Randomized Distributed Real-Time Systems. Technical report, Cambridge, MA, USA, 1996.

[38] M. M. Srinivasan. Nondeterministic polling systems. Management Sci-ence, 37(6):667–681, 1991.

[39] M. Stoelinga. An introduction to probabilistic automata. Bulletin ofthe European Association for Theoretical Computer Science, 78:176–198,2002.

82

[40] M. Timmer. Scoop: A tool for symbolic optimisations of probabilisticprocesses. In C. Palamidessi and A. Riska, editors, Proceedings of the8th International Conference on Quantitative Evaluation of SysTems,QEST 2011, Aachen, Germany, pages 149–150, Los Alamitos, USA,September 2011. IEEE Computer Society.

[41] M. Timmer, J.-P. Katoen, J. P. van de, and M. Stoelinga. Efficientmodelling and generation of markov automata. In 10th Workshop onQuantitative Aspects of Programming Languages, QAPL 2010, Pisa,Ialy, March 2012. Istituto di Scienza e Tecnologie dell’Informazione.

[42] N. Wolovick and S. Johr. A characterization of meaningful schedulers forcontinuous-time markov decision processes. In E. Asarin and P. Bouyer,editors, Formal Modeling and Analysis of Timed Systems, volume 4202of Lecture Notes in Computer Science, pages 352–367. Springer Berlin/ Heidelberg, 2006.

[43] R. Wunderling. Paralleler und objektorientierter Simplex-Algorithmus.PhD thesis, Technische Universitat Berlin, 1996. http://www.zib.de/

Publications/abstracts/TR-96-09/.

[44] L. Zhang and M. Neuhuer. Model checking interactive markov chains.In J. Esparza and R. Majumdar, editors, Tools and Algorithms for theConstruction and Analysis of Systems, volume 6015 of Lecture Notes inComputer Science, pages 53–68. Springer Berlin / Heidelberg, 2010.

83