eecs 144/244: system modeling, analysis, and optimization · adequate for modelling many real-life...

EECS 144/244: System Modeling, Analysis, andOptimization

Stochastic SystemsLecture: Continuous Time Stochastic Systems

Alexandre Donze

University of California, Berkeley

April 26, 2013

EECS 144/244 – Continuous Time Stochastic Systems 1 / 36

Probabilistic Models

Fully Probabilistic Nondeterministic

Discrete TimeDiscrete-TimeMarkov Chains(DTMCs)

Markov DecisionProcesses (MDPs)

Continuous TimeContinuous-TimeMarkov Chains(DTMCs)

CTMDPs, Prob-abilistic TimedAutomaton (PTAs)

EECS 144/244 – Continuous Time Stochastic Systems Introduction 2 / 36

From DTMC to CTMCs

Time in a DTMC proceeds in discrete steps

I accurate model of (discrete) time units (e.g. clock ticks)

I or, no information assumed about the time transitions take

Continuous-time Markov chains (CTMCs): dense model of time

I transitions can occur at any (real-valued) time instant

I modelled using exponential distributions

I suits modelling of: performance/reliability (e.g. of computernetworks, manufacturing systems, queueing networks), biologicalpathways, chemical reactions, ...


1 Exponential Distribution

2 Continuous Time Markov Chains

3 Specifying Probabilistic Properties





EECS 144/244 – Continuous Time Stochastic Systems Exponential Distribution 5 / 36

Continuous Probability DistributionDefined by

I a cumulative distribution function : F (t) = Pr(X ≤ t) =∫ t

−∞ f(x)dx

I where f is the probability density function

I Note: for all t, Pr(X = t) =

0 !

Example: uniform distribution

f(t) =

{1

b−a if a ≤ t ≤ b ,0 otherwise.

F (t) =

0 if t ≤ a,t−ab−a if a ≤ t ≤ b,1 if t > b.

a b

0

1/(b−a)

a b

0

0.2

0.4

0.6

0.8

1




−∞ f(x)dx


I Note: for all t, Pr(X = t) = ?

0 !


f(t) =

{1


F (t) =


a b

0

1/(b−a)

a b

0

0.2

0.4

0.6

0.8

1




−∞ f(x)dx


I Note: for all t, Pr(X = t) = 0 !


f(t) =

{1


F (t) =


a b

0

1/(b−a)

a b

0

0.2

0.4

0.6

0.8

1


Exponential Distribution

A continuous random variable X is exponential with parameter λ if itsdensity functions is

f(t) =

{λ · e−λt if t > 0 ,

0 otherwise .

Cumulative distribution function

F (t) = Pr(X ≤ t) =∫ t

0λ · e−λxdx = 1− e−λt

Other properties

I negation : Pr(X > t) = e−λt

I mean : E[X] =∫∞0 x · λ · e−λxdx = 1

λ

I variance: V ar(X) = 1λ2


Exponential Distribution

A continuous random variable X is exponential with parameter λ if itsdensity functions is

f(t) =

{λ · e−λt if t > 0 ,

0 otherwise .λ= “rate”


F (t) = Pr(X ≤ t) =∫ t

0λ · e−λxdx = 1− e−λt

Other properties

I negation : Pr(X > t) = e−λt

I mean : E[X] =∫∞0 x · λ · e−λxdx = 1

λ

I variance: V ar(X) = 1λ2


Exponential Distribution - Examples

Probability distribution function

0 1 2 3 40

1

2

3

4

5

λ=5λ=1λ=.5


0 1 2 3 40

0.2

0.4

0.6

0.8

1

λ=5λ=1λ=.5

The more λ increases, the faster the c.d.f. approaches 1


Exponential distribution

Adequate for modelling many real-life phenomena

I failuresI e.g. time before machine component fails

I inter-arrival timesI e.g. time before next call arrives to a call center

I biological systemsI e.g. times for reactions between proteins to occur


Useful Properties

The exponential distribution is memoryless

I Pr(X > t1 + t2|X > t1) = Pr(X > t2)

I it is the only memoryless continuous distribution

I the discrete time equivalent is the geometric distribution:

Probability of a failure after k trials knowing the probability p offailure of one trial:

P (X = k) = (1− p)kp


Useful Properties

The minimum of two exponential distributions is an exponentialdistribution

I X1 ∼ Exp(λ1), X2 ∼ Exp(λ2)I Y = min(X1, X2) ∼ Exp(λ1 + λ2)

I generalises to minimum of n distributions

Comparison of two exponential distributionsThe probability of X1 < X2 is given by

Pr(X1 < X2) =λ1

λ1 + λ2





EECS 144/244 – Continuous Time Stochastic Systems Continuous Time Markov Chains 12 / 36

Continuous-time Markov Chains

Informally

I labelled transition systems, augmented with rates

I continuous time delays, exponentially distributed

Definition

A CTMC is a tuple (S, s0, R, L) where

I S is a set of states

I s0 ∈ S is the initial state

I R : S × S → R+ is the transition rate matrix

I L : S → 2AP is a labelling with atomic propositions in AP


CTMCs Semantics

The transition rate matrix assigns rates to each pair of states

I used as a parameter to an exponential distribution

I transition between s and s′ when R(s, s′) > 0

I probability triggered before t time units: 1− e−R(s,s′)t

Race conditionIf there exists multiple s′ with R(s, s′) > 0, the first transition to triggerdetermines the next state.


Example - Modeling a queue of jpbs

I Initially the queue is empty

I jobs arrive with rate 3/2 (i.e. mean inter-arrival time is 2/3

I jobs are served with rate 3 (i.e. mean service time is 1/3)

I maximum size of the queue is 3

I state-space: S : {si}i=0...3 where si indicates i jobs in the queue

s0start

{empty}

s1 s2 s3

{full}3/2 3/2 3/2

333


Interesting Questions for CTMCs

How long is spent in s before a transition occurs ?

I minimum of exponential distributions of outgoing transitions

I i.e. exponential distribution with sum of outgoing rates

Exit rate E(s) =∑s′∈S

R(s, s′)

Note:

I the probability of leaving a state s within [0, s] is 1− e−E(s)t

I s is called absorbing if E(s) = 0 (no outgoing transitions)


Interesting Questions for CTMCs

Which transition is eventually taken from state s ?

I Recall that Pr(X1 < X2) =λ1

λ1+λ2

I This can generalized to Pr(X1 = mini=1...nXi) =λ1∑λi

I Thus the probability that next state from s is s′ is given by

PR(s, s′) =

R(s,s′)E(s) if E(s) > 0,

1 if E(s) = 0 and s = s′,

0 if E(s) = 0 and s 6= s′.


Embedded DTMC

The transition target state is independent from the time the transition occurs.

I.e. we can define a DTMC that abstracts a CTMC by describing only statestransitions without time information

Definition

The embedded DTMC of a CTMC (S, s0, R, L) is the DMTC (S, s0, PR, L)


Embedded DTMC - Example

What is the embedded DTMC of

s0start

{empty}

s1 s2 s3

{full}3/2 3/2 3/2

333

?

s0start

{empty}

s1 s2 s3

{full}




s0start

{empty}

s1 s2 s3

{full}3/2 3/2 3/2

333

?

s0start

{empty}

s1 s2 s3

{full}1




s0start

{empty}

s1 s2 s3

{full}3/2 3/2 3/2

333

?

s0start

{empty}

s1 s2 s3

{full}13/2

3/2+3




s0start

{empty}

s1 s2 s3

{full}3/2 3/2 3/2

333

?

s0start

{empty}

s1 s2 s3

{full}1 13




s0start

{empty}

s1 s2 s3

{full}3/2 3/2 3/2

333

?

s0start

{empty}

s1 s2 s3

{full}1 13

13

23

23

23


Interesting Question

What is the probability of being in state sj at time t starting in si ?Define

P (t) =

P11(t) . . . P1n(t)... . . .

...Pn1(t) . . . Pnn(t)

with Pij(t) = Pr(s(t) = sj |s(t = 0) = si)

and the infinitesimal generator matrix Q

Q(t) =

Q11(t) . . . Q1n(t)... . . .

...Qn1(t) . . . Qnn(t)

with Qij =

{R(si, sj) if i 6= j,

−∑

k 6=iR(si, sk) if i = j.

Then one can show that P (t) satisfies the linear ODE:

P (t) = P (t)′Q,P (0) = I


Simple Example


CTMC path

A path ω is a sequence s0t0s1t1s2t2... such that

I ∀i, R(si, si+1) 0 and ti ∈ R>0

I ti is the time spent in si

The path ω is finite if for some k, the state sk is absorbing (i.e. R(s, s′) =

Path(s) denotes all paths starting in s.


Simulation Algorithm

Main IdeaAs the next state probability is independent from the time when the transitiontakes place, use two independent stochastic processes for si and ti.

1. Init i = 0, si = s0

2. loop

3. Pick ti ∈ R>0 using exponential distribution with rate E(s)

4. Pick si+1 using discrete distribution PR(si, s′) of embedded DTMCs

5. i = i+ 1

6. end loop

Sometimes referred as Gillespie’s algorithm, and used by its author for stochasticsimulation of chemical reactions


Example: A Chemical Reaction

I Three species: A, B and AB, three reactions:

I A and B collide and produce AB A+Bk1−→ AB

I AB breaks into A and B ABk2−→ A+B

I degradation of A Ak3−→ ∅

I CTMC with state-space(]A, ]B, ]AB)

I A and B collide at rate k1]A]B

I AB breaks at rate k2]AB

I A degrades at rate k3]A

2,2,0 1,1,1 0,0,2

1,2,0 0,1,1

0,2,0

4k1 k1

2k2k2

2k32k1

k2

k3

k3











2,2,0 1,1,1 0,0,2

1,2,0 0,1,1

0,2,0

4k14k1 k1k1

2k2k2

2k32k12k1

k2

k3

k3











2,2,0 1,1,1 0,0,2

1,2,0 0,1,1

0,2,0

4k1 k1

2k22k2k2k2

2k32k1

k2k2

k3

k3











2,2,0 1,1,1 0,0,2

1,2,0 0,1,1

0,2,0

4k1 k1

2k2k2

2k32k32k1

k2

k3k3

k3k3


CTMCs for Chemical Reactions

How many states does a CTMC of a chemical reaction has ?It depends on

I The number of reactions

I The number species types

I The initial number of each specie

If there is a production reaction, e.g., ∅ → A, the number can be infinite


Stochastic versus deterministic models of chemicalreactions

Recall that we can formulate ODE to describe a deterministic evolution ofthe number of molecules (using mass-action laws)

The stochastic (CTMC) model is believed is to be more realistic, but canbe quickly intractable.In general,

I For large populations of molecules the deterministic model is used

I For small populations use the stochastic model





EECS 144/244 – Continuous Time Stochastic Systems Specifying Probabilistic Properties 27 / 36

CSL (slides: David Parker)


CSL Syntax (slides: David Parker)


CSL Semantics (slides: David Parker)


CSL Example (slides: David Parker)


CSL Model Checking

(See PRISM litterature for more details...)

I For untimed operators, equivalent to PCTL on embedded DTMCs

I For timed operators, can be reduced to computation of transientprobabilities (such as matrix P (t)) -¿ complex

I An alternative is using Statistical Model Checking, approximate butmore scalable


Statistical Model CheckingAssume we can decide ω |= φ. Based on simulations, decide hypothesisH1 : Pr≥θ(ω |= φ) against H0 : Pr<θ(ω |= φ)

Bounds on the error of choosing H1 instead of H0, depending on thenumber of positive runs


eecs 144/244: system modeling, analysis, and optimization · adequate for modelling many real-life...

Documents