computing probabilistic bisimilarity distances via policy ... · wolfgang doeblin intro-duced the...
Post on 02-Aug-2020
0 Views
Preview:
TRANSCRIPT
Computing Probabilistic Bisimilarity Distancesvia Policy Iteration
Franck van Breugel
(joint work with Qiyi Tang)
October 4, 2016
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 1 / 59
Table of Contents
1 Labelled Markov ChainsProbabilistic bisimilarityCouplingsProbabilistic bisimilarity distances
2 Computing probabilistic bisimilarity distancesThree algorithmsSimple stochastic gamesB2LM algorithm is exponentialPerformance comparison
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 2 / 59
Behavioural Equivalences
Fundamental questionDo two states of a systems behave thesame?
Behavioural equivalence is an equivalence re-lation.
Robin Milner introduced
bisimilarity, the most
well-known behavioural
equivalence, in 1979.
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 3 / 59
Behavioural Equivalences
Fundamental questionDo two states of a systems behave thesame?
Behavioural equivalence is an equivalence re-lation.
Robin Milner introduced
bisimilarity, the most
well-known behavioural
equivalence, in 1979.
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 3 / 59
Model of probabilistic system
12
12
12
12
1 1
1
Labelled Markov chain
Andrey Markov pro-
duced the first results for
Markov chains in 1906.
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 4 / 59
Transitions
τ ∈ S → Dist(S)
For each state s, the transitions of s are presented by aprobability distribution τ(s) on S.
u v
s12
12
τ(s)(w) =
12 if w = u12 if w = v0 otherwise
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 5 / 59
Probabilistic bisimulation
DefinitionAn equivalence relation R is a probabilisticbisimulation if for all (s, t) ∈ R,
`(s) = `(t) and(τ(s), τ(t)) ∈ R.
DefinitionProbabilistic bisimilarity is the largestprobabilistic bisimulation.
Kim Larsen and Arne
Skou introduced
probabilistic bisimilarity
in 1989.
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 6 / 59
Lifting
DefinitionLet R ⊆ S × S be an equivalence relation. The lifting of R,R ⊆ Dist(S)× Dist(S), is defined by
(µ, ν) ∈ R if µ([s]) = ν([s]) for all s ∈ S
Next, we will provide an alternative characterization of lifting.
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 7 / 59
Coupling
DefinitionA coupling of probability distributions µ and νon S is a probability distribution ω on S × Swith marginals µ and ν, that is, for all u,v ∈ S, ∑
v∈S
ω(u, v) = µ(u)∑u∈S
ω(u, v) = ν(v)
The set of couplings of µ and ν is denoted byΩ(µ, ν).
Wolfgang Doeblin intro-
duced the notion of a
coupling in 1936 (pub-
lished in 1938).
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 8 / 59
Coupling
s
s
t
t
s
t
12
12
12
12
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 9 / 59
Coupling
s
s
t
t
s
t
12
12
12
12
12
12
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 10 / 59
Coupling
s
s
t
t
s
t
12
12
12
12
12
12
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 11 / 59
Alternative characterization of lifting
TheoremLet R ⊆ S × S be an equivalence relation.
(µ, ν) ∈ R iff ∃ω ∈ Ω(µ, ν) : support(ω) ⊆ R
Bengt Jonsson and
Kim Larsen pro-
vided the alternative
characterization in 1991.
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 12 / 59
Coupling
There are infinitely many couplings (r ∈ [0, 12 ]).
s
s
t
t
s
t
12
12
12
12
12 − r
12 − r
r
r
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 13 / 59
Coupling
Proposition
Ω(τ(s), τ(t)) is a convex polytope.
PropositionA concave function on a convex polytope attains its minimum ata vertex.
Proposition
The set V (Ω(τ(s), τ(t))) of vertices of Ω(τ(s), τ(t)) is finite.
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 14 / 59
Coupling
There are two vertices (r ∈ 0, 12).
s
s
t
t
s
t
12
12
12
12
12 − r
12 − r
r
r
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 15 / 59
Alternative characterization of lifting
Theorem (TB 2016)Let R ⊆ S × S be an equivalence relation.
(µ, ν) ∈ R iff ∃ω ∈ V (Ω(µ, ν)) : support(ω) ⊆ R
Proof sketchOrder the states s1, . . . , sn such that equivalent states areconsecutive.Apply the North-West corner method.Prove some loop invariants (by means of Dafny).
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 16 / 59
Behavioural pseudometric
Fundamental problemBehavioural equivalences are not robust forsystems with real-valued data.
12
12
1 1
0.51 0.49
1 1
Alessandro Giacalone,
Chi-Chang Jou and
Scott Smolka observed
that probabilistic
bisimilarity, the most
well-known behavioural
equivalence for proba-
bilistic systems, is not
robust in 1990.
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 17 / 59
Behavioural pseudometric
Fundamental problemBehavioural equivalences are not robust for systems withreal-valued data.
Robust alternativeInstead of an equivalence relation
∼ : S × S → true, false
use a pseudometric
d : S × S → [0,1].
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 18 / 59
Probabilistic bisimilarity
s
s
t
u
v
t
s
t
u
v
23
13
13
13
13
support(ω) ⊆ R
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 19 / 59
Probabilistic bisimilarity
s
s
t
u
v
t
s
t
u
v
23
13
13
13
13
support(ω) ⊆ R
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 20 / 59
Probabilistic bisimilarity
s
s
t
u
v
t
s
t
u
v
23
13
13
13
13
13
13
13
support(ω) ⊆ R
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 21 / 59
Probabilistic bisimilarity
Let us represent the equivalence relation R with the followingdistance function.
r(s, t) =
0 if (s, t) ∈ R1 otherwise
Then the conditionsupport(ω) ⊆ R
is equivalent to ∑u,v∈S
ω(u, v) r(u, v) = 0
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 22 / 59
Quantitative generalization of probabilistic bisimilarity
s
s
t
u
v
t
s
t
u
v
23
13
13
13
13
minimize∑
u,v∈S
ω(u, v) d(u, v)
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 23 / 59
Quantitative generalization of probabilistic bisimilarity
DefinitionProbabilistic bisimilarity is the largest equivalence relation ∼such that s ∼ t implies
`(s) = `(t) and∃ω ∈ V (Ω(τ(s), τ(t))) : support(ω) ⊆ ∼.
DefinitionThe probabilistic bisimilarity pseudometric is the smallestd : S × S → [0,1] such that
d(s, t) =
1 if `(s) 6= `(t)
minω∈V (Ω(τ(s),τ(t)))
∑u,v∈S
ω(u, v) d(u, v) otherwise
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 24 / 59
Probabilistic bisimilarity pseudometric
Josee Desharnais, Vineet Gupta, Radha Jagadeesan andPrakash Panangaden. Metrics for Labeled Markov Systems.CONCUR 1999.
Theorem (DGJP 1999)
s ∼ t if and only if d(s, t) = 0.
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 25 / 59
Kantorovich metric
Let µ, ν ∈ Dist(S) and d : S × S → [0,1].
maxf∈(S,d)---<[0,1]
∑s∈S
f (s) (µ(s)− ν(s))
= minω∈Ω(µ,ν)
∑u.v∈S
ω(u, v) d(u, v)
= minω∈V (Ω(µ,ν))
∑u.v∈S
ω(u, v) d(u, v) Leonid Kantorovich first
published this metric in
1942.
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 26 / 59
Table of Contents
1 Labelled Markov ChainsProbabilistic bisimilarityCouplingsProbabilistic bisimilarity distances
2 Computing probabilistic bisimilarity distancesThree algorithmsSimple stochastic gamesB2LM algorithm is exponentialPerformance comparison
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 27 / 59
Algorithm
QuestionHow to compute the probabilistic bisimilarity distances for alabelled Markov chain?
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 28 / 59
Algorithm to compute the bisimilarity distances
Express d(s, t)< q in the first ordertheory over the reals.Use the binary search method toapproximate d(s, t).
Babita Sharma, Franck van Breugel andJames Worrell. Approximating a BehaviouralPseudometric without Discount for Probabilis-tic Systems. FoSSaCS 2007.
Alfred Tarski showed that
the first order theory over
the reals is decidable in
1948.
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 29 / 59
Algorithm to compute the bisimilarity distances
Express d(s, t) as a linear program.Use the ellipsoid method to computed(s, t).
As separation algorithm, to solve aminimum cost flow problem, use thenetwork simplex algorithm.
Di Chen, Franck van Breugel and James Wor-rell. On the Complexity of Computing Proba-bilistic Bisimilarity. FoSSaCS 2012.
Leonid Khachiyan
proved the polynomial-
time solvability of linear
programs in 1979.
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 30 / 59
Algorithm to compute the bisimilarity distances
Giorgio Bacci, Giovanni Bacci, Kim Larsen and Radu Mardare.On-the-Fly Exact Computation of Bisimilarity Distances. TACAS2013.
B2LM algorithm = basic algorithm︸ ︷︷ ︸this talk
+ optimization︸ ︷︷ ︸on-the-fly
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 31 / 59
Simple stochastic game (SSG)
0 1
avg avg avg avg
maxmax maxmax
minmin
Anne Condon was the
first to study simple
stochastic games from
a computational point of
view in 1992.
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 32 / 59
Values of a SSG
DefinitionThe value of a vertex is the probability that the max player winsthe game (reaches 1) provided that both players use optimalstrategies (the min player tries not to reach 1).
0 1
avg avg avg avg
maxmax maxmax
minmin
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 33 / 59
From LMCs to SSGs
For each labelled Markov chain we constructa corresponding simple stochastic game suchthat
LMC SSGdistance valuealgorithm simple policy iteration
Ronald Howard intro-
duced policy iteration in
1958.
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 34 / 59
From LMCs to SSGs
With every pair of states (s, t) of the LMC we associate a vertexof the SSG.
If `(s) 6= `(t) then d(s, t) = 1.
1
If s ∼ t then d(s, t) = 0.
0
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 35 / 59
From LMCs to SSGs
Otherwise,
d(s, t) = minω∈V (Ω(τ(s),τ(t)))
∑u,v∈S
ω(u, v) d(u, v)
st
ω1 · · · ωn
u1v1 · · · umvm
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 36 / 59
From LMCs to SSGs
Otherwise,
d(s, t) = minω∈V (Ω(τ(s),τ(t)))
∑u,v∈S
ω(u, v) d(u, v)
st
ω1 · · · ωn
u1v1 · · · umvm
ω1(u1, v1)
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 37 / 59
Correctness of simple policy iteration
Simple policy iteration
choose a random initial policywhile exists a vertex which is not locally optimal
adjust the policy at that vertex
Theorem (Condon 1992)Simple policy iteration computes the value function if the simplestochastic game terminates with probability one (no matterwhich strategy the players use).
PropositionIf we do not map a pair of probabilistic bisimilar states to a zerosink, then the resulting simple stochastic game may notterminate with probability one.
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 38 / 59
Correctness of simple policy iteration
The labelled Markov chain
s1s1 s2s2
s0
t1t1 t2t2
t0
is mapped to the simple stochastic game
3× 0 +6× 1
PropositionThis simple stochastic game terminates with probability one.
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 39 / 59
Correctness of simple policy iteration
If we do not map a pair of probabilistic bisimilar states to a zerosink, the labelled Markov chain
s1s1 s2s2
s0
t1t1 t2t2
t0
is mapped to the simple stochastic game
1
1• •
PropositionThis simple stochastic game does not terminate with probabilityone.
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 40 / 59
B2LM algorithm is exponential
Simple policy iteration
choose a random initial policywhile exists a vertex which is not locally optimal
adjust the policy at that vertex
TheoremFor each n ∈ N, there exists a labelled Markov chain of sizeO(n) such that simple policy iteration takes Ω(2n) iterations.
Proof idea: Implement an “n-bit counter.”
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 41 / 59
B2LM algorithm is exponential
We start with the following labelled Markov chain.
1 1
1
1 14
34
1 1
1
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 42 / 59
B2LM algorithm is exponential
The labelled Markov chain corresponds to the following simplestochastic game.
1
1
1
1
1 0
1
1
1
0 014
14
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 43 / 59
B2LM algorithm is exponential
1
1
1
1
1 0
1
1
1
0 014
14
000
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 44 / 59
B2LM algorithm is exponential
1
1
1
1
1 0
1
1
1
1
0 014
14
000
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 45 / 59
B2LM algorithm is exponential
1
1
1
1
1
1 0
1
1
1
1
0 014
14
000
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 46 / 59
B2LM algorithm is exponential
1
1
1
1
1
1
1 0
1
1
1
1
0 014
14
000
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 47 / 59
B2LM algorithm is exponential
1
1
1
1
1
1 12
1 0
1
1
1
1
0 014
14
000
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 48 / 59
B2LM algorithm is exponential
1
1 78
1
1
1
1 12
1 0
1
1
1
1
0 014
14
000
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 49 / 59
B2LM algorithm is exponential
1516
1 78
1
1
1
1 12
1 0
1
1
1
1
0 014
14
100
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 50 / 59
B2LM algorithm is exponential
1516
1 78
1
1
34
1 12
1 0
1
1
1
1
0 014
14
110
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 51 / 59
B2LM algorithm is exponential
78
1 78
1
1
34
1 12
1 0
1
1
1
1
0 014
14
010
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 52 / 59
B2LM algorithm is exponential
78
1 58
1
1
34
1 12
1 0
1
0
1
1
0 014
14
011
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 53 / 59
B2LM algorithm is exponential
1316
1 58
1
1
34
1 12
1 0
1
0
1
1
0 014
14
111
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 54 / 59
B2LM algorithm is exponential
1316
1 58
1
1
12
1 12
1 0
1
0
1
1
0 014
14
101
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 55 / 59
B2LM algorithm is exponential
34
1 58
1
1
12
1 12
1 0
1
0
1
1
0 014
14
001
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 56 / 59
Performance comparison
Aron Itai and Michael Rodeh. Symmetry breaking in distributednetworks. Information and Computation, 88(1):60–87, 1990.
SBW: –CBW: more than 10 hours for N = 3 and K = 2B2LM:
without bisimilarity with bisimilarity
N K µ σ µ σ
3 2 4.02 0.15 2.70 0.08
4 2 478.67 1.49 399.833 0.82
3 3 1151.10 0.36 753.73 0.58
5 2 62126.78 1264.50 58862.35 1512.58
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 57 / 59
Conclusion
The basic B2LM algorithm is simple policy iteration.To define the simple stochastic game we need to decideprobabilistic bisimilarity.
In the worst case, the (basic) B2LM algorithm isexponential.In practice, the (basic) B2LM algorithm performs muchbetter than all other algorithms.
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 58 / 59
Related and future work
Use general policy iteration to compute the probabilisticbisimilarity distances for labelled Markov chains.Use (simple/general) policy iteration to compute theprobabilistic bisimilarity distances for probabilisticautomata.
Franck van Breugel (joint work with Qiyi Tang) Computing Probabilistic Bisimilarity Distances 59 / 59
top related