a coverage analysis for safety property lists
DESCRIPTION
A Coverage Analysis for Safety Property Lists. Koen Lindström Claessen Chalmers University Gothenburg, Sweden. FMCAD 2007, Austin, Texas. Property-based Verification. Properties. Design. ?. Dynamic Verification. Checkers/ Monitors. PSL/SVA properties. Properties. Design. stimuli. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/1.jpg)
A Coverage Analysis for Safety Property ListsKoen Lindström ClaessenChalmers UniversityGothenburg, Sweden
FMCAD 2007, Austin, Texas
![Page 2: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/2.jpg)
Property-based Verification
Design Properties?
![Page 3: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/3.jpg)
Dynamic Verification
Design Properties
stimuli ? ? ?
Checkers/Monitors PSL/SVA
properties
![Page 4: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/4.jpg)
Simulation Coverage
Design Properties
stimuli
Statement coverage
Gate coverage
State space coverage
Path coverage
Register coverage
95%
75%
98%
66%
83%
In Formal Verification:
100%
![Page 5: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/5.jpg)
How About the Properties?
Design1. A => B
2. B & C => next A3. not (A and C)4. next A => B
?
Removing / adding one of these does not
affect coverage…
![Page 6: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/6.jpg)
Property Coverage
We need coverage for propertiesHow much is the design constrained by
properties?Do the properties express what we want to
say?Are any properties missing?(lots of work done on this)
The more the merrier!
![Page 7: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/7.jpg)
!
Mutation Coverage
Design PropertiesDesign
?
“Point not covered”
![Page 8: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/8.jpg)
Mutation Coverage Expensive
Many (failing) calls to model checker Involves the whole design
SolutionsSymbolically; still expensiveApproximations
Look at proof Subset of mutations
[Chockler, Kupferman,Vardi]
![Page 9: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/9.jpg)
This Talk
A property coverage analysis Independent of the design
Finds “forgotten cases” …… which indicate forgotten properties
Relatively cheapOnly involves properties Independent of design size
NaNNot a quantitative analysis
Design does not have to exist
There may exist multiple designs
Design may change after
analysis
![Page 10: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/10.jpg)
Forgotten Case Analysis
Design1. prop12. prop23. prop34. prop4
![Page 11: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/11.jpg)
“Forgotten Case”?
Properties specify a function : In Out?Too strictMeans complete specificationSynthesizable
BalanceStrict enough: to catch forgotten casesLoose enough: to be practically usable
![Page 12: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/12.jpg)
Choice: Forgotten Case
in1 1 0 1 0 0 0 1 1 1 0 0 …
in2 1 0 0 0 1 1 0 0 1 1 0 …
in3 0 0 0 1 1 0 0 0 0 1 1 …
out1 1 1 0 0 0 1 1 0 1 1 0 …
out2 0 1 1 1 0 0 0 0 1 0 1 …
PropertiesOK
0
Properties still OK
In this trace:out1 is not constrained by properties at time 6:
Forgotten case!
![Page 13: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/13.jpg)
Example: a FIFO
FIFO
in
put
get fst
num
err
put overrides
get
fst always outputs the first element
err signals for 1 clock cycle when something goes wrong – the FIFO does not break
![Page 14: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/14.jpg)
A First Property List
•always (put=1 & num=n next err=1)•always (put=1 & num<n next num=num+1)•always (put=1 & num=0 next fst=in)•always (put=1 & 0<num<n next fst=fst)•always (get=1 & put=0 & num=0 next err=1)•always (get=1 & put=0 & 0<num next num=num-1)
Safety properties
No restriction on the logic…
n = max. FIFO size
![Page 15: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/15.jpg)
Analyzing err
get 0 …
put 0 …
in 0 …
num 0 …
fst 0 …
err ? …
Not constrained at time 0
![Page 16: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/16.jpg)
Analyzing err: A Fix
•err=0
Holds at initial point in time
Add:
![Page 17: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/17.jpg)
Analyzing err
get 0 0 …
put 0 0 …
in 0 0 …
num 0 0 …
fst 0 0 …
err 0 ? …
Nothing goes wrong…
![Page 18: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/18.jpg)
Analyzing err: A fix
•always (get=0 & put=0 next err=0)
Add:
![Page 19: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/19.jpg)
Analyzing err
get 0 0 …
put 1 0 …
in 1 0 …
num 0 1 …
fst 0 1 …
err 0 ? …
Nothing goes wrong…
![Page 20: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/20.jpg)
Analyzing err: A fix
•always (put=1 & num<n next num=num+1)•always (get=1 & put=0 & 0<num next num=num-1)
•always (put=1 & num<n next num=num+1 & next err=0)
•always (get=1 & put=0 & 0<num next num=num-1 & next err=0)
Change to:
Analysis does not complain about err
anymore…
![Page 21: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/21.jpg)
Analyzing num
get 0 …
put 0 …
in 0 …
num ? …
fst 0 …
err 0 …
Not constrained at time 0
![Page 22: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/22.jpg)
Analyzing num: A Fix
•num=0
Holds at initial point in time
Add:
![Page 23: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/23.jpg)
Analyzing num
get 0 0 …
put 0 0 …
in 0 0 …
num 0 ? …
fst 0 0 …
err 0 0 …Not
constrained
![Page 24: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/24.jpg)
Analyzing num: A fix
•always (get=0 & put=0 next num=num)
Add:
![Page 25: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/25.jpg)
Analyzing num
get 1 0 …
put 0 0 …
in 0 0 …
num 0 ? …
fst 0 0 …
err 0 1 …
An error has occurred
![Page 26: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/26.jpg)
Analyzing num: A fix
•always (get=0 & put=0 next num=num)
Change to:
•always ((get=0 & put=0) v next err=1 next num=num)
The analysis is now happy about
num…
Dependencies between outputs…
![Page 27: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/27.jpg)
Analyzing fst
get 0 …
put 0 …
in 0 …
num 0 …
fst ? …
err 0 …
Not restricted
But we don’t want to specify it!
![Page 28: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/28.jpg)
Analyzing fst: A fix
Add:
•always (num=0 free fst)
A new keyword:free
Logically vacuous: “fst=fst”
For the analysis:free x means that x
is constrained: “don’t complain!”
![Page 29: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/29.jpg)
Analyzing fst
get 0 0 1 0 …
put 1 1 0 0 …
in 3 5 0 0 …
num 0 1 2 1 …
fst 0 3 3 ? …
err 0 0 0 0 …Not restricted:
FIFO behavior not specified in properties
![Page 30: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/30.jpg)
Unconstrained Outputs
Three causes: It is supposed to be unspecifiedWe decide not to specify it (complicated)We have forgotten to specify it
Cases we want to catch
Cases we have to be explicit
about
![Page 31: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/31.jpg)
Analyzing fst: A fix
•always (get=1 & put=0 & 0<num next num=num-1 & next err=0)
Change to:
•always (get=1 & put=0 & 0<num next num=num-1 & next err=0 & next free fst)
![Page 32: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/32.jpg)
Analyzing fst
get 0 0 …
put 0 0 …
in 0 0 …
num 0 0 …
fst 0 ? …
err 0 0 … Real forgotten case
![Page 33: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/33.jpg)
Analyzing fst: A fix
•always ((get=0 & put=0) v next err=1 next num=num)
Change to:
•always ((get=0 & put=0) v next err=1 next num=num & next fst = fst)
The analysis is now happy about
fst…
Luckily, we used free…
![Page 34: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/34.jpg)
New & changed properties
•err=0 & num=0
•always (get=0 & put=0 next err=0)
•always (put=1 & num<n next num=num+1 & next err=0)
•always (get=1 & put=0 & 0<num next num=num-1 & next err=0 & next free fst)
•always ((get=0 & put=0) v next err=1 next num=num & next fst = fst)
initial values
implicit behavio
rconstan
t behavio
r
![Page 35: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/35.jpg)
Implementation For output s, find a trace where
Props(s) is OKProps(s’) is OKExists exactly one t where s[t]≠s’[t]free(s)[t] should be false
(this trace must be infinite)
![Page 36: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/36.jpg)
Implementation (I)
Find a trace satisfying:Props(s) & Props(s’) & ◊!(s≠s’)
With the use of free:Props(s,free_s) & Props(s’,free_s) & ◊!
(~free_s & s≠s’)
Using a standard LTL model
checker (NuSMV)
![Page 37: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/37.jpg)
Implementation (II)
Property observer for safety property phi:□OK holds iff. phi holds
Analysis for property observers:□OK(s) & □OK(s’) ◊!(s≠s’)
![Page 38: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/38.jpg)
Discussion (I)
Forgotten caseGiven an output signal s and a time t, and
given the values of all other signals at all points in time, and given all values of s not at time t, do the properties force the value of s at time t?
Alternatives…
![Page 39: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/39.jpg)
Discussion (II)
Freeness“free s” does not indicate that s can take on
any valuerather, it is an artifical way of constraining s
for the sake of the analysis, without actually restricting it logically
![Page 40: A Coverage Analysis for Safety Property Lists](https://reader036.vdocuments.us/reader036/viewer/2022062500/56814f73550346895dbd258f/html5/thumbnails/40.jpg)
Conclusion: This Analysis Identifies forgotten cases
Which inspire forgotten properties Forces to specify when outputs are free
Distinction between forgotten cases and underconstrainedness
Is design-independentpre-design / multiple implementationscost