verification of intelligent softwarepecheur/talks/vis-ksc-nov00.pdf · intelligent software, how to...
TRANSCRIPT
![Page 1: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/1.jpg)
©Charles Pecheur, RIACS / NASA Ames 1KSC Nov 2000
Verification of Intelligent Software
Charles Pecheur (RIACS / NASA Ames)
![Page 2: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/2.jpg)
©Charles Pecheur, RIACS / NASA Ames 2KSC Nov 2000
Contents
Model Checking for Intelligent Software
• Why?Intelligent software, how to verify it?
• What?A bird's-eye view of model checking
• How?Experiences in the ASE Group
![Page 3: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/3.jpg)
©Charles Pecheur, RIACS / NASA Ames 3KSC Nov 2000
Autonomous Systems
"Faster, better, cheaper" spacecrafts
=> add on-board intelligence
• From self-diagnosisto on-board science.
• Smaller mission control crews=> reduced cost
• Less reliance on control link=> OK for deep space
![Page 4: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/4.jpg)
©Charles Pecheur, RIACS / NASA Ames 4KSC Nov 2000
Model-Based Autonomy
• Based on AI technology
• General reasoning engine +application-specific model
• Use model to respond tounanticipated situations
ReasoningEngine
Model
commands status
Spacecraft
Autonomous controller
model of
![Page 5: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/5.jpg)
©Charles Pecheur, RIACS / NASA Ames 5KSC Nov 2000
Example: Remote Agent
• From Ames ARA Group (+ JPL)
• On Deep Space One in May 1999 (1st AI in space!)
Model Model
![Page 6: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/6.jpg)
©Charles Pecheur, RIACS / NASA Ames 6KSC Nov 2000
Controlled vs. Autonomous
Controller
“Valve 1 stuck” “Open valve 2”
Tester
“Here we are”“Go to Saturn” Tester
?
Controller
Planner MIRExec
![Page 7: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/7.jpg)
©Charles Pecheur, RIACS / NASA Ames 7KSC Nov 2000
Testing intelligent software?
• Programs are much more complex
• Many more scenarios
=> testing gives low coverage
• Concurrency!Due to scheduling,the same inputs (test) can givedifferent outputs (results)
=> test results are not reliable
A.read?0
A.write!1
B.read?1
B.write!2
B.read?0
B.write!1
A.write!1
0
0
01
1 1
12
A.read?x;A.write !x+1;
B.read?y;B.write !y+1;
0
![Page 8: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/8.jpg)
©Charles Pecheur, RIACS / NASA Ames 8KSC Nov 2000
Contents
Model Checking for Intelligent Software
• Why?Intelligent software, how to verify it?
• What?A bird's-eye view of model checking
• How?Experiences in the ASE Group
![Page 9: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/9.jpg)
©Charles Pecheur, RIACS / NASA Ames 9KSC Nov 2000
Model Checking
Check whether a system S satisfies a property Pby exhaustive exploration of all executions of S
• Controls scheduling => better coverage
• Can be done at early stage => less costly
• Widely used in hardware, coming in software
• Examples: Spin (Bell Labs), Murphi (Stanford)
![Page 10: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/10.jpg)
©Charles Pecheur, RIACS / NASA Ames 10KSC Nov 2000
Model ...
Controller
Planner MIRExec
ModelingAbstraction
Verification
![Page 11: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/11.jpg)
©Charles Pecheur, RIACS / NASA Ames 11KSC Nov 2000
Model Checking
Controller
Planner MIRExec
“Valve is closed whenTank is empty”
AG (tank=empty=> valve=closed)
ModelingAbstraction
Verification
![Page 12: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/12.jpg)
©Charles Pecheur, RIACS / NASA Ames 12KSC Nov 2000
State Space Explosion
K processes with N local states ≤ NK global states
Theory:
Practice:
Controller
Planner MIRExec
“Valve is closed whenTank is empty”
Model CheckerRun
Yes/No because ...
Controller
Planner MIRExec
“Valve is closed whenTank is empty”
Model CheckerRun
No morememory
![Page 13: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/13.jpg)
©Charles Pecheur, RIACS / NASA Ames 13KSC Nov 2000
Modeling
This is the tough job!
• Translation: to model checker's syntaxe.g. C —> Promela (Spin)
• Abstraction: ignore irrelevant partse.g. contents of messages
• Simplification: downsize relevant partse.g. number of processes, size of buffers
Controller
Planner MIRExec
“Valve is closed whenTank is empty”
Model CheckerRun
Yes/No because ...TranslationAbstraction
Simplification
![Page 14: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/14.jpg)
©Charles Pecheur, RIACS / NASA Ames 14KSC Nov 2000
Temporal Logic
• Propositional logic + quantifiers over executions
• Example: "every request gets a response"AG (Req => AF Resp)
Always Globally, if Req then Always Finally Resp
• Branching (CTL) vs. linear (LTL)– different verification techniques
– neither is more general than the other
• Model checking without TL– Assertions, invariants
– Compare systems, observers
![Page 15: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/15.jpg)
©Charles Pecheur, RIACS / NASA Ames 15KSC Nov 2000
Symbolic Model Checking
• Manipulates sets of states,Represented as boolean formulas,Encoded as binary decision diagrams.
• Can handle larger state spaces (1050 and up).
• BDD computations:– Good in average but exponential in worst case.
– Computation time depends on BDD size=> number of variables, complexity of formulas,but not directly state space size.
• Example: SMV (Carnegie Mellon U.)
x
y
0 1 2 ...0
1...
x=2 ∨ y=1
1 0
x=2
y=1
![Page 16: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/16.jpg)
©Charles Pecheur, RIACS / NASA Ames 16KSC Nov 2000
Real-Time and Hybrid
• "Classic" model checking: finite state, un-timed
• Real-time model checking: add clockse.g. Khronos (Verimag), Uppaal (Uppsala/Aalborg)
• Hybrid model checking: add derivativese.g. Hytech (Berkeley)
More complex problems & less mature tools
cl<5 cl≥4cl:=0
dx/dt=2 x≥4x:=0
![Page 17: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/17.jpg)
©Charles Pecheur, RIACS / NASA Ames 17KSC Nov 2000
Contents
Model Checking for intelligent software
• Why?intelligent software, how to verify it?
• What?A bird's-eye view of model checking
• How?Experiences in the ASE Group
![Page 18: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/18.jpg)
©Charles Pecheur, RIACS / NASA Ames 18KSC Nov 2000
Verification ofRemote Agent Executive
• Smart executive system with AI features (Lisp)
• Modeled (1.5 month) andModel-checked with Spin (less than a week)
• 5 concurrency bugs found, that would have beenhard to find through traditional testing
(Lowry, Havelund and Penix)
![Page 19: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/19.jpg)
©Charles Pecheur, RIACS / NASA Ames 19KSC Nov 2000
Hunting the RAX Bug
• 18 May 1999: Remote Agent Experimentsuspended following a deadlock in RA EXEC=> Q: could V&V have found it?
• Over-the-week-end "clean room" experiment• => A: V&V found it... two years ago!
Similar to one of the 5 bugs found before (elsewhere)– Highly unlikely to occur– Never occurred during thorough testing– Occurred in flight!
• Morale: Testing not enough for concurrency bugs!
(Lowry, White, Havelund, Pecheur, ...)
![Page 20: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/20.jpg)
©Charles Pecheur, RIACS / NASA Ames 20KSC Nov 2000
Verification ofModel-Based Autonomy
Reasoning Engine• Relatively small, generic
algorithm => use prover
• Requires V&V expert levelbut once and for all
• At application level,assume correctness(cf. compiler)
ReasoningEngine
Model
Autonomous Controller
Model• Complex assembly of
interacting components=> model checking
• Avoid V&V experts=> automated translationNot too hard because modelsare abstract
Reasoning Engine + Model ???
![Page 21: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/21.jpg)
©Charles Pecheur, RIACS / NASA Ames 21KSC Nov 2000
Verification ofPlanner/Scheduler Models
• Model-based planner from Remote AgentModels: constraint style, real-time
• Small sample model translated by handSubset of the full modeling language, untimed
• Compare 3 model checkers: Spin, Murphi, SMV=> SMV much easier and faster (≈0.05s vs. ≈30s)
• Continuation (Khatib): handle timed propertiesusing real-time model checker (Uppaal)
(Penix, Pecheur and Havelund)
![Page 22: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/22.jpg)
©Charles Pecheur, RIACS / NASA Ames 22KSC Nov 2000
MRMI
Co
mm
an
d
DiscretizedObservations
Mode updates
GoalsModel
Reconfig Command
current state
Plan Execution System
High level operational plan
Livingstone
Courtesy Autonomous Systems Group, NASA Ames
The Livingstone MIR
Remote Agent's model-based fault recovery sub-system
![Page 23: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/23.jpg)
©Charles Pecheur, RIACS / NASA Ames 23KSC Nov 2000
Verification ofLivingstone Models
LivingstoneModel
SMVModel
LivingstoneRequirement
SMVRequirement
LivingstoneTrace
SMVTrace
Livingstone
SMV
TRANSLATOR
Autonomy Verification
![Page 24: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/24.jpg)
©Charles Pecheur, RIACS / NASA Ames 24KSC Nov 2000
Livingstone to SMVTranslation
MODULE valveVAR mode: {Open,Closed,
StuckO,StuckC};cmd: {open,close};
DEFINE faults:={StuckO,StuckC};TRANS (mode=Closed & cmd=open) -> (next(mode)=Open | next(mode) in faults)
ClosedClosed
ValveValveOpenOpen StuckStuck
openopen
StuckStuckclosedclosed
openopen closeclose
Livingstone Model SMV Model(defcomponent valve () (:inputs (cmd :type valve-cmd)) ... (Closed :type ok-mode :transitions ((do-open :when (open cmd) :next Open) ...)) (StuckC :type :fault-mode ...) ...)
LivingstoneAutonomousController
SMVSymbolic
Model Checker
![Page 25: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/25.jpg)
©Charles Pecheur, RIACS / NASA Ames 25KSC Nov 2000
From Livingstone Modelsto SMV Models
Translation program developed by CMU and Ames
• 4K lines of Lisp
• Similar nature => translation is easy
• Properties in temporal logic + pre-defined patterns
• In progress:– more property patterns
– translate results back to Livingstone
(Simmons, Pecheur)
![Page 26: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/26.jpg)
©Charles Pecheur, RIACS / NASA Ames 26KSC Nov 2000
• Use atmosphere from Mars tomake fuel for return flight.
• Livingstone controller developedat NASA KSC.
• Components are tanks, reactors,valves, sensors...
• Exposed improper flow modeling.
• Latest model is 1050 states.
Application In-Situ Propellant Production
CO2 + 2H2 —> CH4 + O2
Marsatmosphere
oxidizerfuel
on-board
![Page 27: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/27.jpg)
©Charles Pecheur, RIACS / NASA Ames 27KSC Nov 2000
Beyond Model-BasedVerification
• correct engine + correct model≠> correct control !– heuristic search strategies
– enough sensors/actuators?
– model approximations
• Model check everything?Very hard!
Need (abstract) V&V model ofengine + model + spacecraft + ...
=> complex, error-prone, huge state space
Controller
MIRExecVerified
Trusted=> Reliable???
![Page 28: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/28.jpg)
©Charles Pecheur, RIACS / NASA Ames 28KSC Nov 2000
Closed-Loop Verification
• Real system => accuracy.• More control => more coverage.• For any discrete-event controller (not only model-based).
• single step• backtrack• select choices• get/set state• ...
Engine Model
Autonomous Controller ModelCheckingEngine
SpacecraftSimulator
Driver ...
TESTBED Exec Control API
![Page 29: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/29.jpg)
©Charles Pecheur, RIACS / NASA Ames 29KSC Nov 2000
Model Checking JavaJava PathFinder
• Java PathFinder 1– Translates from Java to Promela (Spin)
• Java PathFinder 2– Explicit-state model checking.– Works with bytecodes => handle all of Java.– Based on custom Java Virtual Machine
• Written in Java (rapid prototyping).
• Emphasis on memory management not speed.
– Efficient encoding of states (heap, GC).
(Visser, Havelund)
![Page 30: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/30.jpg)
©Charles Pecheur, RIACS / NASA Ames 30KSC Nov 2000
Generic VerificationEnvironment
• Principle: uncouple V&V subject from V&V algo.
• Common denominator of several V&V projects.
• Current VMs: Java, Livingstone.
TestModel Check
![Page 31: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/31.jpg)
©Charles Pecheur, RIACS / NASA Ames 31KSC Nov 2000
Conclusions
Model checking:• Autonomy needs it – testing is not enough
• General pros&cons apply:– exhaustive... if model is small enough
– automatic verification... but tough modeling
• Works nicely on autonomy models
• Solutions inbetween testing and model checking
• Not short of tough problems:– Real-time, hybrid, AI
– Learning/adaptive systems: after training/including training
![Page 32: Verification of Intelligent Softwarepecheur/talks/VIS-KSC-Nov00.pdf · Intelligent software, how to verify it? • What? A bird's-eye view of model checking • How? Experiences in](https://reader030.vdocuments.us/reader030/viewer/2022040407/5eabf9c518986851ed0439e7/html5/thumbnails/32.jpg)
©Charles Pecheur, RIACS / NASA Ames 32KSC Nov 2000
Pointers
• My home pagehttp://ase.arc.nasa.gov/pecheurhttp://ase.arc.nasa.gov/pecheur/publi.htmlhttp://ase.arc.nasa.gov/pecheur/talks.html
• JavaPathFinderhttp://ase.arc.nasa.gov/jpf
• Model-Based Verification of IntelligenceAAAI Spring Symposium, Stanford, March 2001
http://ase.arc.nasa.gov/mvi