open-do update
DESCRIPTION
A review of the latest news and developments with the Open-DO effort.TRANSCRIPT
Summary
• Reminder on Open-DO Concepts
• What has been Happening within Open-Do
• Couverture & Formal Methods
• Hi-Lite
Reminder on Open-DO Concepts
• How to solve the “big Freeze” problem ?
• How to manage exposed life-cycle Data ?
• How to reduce cost & time-to-market ?
• How to augment quality & reduce residual problems ?
Open–DO
FLOSSFreely Licensed
Open SourceSoftware
High-IntegrityCertification
AgileLean
The meeting of 3 worlds
FLOSSFreely Licensed
Open SourceSoftware
High-IntegrityCertification
AgileLean
Longevity
Visibility
Resilience
Cost-sharing
Reuse
Iterative requirements
Continuous integration
Test driven development
Executable specifications
Reducing waste
Qualified tools
Life cycle traceability
Req based testing
The meeting of 3 worlds
FLOSSFreely Licensed
Open SourceSoftware
High-IntegrityCertification
Security
AgileLean
Longevity
Visibility
Resilience
Cost-sharing
Reuse
Iterative requirements
Continuous integration
Test driven development
Executable specifications
Reducing waste
Qualified tools
Life cycle traceability
Requirement based
testing
The meeting of 3 worlds
Formal methods
verification
verification
4
?
Open-DO
DocumentTemplates
QualifiableTools
EducationMaterials
CertifiableComponents
Open DO Components
Life Cycle Management
What has been Happening within Open-Do ?
Awareness
Stages
Infrastructure
Initial projects
Community management
Kick-off projects
Awareness
Stages
Infrastructure
Initial projects
Community management
• Conferences & Papers
- OpenCert conference- Avionics 2009 - DASIA 2009 - Safety-Critical Systems Club- Ada Europe 2009- International SPICE days- Eclipse Embedded Day- 2009 IET System Safety Conf.- Ada UK Conference 2009- Agile Tour 2009
• Presentations- SC-205 (DO-178c)- Boeing/Airbus/Embraer
• Social networks- Linked-In (80 members)
Kick-off projects
Awareness
Stages
Infrastructure
Initial projects
Community management
Kick-off projects
• Website- increasing #s of visits
• Forge- ½ dozen hosted projects
• Mailinglists/forums- more than a 100 registrations- mostly from mil-aero
Awareness
Stages
Infrastructure
Kick-off projects
Community management
• Couverture• Qualification Machine• Hi-Lite• … and a few others
Awareness
Stages
Infrastructure
Kick-off projects
Community management
• Still a bit early…
• Concentrate on relationship with related initiatives
Couverture …
Hi-Lite …
The Qualifying Machine
An agile infrastructure to support:- Delta qualification
- Continuous qualification
- Internally used at AdaCore
Availability of partial qualification material for GNATcheck- A coding standard tools
-Tool Qualification Plan
- Quality Assurance Plan
- Configuration Management Plan
- Example of Tool Operational Requirements
Some Open-DO Projects (1)
Other projects
HiberSource- Configuration Management System
- Support for full life cycle
- DO-178 compliant
Gene-Auto/Ada- A model compiler for data-flow and state machine languages
- Supports Simulink and Stateflow
- Generates Ada 2005
- Final goal: qualification as DO-178C development tool
Some Open-DO Projects (2)
• Embarquez Agile (Embed Agility)
Bordeaux- March 18th, 2010
Cyrille Comar, AdaCore: “Open-DO: open source and agility for critical software”Matteo Bordin, AdaCore: “The Qualifying Machine: agile DO-178 qualification”
• ERTS2 2010: Embedded and Real-Time Systems 2010
Toulouse, May 19th-21st
• FM+AM 2010
Pisa – September 17th, 2010
2nd Internation Workshop on Formal Methods and Agile MethodsCo-located with the 8th IEEE Conference on Software Engineering and Formal Methods
Upcoming Events
• Good visibility in the avionics industry
• Open Development in a certification context is a challenge
• Importance of the quick-off projects
Remarks
Couverture & Formal Methods
Couverture provides either Object or Source coverage
• Source coverage:• Statement• Decision• MC/DC
pros:Simple for user DO-178
• Object coverage:• Instruction• Branch
pros:on the final codebounded traceslang independent
Object Branch coverage output example
function P (A, B, C : Boolean) return Boolean isbegin
if ( A and then B ) or else C then
return True;
end if;end P;
A B C if statement
T T ? T
F ? F F
A B C if statement
T T ? T
F ? F F
F ? T T
T F F F
Decision
Conditions
Decision Coverage
At least n+1 testsn = number of conditions
A
C
Decision and MC/DC coverage
B
MC/DC Coverage
statements
A B C if statement
T T ? T
Statement Coverage
• Seems a reasonable assumption when boolean operator branch in the object
• Has been assumed true for years
• Recent FAA study (J. Chelinsky from Boeing) shows experimentally that it is not always the case
So what is the story?
is MCDC implied by object branch coverage?
Counter-example
if ( A and then B ) or else C then
…
end if;
A B C if statement
T T ? T
F ? F F
T F T T
A B C if statement
T T ? T
F ? F F
F ? T T
T F F F
Object Branch Coverage
4 tests
A
C B
MC/DC Coverage
A
B
C
TrueFalse
T
F
F
FT
T
3 tests
Binary Decision Diagram (BDD)
Verify new conjectures
• Only when a single kind of operator?
• No diamond in the BDD ?
• Are the 2 above equivalent?
if A and then B and then C and then D … then
…
end if;
A
B
C
TrueFalse
T
F
F
FT
T
DF
T
• What is Alloy?– a specification language for relational (first order) logic
– specifications are executable
• What does it bring?– exhaustive exploration in (small) user-defined scope
– produces counter-examples
Alloy in the loop
model checking
• Modelisation of BDDs & MC/DC requirements
• Verification of conjectures in a limited scope:– Decisions with less than N conditions – with N = 5, 6, …
• Alloy generated counter-examples were key to find the proper equivalence
Alloy in the loop (2)
• Even in the proper context– Boolean ops limited to: not, and then, or else– one branch in the object per condition– Normalized Decisions (NNF)– …
• Object Branch Coverage MC/DC
• For decisions limited to “and then” (or “or else”) OBC is sufficient for MC/DC … but not necessary…
The results
• There are forms of NNF decisions whereOBC MC/DC
(sub-decision1) and then (sub-decision2)with no “or else” in sub-decision1
(sub-decision1) or else (sub-decision2)with no “and then” in sub-decision1
• Alloy shows this is true for Nb_Conditions <= 7
• Manual Proof was built to show it for any conditions
The results (2)
Hi-Lite
Overlap of existing techniques
Testing
Hi-Lite: Verifying Program Properties (1)
Static Analysis
Formal methodsSPARK: decades of experience in proof of: - absence of RT errors - functional properties
Programming by contract in Ada
CodePeer: - detection of RT errors - implicit contracts
• Properties:
• Absence of classes of errors
• Invariants maintained
• Function contracts
• Verification:
• Testsuite passes ok
• No critical warnings (compiler, static analyzer)
• 100% VC proved (VC = Verification Condition)
Hi-Lite: Verifying Program Properties (2)
NO method-specific expression of properties ex:
• Oracles for tests
• Annotations for static analysis
• Logical formulas for proof
Instead, ONE executable annotation languageex : assertions
TOOLS do the translation
Hi-Lite: Verifying Program Properties (3)
ONE artifact for program and properties
ONE language for program and properties
MANY eyes for reviewing both
MANY ways to contribute properties
• Manually added
• Inferred by static analyzer
• Generated from higher-level description (model)
MANY different workflows
• Dynamic vs. static verification
• Various techniques to generate and prove formulas
Hi-Lite: Verifying Program Properties (4)
Conclusion
• Formal methods are useful in various ways
• They need to be democratized
• They need to be integrated in the Dev Cycle … in an agile way