advanced technology center slide 1 requirements-based testing dr. mats p. e. heimdahl university of...
TRANSCRIPT
Advanced Technology Center Slide 1
Requirements-Based TestingRequirements-Based Testing
Dr. Mats P. E. HeimdahlUniversity of Minnesota Software Engineering Center
Dr. Steven P. Miller
Dr. Michael W. WhalenAdvanced Computing Systems
Rockwell Collins
400 Collins Road NE, MS 108-206
Cedar Rapids, Iowa 52498
Advanced Technology Center Slide 2
Outline of PresentationOutline of Presentation
Motivation
Validation Testing
Conformance Testing
What’s Next
Advanced Technology Center Slide 3
How We Develop SoftwareHow We Develop Software
SW High-Level Reqs. Development
SW Design Description Dev. (SW Low-Level
Reqs. & SW Arch.
SW Source Code Dev.
SW Integration (Executable Code Production)
SW Low-Level Testing
SW Integration Testing
HW/SW Integration Testing
Advanced Technology Center Slide 4
How we Will Develop SoftwareHow we Will Develop Software(From V to a Y)(From V to a Y)
SW High-Level Reqs. Development
Software Model
SW Integration (Executable Code Production)
SW Integration Testing
HW/SW Integration Testing
Can we trust the code
generator?
How do we know our model is
correct?
Validation TestingFormal
Verification
Conformance Testing
Advanced Technology Center Slide 5
Outline of PresentationOutline of Presentation
Motivation
Validation Testing
Conformance Testing
What’s Next
Advanced Technology Center Slide 6
How we Will Develop SoftwareHow we Will Develop Software(From V to a Y)(From V to a Y)
SW High-Level Reqs. Development
Software Model
SW Integration (Executable Code Production)
SW Integration Testing
HW/SW Integration Testing
How do we know our model is
correct?
Advanced Technology Center Slide 7
Modeling ProcessModeling Process
SW High-Level Reqs. Development
Software Model
SW Integration (Executable Code Production)
Desired Model Properties
High-Level Requirements
Low-Level Requirements
Advanced Technology Center Slide 8
Problem—Modeling FrenzyProblem—Modeling Frenzy
SW High-Level Reqs. Development
Software Model
SW Integration (Executable Code Production)
Desired Model Properties
Headfirst into m
odeling
How do we know the model is
“right”?How do we test the
model?
Advanced Technology Center Slide 9
One Solution: Redefine RequirementsOne Solution: Redefine Requirements
System Reqs. Development
Software Model
SW Integration (Executable Code Production)
SW Integration Testing
HW/SW Integration Testing
Software Development
Processes(DO-178B)
Software Development
Processes(DO-178B)
System Development
Processes(ARP4754)
System Development
Processes(ARP 4754)
The model is the requirements
Use Engineering Judgment when
Testing
Advanced Technology Center Slide 10
One Solution: Redefine RequirementsOne Solution: Redefine Requirements
System Reqs. Development
Software Model
SW Integration (Executable Code Production)
SW Integration Testing
HW/SW Integration Testing
Software Development
Processes(DO-178B)
Software Development
Processes(DO-178B)
System Development
Processes(ARP4754)
System Development
Processes(ARP 4754)
The model is the requirements
Use Engineering Judgment when
Testing
My Com
ment
Advanced Technology Center Slide 11
Testing Does not go AwayTesting Does not go Away
System Reqs. Development
Software Model
SW Integration (Executable Code Production)
SW Integration Testing
HW/SW Integration Testing
Extensive Testing (MC/DC)
Advanced Technology Center Slide 12
It Simply MovesIt Simply Moves
System Reqs. Development
Software Model
SW Integration (Executable Code Production)
SW Integration Testing
HW/SW Integration Testing
Extensive Testing (MC/DC)
Advanced Technology Center Slide 13
Do it Right!Do it Right!
SW High-Level Reqs. Development
Software Model
SW Integration (Executable Code Production)
Desired Model Properties
Analysis (Model Checking,Theorem Proving)Specification Test –
Is the Model Right?
Advanced Technology Center Slide 14
How Much to Test?How Much to Test?
State Coverage
Masking MC/DC?
Transition
Coverage?
Decision Coverage
?
Def-Use Coverage
?
Something New??
MC/DC
Where Do the TestsCome From?
Advanced Technology Center Slide 15
Properties are Requirements…
Requirements Based TestingRequirements Based Testing
SW High-Level Reqs. Development
Software Model
SW Integration (Executable Code Production)
Desired Model Properties
Cover the Properties!
Advanced Technology Center Slide 16
Properties are RequirementsProperties are Requirements
Advanced Technology Center Slide 17
Requirements Based Testing Requirements Based Testing AdvantagesAdvantages
Objective Measurement of Model Validation Efforts– Requirements Coverage in Model-based Development
– Help Identify Missing Requirements• Measure converge of model
Basis for Automated Generation of Requirements-based Tests– Even If Properties Are Not Used for Verification, They Can Be
Used for Test Automation
How Are Properties “Covered” with Requirements-based Tests?
Advanced Technology Center Slide 18
Property CoverageProperty Coverage
“If the onside FD cues are off, the onside FD cues shall be displayed when the AP is engaged”– G(((!Onside_FD_On & !Is_AP_Engaged) -> X(Is_AP_Engaged -> Onside_FD_On))
Property Automata Coverage– Cover a Synchronous Observer Representing the
Requirement (Property)
Structural Property Coverage– Demonstrate Structurally “Interesting” Ways in Which
the Requirement (Property) Is Met
Advanced Technology Center Slide 19
Property Automata CoverageProperty Automata Coverage
Cover Accepting State Machine As Opposed to Structure of Property
Büchi Coverage– State Coverage, Transition Coverage, Lasso
Coverage…
S1S0
Is_AP_Engaged vOnside_FD_On
not Is_AP_Engaged ^not Onside_FD_On
not Is_AP_Engaged ^not Onside_FD_On
Onside_FD_On
1
2
3
4
Advanced Technology Center Slide 20
Alternative MachineAlternative Machine
Different synthesis algorithms give different automata– Will affect the test cases
required for coverageS0 S1
S4 S2 S5
S3
S6 S7
Init
a
a
b
a
bb
b
a, b
a, b
a, b
abb
a
!a, b
!a, b
!a, b
b
b
b
!a
!a
!a
b
b
b
a
!a, !b
!a, !b
!a, !b !a, !b
!a, !b
!a, !b
Advanced Technology Center Slide 21
Structural Property CoverageStructural Property Coverage
Define Structural Coverage Criteria for the Property Specification– Traditional Condition-based Criteria such as MC/DC
Prime Candidates
Property Coverage Different than Code Coverage– Coverage of Code and Models
• Evaluate a decision with a specific combination of truth values in the decision
– Coverage of Properties• Run an execution scenario that illustrates a specific way a
requirement (temporal property) is satisfied
Advanced Technology Center Slide 22
ExampleExample
– G(((!Onside_FD_On & !Is_AP_Engaged) -> X(Is_AP_Engaged -> Onside_FD_On))
Demonstrate That Somewhere Along Some Execution Trace Each MC/DC Case Is Met– Only the “positive” MC/DC cases
• The negative cases should have no traces
In the Case of G(p)—Globally p Holds—we Need to Find a Test Where– in the prefix the requirement p is met
– we reach a state of the trace where the requirement p holds because of the specific MC/DC case of interest – let us call this case a
– then the requirement p keeps on holding through the remainder of the trace
p U ( a U X(G p))
p p a p p p
Advanced Technology Center Slide 23
SummarySummary
Objective Measurement of Model Validation Efforts– Requirements Coverage in Model-based Development– Help Identify Missing Requirements
Basis for Automated Generation of Requirements-based Tests– Even If Properties Are Not Used for Verification, They Can Be Used for
Test Automation and Test Measurement
Challenges – How Are Properties Specified?
• Combination of Observers and Temporal Properties
– What Coverage Criteria Are Suitable?– How Is Automation Achieved?– How Do We Eliminate “Obviously” Bad Tests? Should We?– How Do We Generate “Realistic” Test-cases?– Rigorous Empirical Studies Badly Needed
Advanced Technology Center Slide 24
Outline of PresentationOutline of Presentation
Motivation
Validation Testing
Conformance Testing
What’s Next
Advanced Technology Center Slide 25
How we Will Develop SoftwareHow we Will Develop Software(From V to a Y)(From V to a Y)
SW High-Level Reqs. Development
Software Model
SW Integration (Executable Code Production)
SW Integration Testing
HW/SW Integration Testing
Can we trust the code
generator?
Advanced Technology Center Slide 26
““Correct” Code Generation—How?Correct” Code Generation—How?
Provably Correct Compilers– Very Hard (and Often Not
Convincing)
Proof Carrying Code
Generate Test Suites From Model– Compare Model Behavior
With Generated Code
– Unit Testing Is Now Not Eliminated, but Largely Automated
Specification/Model
Implementation
Output
Output
Specification Based Tests
Generate
Advanced Technology Center Slide 27
Existing CapabilitiesExisting Capabilities
Several Commercial and Research Tools for Test-Case Generation– TVEC
• Theorem Proving and Constraint Solving techniques
– Reactis from Reactive Systems Inc. • Random, Heuristic, and Guided Search
– University of Minnesota• Bounded Model Checking
– NASA Langley• Bounded Model Checking/Decision Procedures/Constraint Solving
Tools Applicable to Relevant Notations– In Our Case Simulink
Advanced Technology Center Slide 28
An Initial ExperimentAn Initial Experiment
Used a Model of the Mode Logic of a Flight Guidance System As a Case Example
Fault Seeding– Representative Faults– Generated 100 Faulty Specifications
Generate Test Suites– Selection of Common (and Not So Common) Criteria
Fault Detection– Ran the Test Suites Against the Faulty Specifications– Recorded the Total Number of Faults Detected
Advanced Technology Center Slide 29
Fault Finding ResultsFault Finding Results
0
10
20
30
40
50
60
70
80
90
100
Variable Domain
Transition
Decision
Decision (use)
Masking MC/DC
Masking MC/DC (use)
MC/DC
MC/DC (use)
Random
Same Effort
Advanced Technology Center Slide 30
Model “Cheats” Test Generator Model “Cheats” Test Generator
FGSR
ModeLogic
ControlLaws
FGSL
ModeLogic
ControlLaws
AutopilotPFDRPFDL
Air DataL
FMSL
Air DataR
FMSR
FCP
ControlSurfacesFCS
Architecture
Advanced Technology Center Slide 31
Effect of Test Set SizeEffect of Test Set Size
0102030405060708090
100
Variable Domain
Transition
Decision
Decision (use)
MC/DC
MC/DC (use)
Full Reduced
Advanced Technology Center Slide 32
SummarySummary
Automated Generation of Conformance Tests– Current Technology Largely Allows This Automation
Challenges – Development of Suitable Coverage Criteria
– Effect of Test Set Size on Test Set Effectiveness
– Effect of Model Structure on Coverage Criteria Effectiveness
– Traceability of Tests to Constructs Tested
– Empirical Studies of Great Importance
Advanced Technology Center Slide 33
Outline of PresentationOutline of Presentation
Motivation
Conformance Testing
Validation Testing
What’s Next
Advanced Technology Center Slide 34
New Challenges for TestingNew Challenges for Testing
Model Validation – Requirements-based Testing– How Do We Best Formalize the Requirements?– What Coverage Criteria Are Feasible?– Which Coverage Criteria Are Effective (If Any)?– How Do We Generate “Realistic” Tests?– Will This Be a Practical (Tractable) Solution?
Conformance Testing– What Coverage Criteria Are Effective?
• Detecting Faults From Manual Coding
• Detecting Faults From Code Generation
– Relationship Between Model Structure and Criteria Effectiveness
– Traceability From Tests to Model– Relationship Between Model Coverage and Code Coverage
• Optimizations in Code Generator Will Compromise Coverage
Advanced Technology Center Slide 35
DiscussionDiscussion
Advanced Technology Center Slide 36
Perfection is Perfection is NotNot Necessary Necessary
Tools and Models Only Need To Be Better Than Manual Processes…– How Do We Demonstrate This?
• Empirical Studies Are of Great Importance
≥Missed Faults
I Think Many Already Are
Advanced Technology Center Slide 37
DO-178B Test ObjectivesDO-178B Test Objectives
1. The executable code complies with the high-level requirements.
2. The executable code complies with the specification (low-level requirements).
3. Test coverage of high-level requirements is achieved
4. Test coverage of specification (low-level requirements) is achieved
5. Test coverage of the executable code is achieved
Requirements-Based Testing
Conformance Testing