1 model testing © 2015 carnegie mellon university model testing: testing executable requirements,...
TRANSCRIPT
1Model Testing © 2015 Carnegie Mellon University
Model Testing:Testing Executable Requirements, Architecture, and Design Models
Software Engineering InstituteCarnegie Mellon UniversityPittsburgh, PA 15213
Donald G. FiresmithPrincipal Engineer Engineer
2Model Testing
Copyright 2015 Carnegie Mellon University
This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense.
NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.
This material has been approved for public release and unlimited distribution except as restricted below.
This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at [email protected].
DM-0002518
3Model Testing
Verification and Validation Methods
Traditionally, V&V ≈ Analysis + Demonstration + “Inspection” + Test.There are actually more methods for verification and validation.Testing is similar to other dynamic V&V methods.
Verification and Validation (V&V) MethodsQuality Control (QC)
Test Evaluation OtherIn Scope
T&E
Analysis
StaticAnalysis
DynamicAnalysis
Demonstration
Desk Checking
Inspection ReviewWalk-
Through
Peer Review
Formal Review
“Inspection” Certification Reuse Warantee
Legend
Static Dynamic
4Model Testing
Model Testing
Model testing tests executable requirements, architecture, and/or design models the same way traditional testing tests software (and systems).A requirements, architecture, or design model of a system is executable if:•It describes the system in sufficient detail (e.g., components,
functions, relations, processes, and status changes) that it can be executed as a simulation of that system.
•A means exists (e.g., modeling tool, simulation engine, execution engine, or human brain) to run the model.
Model testing is NOT the same thing as Model-Based Testing (MBT)!•Model testing uses testing to uncover defects in the models.•MBT uses models to automate testing to uncover defects in the software / systems.
5Model Testing
Model-Based Shift Left Testing
System RequirementsModel Tests
Architectural Model Tests
Component Design
Model Tests
Unit Design Model Tests
User Requirements Model Tests
Production System
Subsystem Integration
Tests
Component Integration
Test
Unit Tests
System Tests
System Requirements
Models
Architectural Models
Component DesignModels
Unit Design Models
User Requirements
Models
Executable Work Products
Tests
Legend
↑ Validation↓ Verification
xMBSLx
User Needs Model Tests
User Needs Models
Operational System
Operational Tests
Units
Components
Subsystems
Integrated System
xMBSLx
xMBSLx
xMBSLx
Model-Based Shift Left (MBSL)x
MBSL
Acceptance Tests
6Model Testing
Shift Left Testing
Major trend in software testing.
Move testing earlier in development cycle (left on timeline)
Four types of shift left testing:
•Traditional Shift Left Testing
Move emphasis from system-level GUI testing via record/playback to API, component integration, and unit testing
• Incremental Shift Left Testing
Few deliveries/releases with testing of multiple large increments
•Agile/DevOps Shift Left Testing
Many deliveries/releases with testing of multiple small increments
•Model-Based Shift Left Testing ←Test executable requirements, architecture, and/or design models(move testing left to models)
7Model Testing
Model Testing
Model Testing
Requirements ModelTesting
Design Model Testing
Architecture Model Testing
Model Testing via Execution
Engine
Testing via Symbolic Execution
Model Testing via Simulation
Engine
Model Testing with Executionby Computer
Model Testing with Execution
by Human
Test Model Testing
8Model Testing
Model TestingClient Actors
(roles + systems)real or simulated
Actor 1
Actor 2
Test Inputs
Server Actors(roles + systems)real or simulated
Software Under Test
(SUT)Test Outputs
Test InputsTest Outputs
Test Outputs
Test Input(Exception)
StubTest Outputs
Actor 3
Client Actors(roles + systems)
simulated
Actor 1
Actor 2
Test Inputs
Server Actors(roles + systems)
simulated
Model Under Test
(MUT)Test Outputs
Test InputsTest Outputs
Test Outputs
Test Input(Exception)
StubTest Outputs
Actor 3
Software Testing
Model Testing
9Model Testing
Model Testing
verify the creates
the
is input for the
1
Executable Requirements, Architecture, or Design Model
Under Test (MUT)
Testeror
Test Case Generator Tool
2
Test Cases
3
Simulation Engine, Execution Engine, or
Testerexecutes the
executes the
4
10Model Testing
Testable Models – Requirements Models
Example executable models include:•ConOps storyboards (simulations)
•Use case path sequence diagrams
•Decision trees, finites state machines, and Petri nets
•Executable requirements languages
•Requirements prototypes
11Model Testing
Testable Models – Architecture/Design Models
Examples executable models include:•Architecture Analysis and Design Language (AADL) models
•Unified Modeling Language (UML) models and System Modeling Language (SysML) models:
–Communication diagrams, interaction diagrams, sequence diagrams, state machine diagrams, activity diagrams, and timing diagrams,
–Object Constraint Language (OCL) or Action Semantic Language (ASL)
•Unified Profile for DODAF and MODAF (UPDM 2.0)
•Hardware Description Languages
•Architecture prototypes
•Design models for automatic code generation