se300 swe practices - embry–riddle aeronautical...
TRANSCRIPT
March 31, 2015 Sam Siewert
SE300
SWE Practices
Lecture 12 – Design and Prototype
Testing Strategies
Testing Strategies
Validation – Did we
Design/Code the Right
Thing?
Verification – Did we
Design/Code the Thing
Right?
Testing Should be
Concurrent with
Analysis, Design and
Development
Sam Siewert 2
Challenge of Waterfall and V
Requirements Generation and Acceptance Tests Require Experience
Customer May Not Be Forthcoming
Candidate or Reference Design and Code Re-Use May Be Best Starting Point to Drive Process
Sam Siewert 3
Regre
ssio
n T
esting
Can Drive Top-Down (from concept to code)
or Bottom-Up (from candidate code, design, system)
Requirements Verification and
Validation UC Traceability
UC and Requirements Walk-throughs and Formal
Reviews
Acceptance Test Walk-throughs – All R’s Testable?
Refinement Based on Next Steps
1. System Design – UML Domain Model and OIM
2. Architecture – Transformation, Event-Driven, Transaction,
Interactive Behavioral and Structural Models
3. Module Design – Decomposition into Classes or Modules
Based on Cohesion and Loose Coupling
Sam Siewert 4
System Design Validation & Verification
UML Domain Model – Generate C++ Code
OM Instantiation with C++ Driver Stub
OIM Protocol Tests – Main Driver
Validate and Verify Expected Protocols Between Objects
Validate and Verify Unexpected, Error Protocols
Between Objects (Negative Testing)
Design Simulation, Stubbed Code Compilation and
Execution
Sam Siewert 5
Architecture Design Verification and
Validation Focus on Unique Architecture Aspects
Behavior Models – OIM (Interactive, Transaction), State-
Machine (Event-Driven), Activity
(Event+Transformation), CFD/DFD
(Event+Transformation)
Prototype or PoC Code for Key Architectural Verification
1. Performance Assumptions
2. Feasibility
3. Mock-Ups or Early GUI/HCI Feedback
4. Candidate Platforms (System and Derived Requirements)
5. Code Re-Use (e.g. specialized libraries)
6. DBMS Selection and Prototype
Sam Siewert 6
Module Design Validation & Verification
Detailed Design Walk-throughs
Detailed Design Reviews
Module Interface Specification Review
– Header Files (Public)
– Class (Public, Private, Protected)
– Method Prototypes
Function or Method Bodies Stubbed Out or Prototyped
Module Unit Test Development
Sam Siewert 7
Fagan Inspections Michael Fagan, IBM – 1976, Design and code
inspections to reduce errors in program development
Process for Design and Code Inspections – Formal
Reviewers Have Specific Roles
– Adherence to Coding Standards (Style, Comments, Common
Coding Pitfall Avoidance)
– Readability and Maintainability
– Correctness (Validation and Verification)
Defects Noted Formally with Write-up
Defects Must be Formally Addressed and Resolved
Sam Siewert 8
http://en.wikipedia.org/wiki/Fagan_inspection
Code Testing Strategy Code Walkthroughs and Inspections – At Bottom of V
May Iterate, but We Must have Some Code!
Walk-through is Informal (Progress, Improvement, Peers)
Inspection is Formal (Review Panel, Defect Write-Ups, Resolutions)
Sam Siewert 9
Regre
ssio
n T
esting
walkthroughs inspections
Code Walkthroughs
Informal
With a Peer Engineer
– Explain your Code Out Loud
– Comment and Improve on the Spot (Extreme Programming)
Explain Each Function and Each Basic Block
– Function by Function
– Function, Loop, Switch/If Preconditions and Post-conditions
Run in Debugger as Needed
Prepare Simple Unit Test Driver to Assist
Sam Siewert 10
Class Exercise – Example Walk-through(s)
RAID 5 Code – Walk-through improvements here
– Separation of 64-bit XOR from simple Byte-by-Byte XOR
– Static Initializers replaced with Set all test buffers for loop
– Test Driver main() has conditional compile for 64-bit or 8-bit
– Assert used properly on memcmp
– TEST CASE #0 simple single LBA encode and rebuild
– printBuffer added (dumpBuffer for Hex) for buffer inpection
– PRECONDITIONS, POST-CONDITIONS for xorLBA (add for
rebuildLBA as well)
– Makefile64 for 64-bit version
– Makefile for 8-bit version
64-bit runs faster without SSE and O3 Optimization
With –O3 and –msse3 –malign-double – both are same
Sam Siewert 11
RAID test 8-bit
Sam Siewert 12
8-bit has 120K IOPs for 1000 iteration test, un-optimized
Passes Data Compare Tests after Rebuild
RAID test 64-bit
64-bit has 665K IOPs for 1000 iterations, un-optimized
Passes Data Compare Tests after Rebuild
Sam Siewert 13
Discussion – is 64-bit Better?
64-bit Has Confusing Pointer Aliases and Arithmetic
64-bit Runs Faster with No Compiler Optimization
Intel Architecture Provides SSE (SIMD) Code
Generation
What if We Try that?
CFLAGS= -O3 -msse3 -malign-double -g $(INCLUDE_DIRS) $(CDEFS)
Sam Siewert 14
RAID test 8-bit - Optimized
Sam Siewert 15
8-bit has 1,366K IOPs for 1000 iteration test, optimized
10x Speed-Up
SSE and O3 Speed-Up Simpler Code, Still Passes Tests
RAID test 64-bit - Optimized
64-bit has 974K IOPs for 1000 iterations, optimized
50% Speed-Up, But Slower than 8-Bit Optimized!
So, Testing Shows 8-Bit is Easier to Read and Faster
Sam Siewert 16
Longer Test? If we Try 100,000,000 Iterations of XOR Encode and Rebuild, Best is about 5.65 Million IOPs for 8-bit – 17.7 Seconds
Covering more Unique Data Patterns and Passing Assert Too!
Sam Siewert 17
Code Inspections Formal – A Gate to Entering System Integration and Test
With a Review Panel (Engineers & Management)
– Submit Code for Review
– Reviewers Read and Execute Code in Advance
– Explain your Code Out Loud and Answer Review Questions
– Defects are Noted and Written Up
Can Be Reviewer Question Driven or Walk-through
– Ideally Reviewers Come with Questions
– Concerns are Focus and Defects are Identified
Defects Must Be Formally Fixed (Addressed)
Follow-Up Inspections to Cover Defect Corrective Actions
Sam Siewert 18
Formal Review Process
Requirements Review
Preliminary Design Review (Architecture)
Critical Design Review (Detailed Design, Proofs-of-
Concept)
– Defect Tracking and Resolution
– Must Pass CDR Prior to Implemenation(s) – Funding Gate
Code Walk-throughs – Informal
Code Inspections
– Defect Tracking
– Defect Resolution, Follow Up Inspection to Verify
Delivery of Unit Tests and Results
System Integration and Test Results
Acceptance Test, Delivery, Release Notes
Sam Siewert 19
When are We Done Testing?
Based on a Metric
– Code Coverage – Path, Statement
– Hours of Continuous Testing – Soak Time
– Number of Test Cases
– Test Vectors (Variation of Inputs to Functions and Units)
– Other?
Path Coverage Tools and Metrics
– E.g. Enigma code Function and Path Coverage using Gcov
SE420, SQA (Software Quality Assurance)
Sam Siewert 20
Take-Away
Transition from Design/Code Phase to Integration and Test
Could Be Iterative Spiral or Extreme Programming Process, but Could be Waterfall Too
Walk-throughs are Informal Quality Improvement
Inspections are Formal, Defect Prevention, Design V&V Focus
Both are Critical – Walk-throughs in Advance Improve Inspections
Sam Siewert 21
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 22
Chapter 22
Software Testing Strategies
Slide Set to accompany
Software Engineering: A Practitioner’s Approach, 8/e by Roger S. Pressman and Bruce R. Maxim
Slides copyright © 1996, 2001, 2005, 2009, 2014 by Roger S. Pressman
For non-profit educational use only May be reproduced ONLY for student use at the university level when used in conjunction
with Software Engineering: A Practitioner's Approach, 8/e. Any other reproduction or use is
prohibited without the express written permission of the author.
All copyright information MUST appear if these slides are posted on a website for student
use.
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 23
Software Testing
Testing is the process of exercising
a program with the specific intent of
finding errors prior to delivery to the
end user.
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 24
What Testing Shows
errors
requirements conformance
performance
an indication of quality
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 25
Strategic Approach
To perform effective testing, you should conduct effective technical reviews. By doing this, many errors will be eliminated before testing commences.
Testing begins at the component level and works "outward" toward the integration of the entire computer-based system.
Different testing techniques are appropriate for different software engineering approaches and at different points in time.
Testing is conducted by the developer of the software and (for large projects) an independent test group.
Testing and debugging are different activities, but debugging must be accommodated in any testing strategy.
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 26
V & V
Verification refers to the set of tasks that ensure
that software correctly implements a specific
function.
Validation refers to a different set of tasks that
ensure that the software that has been built is
traceable to customer requirements. Boehm
[Boe81] states this another way:
Verification: "Are we building the product right?"
Validation: "Are we building the right product?"
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 27
Who Tests the Software?
developer independent tester
Understands the system
but, will test "gently"
and, is driven by "delivery"
Must learn about the system,
but, will attempt to break it
and, is driven by quality
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 28
Testing Strategy
System engineering
Analysis modeling
Design modeling
Code generation Unit test
Integration test
Validation test
System test
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 29
Testing Strategy
We begin by ‘testing-in-the-small’ and move
toward ‘testing-in-the-large’
For conventional software
The module (component) is our initial focus
Integration of modules follows
For OO software
our focus when “testing in the small” changes from
an individual module (the conventional view) to an
OO class that encompasses attributes and
operations and implies communication and
collaboration
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 30
Strategic Issues
Specify product requirements in a quantifiable manner long before testing commences.
State testing objectives explicitly.
Understand the users of the software and develop a profile for each user category.
Develop a testing plan that emphasizes “rapid cycle testing.”
Build “robust” software that is designed to test itself
Use effective technical reviews as a filter prior to testing
Conduct technical reviews to assess the test strategy and test cases themselves.
Develop a continuous improvement approach for the testing process.
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 31
Unit Testing
module to be
tested
test cases
results
software engineer
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 32
Unit Testing
interface
local data structures
boundary conditions
independent paths
error handling paths
module to be
tested
test cases
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 33
Unit Test Environment
Module
stub stub
driver
RESULTS
interface
local data structures
boundary conditions
independent paths
error handling paths
test cases
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 34
Integration Testing Strategies
Options:
• the “big bang” approach
• an incremental construction strategy
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 35
Top Down Integration
top module is tested with stubs
stubs are replaced one at a time, "depth first"
as new modules are integrated, some subset of tests is re-run
A
B
C
D E
F G
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 36
Bottom-Up Integration
drivers are replaced one at a time, "depth first"
worker modules are grouped into builds and integrated
A
B
C
D E
F G
cluster
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 37
Sandwich Testing
Top modules are
tested with stubs
Worker modules are grouped into builds and integrated
A
B
C
D E
F G
cluster
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 38
Regression Testing
Regression testing is the re-execution of some subset of tests that have already been conducted to ensure that changes have not propagated unintended side effects
Whenever software is corrected, some aspect of the software configuration (the program, its documentation, or the data that support it) is changed.
Regression testing helps to ensure that changes (due to testing or for other reasons) do not introduce unintended behavior or additional errors.
Regression testing may be conducted manually, by re-executing a subset of all test cases or using automated capture/playback tools.
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 39
Smoke Testing
A common approach for creating “daily builds” for product software
Smoke testing steps: Software components that have been translated into code are
integrated into a “build.” • A build includes all data files, libraries, reusable modules, and engineered
components that are required to implement one or more product functions.
A series of tests is designed to expose errors that will keep the build from properly performing its function.
• The intent should be to uncover “show stopper” errors that have the highest likelihood of throwing the software project behind schedule.
The build is integrated with other builds and the entire product (in its current form) is smoke tested daily.
• The integration approach may be top down or bottom up.
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 40
General Testing Criteria
Interface integrity – internal and external module interfaces are
tested as each module or cluster is added to the software
Functional validity – test to uncover functional defects in the
software
Information content – test for errors in local or global data
structures
Performance – verify specified performance bounds are tested
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 41
Object-Oriented Testing
begins by evaluating the correctness and consistency of the analysis and design models
testing strategy changes the concept of the ‘unit’ broadens due to
encapsulation
integration focuses on classes and their execution across a ‘thread’ or in the context of a usage scenario
validation uses conventional black box methods
test case design draws on conventional methods, but also encompasses special features
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 42
MobileApp Testing
User experience testing – ensuring app meets
stakeholder usability and accessibility expectations
Device compatibility testing – testing on multiple devices
Performance testing – testing non-functional
requirements
Connectivity testing – testing ability of app to connect
reliably
Security testing – ensuring app meets stakeholder
security expectations
Testing-in-the-wild – testing app on user devices in
actual user environments
Certification testing – app meets the distribution
standards
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 43
High Order Testing Validation testing
Focus is on software requirements
System testing Focus is on system integration
Alpha/Beta testing Focus is on customer usage
Recovery testing forces the software to fail in a variety of ways and verifies that recovery is
properly performed
Security testing verifies that protection mechanisms built into a system will, in fact, protect it
from improper penetration
Stress testing executes a system in a manner that demands resources in abnormal quantity,
frequency, or volume
Performance Testing test the run-time performance of software within the context of an integrated
system
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 44
Debugging: A Diagnostic Process
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 45
The Debugging Process
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 46
Debugging Effort
time required to diagnose the symptom and determine the cause
time required to correct the error and conduct regression tests
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 47
Symptoms & Causes
symptom
cause
symptom and cause may be geographically separated
symptom may disappear when another problem is fixed
cause may be due to a combination of non-errors
cause may be due to a system or compiler error
cause may be due to assumptions that everyone believes
symptom may be intermittent
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 48
Consequences of Bugs
damage
mild annoying
disturbing
serious
extreme
catastrophic
infectious
Bug Type
Bug Categories: function-related bugs,
system-related bugs, data bugs, coding bugs, design bugs, documentation bugs, standards violations, etc.
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 49
Debugging Techniques
brute force / testing
backtracking
induction
deduction
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 50
Correcting the Error
Is the cause of the bug reproduced in another part of the
program? In many situations, a program defect is caused by an
erroneous pattern of logic that may be reproduced elsewhere.
What "next bug" might be introduced by the fix I'm about to
make? Before the correction is made, the source code (or,
better, the design) should be evaluated to assess coupling of
logic and data structures.
What could we have done to prevent this bug in the first place?
This question is the first step toward establishing a statistical
software quality assurance approach. If you correct the
process as well as the product, the bug will be removed from
the current program and may be eliminated from all future
programs.
These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 8/e
(McGraw-Hill 2014). Slides copyright 2014 by Roger Pressman. 51
Final Thoughts
Think -- before you act to correct
Use tools to gain additional insight
If you’re at an impasse, get help from someone
else
Once you correct the bug, use regression
testing to uncover any side effects