vv07
DESCRIPTION
TRANSCRIPT
Testing AutomationCFICSE VV07 October 1999
Dr. Terry Shepard
Royal Military College of Canada
Electrical and Computer Engineering
613-541-6000 ext. 6031
Testing Automation Topics
• Motivation for Automation
• Design for Testability
• What can be automated?
• Context and Needs
• Intrusive vs. non-intrusive
• Software Test Technologies Report– Classification of testing tools
Motivation for Testing Automation
• reduce costs
• ensure repeatability of tests
• ensure that test scenarios are followed exactly
• increase number of tests that can be run by a deadline
• improve reliability estimation
Design for Testability
• Common for computer hardware
• Primarily means– Controllability– Observability– Well behaved interfaces between modules
• Determines what can easily be automated– e.g. instrumentation code built into source code
What can be automated?
• Test execution
• Comparison of actual and expected results
• Logging of state at the time of a failed test– recovery for debugging purposes
• Management of test logs
• Collection of metrics
• Version control (in part)
• Test case selection: hard!
Automated support can be provided for other tasks
• Test planning and budgeting
• Test setup procedures
• Test adequacy decisions
• Test scenarios with external dependencies
Context and Need
• In a given project, the context will determine candidate testing tools– constraints include language, platform, OS, …
• Given a set of constraints, a market search and evaluation can be undertaken
• Broad range of applicability and variety of tools makes needs assessment hard
• Cost justification should be applied
Intrusive vs. Non-Intrusive
• Intrusion is generally not possible in an operational setting
• Build black box recorder capabilities into the software?
• Continuing hardware performance improvements will make this increasingly feasible
Software Test Technologies Report
• August 1994
• Software Technology Support Center, Hill AFB, Air Logistics Command, Ogden, Utah– Online at www.stsc.hill.af.mil– STSC has an Annual Conference, publishes
Crosstalk, …
• No update in classification since 1994
1994 STSC Test Tools Classification
• Test Resource Management
• Requirements and Design Test Support
• Implementation and Maintenance Test Support– Compilers– Source Code Static Analyzers– Test Preparation– Test Execution (Dynamic Analyzers)– Test Evaluators
Test Resource Management Tools
• Configuration Managers– test version control– defect list management– change control management
• Project Managers– can help to elevate the visibility of testing early in a
project
Requirements and Design Test Support Tools• Analyzers for Software Plans,
Requirements, and Designs
• System/Prototype Simulators
• Requirements Tracers
• Requirements-Based Test Case Generators
• Test Planners
Implementation and Maintenance Test Support Tools
• Compilers (syntax checking)
• Source Code Static Analyzers
• Test Preparation Tools
• Test Execution Tools (Dynamic Analyzers)
• Test Evaluators
Implementation and Maintenance Test Support Tools
• Compilers (syntax checking)
• Source Code Static Analyzers
• Test Preparation Tools
• Test Execution Tools (Dynamic Analyzers)
• Test Evaluators
Source Code Static Analyzers
• Auditors (enforce style manuals)
• Complexity Measurers
• Cross Referencing Tools– e.g. to verify uses
• Size Measurers
• Structure Checkers
• Syntax and Semantics Analyzers
Implementation and Maintenance Test Support Tools
• Compilers (syntax checking)
• Source Code Static Analyzers
• Test Preparation Tools
• Test Execution Tools (Dynamic Analyzers)
• Test Evaluators
Test Preparation Tools
• Data Extractors
• Requirements-Based Test Case Generators
• Test Data Generators
• Test Planners
Implementation and Maintenance Test Support Tools
• Compilers (syntax checking)
• Source Code Static Analyzers
• Test Preparation Tools
• Test Execution Tools (Dynamic Analyzers)
• Test Evaluators
Test Execution Tools (Dynamic Analyzers) - Part 1
• Assertion Analyzers
• Capture-Replay Tools
• Coverage/Frequency Analyzers
• Debuggers
• Emulators
• Network Analyzers
• Performance/Timing Analyzers
Test Execution Tools (Dynamic Analyzers) - Part 2
• Run-Time Error Checkers
• Simulators
• Status Displayer/Session Documenters
• Test Execution Managers
• Validation Suites
Implementation and Maintenance Test Support Tools
• Compilers (syntax checking)
• Source Code Static Analyzers
• Test Preparation Tools
• Test Execution Tools (Dynamic Analyzers)
• Test Evaluators
Test Evaluators
• Comparators
• Data Reducers and Analyzers
• Defect/Change Trackers
Other sources for testing tool info
• comp.software.testing
• www.cs.queensu.ca/shepard
• www.soft.com
• www.sqe.com
• www.methods-tools.com
• article by Robert Poston: A Guided Tour of Software Testing Tools
Testing Tool Market (Ovum - www.ovum.com)
• Currently $450M, growing at 30% per year
• dominant players (with 60% of the total market) are:– Mercury Interactive– Rational– Compuware– Segue