vv07
DESCRIPTION
TRANSCRIPT
![Page 1: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/1.jpg)
Testing AutomationCFICSE VV07 October 1999
Dr. Terry Shepard
Royal Military College of Canada
Electrical and Computer Engineering
613-541-6000 ext. 6031
![Page 2: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/2.jpg)
Testing Automation Topics
• Motivation for Automation
• Design for Testability
• What can be automated?
• Context and Needs
• Intrusive vs. non-intrusive
• Software Test Technologies Report– Classification of testing tools
![Page 3: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/3.jpg)
Motivation for Testing Automation
• reduce costs
• ensure repeatability of tests
• ensure that test scenarios are followed exactly
• increase number of tests that can be run by a deadline
• improve reliability estimation
![Page 4: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/4.jpg)
Design for Testability
• Common for computer hardware
• Primarily means– Controllability– Observability– Well behaved interfaces between modules
• Determines what can easily be automated– e.g. instrumentation code built into source code
![Page 5: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/5.jpg)
What can be automated?
• Test execution
• Comparison of actual and expected results
• Logging of state at the time of a failed test– recovery for debugging purposes
• Management of test logs
• Collection of metrics
• Version control (in part)
• Test case selection: hard!
![Page 6: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/6.jpg)
Automated support can be provided for other tasks
• Test planning and budgeting
• Test setup procedures
• Test adequacy decisions
• Test scenarios with external dependencies
![Page 7: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/7.jpg)
Context and Need
• In a given project, the context will determine candidate testing tools– constraints include language, platform, OS, …
• Given a set of constraints, a market search and evaluation can be undertaken
• Broad range of applicability and variety of tools makes needs assessment hard
• Cost justification should be applied
![Page 8: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/8.jpg)
Intrusive vs. Non-Intrusive
• Intrusion is generally not possible in an operational setting
• Build black box recorder capabilities into the software?
• Continuing hardware performance improvements will make this increasingly feasible
![Page 9: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/9.jpg)
Software Test Technologies Report
• August 1994
• Software Technology Support Center, Hill AFB, Air Logistics Command, Ogden, Utah– Online at www.stsc.hill.af.mil– STSC has an Annual Conference, publishes
Crosstalk, …
• No update in classification since 1994
![Page 10: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/10.jpg)
1994 STSC Test Tools Classification
• Test Resource Management
• Requirements and Design Test Support
• Implementation and Maintenance Test Support– Compilers– Source Code Static Analyzers– Test Preparation– Test Execution (Dynamic Analyzers)– Test Evaluators
![Page 11: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/11.jpg)
Test Resource Management Tools
• Configuration Managers– test version control– defect list management– change control management
• Project Managers– can help to elevate the visibility of testing early in a
project
![Page 12: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/12.jpg)
Requirements and Design Test Support Tools• Analyzers for Software Plans,
Requirements, and Designs
• System/Prototype Simulators
• Requirements Tracers
• Requirements-Based Test Case Generators
• Test Planners
![Page 13: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/13.jpg)
Implementation and Maintenance Test Support Tools
• Compilers (syntax checking)
• Source Code Static Analyzers
• Test Preparation Tools
• Test Execution Tools (Dynamic Analyzers)
• Test Evaluators
![Page 14: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/14.jpg)
Implementation and Maintenance Test Support Tools
• Compilers (syntax checking)
• Source Code Static Analyzers
• Test Preparation Tools
• Test Execution Tools (Dynamic Analyzers)
• Test Evaluators
![Page 15: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/15.jpg)
Source Code Static Analyzers
• Auditors (enforce style manuals)
• Complexity Measurers
• Cross Referencing Tools– e.g. to verify uses
• Size Measurers
• Structure Checkers
• Syntax and Semantics Analyzers
![Page 16: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/16.jpg)
Implementation and Maintenance Test Support Tools
• Compilers (syntax checking)
• Source Code Static Analyzers
• Test Preparation Tools
• Test Execution Tools (Dynamic Analyzers)
• Test Evaluators
![Page 17: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/17.jpg)
Test Preparation Tools
• Data Extractors
• Requirements-Based Test Case Generators
• Test Data Generators
• Test Planners
![Page 18: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/18.jpg)
Implementation and Maintenance Test Support Tools
• Compilers (syntax checking)
• Source Code Static Analyzers
• Test Preparation Tools
• Test Execution Tools (Dynamic Analyzers)
• Test Evaluators
![Page 19: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/19.jpg)
Test Execution Tools (Dynamic Analyzers) - Part 1
• Assertion Analyzers
• Capture-Replay Tools
• Coverage/Frequency Analyzers
• Debuggers
• Emulators
• Network Analyzers
• Performance/Timing Analyzers
![Page 20: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/20.jpg)
Test Execution Tools (Dynamic Analyzers) - Part 2
• Run-Time Error Checkers
• Simulators
• Status Displayer/Session Documenters
• Test Execution Managers
• Validation Suites
![Page 21: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/21.jpg)
Implementation and Maintenance Test Support Tools
• Compilers (syntax checking)
• Source Code Static Analyzers
• Test Preparation Tools
• Test Execution Tools (Dynamic Analyzers)
• Test Evaluators
![Page 22: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/22.jpg)
Test Evaluators
• Comparators
• Data Reducers and Analyzers
• Defect/Change Trackers
![Page 23: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/23.jpg)
Other sources for testing tool info
• comp.software.testing
• www.cs.queensu.ca/shepard
• www.soft.com
• www.sqe.com
• www.methods-tools.com
• article by Robert Poston: A Guided Tour of Software Testing Tools
![Page 24: VV07](https://reader034.vdocuments.us/reader034/viewer/2022052522/5480b027b4af9fa2088b45f8/html5/thumbnails/24.jpg)
Testing Tool Market (Ovum - www.ovum.com)
• Currently $450M, growing at 30% per year
• dominant players (with 60% of the total market) are:– Mercury Interactive– Rational– Compuware– Segue