automated testing nathan weiss april 23, 2007. overview history of testing advantages to automated...
TRANSCRIPT
Overview
History of Testing Advantages to Automated Testing Types of Automated Testing Automated Testing Tools Disadvantages to Automated Testing
History of Testing
Four main phases in testing methodology: Debugging (1950’s) Exhaustive Testing/Prove it Works
(1960’s) Prove it Does Not Work (1970’s) Defect Prevention & Testing Process
(1980’s)
Debugging
No distinction between debugging and testing.
Consisted of making code compile and changing code so that it would not crash.
Exhaustive Testing/Prove it Works
Exhaustive testing in terms of code paths and given input. Found to be impossible
Write tests to prove the program works.
Testing defined as “what is done to demonstrate correctness of a program.”
Prove it Does Not Work
Testing not for proving correctness, but for finding defects.
Started testing to see if a program did what it was not suppose to do, not just test to prove that it does what it was suppose to do.
Defect Prevention & Testing Process
“Testing” traditionally referred to what was done to the system after working code was produced.
Testing moved towards an activity done during a phase, not a phase by itself.
Studies show that 50% of bugs are created in requirements and design stages.
Advantages to Automated Testing
Test scenarios impossible/unfeasible using manual testing.
Lowers testing time and costs in an iterative environment.
Remove the human error factor. Introduces stability in regression tests. Increase in testers’ morale.
Types of Automated Testing
Split into two categories Black box tests White box tests
PC applications Embedded Systems
Black Box Automated Testing
GUI testing Always been prime candidates for
manual testing. Need to automate has created many
commercial tools. Two ways to create the tests:
Scripts Recordings
Black Box Tests - Scripts
Less of a testing tool/environment and more of a testing library.
Flexible. Easy to run batch scripts to run
multiple tests.
Black Box Tests - Recordings
Full testing tool/environment. Tester performs a test once while tools
is recording his actions. Tester can add ending conditions to
test the success of a test. Very useful for testing GUIs of stand
alone applications.
White Box Automated Testing
Module testing. More APIs then environments. Embedded systems introduce more
complexity to white box testing.
White Box Tests – PC Apps
Tests usually written in the same language as the system being tested.
Tools usually come in the form of an API containing asserts and other testing utility methods.
Tests can be performed on any level of abstraction required by the project.
White Box Testing – Embedded Systems
Test code usually not stored/run on the same hardware as the system code.
Do not have direct access to system code or memory.
Requires an interface with an embedded system debugger.
xUnit
Compilation of popular code driven testing frameworks.
Frameworks based on the design by Kent Beck.
Originally implemented for SmallTalk as SUnit.
xUnit
Partial List of Frameworks NUnit (.net) JUnit (java) CUnit ( C ) CPPUnit (C++) SUnit (SmallTalk)
SilkTest
Created by Borland. Works with:
Web Browsers(IE and Firefox) Java GUIs .Net GUIs Win32 GUIs
Price : 4-6k
Selenium
Testing tool for web applications. Comes in API and tool form. Supports:
Windows: IE, Firefox, Seamonkey, Opera Mac OS X: Safari, Firefox, Camino,
Seamonkey Linux: Firefox, Konqueror
Selenium
Can create tests using the Selenium library in many languages (Java, .NET, Perl, Python and Ruby)
Or you can create tests using their Selenium IDE which is implemented as a Firefox extension. Once created in Firefox, the tests can be
run in any of the suported browsers.
Custom
Write your own custom testing environment or API.
Used when you have very specific testing needs.
Often used for embedded systems.
References
Borland. (2007). Borland SilkTest: An Automated Regression and Functional Software Testing Tool. Retrieved April 2, 2007, from http://www.borland.com/us/products/silk/silktest/index.html
[2] Dustin, E., Rashka, J., Paul, J. (1999). Automated Software Testing: Introduction, Management, and Performance. Addison-Wesley.
[3] Lewis, E., W., (2005). Software Testing and Continuous Quality Improvement. CRC Press LLC.
[4] NUnit. (2006). Nunit. Retrieved April 2, 2007, from http://www.nunit.org
[5] OpenQA. (2006). OpenQA: Selenium. Retrieved April 2, 2007, from http://www.openqa.org/selenium/index.html