automated software testing using open source testing tools by elfriede dustin
TRANSCRIPT
1
Automated Software Testing using Open Source Testing Tools
By Elfriede Dustin
2
http://www.idtus.com
Agenda
• Who is IDT• State of Software Testing• What is Automated Test and Re-test (ATRT) • Approach to ATRT• Advantage of open source• ATRT and open source – Our IDE and solution• ATRT Challenges
3
http://www.idtus.com
Who is IDT?
• IDT specializes in the design, development, and implementation of Automated Software Testing and Re-Test (ATRT) and Quality Assurance (QA) solutions– Deliver turn key automated test suite
• Automated test strategy and identification of highest payoff areas to apply automation for your project
• Selection of best automation tools for your project
• Using your existing test cases or develop test cases for you
• Test results documented in requirements traceability matrix
– Training and pilot project implementation with companies
4
http://www.idtus.com
Who is IDT?
• IDT specializes in the design, development, and implementation of Automated Software Testing and Re-Test (ATRT) and Quality Assurance (QA) solutions– Deliver turn key automated test suite
• Automated test strategy and identification of highest payoff areas to apply automation for your project
• Selection of best automation tools for your project
• Using your existing test cases or develop test cases for you
• Test results documented in requirements traceability matrix
– Training and pilot project implementation with companies
•We are currently hiring Java Developers
•Plus we’d like to hire 2 or 3 grad students to work for us part time ( i.e. 20 hours week )
•must be US citizen
5
http://www.idtus.com
State of Software Testing - Why ATRT?
?
6
http://www.idtus.com
State of Software Testing - Why ATRT?
Utilize Automated Testing Strategies and Technology to Improve Productivity and Quality
Man
ual
Automated
Reduction in Test Days
IDT Provides Automated Software Testing Solutions
Increased Test
Coverage
Manual vs. Automated
Spiral / Release 1
Spiral / Release 2
Spiral / Release 3
Spiral / Release 4
Siz
e &
Co
mp
lexi
ty o
f S
W B
ase
lin
e
Test Days
Test Days
Test Days
Test Days
50% or More of Overall Development Cost is Typically Spent on Testing
7
http://www.idtus.com
What is ATRT?
• Our definition of ATRT is:
Application and implementation of software
technology throughout the entire Software
Testing Life Cycle (STL) and QA lifecycle;
with the goal to improve STL efficiencies and
effectiveness
8
http://www.idtus.com
ATRT Spans the Software DevelopmentLife Cycle – Visual Modeling
Modeling and test case generation Modeling and test case generation
Development Tools
ComponentsUse cases
Design Build Assemble
Round-Trip Engineering
TestTest
9
http://www.idtus.com
ATRT Spans the Software DevelopmentLife Cycle – Requirements
Organizes, tracks, & controls requirementsOrganizes, tracks, & controls requirements
Requirements Traceability Matrix (RTM)Requirements Traceability Matrix (RTM)
Development Tools
Components
Visual Modeling
Requirements Management andProcess Automation
10
http://www.idtus.com
Automated Software Testing
ATRT Spans the Software DevelopmentLife Cycle – Automated Test Tools
Development Tools
Components
Visual Modeling
Execution
Management
Development
Requirements Management andProcess Automation
Automates test cases using vendor-provided, Automates test cases using vendor-provided, open-source tools or in-house developmentopen-source tools or in-house development
Automated Testing Tools
11
http://www.idtus.com
ATRT Spans the Software DevelopmentLife Cycle – Integrated Suite of Tools
Additionally:Additionally:
MiddlewareMiddleware
InfrastructureInfrastructure
Defect TrackingDefect Tracking
Configuration Configuration ManagementManagement
Memory Leak Memory Leak DetectorsDetectors
Performance Performance Testing ToolsTesting Tools
Documentation Documentation ToolsTools
othersothers
Development Tools
Components
Visual Modeling
Automated Testing Tools
Requirements Management andProcess Automation
Software Configuration Management
Defect Tracking
Middleware
Infrastructure
12
http://www.idtus.com
Types of Software Test Life-cycle Support Tools
• Which tools are you familiar with?
13
http://www.idtus.com
Types of Automated Test Tools
Life-CyclePhase
Type of Tool Tool Description
BusinessAnalysis
Phase
Business Modeling Records definitions of user needs and automates rapid construction of
flexible, graphical, client-server applications
ConfigurationManagement
Baselines important data repositories
Defect Tracking Manages system life-cycle defects
Technical ReviewManagement
Facilitates communication, while automating the technical review/inspection process
DocumentationGenerators
Automate document generation
RequirementsDefinition
Phase
RequirementsManagement
Manages and organizes requirements; allows for test procedure design and test progress reporting
RequirementsVerifiers
Verify syntax, semantics, and testability
Use CaseGenerators
Create use cases
14
http://www.idtus.com
Types of Tools (Cont’d)
Programming Phase
Memory Leak and Runtime Error
Detection
Detects runtime errors and memory leaks
Source Code Testing Verifies maintainability, portability, complexity, and standards compliance
Static and Dynamic Analyzers
Depict quality and structure of code
Life-CyclePhase
Type of Tool Tool Description
Analysis and Design Phase
Database Design Develops second generation enterprise client-server systems
Structure Charts, Flowcharts, and
Sequence Diagrams
Manage processes
Test Procedure Generators
Generate test procedures from requirements, design, or data and object models
Syntax Checkers/ Debuggers
Perform syntax checking and have debugging capability; usually available with built-in programming language compiler
Metrics Tools Code (Test) Coverage Analyzers or Code
Instrumentors
Identify untested code and support dynamic testing
Usability Measurements Provide usability testing as conducted in usability labs
15
http://www.idtus.com
Types of Tools (Cont’d)
Life-CyclePhase
Type of Tool Tool Description
Other Testing Life-Cycle Support Tools
Data Extraction Tool Extract Data from various formats into various formats
Test Data Generators Generate test data
File Compare Utilities Find discrepancies between files that should be identical in content
Simulation Simulates application to measure for scalability, among other tasks
Test Management Tests management
Network Testing Monitors, measures, tests, and diagnoses performance across the entire network
GUI Testing (Capture/Playback)
Conducts automated GUI tests; capture/playback tools record user interactions with online systems so they may be replayed automatically
Load/Performance Testing Conducts load/performance and stress testing
Security Testing Performs security testing and vulnerability scanning at the application or network level; plus debuggers will allow to check for security coding errors (source code checkers)
16
http://www.idtus.com
Deployment Approach – Adapted ATLM
Collect/Create:• Requirements• Test Cases• Test Procedures• Expected Results• Interface Specifications• System/Component Configuration Description
Identify or Develop & Pilot• Test Manager• Test Scripts• Data Comparators• Performance Test Tools• GUI Record/Playback• Storing/Retrieving Results
Populate Test Manager with Test Cases and Requirements
Develop Test Scripts from Test Procedures
Test Results:• Pass/Fail by Test Case• Pass/Fail Summary• Test Time Required
17
http://www.idtus.com
Why Open Source
?
18
http://www.idtus.com
Why Open Source?
• Advantage of Open Source:– no licensing issues; i.e. license cost; maintenance;
etc.– easily modifiable– adaptable– lightweight– flexibility - not tied to one vendor
19
http://www.idtus.com
ATRTs IDE - Eclipse
Eclipse – this site lists all the plugins
http://www.eclipse-plugins.info/eclipse/plugins.jsp?category=SCM&pager.offset=0&firstItem=1
Eclipse/TPTP – Open Source - Developed by IBM– Eclipse/Test and Performance Tools Platform (TPTP)
offers a common extensible framework for the following functions:
• Requirements Management http://sourceforge.net/projects/osrmt/
• Configuration Management - Subversion
• Testing Tools (http://www.nabble.com/Eclipse-TPTP---Testing-Tools-f2262.html)
• Profiling: default Java applications, but can be extended to other apps
20
http://www.idtus.com
ATRTs IDE - Eclipse
Eclipse/Test and Performance Tools Platform (TPTP) offers a common extensible framework for the following functions (cont):
• Static Analysis: default for C++ and Java, but can be extended to other apps
• Application monitoring and log analysis
• Build tool – ant
• Xml input and output - using an open source Java package http://jakarta.apache.org/ecs/ , which is a Java library that has an API for constructing and outputting XML.
• Test Manager – STAF/STAX
21
http://www.idtus.com
Current Test Tool Implementation – Example and Challenges
Vendor-provided (Capture/Playback) Tool:
Automated test tools mimic actions of the test engineer.
During testing, the engineer uses the keyboard and mouse to perform some type of test or action.
Testing tool captures all keystrokes and subsequent results, which are baselined in an automated test script.
During test playback, scripts compare latest outputs with previous baseline.
Testing tools have built-in, reusable test functions.
Most test tools provide for non-intrusive testing; i.e., they interact with the “application-under-test” as if the test tool was not involved.
22
http://www.idtus.com
Current Testing Tools – Challenges (cont)
In-House Software Development required:
Capture/Playback tools generate hard-coded values; test scripts are not reusable, nor do they implement software development best practices right out of the box; scripts need to be modified.
Capture/Playback tools don’t necessarily provide all testing features required; code enhancements are often required to meet testing needs.
Capture/Playback tools are not necessarily compatible with system engineering environment, and software testing scripts need to be developed in-house.
Developed testing scripts are not cross-vendor compatible, even if the same scripting language is used
23
http://www.idtus.com
Linux Capture/Playback Tools IDT Findings
Replay Xcessory http://www.scl.com/products/ics/motif/testing/replay/www.ics.com
QF-Test http://www.qfs.de/en/qftestJUI/index.html
Froglogic http://www.froglogic.com/pg?id=Google&category=squishjava
Redstone Software Eggplant
KD Executor http://www.kdab.net/?page=products&sub=kdexecutor
Jameleon http://sourceforge.net/projects/jameleon/
ShUnit2 for shell scripts testing
GNU Linux Desktop Project http://ldtp.freedesktop.org/wiki/Home
Cantana http://www.ipl.com/pdf/p0003.uk.pdf
crontab Part of Unix
crontab, cruisecontrol, www.adminschoice.com/docs/crontab.htm
Dogtail http://people.redhat.com/zcerza/dogtail/about.htmlhttp://www.cyberciti.biz/tips/linux-automated-gui-testing-with-python-based-dogtail-tool.html
Expect http://expect.nist.gov/
24
http://www.idtus.com
Automated Test and Re-Test (ATRT)
• ATRT includes a common framework comprised of re-usable software services– Open Source Software Test Automation Framework
(STAF)
• Provides automated test procedures, test inputs, and test results for SW components– Suitable for use by 3rd parties not associated with
development of the component
25
http://www.idtus.com
STAX is an automation systemWith STAX you can automate and monitor your entire test environment
ƒ System/product setupƒ Testcase distributionƒ Testcase executionƒ Testcase results collection and analysis
STAX consists ofƒ A programming language designed for automation
–Programs written in this language are called jobsƒ An execution engine (similar to an interpreter for other programming/scripting
languages) which manages the execution and runtime behavior of the jobƒ A GUI application, called the STAX Monitor, which provides a dynamically
updated view of your executing jobsƒ A tool (STAXDoc) that is used to generate documentation for your STAX jobs
What is STAX?
http://www.idtus.com
* Windows 95 * Windows 98 (and 98 SE) * Windows Millenium Edition * Windows NT Version 4.0 * Windows 2000 * Windows XP * Windows Server 2003 (IA32, IA64, AMD64) * Windows Vista (IA32, AMD64) * Linux (Intel32/64, AMD64, PPC32/64) * Linux on zSeries (31-bit, 64-bit) * AIX Version 4.3.3.0 or higher (32-bit, 64-bit) * Solaris (Sparc) 2.6 and higher * Solaris AMD Opteron 64-bit * HP-UX 11.00 and higher (PA-RISC, IA64 32-bit and 64-bit) * OS/400 V5R2 or higher * z/OS UNIX V1.4 and higher
STAF can run on:
27
http://www.idtus.com
STAF Internal Services
• DIAG Provides diagnostics services Internal ("DIAG")• DELAY Provides a means to sleep a specified amount of time Internal ("DELAY")• ECHO Echos back a supplied message Internal ("ECHO")• FILE SYSTEM Allows you to get and copy files across the network Internal ("FS")• HANDLE Provides information about existing STAF handles Internal ("HANDLE")• HELP Provides Help on STAF error codes Internal ("HELP")• MISC Handles miscellaneous commands such as displaying the version of STAF• that is currently running Internal ("MISC")• PING Provides a simple is-alive message Internal ("PING")• PROCESS Allows you to start, stop, and query processes Internal ("PROCESS")• QUEUE Provides a network-enabled IPC mechanism for STAF Programs Internal ("QUEUE")• SEMAPHORE Provides network-enabled named event and mutex semaphores Internal ("SEM")• SERVICE Allows you to list services available on a machine and to examine the• Requests that have been submitted on a machineInternal ("SERVICE")• SHUTDOWN Provides a means to shutdown STAF and register for shutdown• Notifications Internal ("SHUTDOWN")• TRACE Provides tracing information for STAF services Internal ("TRACE")• TRUST Interfaces with STAF's security Internal ("TRUST")• VARIABLEProvides a method for maintaining configuration and runtime data• (variables) Internal ("VAR")
28
http://www.idtus.com
STAF External Services
The executable code for external STAF services resides outside of STAFProc, for example in a Java jar file, a C++ DLL file, or a Rexx script file.
• CRON Calls into STAF services at a specified time interval External (Java)• EMAIL Allows you to send email messages External (Java)• EVENT Provides a publish/subscribe notification system External (Java)• EVENTMANAGER Allows you to call STAF services when a specified Event
occurs External (Java)• HTTP Allows you to make HTTP requests which can be grouped together in a
session External (Java)• LOG Provides a full-featured logging facility External (C++)• MONITOR Allows a testcase to publish its current running execution status for
others to read External (C++)• RESOURCE POOL Allows you to manage exclusive access to pools of elements,
e.g. VM UserIDs or Software Licenses External (C++)• STAX Provides an XML-based execution engine External (Java)• ZIP Provides a means to zip/unzip/list/delete PKZip/WinZip compatible archives
External (C++)
29
http://www.idtus.com
STAFƒ STAF provides the infrastructure on which STAX builds. The full power of
STAF and its services is exposed for use within STAX jobs.XML
ƒ The STAX programming language is based on XML. This provides built-in structure to your jobs, as well as providing a set of existing tools for use in constructing your jobs, such as XML (aware) editors and XSLT.
Pythonƒ The STAX programming language builds on Python to provide a rich and
accessible data modelƒ STAX's Python integration also allows you to access the wealth of existing
Python librariesJava
ƒ The STAX programming language allows you to access existing Java classes/libraries, providing another source for reuse
Technologies used in STAX
30
http://www.idtus.com
Automation Tasks
Execution
Synchronization
Monitoring
Resource Management
Automation Completion
Testcase Output Analysis
Results Notification
Testcase Cleanup
Testcase Execution
System Setup
Automation Startup
31
http://www.idtus.com
End-to-End Automation with STAF and STAX
Event, EventManager, Cron
FS, Process
Process, Monitor, Log, Variable, Queue, ResPool
Log, Process
Process
Email, HTTP
STAX
Job
<function><parallel><sequence><process><stafcmd><testcase><timer><block><loop><message><log><import><job><iterate><paralleliterate><hold><release><terminate>Automation Completion
Results Notification
Testcase Cleanup
Testcase Output Analysis
Testcase Execution
System Setup
Automation Startup
32
http://www.idtus.com
Key ATRT Architecture Requirements
• Support applications running on multiple computers• Support applications developed in different languages• Support applications running on different types of OS’s• Support applications which have GUI and those which do not (for
example Interface testing)• Support applications which use different types of network
protocols such as TCP/IP, DDS, etc• Support integration of multiple commercial testing tools from
different vendors ( allowing as new or better products emerge in the market they can be utilized )
• Support testing w/o having to install ATRT on the samecomputers as the application under test and be able to be
distributed across computers
33
http://www.idtus.com
Display Application Display Application Analysis & Display Application
Simulation of Application Processing Messages
Simulation of Application Processing Messages
System Under Test
Test ManagerRTMBugzilla
ATRT
EggplantR/P
VNC RobotR/P
STAF/STAX
Prototype – Proof of Concept
WindowsBased
Linux Based
34
http://www.idtus.com
• Able to successfully execute ATRT for 48 hours without operator intervention
• During the tests:
VNCRobot - Approximately 25,000 operator key strokes were replayed STAF/STAX - More than 1,000,000 messages processed and verified by ATRT
ATRT Prototyping Effort
35
http://www.idtus.com
Additional Requirement
• Pristine test environment required – automated testing tool cannot be installed on Application-Under-Test PC
Eggplant – allows for remote capture/playback via VNCserver and KVM switch
VNCRobot
36
http://www.idtus.com
Automated Software Testing
• Software testing paradox
• What is it?
37
http://www.idtus.com
Effective Automated Software Testing
Current State: Developing software in order to test software
Automate the test support development: automated IDL to code generation; test data generation (using FireEye from NIST); compare utilities; log file analyzers;
Our Automated Software Testing Goal: • Minimize Human interaction
– Possibly get on par with Hardware Automated Testing?
38
http://www.idtus.com
Hardware Testing: current state
• Many automated test generators for manufacturing/ logic devices, circuit testing, etc., available i.e. SEI/ATG, uMaster, etc.
• Automated testing without human interaction is the norm for hardware testing
39
http://www.idtus.com
Hardware vs. SoftwareAT
How is hardware testing different?• Software Challenge:
– Changing requirements– GUI interfaces– Code access/availability– Defects uncovered earlier in the software
development lifecycle are cheaper to fix
• Solution flexibility required – problem dependent
40
http://www.idtus.com
Proposed solutions
Focus on Reusable Components– Test Data / Expected Results– Interfaces – Middleware
Assuming code and models are available:• Model based testing; automated test case generators based on models• Eclipse – effective ATRT IDE• Probes – see Aprobe – Code instrumentation• Building self-testable components• Using standard language MOF to Text to generate automate test casesGUI Testing tool ideasNew tools - other• Working with NIST to produce additional tools
41
http://www.idtus.com
GUI Testing
• Jemmy - java library to test UI
• http://jemmy.netbeans.org/
• http://abbot.sourceforge.net/doc/overview.shtml
• Perl modules
• run via VNCServer?
43
http://www.idtus.com
IDT
We are hiring…….
Please come see me now if interested or sendemail [email protected] with any follow-up