software testing
TRANSCRIPT
Software Development Conference 2007
IBM RATIONAL TOOLS FOR TESTING
Presentation: Vítek LinduškaDemo: Václav Novotný
Software Development Conference 2007
2
Agenda
Manual Tester
Functional Tester
Performance Tester
ClearQuest
Software Development Conference 2007
3
Manual Tester
Software Development Conference 2007
4
Why Teams use Manual Testing?
Recruitment of subject-matter experts and end-users to test
Poor or non-existent test automation-ready interfaces
Strict regulatory or process requirements aren’t met with automation tools
Software Development Conference 2007
5
Challenges of Manual TestingChallenges of Manual Testing
Labor Intensive Too much time is spent creating
and maintaining manual test documentation
Labor Intensive Too much time is spent creating
and maintaining manual test documentation
“More time is spent maintaining test
documentation than on executing tests.”
“We struggle to document even a
minimum set of tests.”
Software Development Conference 2007
6
Challenges of Manual TestingChallenges of Manual Testing
Labor Intensive Too much time is spent creating
and maintaining manual test documentation
Unreliable
Inadvertent human-error often skews test results
Labor Intensive Too much time is spent creating
and maintaining manual test documentation
Unreliable
Inadvertent human-error often skews test results
“Our results are unreliable due to
human error.”
“Our tests aren’t detailed enough for the testers to consistently
execute them.”
Software Development Conference 2007
7
Challenge 1: Labor Intensive
Challenge Creating and maintaining manual tests is labor intensive and difficult
with tools such as Microsoft Word or Excel.
Challenge Creating and maintaining manual tests is labor intensive and difficult
with tools such as Microsoft Word or Excel.
Resolution Implement a solution that
promotes the best practice of modular tests which can share common content
Leverage existing Word or Excel test documents to quick-start test development
Resolution Implement a solution that
promotes the best practice of modular tests which can share common content
Leverage existing Word or Excel test documents to quick-start test development
Share test
content
Software Development Conference 2007
8
Challenge Human error often skews tests results
Challenge Human error often skews tests results
Resolution Assist testers with the tasks
most prone to error – data entry and data verification
Build manual tests that are clear, concise and easy for the tester to follow
Resolution Assist testers with the tasks
most prone to error – data entry and data verification
Build manual tests that are clear, concise and easy for the tester to follow
Challenge 2: UnreliableChallenge 2: Unreliable
Software Development Conference 2007
9
Leverage Existing Test Documents Import from Word or Excel Microsoft Word
documents
Microsoft Excel spreadsheets
Jump-start adoption of Rational Manual Tester by leveraging existing assets
Bulk import manual tests documented in Word or Excel
Jump-start adoption of Rational Manual Tester by leveraging existing assets
Bulk import manual tests documented in Word or Excel
Rational Manual Tester test
Software Development Conference 2007
10
Assisted Data Entry and Data VerificationReduces human error
Store critical data used during the test as part of the test instructions Store critical data used during the test as part of the test instructions
Test execution window
Software Development Conference 2007
11
Assisted Data Entry and Data VerificationReduces human error
Store critical data used during the test as part of the test instructions
Automate data entry during test execution to speed entry and reduce errors
Store critical data used during the test as part of the test instructions
Automate data entry during test execution to speed entry and reduce errors
Test execution window
Application under test
Software Development Conference 2007
12
Assisted Data Entry and Data VerificationReduces human error
Store critical data used during the test as part of the test instructions
Automate data entry during test execution to speed entry and reduce errors
Automate data validation during test execution to ensure accurate results
Store critical data used during the test as part of the test instructions
Automate data entry during test execution to speed entry and reduce errors
Automate data validation during test execution to ensure accurate results
Software Development Conference 2007
13
Enables Shared Content Across Multiple TestsReduces test maintenance and ensure consistency
Create commonly used test statements once
Reuse statements across multiple tests through simple drag and drop
Maintenance of shared content is efficient with single-point updating of shared content
Ensures consistency and reduces test maintenance
Create commonly used test statements once
Reuse statements across multiple tests through simple drag and drop
Maintenance of shared content is efficient with single-point updating of shared content
Ensures consistency and reduces test maintenance
Software Development Conference 2007
14
On-Screen Prompted Test ExecutionImmediate and accurate results collection
On-screen prompting of test steps
Immediately record test results and attach images or files for clarity
Tool will assist with data entry and data verification
Results recorded in permanent log
On-screen prompting of test steps
Immediately record test results and attach images or files for clarity
Tool will assist with data entry and data verification
Results recorded in permanent log
Software Development Conference 2007
15
Spread-Sheet Ready Test ResultsUse with any 3rd party reporting tool
Results easily exported to common file format
Import into any 3rd party tool for evaluation and consolidation with other test data
Results easily exported to common file format
Import into any 3rd party tool for evaluation and consolidation with other test data
Software Development Conference 2007
16
Functional Tester
Software Development Conference 2007
Manual testing can’t keep pace!Manual testing
can’t keep pace!Rerun the first tests and...
Time
Tests
...take more time
...take more time
...the new tests...
Build 1 Build 2 Build 3, 4, 5 …
Challenge of Manual Testing with Short Test Cycles
Manual testing alone can’t keep pace with the rapid development of applications Quality will be sacrificed to keep on schedule
Software Development Conference 2007
18
Challenges of Functional TestingChallenges of Functional Testing
Controlling cost of test automation Inexperienced testers face steep
learning curve for tools
Test maintenance outpaces testing
Controlling cost of test automation Inexperienced testers face steep
learning curve for tools
Test maintenance outpaces testing
“Our testers are not experienced in building
automated tests.”
“Application UI changes are always
breaking our test scripts.”
Software Development Conference 2007
19
Challenges of Functional TestingChallenges of Functional Testing
Control cost of test automation Inexperienced testers face steep
learning curve for tools
Test maintenance grows with project
Testing Tools Lack Flexibility Power users hands are tied by tool
limitations
Control cost of test automation Inexperienced testers face steep
learning curve for tools
Test maintenance grows with project
Testing Tools Lack Flexibility Power users hands are tied by tool
limitations
“I need one tool for both my expert and
novice testers.”
“Our script language limits our ability to overcome some
automation obstacles.”
Software Development Conference 2007
20
Challenges of Functional TestingChallenges of Functional Testing
Control cost of test automation Inexperienced testers face steep
learning curve for tools
Test maintenance grows with project
Testing Tools Lack Flexibility Power users hands are tied by tool
limitations
Test Team Integration Managing script changes with
parallel development
Sharing test assets with the larger software team
Control cost of test automation Inexperienced testers face steep
learning curve for tools
Test maintenance grows with project
Testing Tools Lack Flexibility Power users hands are tied by tool
limitations
Test Team Integration Managing script changes with
parallel development
Sharing test assets with the larger software team
“How do I keep my team in connected with
the project team?”
“We’re geographically distributed and we’re worried about parallel script development .”
Software Development Conference 2007
21
Challenge Overcome the complexity of test automation to ensure teams spend more
time on high value quality assessment rather than on low return test maintenance.
Challenge Overcome the complexity of test automation to ensure teams spend more
time on high value quality assessment rather than on low return test maintenance.
Resolution Provide technologies to enable
novice testers to craft complex tests with no programming Data driven wizard
Dynamic data verification
Reduce script maintenance Wizard driven global updates to
centralized object map
Resolution Provide technologies to enable
novice testers to craft complex tests with no programming Data driven wizard
Dynamic data verification
Reduce script maintenance Wizard driven global updates to
centralized object map
Challenge 1: Controlling the cost of AutomationChallenge 1: Controlling the cost of Automation
Software Development Conference 2007
22
Challenge Fulfill both standard test needs as well as expert
requirements to ensure usage limited only by the imagination of the project team.
Challenge Fulfill both standard test needs as well as expert
requirements to ensure usage limited only by the imagination of the project team.
Resolution Provide two test script
development languages Java
Visual Basic .NET
Give professional testers the flexibility to overcome testing challenges in the language of their choice
Resolution Provide two test script
development languages Java
Visual Basic .NET
Give professional testers the flexibility to overcome testing challenges in the language of their choice
Challenge 2: Tools Lack FlexibilityChallenge 2: Tools Lack Flexibility
Software Development Conference 2007
23
Challenge 3: Test Team IntegrationChallenge 3: Test Team Integration
Challenge Accommodate the requirements of both dedicated,
independent QA teams as well as integrated project teams to ensure high operational integrity.
Challenge Accommodate the requirements of both dedicated,
independent QA teams as well as integrated project teams to ensure high operational integrity.
Resolution Provide version control
support in the box with ClearCase LT
Provide integrations for standard version control features
Check In/Out
Compare withPrevious Versions
Show change history
Resolution Provide version control
support in the box with ClearCase LT
Provide integrations for standard version control features
Check In/Out
Compare withPrevious Versions
Show change history
Software Development Conference 2007
24
Reduce Test Script MaintenanceReliable Playback with Script Assure
DeterminesMatch
Version 1.0 Version 2.0
TesterSees
ToolSees
No User No User Intervention Intervention
Required With Required With ScriptAssureScriptAssureTMTM
Software Development Conference 2007
25
Performance Tester
Software Development Conference 2007
26
Challenges of Performance TestingChallenges of Performance Testing
Complex Tools & Complex Task Tool complexity and lack of
experience intimidates many first time users
Complex Tools & Complex Task Tool complexity and lack of
experience intimidates many first time users
“We don’t have a lot of experience building
good load tests.”
“Most load test tools are much more
complicated than we require.”
Software Development Conference 2007
27
Challenges of Performance TestingChallenges of Performance Testing
Complex Tools & Complex Task Tool complexity and lack of
experience intimidates many first time users
Testing Tools Lack Flexibility Power users hands are tied by tool
limitations
Complex Tools & Complex Task Tool complexity and lack of
experience intimidates many first time users
Testing Tools Lack Flexibility Power users hands are tied by tool
limitations
“We don’t have a good understanding of what
exactly is occurring during a load test.”
“We have highly technical performance
testers that need to customize their tests.”
Software Development Conference 2007
28
Challenge 1: Complex Tools & Complex TaskChallenge 1: Complex Tools & Complex Task
Challenge Tool complexity and lack of experience intimidates many first time
users
Challenge Tool complexity and lack of experience intimidates many first time
users
Resolution Remove the need for
programming
Tests are represented in a tree view as a sequential flow web pages
User interface enables
Data pooling
Data correlation
Looping
Conditional events
Resolution Remove the need for
programming
Tests are represented in a tree view as a sequential flow web pages
User interface enables
Data pooling
Data correlation
Looping
Conditional events
Software Development Conference 2007
29
ChallengePower users hands are tied by tool limitations
ChallengePower users hands are tied by tool limitations
Resolution Tests may call custom Java
code provided by the user Offers flexibility for power users
to perform complex tasks
Expose behind the scenes details which are hidden from novice users by default Provide the flexibility to
overcome any testing obstacles
Resolution Tests may call custom Java
code provided by the user Offers flexibility for power users
to perform complex tasks
Expose behind the scenes details which are hidden from novice users by default Provide the flexibility to
overcome any testing obstacles
Challenge 2: Tools Lack FlexibilityChallenge 2: Tools Lack Flexibility
Software Development Conference 2007
30
Performance Problem Identification During Test
Performance Testing finds bottlenecks Next logical question is Why?
Root Cause Analysis provides to tools to answer this question
Page Performance ReportShows average response time
per page.
Highest bar = Performance Problem
Software Development Conference 2007
31
Performance Problem Identification During Test
Performance Testing finds bottlenecks Next logical question is Why?
Root Cause Analysis provides to tools to answer this question
Software Development Conference 2007
32
Response Time Breakdown
Feature: Breakdown page response times into composite element response time
Benefit: Breakdown data drills down to highlight slowest components of a given page.
Breakdown into tiers and components is a unique competitive advantage for RPT.
Response data broken down by tier and by transaction component (JDBC, JSP,
Servlet, etc…)
Software Development Conference 2007
33
Deep Diagnostic DataProviding the details you need to solve the performance problem mystery
UML Sequence diagram showing method calls between
classes with timing information. User can drill
down to source code.
Execution Statistics Shows response time for
every method
Software Development Conference 2007
34
ClearQuest
Software Development Conference 2007
35
Eclipse Based Test Management Joins Functional and Performance
Tester to complete transition of testing tools to eclipse platform
Native and web clients also available
Key Functional Areas Test Planning
Create test plans, test cases and configured test cases
Test Execution Create suites or run individual
test cases
Test Reporting Extensive query and reporting
facilities to measure test progress
Test Management with ClearQuestEclipse, Windows and Web Clients for test project tracking
Software Development Conference 2007
36
Lifecycle TraceabilityQuery one source for all aspects of project progress
Query to find Test results associated to requirements
Defects associated to test results
Defects associated to tests associated to requirements
Traceability from requirements to tests and defects closes the loop on the software development cycle
Requirements Tests Defects
Software Development Conference 2007
37
Test Case Trending
Test Verdicts
Lifecycle TraceabilityQuery one source for all aspects of project progress
Leverage existing reports
orCreate new reports with Crystal Reports
To provide the information you need for decision making
Software Development Conference 2007
38
ClearQuest Web InterfaceDistributed access for test planning and reporting
Instant access for remote users
Access for alternative platforms
No additional licenses required
Perfect for test planning and reporting
Software Development Conference 2007
39
`
ClearQuest MultiSiteReplication and synchronization for geographically distributed teams
Automatic replication and synchronization of ClearQuest repositories
Access current information, regardless of location
Automatic recovery of repositories in the event of system failure maintains data integrity
Transmission of only incremental changes for improved network performance
Software Development Conference 2007
40
Centralized User ManagementLDAP authentication for common user id support
Reduce administration with centralized user management – maintain only one list of user ids
LDAP authentication supports common user ids for ClearQuest
and RequisitePro
Rational RequisitePro Authorization Rational
RequisitePro
Rational ClearQuest
LDAP 3.0 Compliant Server
Rational ClearQuest
Authorization
Rational ClearQuest User DBLogon Dialog
LDAP
Software Development Conference 2007
41
Compliance and ControlUsing Audit Trails and eSignatures to track change
Audit Trails can track changes to individual records Provides immutable change log of all
record change activity
e-Signature requirement can be added to any event on any artifact
Will record who changed a record, when and how the record was changed
e-Signature provides immutable evidence of process completion
ValidateResults
TestExecutionTest CaseRequirement Approve
e-Signature Approval
Software Development Conference 2007
42
Configure and Enforce Your ProcessCustomize appearance and behavior of all forms and processes
Customize everything Modify forms
Configure State Transition
Add fields and control behavior
Extend your system with custom coded hooks
Extend functionality using pre-defined Packages eSignatures
Audit Trail
Many, many more…
Software Development Conference 2007
43
Extensible Test EcosystemIntegrated with Rational, open source and 3rd party test automation tools
Collect and report on results from Rational test execution engines Rational Manual Tester
Rational Functional Tester
Rational Performance Tester
Collect and report on results from Eclipse Test and Performance Tools Project jUnit tests
Manual tests
URL tests
Open API Extensibility SPI Dynamics QAInspect
Software Development Conference 2007
44
Rational Software Quality Solutions
Developer Test Functional Test
Automated Manual
Rational RequisitePro
Rational ClearQuest
Rational ClearQuest
Defects
Project Dashboards
Detailed Test Results
Quality Reports
Performance Test
SOFTWARE QUALITY SOLUTIONS
Test and Change Management
Test Automation
Quality Metrics
DE
VE
LO
PM
EN
T
OP
ER
AT
OIN
S
BUSINESS
Rational ClearQuest
Requirements Test Change
Rational PurifyPlus
Rational Test RealTime
Rational Functional Tester Plus
Rational Functional Tester
Rational Robot
Rational Manual Tester
Rational Performance Tester
Software Development Conference 2007
45
WAKE UP, IT’S DEMO TIME!