® ibm software group © 2004 ibm corporation ensuring application reliability and scalability...

ASQ Seminar PresentationBy Denice Wong
ASQ Seminar Deck. Contact [email protected]
Seminar is 3 hours, includes 3 demos for RMT, RFT and RPT. Demos run in seminar VM from ftp://pokgsa.ibm.com/projects/v/vm_images/050610_BBryson_ASQ_Seminar.
IBM Software Group | Rational software
Challenges Associated With Testing
Unable to perform full regression testing
Inaccurate performance prediction on production environment
Here are some of the challenges of testing – automated or otherwise:
Insufficient Requirements – I think we’ve all experienced this…and especially for newcomers to a testing team. We’ve all be assigned to a project where we’ve been given a PC and told to “test the app”…no help, no requirements…well the results of those tests, as you would expect are not usually great. Without some guidance on what to test, and what results should be expected for a given set of inputs, the tests are not likely to find any significant problems beyond “Alt+F doesn’t open up the File menu”.
“Changing requirements” is the next variation of that problem. When system specs are changing so rapidly that the QA team can’t keep up, tests that used to be perfectly valid may be of little or no use – or worse they may find “false positives or false defects” – defects that aren’t true defects that waste time as they get entered into the defect tracking system, and consume project resources.
The waterfall process model is a development practice that leaves all QA to the end of a project. This cultivates multiple issues. First of all, there is no validation of the quality as you go along. It is not until very late in the cycle where you will discover potentially critical bugs and architectural issues that might take a lot of rework to correct. If it weren’t for schedule slips, this might be manageable, but virtually all projects experience some schedule slip that inevitably takes away from the time allotted for testing.
Poor unit testing is frustrating for QA teams, as they often cause road-blocking defects. Defects that should have been found earlier and were not impede other integrated tests.
And, of course, time. I’ve already brought up the schedule slip, and it’s worth it’s own point here – as no QA team ever really has the time they need to do the job they want.
An Example: Challenges of Siebel Implementations
a
Testing costs vary widely depending on project type and complexity
“Customers are spending close to $½ Million on testing complex projects”1
1 Source: Siebel Test Automation Datasheet
Survey of more than 130 Siebel projects showed that:
Not unusual for customers to spend ~25% of total project costs on testing
Testing costs vary widely depending on project type and complexity
Customers are spending close to $½ Million on testing complex projects
Chart1
109998
212400
462000
10%
9.17
17.70
38.50
8.25
10%
16.82
5%
37.54
3%
2.44
73%
10.09
43%
33.65
13%
10%
16.00
16.67
17.34
14.40
10%
15.83
5%
16.90
3%
4.25
73%
9.50
43%
15.15
13%
10%
9.34
14.50
53.50
8.40
10%
13.78
5%
52.16
3%
2.48
73%
8.27
43%
46.76
13%
5%
4.67
7.25
26.75
4.20
10%
6.53
10%
24.08
10%
2.94
37%
4.57
37%
16.85
37%
Total
93.35
145.00
535.00
86.51
7%
137.28
5%
511.86
4%
43.55
53%
79.85
45%
339.64
37%
ahan: This number is estimated since there was no survey data in this group. Estimated based on linear extrapolation from medium and complex.
ahan: Percentage breakdown based on survey responses
Cost Model by Feature
10%
9.17
17.70
38.50
9.17
17.70
38.50
calc
0.92
8.85
34.65
9.17
17.70
38.50
9.17
17.70
38.50
95%
8.71
16.82
36.58
9.17
17.70
38.50
80%
7.33
14.16
30.80
9.17
17.70
38.50
10%
16.00
16.67
17.34
16.00
16.67
17.34
calc
1.60
8.33
15.60
16.00
16.67
17.34
16.00
16.67
17.34
95%
15.20
15.83
16.47
16.00
16.67
17.34
80%
12.80
13.33
13.87
16.00
16.67
17.34
10%
9.34
14.50
53.50
9.34
14.50
53.50
calc
0.93
7.25
48.15
9.34
14.50
53.50
9.34
14.50
53.50
95%
8.87
13.78
50.83
9.34
14.50
53.50
80%
7.47
11.60
42.80
9.34
14.50
53.50
5%
4.67
7.25
26.75
95%
4.43
6.89
25.41
4.67
7.25
26.75
4.67
7.25
26.75
4.67
7.25
26.75
4.67
7.25
26.75
90%
4.20
6.53
24.08
4.67
7.25
26.75
90%
4.20
6.53
24.08
1
2
93.35
145.00
535.00
83.64
129.92
479.36
75.87
129.92
523.87
76.63
119.03
439.17
90.62
140.76
519.35
90.55
140.65
518.95
88.22
137.03
505.58
84.95
131.95
486.85
90.25
140.18
517.21
TOTAL
93.35
145.00
535.00
10%
10%
10%
19%
10%
2%
18%
18%
18%
3%
3%
3%
3%
3%
3%
5%
6%
6%
9%
9%
9%
3%
3%
3%
ahan: These Numbers are based on Testing customer survey. 130 total respondents
ahan: Need PS to validate these numbers
ahan: 90% savings for simple 50% savings for medium 10% savings for complex
Cost Model by Type
10%
9.17
17.70
38.50
10%
16.00
16.67
17.34
10%
9.34
14.50
53.50
Cost Model by Type
0
0
0
Do you ever have enough time to test everything???
Trade coverage for risk
Reduce scope of testing
Introduce / enhance automation
…and that’s often why people come to test automation.
They run out of time, and look for ways to get more done with less…usually they’ll examine ways of cutting tests (“Just test what is really REALLY important.”), cutting cycles (“We only have time to test every other milestone build.”), or configurations (“We’ll test on Windows XP, and just assume it works on Windows 2000”) – or they’ll consider adding people to a project (few projects have the resources or budget to simply increase staff, however) – or they’ll look to become more efficient by using automation.
Automation Adds Options!
Improve human productivity
Broader test coverage
Tie testing to requirements
…And automation certainly offers the hope of becoming more efficient with the resources you have. You have the opportunity to improve human productivity – by running test overnight. You can remove the tedium: Let the computer handle repetitive tasks and thus enable your team to spend more time on complex tests and test designs. You also remove the human error factor, which is bound to creep into any human-intensive process.
Automation also offers the hope of process improvements – enabling you to execute more tests on every build of your application. Continuously regression testing each build helps ensure that things that worked in one build are not broken by new features or bug fixes in the next build.
So – there are obviously some great benefits to automation – the devil, as they say, is in the details…so let’s dive into the details of implementing test automation into your organization.
What Does Test Automation Mean?
Do my testing for me!
Of course, this is what we’d like test automation to mean. Wouldn’t that be nice?! Unfortunately, the tool has yet to be invented that will read the mind of the designer to determine how he intended to build it and read the mind of the end user to determine how he intended to use it. This is just one misconception of automation. Automated test tools do not simply do your testing for you. They help to automate the time-consuming, error-prone, mundane tasks, but there is certainly still some thinking required.
What is Automated Functional
Using automation tools to record actions taken against an application.
Test scripts are derived from these recordings which are then used for subsequent test execution.
Record
Enhance
Execute
BBryson: Introduce here the three main steps of creating a script. Note that few vendors talk about the enhance step. It’s a critical stage, but often overlooked and underestimated by those taking a first step into automation. Hopefully today we can steer you in the right direction and set some proper expectations for you. [WE’RE ATTEMPTING TO BUILD CREDIBILITY AGAINST MERCURY’S RECOCRD-PLAYBACK CLAIMS, BY BEING MORE UP FRONT AND HONEST ABOUT THE REALITY OF AUTOMATION].
What Is Automated Performance Testing?
The process of exercising an application by emulating actual users
with a load generation tool for the purpose of finding system bottlenecks
Controller
First, what is performance testing?
It is the process of exercising an application by emulating actual users putting a load on your system for the purpose of finding system bottlenecks. This load is generally created by a load generation tool. You typically have a master computer that is controlling other computers – as well as ‘virtual testers’ – to emulate this production load.
Once the test run is complete all the data is returned to the master where the user analyzes the data to find the performance bottlenecks.
There are probably a million different definitions which we could use, but I’m hoping we can just start from here and build on this. From here we’ll move forward take a look at the various challenges of performance testing.
What is Automated Test Management?
Using automation tools to accelerate the process of test planning, test creation, test execution and test results analysis
Plan
Create
Execute
Analyze
There’s probably as many valid and arguable definitions for automated test management as there are people in this room today, but just for the purposes of out presentation, let’s go with the one we have here. [READ DEFINITION FROM SLIDE]
It’s quite a narrow definition, and I’m doing myself somewhat of a disservice by using this – as it ignores many of the areas of automation that IBM Rational can help out with, but we want this narrow focus for today [YOU’RE TEASING AND ALLUDING TO ROBOT, PURIFYPLUS, ANY OTHER TOOL THAT THEY MAY BE INTERESTED IN HERE BY SAYING THIS] , so that we can drill a little deeper on just plan-create-execute-analyze areas.
Challenges of Functional Testing
Inexperienced testers face steep learning curve for tools
Test maintenance outpaces testing
“Our testers are not experienced in building automated tests.”
“Application UI changes are always breaking our test scripts.”
The main point on this slide is that test maintenance becomes a significant cost for test teams as they grow their testing projects. Test maintenance is the death of automation projects.
The text in the bubbles illustrates sample comments/complaints that your customers may have heard from their teams.
Challenges of Functional Testing
Inexperienced testers face steep learning curve for tools
Test maintenance grows with project
Testing Tools Lack Flexibility
Power users hands are tied by tool limitations
“I need one tool for both my expert and novice testers.”
“Our script language limits our ability to overcome some automation obstacles.”
The main point on this slide is that in other test tools (like Mercury’s QuickTestPro) the product’s efforts to simplify testing make it difficult for power users to overcome inevitable challenges of automation.
The text in the bubbles illustrates sample comments/complaints that your customers may have heard from their teams.
Functional Testing with IBM Rational Functional Tester
Test automation for the novice and the professional
IBM Rational Functional Tester
Key Benefits
Minimize test maintenance with scripts resilient to application changes
Wizard enhanced automation to speed test creation for the new user
Powerful scripting language and IDE for the professional tester
Supports Team oriented parallel development
Effective Test Automation
Execute
Script Enhancing
Script Execution
Enhance
Record
BBryson: When it comes to script execution, you can run locally on your test development machine or leverage TestManager and run remotely on distributed machines, even Linux machines.
Recording Scripts
Separates test data from test script
Enables a single script to run multiple tests by using multiple data sets
Wizard driven process
No programming involved
Recording Scripts
Validating Dynamic Data
Dynamic Data/Content Matching
Use pattern matching technique to verify dynamic data and create robust tests
E.g. Instead of validating “Order ID 230”, validate “Order ID ###”
or Order ID 2##, etc.
This allows for a wide variety of acceptable responses as well as restrictions on acceptable responses when validating the application’s behavior
Order ID changes with each order placed
Test Fails when comparing static baseline to dynamic data
Test Passes when comparing pattern baseline to dynamic data
Dynamic Data Validation solves another very common problem with java and web applications.
In many applications, there will be dynamic data that changes with every run of the application. In the case we have here, you can see that every time an order is placed a new order number is generated. Any attempt to compare this order number to a static baseline will result in a test failure.
So how does one validate ever changing data in an application.
<<< CLICK >>>
The answer is to use Pattern Matching technology. Instead of trying to match a dynamic value to a static baseline, you match it to a pattern. In our case here, we match simply against 3 digits. If we get a three digit number, the test passes. This of course is a very simple pattern. Using industry standard Regular Expression notation (common in popular parsing languages such as PERL), you can create any imaginable pattern to validate any imaginable type of dynamic data!
Enhancing Scripts
Visual Studio .net
Code assist editors
Reduce Test Script Maintenance
Object Map
The object map is a collection of descriptions of test objects
It is automatically populated during test script recording
It contains properties for each test object
Recognition properties enable Rational XDE Tester to find the test object during playback
Changes to an object need only be made once – in the map
Central Object Map
Panel
The object map is a collection of descriptions of test objects that is automatically populated during test script recording – but also allows you to manually add objects later if you so desire.
For each object, the map stores properties and values that identify the object. It is the recognition properties that enable Rational XDE Tester to find the test object during execution of a test script.
In the event the application changes substantially and and changes to an object‘s identification are needed, the change can be made in the central object map and all test scripts that reference that object are automatically updated. Keeping script maintenance, when required, to a minimum.
So, now we know how objects are stored, and how centralized object maintenance can be achieved. The next question is how ScriptAssure can virtually eliminate the need for script maintenance?
Objective
Key Points
Recognition properties are used to find (recognize) the object.
Administrative properties are all the properties not used to find the object (mostly used to specify what can be done to the object).
Reduce Test Script Maintenance
Version 1.0
Version 2.0
No User
DeterminesMatch
Tester
Sees
Tool
Sees
Let’s take a closer look at the ScriptAssure technology. ScriptAssure is the technology that makes test scripts immune to object name changes between software builds. This is a critical feature to reduce script maintenance and increase script reliability.
Here we have two builds of the same application.
<<< CLICK >>>
The scripting tool sees the underlying attributes which allow the tool to uniquely identify the objects being testing. The problem is, when the attributes are changed build to build as the application gets constructed, the test scripts can no longer uniquely identify the object to be tested, thus breaks. In the past, the Tester would need to manually intervene using a GUI Mapper to identify to the script what it is was supposed to test.
<<< CLICK >>>
Unique to Rational, the new technology tool can identify objects even after the identifiers have changed. It uses many different attributes, weighs them proportionately, then uses a sensitivity setting to identify the object. This way, should one or a few properties – such as an object name – change between builds, it can still recognize the correct object.
<<< WAIT FOR CHECK MARK TO APPEAR >>>
The benefit is that even as object properties change between builds, the scripts will still playback. This decreases script failures by making scripts more resilient to change. Of course, all of this translates to less script maintenance and more time for the Tester.
Summarizing Key Features in IBM Functional Tester
Broad Environment Support
Web, Java. .net
Static data and properties verification
Dynamic data validation without coding
What can’t or shouldn’t be automated?
Flexible and Extensible Scripts
Challenges of Performance Testing
Complex Tools & Complex Task
Tool complexity and lack of experience intimidates many first time users
“We don’t have a lot of experience building good load tests.”
“Most load test tools are much more complicated than we require.”
BBryson: Emphasize this point to sell our ease of use benefit. Visual editor is the feature we’re getting at. The though bubbles are common symptoms that audience may have heard and resonate with.
Power users hands are tied by tool limitations
“We don’t have a good understanding of what exactly is occurring during a load test.”
“We have highly technical performance testers that need to customize their tests.”
BBryson: Emphasize this point to sell benefit of being able to get at all page details and add custom java code
Hidden Costs
Hardware intensive tools and inability to share information with team mates increases cost of testing
“Our developers have no way of performing load tests.”
“We don’t have enough hardware to run our tests.”
BBryson: Emphasize this point to sell benefit of low memory footprint of tool
Performance Testing with IBM Rational Performance Tester
Test automation for the novice and the professional
IBM Rational Performance Tester
Key Benefits
Mask complexity to get the job done
Advanced Data Access & Manipulation
Low resource requirements
Desktop IDE integration
Creating a Performance Test
Execute & Analyze
Schedule Workload
Build Scripts
Scheduling Considerations
Execute and Analyze Considerations
BBryson: Analysis is relatively strong in the tool.
No-Code Tests
Tests are represented in a tree view as a list of web pages and their elements, such as images.
No programming necessary to create a test
Underlying detail is accessible but hidden to simplify test creation and maintenance
Literal values used during recording are replaced with variables
At playback time, variables are populated with data from a datapool
Data for playback can be imported from external data sources
No user coding involved
User Name: JSmith
User Name: CBryson
User Name: TJones
Session No. 100
Session No. 101
Session No. 102
Datapooling is the process of making sure that each user submits unique data to the server. If I run a test for 1,000 users, I want to make sure that each user submits a different user name, e-mail, zip code etc… Why? If I don’t – the server may cache the response resulting in overly optimistic response times. The good news is that Performance Tester handles this automatically for me. It generates scripts that are already pre-programmed to feed off of a datapool of values. I just need to create that data and attach it to the test.
LoadSmart scheduling
scheduling
Historically, load testing has been very difficult. That’s one of the reasons more companies don’t do it. You’ve got very complex scenarios to model and multiple transactions and data exchanges to emulate. Rational Suite has taken the ‘difficult’ out of this task.
Rational Suite TestStudio provides a graphical point-and-click scheduler that lets you model your user population and activity quickly and easily.
Your schedule can include already-recorded functional scripts (as you might find in your test plan) and/or virtual user scripts used solely for load testing..
And because all user groups are percentages of the whole, changing your load test from 100 to 10,000 plus users is as easy as changing one number. Any data that is passed during these scripts can be specific (read from existing data tables) or parameterized, i.e., read from a random table of data that is created on the fly. There is no programming required here and that will help you get started much faster.
Coordinate timing and dependencies
Dynamically increase the load during the run
BBryson: A nice, but not unique, feature of the tool is the ability to dynamically add users during a test run. Other tools can do this, so don’t over sell…also, we cannot schedule additional users – it has to be done ad hoc during test execution…however, it does give prospects the ability to crank up the load during a test to break a server.
All activity coordinated and monitored
from master console
Key considerations for analysis
Find the slow pages
Find resource bottlenecks
BBryson: Resource monitoring is better in RPT v6.1 than it was with Robot. We now have access to all perfmon counters. Provides lots of data.
Find Slow Pages
Easily determine slow performing pages of your web site
Reports are available live during the run and for post run analysis
BBryson: This is the first report prospects will see when they use the tool. It shows the average page load response times for all users.
Find Resource Bottlenecks
Capture and analyze a long list of performance monitoring statistics
Determine your resource bottlenecks
BBryson: This is a screenshot of the resource monitoring ability. On the left is a list of counters to monitor. The graph shows which ones have been selected and are being measured.
Identify the Root Cause of Performance Problems
IBM Performance Optimization Toolkit for Rational Performance Tester
System
under
test
Analyze
Speed delivery
Developer
Tester
Speed changes by providing detailed data to developers
User
Load
1
2
b
c
[NOTE TO SPEAKER] The IBM Performance Optimization toolkit is a free developerWorks download for all customer with Rational Performance Tester 6.1
KEY POINTS FOR IBM PERFORMANCE OPTIMIZATION TOOLKIT:
Monitors web applications running on the application server during performance testing under load
Provides analysis of the collected performance trace data in Rational Performance Tester to identify where the root cause of the application bottleneck is located
Enables analysis and correlation of log files of the tiers of the tested J2EE system leveraging the Self -Managing Autonomic Capabilities from Tivoli
------------------------------------------------------
GENERAL PROBLEM STATEMENT FOR BOTH TOOLKITS:
Once an application goes to production, it becomes increasing difficult and expensive to isolate any performance or functionality problems that may come up. Up until now, the best the IT Operator could do was attempt to appropriately tune the system to "alleviate" the problem as best as they could, then point to a suspect transaction, and take a guess at what was happening at the point in time the Performance or Functional flaw surfaced and provide this information back to the development team. The development team would eventually pick up on the problem and take a shot at reproducing it in their test environment. It could take days or weeks to isolate the problem, especially those related to memory leaks or code bottlenecks that only become "problematic" under load, or after hours of continuous execution - both of which happen in production.
IBM SOLUTION (S)
IBM Problem Resolution Toolkit for Rational Application Developer links the developer directly to data that has been collected in a production environment, allowing the developer to quickly and efficiently identify the root cause of performance bottlenecks and provide a resolution.
and
IBM Performance Optimization Toolkit for Rational Performance Tester collects performance trace data for the applications running on the application server in the test lab. It provides analysis capabilities of the collected data and makes them available for the developer for efficient root cause analysis and resolution.
IBM Problem Resolution Toolkit helps reduce problems once they entered production and IBM Performance Optimization Toolkit helps the Performance Tester to detect and with the help of the Developer fix the problems before they get to the production.
These toolkits enable a faster, automated and more coherent process for application problem resolution. These toolkits link the developer directly to data recorded in either the production environment or the test lab.
KEY POINTS FOR IBM PERFORMANCE OPTIMIZATION TOOLKIT:
Monitors web applications running on the application server during performance testing under load
Provides analysis of the collected performance trace data in Rational Performance Tester
Enables analysis and correlation of log files of the tiers of the tested J2EE system leveraging the Self -Managing Autonomic Capabilities from Tivoli
Enables sharing of the recorded performance trace data with the developer using Rational Application Developer or Rational Software Architect
* Includes self-managing autonomic capabilities
HOW DOES IBM PERFORMANCE OPTIMIZATION TOOLKIT WORK:
Tivoli Monitoring for Transaction Performance (TMTP) Software monitors application performance and quality in the production environment, tracing the application execution and saving away application and system details when it identifies that a performance or quality issue exists. The IBM Performance Optimization Toolkit uses Data Collection Infrastructure (DCI) that is based on the TMTP technology in the test lab. This enables the Performance Tester to easily and quickly identify the root cause of performance or quality problems uncovered during the test phase. Through the use of IBM's self-managing Autonomic Capabilities, the toolkit can suggest probable causes and resolution. If the probable cause is linked to the application source code, the recorded data can then be shared with the developer for quick resolution
FLOW:
1: Need to pinpoint/recreate performance problems in test Lab
1a: Tester initiates multi-user test with RPT against system in test lab
1b: Results about slow performing pages and page elements captured and analyzed
1c: Root Cause identification of bottlenecks done to the method level with data collection infrastructure (from Tivoli)
2: Isolated defect submitted into ClearQuest for Developer to take immediate action
In the test lab, the tester is using Rational Performance Tester (RPT) to apply a user load on the target application environment. The user takes advantage of the real time reporting capabilities of RPT to view the response times being achieved. If the response times are acceptable, the user takes advantage of the Dynamic Load capabilities of RPT and dynamically increases the user load until they recognize that certain transactions (page hits) are now exceeding a target response time threshold. The Tester engages IBM Performance Optimization Toolkit ( which is powered by Tivoli Monitoring for Transaction Performance tools) to turn on application tracing, gathering all of the execution data associated with the slow transaction that was observed. Once the data is gathered, the tester reviews the data to identify the root cause. They compare the data they have obtained to data contained in the Symptoms Database for Autonomic Computing. This Symptoms Database is a cross reference to commonly known problems that are linked to specific types of data being observed. In some cases, a tuning recommendation will come out of the Symptoms Database, in other cases, the tester will have to seek additional help to drill into the root cause. Under this scenario, the tester will file a defect in ClearQuest, and link that defect to the application trace data they have recorded/stored. The developer will see the defect and review the trace data recorded by the Performance Tester in the test lab. After analyzing the test results the Developer the Developer starts the debugging/isolation activity just like they did with the data from the production environment. When the developer has drilled down to the line of code that is the root cause, they will resolve the problem and create a new build to push into the test lab.
WHAT IS UNIQUE TO IBM:
Consider these steps used in the two scenarios above:
1. Synthetic Transaction Generation in the production environment: - Keynote, Mercury, IBM and many more
2. Record and store detailed trace data associated with the slow transaction - Wily, Mercury, IBM and more – unlike Wily Tivoli is integrated in the production environment and remains active 24x7
3. Package that data up to be shipped to the Developer - Wiley, IBM
4. Import the data into the Developer IDE and perform Jump to Source Code Analysis - IBM ( Wiley has some jump to code capabilities, but not in the Developer’s IDE). The data recorded by IPOT is ready for the Development environment. That’s a unique IBM capability – both RPT and RAD/RSA use the same data.
5. Use the IDE to fix the problem - IBM (Borland can also do this)
6. Use ClearCase to perform another build for the Change Management tools in the production Space – IBM, CVS (open source)
Identify the Root Cause of Performance Problems
The IBM Performance Optimization Toolkit decomposes poor response times identified during a performance test…
into the amount of time spent within each application component.
Summarizing Key Features in Performance Tester
Built for Day 1 Productivity
Create code free tests
Advanced Data Access & Manipulation
Create code free schedules
Low resource requirements
Desktop IDE integration
BBryson: Wrap up here referring back to the challenges of Complex Tools, Lack of Tool Flexibility and High cost of performance testing.
Why Should I Test?
Establishes a link between the test and the motivator
Motivator can be RequisitePro Requirements, Rose Model, Excel Spreadsheet, or custom
Establishing link enables requirements coverage reporting
Test Input
…let’s examine these last two items in more detail, starting with the “why” – Why should you create and run a given test?
The “why” of the test defines what is driving the need for the test. There are many reasons you could be running the test – but generally the most common reason is because there is a system requirement that says “The system should do X”, therefore you should have a test “for X”.
Establishing this link accomplishes two key objectives for you. First, should the underlying requirement or motivator change – you’ll get a notification in TestManager that the test is now “suspect”. A “suspect” test is one who’s motivator has changed. The idea here is that if the requirement changes, you may need to re-evaluate the test to determine if it needs to change as well.
Establishing the reason for the test case also will enable you to do requirement coverage reporting. With the link established you’ll be able to see which of your requirements are passing and which have failed…and which have not been tested. Invaluable information when it comes to release time.
Suspect Test Cases
Suspect Test Cases show up with red slashes in the Test Inputs View
Suspect Test Cases show up in bold in the Test Plan View
AND
Within TestManager:
Suspect test cases display in both the Test Inputs View and the Test Case view.
Automatically Track Impact of Changing Requirements
Automatic Notification of Changed Requirements that Affect Test Cases
With TestManager and RequisitePro, that’s easy. Any change to a requirement automatically causes the test case to be marked as “suspicious”. That means that the test case need to be re-evaluated to see if the change in the requirement impacts the testing. And TestManager makes that very easy.
There are several ways that TestManager notifies the test team about a change. Here is just one way. This report marks in Red any test case whose associated requirement has been changed. And, by drilling down right from this report, you can find the details of the change to understand what impact this change will have on testing.
By TestManager and RequisitePro working together, no changes to requirements will go undetected. This ensures that your testing team is always working with the latest information – no matter how often or how quickly requirements changes.
Comprehensive Change/Defects Management
Script
Requirement
One of the greatest advantages of using Rational Suite TestStudio is the team unification power you gain. Let’s take the example of requirements. You may have formal requirements or a product spec in a database or a Word doc. You can look at this requirement from Rational TestManager, the test planning and test management tool. From here you can assign or create test scripts that will be used to measure functional compliance against that requirement.
But, requirements change. They are dynamic, right? No problem with Rational Suite TestStudio. It doesn’t matter where you update your requirement - work wherever you are most comfortable. Because the requirement is stored in a central repository, you can edit it from Word, RequisitePro, or TestManager - and it gets automatically updated everywhere.
More importantly, when logging a defect, you can note which requirement this test script was running against. So when the developer or analyst views the defect, they know immediately which business requirement is broken. Rational ClearQuest, the integrated defect management component, automatically pulls in that info when you generate the defect. The script, verification point, input/requirement, log results, etc. are ALL attached automatically to the defect note, ensuring that the developer can reproduce it quickly and easily.
Rational ClearQuest has dozens of built-in reports you can run with a single mouse click, so anyone on the team can find out the status of the project at that moment in time!
=======================
If you are using the built-in defect tracking tool, Rational ClearQuest, to handle your change requests, a simple right-click lets you submit a defect. The nice part here is that when you choose submit defect, any relevant test data that went into generating that fail message automatically gets attached to the defect – including the test script, results, machine configuration, test input or requirement that’s being failed, … everything. Again, extreme traceability is the key here.
Control Your Process
Track Test Thoroughness
Defect Tracking (ClearQuest)
One of the greatest advantages of using Rational Suite TestStudio is the team unification power you gain. Let’s take the example of requirements. You may have formal requirements or a product spec in a database or a Word doc.
You can look at this requirement from Rational TestManager, the test planning and test management tool. From here you can assign or create test scripts that will be used to measure functional compliance against that requirement.
But, requirements change. They are dynamic. No problem with Rational Suite TestStudio. It doesn’t matter where you update your requirement - work wherever you are most comfortable. Because the requirement is stored in a central repository, you can edit it from Word, RequisitePro, or TestManager - and it gets automatically updated everywhere.
Control Your Process
Track Test Thoroughness
Finally, to close the loop, you can generate reports on requirements coverage. Can’t release the product until it meets all its business requirements, right? Either that or you’ve got to start dropping features. Well, with Rational Suite TestStudio you know where you stand at any point in time.
Summarizing Key Features
Simulate Real World Workload
Create real world workload model to make sure the test result is meaningful
Effortless Analysis
test reports
Hopefully I’ve given you a realistic view of what you can expect as you take your first step towards automated test asset management. Let’s review some of the keys to delivering quality on time…
Crucial is the ability to support all types of testing…performance, functional and manual. By automating all phases of the testing life cycle, from planning through execution to data analysis, the tools will ensure your test group runs at maximum efficiency.
Configuration awareness ensures that TestManager only runs tests on the appropriate platform. Tracking this effort manually can consume many needless hours as testers maintain complex checklists of what has and has not been done.
Finally, the tools enable you to effortlessly track your testing progress. By taking care of all the data gathering and reporting for you, TestManager can instantly keep you up to date on test progress reporting, requirement changes and configuration testing … to name a few.
The IBM Rational Software Development Platform
A complete, open, modular, and proven solution for Software Quality Assurance
Customer
Extensions
The IBM Software Development Platform is based on an architectural foundation called the Eclipse universal framework. This framework enables unprecedented tool integration and artifact traceability throughout the development lifecycle.
What is Eclipse? It is both an open source software development project—with over 100 active vendor participants—and a rich foundation for providing shared services across an application tooling environment.
Part of the Eclipse Project¹ is the Eclipse Modeling Framework (EMF), which offers essential capabilities, notably code generation, meta-data management, and editor support.
Eclipse offers customers:
Role-based user interfaces that allows each member of the team to view information in the way that is optimized for their technical specialty.
It provides common models for integration across the lifecycle. This makes it easier for all vendors—including IBM—to integrate their products by mapping meta-data to a shared set of common models.
-----------------------------
“The Eclipse Project is an open source software development project dedicated to providing a robust, full-featured, commercial-quality, industry platform for the development of highly integrated tools. The Eclipse Tools Project provides a focal point for diverse tool builders to ensure the creation of best of breed tools for the Eclipse Platform. The mission of the Eclipse Technology Project is to provide new channels for open source developers, researchers, academics and educators to participate in the on-going evolution of Eclipse.”* Part of the Eclipse project is the Eclipse Modeling Framework (EMF), which offers essential capabilities, notably code generation, meta-data management, default serialization, and editor support. For more information on the eclipse platform, please visit eclipse.org.
*From eclipse.org FAQ on www.eclipse.org
What is the planned architectural direction of the IBM Rational solution?
We intend to invest in and leverage shared components based on the Eclipse open source platform. This architectural direction reflects our deep commitment to open standards– including UML2, Java, J2EE, and XML, and Web service standards. In concert with our customers and partners, we envision a vibrant tools and services ecosystem supported and extended by a rich community of developers, partners, and Independent Software Vendors
What’s Next?
Rapid Web Application Deployment Seminar (26th May, 2006)
Please fill in the feedback form and get IBM Rational evaluation CD kit !!
Cost of testing new Siebel installation
$0
$100,000
$200,000
$300,000
$400,000
$500,000

® ibm software group © 2004 ibm corporation ensuring application reliability and scalability...

Documents