automated testing with silktest: strategies that really work

23
Automated Testing with SilkTest: Strategies That Really Work Santa Clara Valley Software Quality Association September 14, 1999 Presented by John W. Green Automation Expertise

Upload: nell-moran

Post on 31-Dec-2015

30 views

Category:

Documents


2 download

DESCRIPTION

Automated Testing with SilkTest: Strategies That Really Work. Santa Clara Valley Software Quality Association September 14, 1999 Presented by John W. Green Automation Expertise. Introduction. 6 years of automation experience 15 years of software QA - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Automated Testing with SilkTest: Strategies That Really Work

Automated Testing with SilkTest:

Strategies That Really Work

Santa Clara Valley Software Quality AssociationSeptember 14, 1999

Presented by

John W. GreenAutomation Expertise

Page 2: Automated Testing with SilkTest: Strategies That Really Work

Introduction

• 6 years of automation experience• 15 years of software QA• 3.5 years at Segue as Trainer and Sr.

Consultant• Founded Automation Expertise in

1998.• This document and SilkTest examples

and FAQ are available at:

www.automationexpertise.com

You may need to go to

http://207.114.165.79

Follow the link for Resources

Page 3: Automated Testing with SilkTest: Strategies That Really Work

Successful Automation

• Skills

• Tool/Software Marriage

• Organizational Support

• Realistic Goals

• Training

• Good Test Plans

• Examples of Success

Page 4: Automated Testing with SilkTest: Strategies That Really Work

Skills

• Two types of skills are critical to be successful as an automation engineer.

• Testing

• Programming

• Sometimes people have one of the skills sometimes they have both. Good automated testers need to be subject matter experts. That means they understand exactly how the product they are testing should behave. They know how users use the product. They can prioritize between the thousands of possible testcase to find the subset that are likely to: find bugs, provide significant code coverage.

Page 5: Automated Testing with SilkTest: Strategies That Really Work

Programming Skills

• Good automated testers need to be good programmers. They need to be able to create intelligent, reusable, maintainable modules (classes/methods, functions and testcases) that are capable of behaving like a user.

Page 6: Automated Testing with SilkTest: Strategies That Really Work

More about Skills

• Sometimes testers need to become detectives to determine exactly how the product should behave so that they can create a testcase which verifies that the product behaves as expected. This is not quite as simple at a company which doesn’t really have time for specifications. Some QA people at web companies say they’re lucky to do an acceptance test on every third modification to the live web site.

• Most importantly, the tool’s learning curve should be consistent with the time frame in which the team want’s to use the tool. Setting realistic goals is critical.

Page 7: Automated Testing with SilkTest: Strategies That Really Work

Tool/Software Marriage

• Should be a good marriage.• Tool should work with your hardware• Tool should be compatible with your

development environment.• Tool should scale to allow “as close as

possible” simulation of user interaction with the software.

• You should at least believe that the vendor will continue to support your needs.

Page 8: Automated Testing with SilkTest: Strategies That Really Work

Organizational Support

Automated Testing is often brought to a company amidst some controversy. The issues usually are:

•Should we automate or not automate?•Will Automation cost us money?•Will Automation save us money?•Will Automation be too difficult?•Is Automation essential?•Did we choose the right tool?

With mixed organizational support for automation from development, QA and others, the project is likely to fail.

Page 9: Automated Testing with SilkTest: Strategies That Really Work

Support Issues

• An automation team can create some GUI testcases without support. They need a number of issues resolved to be successful. These include:– test data issues– custom object assistance– agreements on correct behavior– access to dedicated machines– unique databases– performance standards– GUI standards, etc.– configurations to be tested

• These are all issues which the team and development should work together to resolve.

Page 10: Automated Testing with SilkTest: Strategies That Really Work

Realistic Goals

• You can’t automate everything!!!• It will take time for the team to become

experts.• Suggestion: Choose 25 testcases to

automate. You can select a lot of tests from one product area or tests from a variety of areas.

• Define them in excruciating detail.• Get assistance, if needed, at

automating the 25 testcases you have selected.

• Use this as a pilot project to see if the more tests can/should be automated.

Page 11: Automated Testing with SilkTest: Strategies That Really Work

More on Goals

• The team will develop skills and the automation “guru” may appear during the pilot project.

• Estimates of the cost of automation range from 1-8 times the cost of manual testing. Maintenance costs should be considered and will be relative to the number of changes to the application in the future and to how well the tests were written in the beginning.

• The tools themselves are very expensive and must also be considered in the cost equation.

Page 12: Automated Testing with SilkTest: Strategies That Really Work

Selecting the Right Tool

• Narrow things down to the vendors who say they will support your environment.

• Choose 3-6 testcases and define them in detail. Select tests which exercise your application in several different ways. Ask each vendor to come in and automate these testcases while you watch.

Page 13: Automated Testing with SilkTest: Strategies That Really Work

Selecting Tools cont.

• Then after they’ve left, rerun the tests, then see how the test perform when you move the window, cause errors, etc.

• Next, compare the code each tool generated to see which one is more: readable, maintainable and flexible. See if your team can learn to use the language of the tool.

Page 14: Automated Testing with SilkTest: Strategies That Really Work

Things Automation Tools Can Do

• Repeat boring tasks without complaining.

• Repeat boring tasks without quitting.• Works all night.• Control several machines at once.• Get data out of a database easily.• Insert data into a database fairly easily.

Page 15: Automated Testing with SilkTest: Strategies That Really Work

More Things Automation Can Do

• Reboot a remote NT machine and login as someone else.

• Find all the links on a page that are outside the current domain and check them.

• Create a test on one browser and play it back on another browser.

Page 16: Automated Testing with SilkTest: Strategies That Really Work

Advanced Automation Tools

• Understand and recognize objects.• Allow intelligent recovery from errors.• Have object-oriented languages built-

in.• Allow DLL calls.• Allow calls to application code.• Allow control of multiple machines.• Are cross-platform.• Are cross-browser.

Page 17: Automated Testing with SilkTest: Strategies That Really Work

Training

• Develop the team’s skills.• Provide an in-house trainer, if possible.• Skilled contractors can perform this.

function if their focus is on providing mentoring, training and support to full-time people.

• All tool vendors offer classes.• It takes time to become expert.• Productivity increases as skill

increases.

Page 18: Automated Testing with SilkTest: Strategies That Really Work

Good Test Plans

• Test planning, I’ve left for next to last, but it is probably the most critical of the success factors for automation.

Define what’s important to test.Decide how you’re going to execute the required test. For some tests, its obvious. If you need to stress test your web server with 500 users, you’ll need a tool to do that. However, for many of tests in your plan, you will have the choice of automation or manual execution.

Page 19: Automated Testing with SilkTest: Strategies That Really Work

Which Tests To Automate

• Tests which are good candidates for automation include: – Time-consuming manual tasks– Functional tests of stable modules– Tests which will need to be repeated often. – Tests which are inexpensive to automate.– Stress/Load Tests– Multi-user scenarios

• Either way, define what tests are important first, execute them and report the results. This should be done effectively no matter what execution method you select.

• Test planning helps you to focus on what’s really critical in your application.

Page 20: Automated Testing with SilkTest: Strategies That Really Work

Success Example

• Using SilkTest, we created a series of multi-machine scenario tests which could randomly select treeview items on each of six machines which were “clustered” together. Each treeview item launched an specific module. The “random” selections generated on each machine were repeatable by using a “random seed”. Also, the random items selected could be recorded and played back, at fairly close to the original sequence. The tests could be run for an unlimited amount of time.

Page 21: Automated Testing with SilkTest: Strategies That Really Work

Example 1 Results

• This process resulted in the identification of more than 10 significant previously unidentified new major defects. Each defect we identified and which was fixed, lead us to another defect at another layer, using the same tests.

Page 22: Automated Testing with SilkTest: Strategies That Really Work

Success Example 2

• A team of manual testers created test plans using Excel, and then passed some of those plans to the automation team to be automated. Using the some advanced features of SilkTest, we created an interpreter that could read and execute the tests defined in the spreadsheets. The manual testers were unfamiliar with SilkTest. The language which the manual testers used to define tests was mutually agreed to, and involved the use of numerous functions defined to perform many of the test activities.

• The focus of the test group shifted from execution of their tests to creation of new and different test cases.

Page 23: Automated Testing with SilkTest: Strategies That Really Work

Conclusions

• Test Automation is development• Tests should be carefully planned• Tests should be carefully selected• Expectations should be very reasonable

(low) for the first project.• Be concerned about hiring, developing

and retaining the skill sets required for automation.

• Let someone on the team learn the tool really well in order to take full advantage of its features.

• Build modular, maintainable, well-documented flexible tests.