2006 squad annual conference february 16-17, 2006 ppa ... · copyright 2006 by cordell vail joe...

Presentation by

Joe Towns & Cordell Vail

2006 SQuAD Annual Conference

February 16-17, 2006 PPA Event CenterDenver, Colorado

1

2006 SQuAD Annual Conference

February 16-17, 2006 PPA Event CenterDenver, Colorado

1

Presentation byCordell Vail & Joe Towns

Testing Tools that Will Increase Your ROI

Copyright 2006 by Cordell Vail


Some of the joke slides you saw in the actual presentation as transition slides, have been removed from this web version of the presentation due to copyright laws. They can only be displayed in a class room setting and not distributed to the public with out permission from the cartoonist. Therefore, the ones I do not have permission to distribute have been removed.

COPYRIGHT NOTICE:


The reason most people never find a 4 leaf clover is thatthey never look for one.Life is a lot like that.

Cordell Vail

Making Things Happen

2


OK! Now I am going to just sit here on thisstool and hang onand I want the cow to jump up and down

Is this your view of improving ROI?

Do you just expect it to happen?

c

INTRODUCTION

3


Was this YOU, when they asked you to find a load testing tool?

Purpose of this Presentation

j4


Help you learn how to find aninexpensive testing tool that will give you predictable results

This presentation is NOT intended to teach you how to do the testing

Purpose of this Presentation

5


• Know Where To Start• Uniformity• Definition Of Terms

Stress TestingLoad TestingVolume TestingPerformance TestingBenchmark TestingBaseline Testing

6

The “Big Picture” (Overview)


• Know the testing objective• Know the testing environment• Only buy what you need• Know how you are going to test• Trusting the test results• Possible solutions• Can we trust market share• Testing tool comparisons

7

The “Big Picture" (Overview)


In testing we would all do well to follow Stephen Covey’s advice:

“Begin with the end in mind”

What is it you need to test?

Know Where to Start

8


Uniformity

c9


Defining terms eliminates confusion

• Stress • Load • Volume • Performance • Benchmark• Baseline

Uniformity

10


It is not so important how you define testing terms as it is that everyone in your organization use the same definitions

Uniformity

11


Stress Testing

Tests the server

Peak volume over a short span of time

12


Load Testing

Tests the database

Largest load the database can handle at one time

13


Volume Testing

Tests the server & the database

Heavy volumes of data over time

Combination of Stress Testing and Load Testing over time

14


Performance Testing

Tests user response time

With web applications this is normally the main consideration

15


Benchmark Testing

Compares your testing standards to the same testing standards in other similar organizations in the industry

16


Baseline Testing

Setting testing standards to be used as a starting point for comparison later within your own organization

17

See the 2006 search engine test tool update on the CD


Will the tool need to test:

Bandwidth MultithreadingConcurrent users Disk capacityMultiple platforms FaultsMultiple browsers MemoryUsers per server User response

Identify the Target of the Test

18


• Number of client workstations• Connectivity to servers• Database availability• Production copy of application• Application and web servers• Bandwidth and LAN• Test tools to monitor results

Know the Testing Environment

19


WSIPC Testing Environment

NOTE: The Client Server and the 2 Web Servers areHP Proliant BL 20 PG2 Dual 3GH Xeon with 2GB Ram

20


To find an economical tool you must know what your testing needs are

Each tool is very different

You are not going to pay $200 and get a tool that will do what an $85,000 tool will do!

Only Buy What You Need

21


Bandwidth RestrictionServer PerformanceMultithreading

DevelopersDevelopers System EngineersSystem

EngineersTest

EngineersTest

EngineersSource Code AnalysisCyclomatic ComplexityMemory Leaks

Error HandlingFault InjectionUser Response Time

Consider Team Composition

The members of your testing team will be able to help determine what tests need to be done withwhich testing tool

22


targets causes Stress FailureStress Test Stress Attack Point(s)

Buffer, cache Resource leak

Data corruption

Unhandled exception

Race condition\deadlock

Async pattern

p\invokes

Load Limit Breakpoint Torture Duration Synchronization Fault Injection

This example taken froma presentation at the Seattle SASQAG meeting 21 Apr 2005 by Keith Stobie. Diagram created by Reji John. Both work at Microsoft.(Used with permission)

Know How You Test

23


With a new testing tool:

Are results accurate?

Are the results Valid?

What are you comparing to?

Trusting The Test Results

24


How big is this equipment?

25


c


Now how big is it?

26



Possible Solutions

Test the tool on an existing application and web server where you already know what the test results should be

27

Are there other solutions?


Test with more than one testing tool and compare the results

Is that feasible with the cost of the tools on the market today?

28

Possible Solutions

c


93% of the market shareis controlledby 7 vendorsall chargingvery high prices

Can We Trust Market Share

29c


Source:http://www.mercury.com/us/pdf/company/newport_load2000.pdf

30

Can We Trust Market Share

http://www.mercury.com/us/pdf/company/newport_load2000.pdf


We decided to look at that “OTHER” 7%mostly because of price

Affordable Testing Tools

31


http://www.vcaa.com/testengineer/links.htm

http://www.testingfaqs.org/t-load.html

http://hammerhead.sourceforge.net/

http://opensourcetesting.org/performance.php

http://www.grove.co.uk/Tool_Information/Choosing_Tools.html

http://www.softwaregatest.com/gatweb1.html#LOAD

http://www.sqa-test.com/toolpage.html

http://www.webservices.org/index.php/ws/content/view/full/102


http://sourceforge.net/projects/dieseltest/

NOTE: See our comparison list on the session CD handout

32

Affordable Testing Tools

http://www.vcaa.com/testengineer/links.htm

http://www.testingfaqs.org/t-load.html

http://hammerhead.sourceforge.net/

http://www.grove.co.uk/Tool_Information/Choosing_Tools.html

http://www.softwaregatest.com/gatweb1.html#LOAD

http://www.webservices.org/index.php/ws/content/view/full/102



http://sourceforge.net/projects/dieseltest/


Testing Tool Vendor Links

Here are links to those 7 larger vendors that are holding 93% of the market share

Check the Prices out for yourself!They are all over $50,000 for 1,000 virtual users

www-306.ibm.com/software/awdtools/tester/performance/index.htmlwww.segue.com/products/load-stress-performance-testing/index.aspwww.mercury.com/us/products/performance-center/loadrunner/www.radview.com/products/WebLOAD.aspwww.quotium.com/qpro_overview_load_testing.htmlwww.empirix.com/default.asp?action=article&ID=418www.compuware.com/products/qacenter/performance.htm

33

http://www-306.ibm.com/software/awdtools/tester/performance/index.html

http://www.segue.com/products/load-stress-performance-testing/index.asp

http://www.mercury.com/us/products/performance-center/loadrunner/

http://www.radview.com/products/WebLOAD.asp

http://www.quotium.com/qpro_overview_load_testing.html

http://www.empirix.com/default.asp?action=article&ID=418

http://www.compuware.com/products/qacenter/performance.htm


TestMaker – PushToTest: FREE http://www.pushtotest.com/Downloads/

WAST – Microsoft: FREEhttp://www.microsoft.com/downloads/details.aspx?FamilyID=E2C0585A-062A-439E-A67D-75A89AA36495&displaylang=en

LoadTester – AppPerfect: FREEhttp://www.appperfect.com/products/devsuite/lt.html

Our Testing Tool Findings

34

http://www.pushtotest.com/Downloads/

http://www.microsoft.com/downloads/details.aspx?FamilyID=E2C0585A-062A-439E-A67D-75A89AA36495&displaylang=en

http://www.microsoft.com/downloads/details.aspx?FamilyID=E2C0585A-062A-439E-A67D-75A89AA36495&displaylang=en

http://www.appperfect.com/products/devsuite/lt.html


Site Tester 1 – Pilot: $29http://www.pilotltd.com/eng/index.html

Portent Supreme – $279Loadtesting.com:www.loadtesting.com

WAPT - Logasoft: $299http://www.loadtestingtool.com

35


http://www.pilotltd.com/eng/index.html

http://www.loadtesting.com/

http://www.loadtestingtool.com/


Webserver Stress $625Tool 7 – Paessler:www.paessler.com

HOLODECK - SISE: $1,500http://www.sisecure.com/holodeck/learn.shtml

NOTE: Holodeck is a fault injection tool-not a normal virtual user testing tool

36


http://www.paessler.com/

http://www.sisecure.com/holodeck/learn.shtml


Other Options

Testing Onshore Outsourcinghttp://www.veritest.com/services/load_stress.asp

Online Hosted Serviceshttp://servers.aplus.net/loadbalance.htmlhttp://www.webpartner.com/products/st_main.htmlhttp://www.keynote.com/http://www.webmetrics.com/loadtesting.html

NOTE: Hosted or online testing may be available free from your ISP

37j

http://www.veritest.com/services/load_stress.asp

http://servers.aplus.net/loadbalance.html

http://www.webpartner.com/products/st_main.html

http://www.keynote.com/

http://www.webmetrics.com/loadtesting.html


QUESTIONS

38


HOW THE TOOLS WORK

The Tool Demonstration HandoutOnly Shows The WAPT Tool

Seeing the Results

39


The tool we selected for our testing was WAPT

40


To use WAPT first you need to createa profile, which becomes the individuallyrecorded test scripts

41


Name the PROFILE (this will by your test script)

42


A profile named TEST 1 is now createdthe actual name in the example below will be605-303-01 - Monroe JH - log in - Allan Shell - SAVE Attendance af

43


You will now have to record the test script

44


When you record the profile you can use thename you want to use

45


It will then ask you to enter the URL for the web page to be recorded from

46


It then begins to record all of the Web page mouse clicks as you navigate through the application

47


Now you have arecorded test scriptthat you can play back

You place the loginscript in the “INITIAL”folder and the logoffscript in the “FINAL”folder so they only runone time in the sequence

48


Each test script (or as they call them - profiles) has set properties that control it and can be changed

49


Now, after you have recorded a whole series of profiles,you can run them in different combinations in a scenario

Each scenario is just a shell that holds profiles

You can run one profile or many in each scenario

50


You can then add the profiles that you have created that you want to have run in this particular scenario

55


The test scripts for each profile you put in the scenario is displayed

57


Each profile has set properties that can be changed that control how the profiles are to be run

58


With the properties in the scenario you can control the test run

59


You can also control where the logs will be sent and set properties for them

60


As the test runs you can watch the results being built

61


The test results are then displayedWAPT has a wonderful reporting system

62


There are numerous test results that you can then display, print or email to management

63


For More Information

[email protected]

64

http://www.vcaa.com/


Joe Towns, Applications Development Manager/ Project Manager. Joe grew up in the Seattle area, attained his degree in Computer Science and has worked in the field of software development for over 18 years. Joe has worked in various phases of software development, with focus on quality assurance, quality control and configuration management. Joe is currently the Applications Development Manager and Project Manager for the Financial Management development team at Washington School Information Processing Cooperative (WSIPC), where he is pursuing PMI certification. Joe brings a development perspective to the presentation.

The AuthorsCordell Vail, cste – Quality Assurance Analyst. Cordell grew up in Utah and graduated from the Brigham Young University and has completed two years of graduate school work at the University ofUtah in Interpersonal Communications. Cordell brings to the presentation a test engineer’s perspective. He is a Certified Software Test Engineer and Certified School Business Specialist with 9 years experience in manual and automated testing. Cordell has made several presentations on “Improving Testing Processes”at both local and national conferences.

65


NOTE: The information contained in this presentation handout and on the handout CD is for use only by the participants who attend our seminar at the Software Quality Association of Denver Conference held 16th

and 17th of February, 2006 in Denver, Colorado.

Distribution of this information to anyone other than those attending the seminar is not authorized by the authors. It is for educational purposes of the seminar attendees only. The jokes in the slide presentation are not printed here in the conference handout in compliance with current copyright laws.

Credits

66


Care enough to be the very best!

THIS IS NOT THE END IT IS JUST THE BEGINNING

2006 squad annual conference february 16-17, 2006 ppa ... · copyright 2006 by cordell vail joe...

Documents