testing tools landscape

© 2011 Forrester Research, Inc. Reproduction Prohibited1

© 2011 Forrester Research, Inc. Reproduction Prohibited2 © 2009 Forrester Research, Inc. Reproduction Prohibited

Testing Tools - How Teams Make It Work

Margo Visitacion, Vice President

February 17, 2011

© 2011 Forrester Research, Inc. Reproduction Prohibited3

Don’t lose before you get into the game……

© 2011 Forrester Research, Inc. Reproduction Prohibited4

Don’t lose before you get into the game

Why team focused testing can make the difference

Planning performance testing today can help budget planning tomorrow

Developing your test game plan

Recommendations

Agenda

© 2011 Forrester Research, Inc. Reproduction Prohibited5

If you don’t plan your quality and build the right team, you lose before you get into the game…….

© 2011 Forrester Research, Inc. Reproduction Prohibited6

More business is transacted online….

© 2011 Forrester Research, Inc. Reproduction Prohibited7

But problems increase your risk and costs …

Performance:– It only takes a moment to lose customers

– 74% of surveyed customers shop online….

– …. Yet 54% abandon shopping carts, a continuing plague for online vendors vendors

Security– Online banking: the future is now

– 90.5 million Americans now bank online…..

– ….Yet less than half of customers feel secure about their institutions’ security

Scalability– 70% of auto policies renew in January

– The global Internet population is now 1.6B and expects to grow another 42% by 2014

Do you have the right team in place?

© 2011 Forrester Research, Inc. Reproduction Prohibited8

In 2010 more money was directed to toward growth and expansion – don’t let poor performance shift dollars away from innovation

Poor quality diverts focus

© 2011 Forrester Research, Inc. Reproduction Prohibited9

Don’t lose before you get into the game

Why team focused testing can make the difference

Planning performance testing today can help budget planning tomorrow

Developing your test game plan

Recommendations

Agenda

© 2011 Forrester Research, Inc. Reproduction Prohibited10

Siloed approach to testing won’t let you answer these questions

Functionality• Are we meeting customer’s expectations• Does the application work the way we want it?

Performance• Are our applications accessible?• Can our clients complete orders in a timely fashion?

Security• Is client data protected?• Are transactions secure?

Scalability• Does the application scale up to meet peak performance

requirements• How stable are services when performance peaks?

© 2011 Forrester Research, Inc. Reproduction Prohibited11

Companies are planning to spend more on securing their customer’s loyalty

© 2011 Forrester Research, Inc. Reproduction Prohibited12

How will you change your behaviors to improve your approach to testing??

How will you “Change your ways”?– Challenges in testing are growing

– Performance, Data privacy, compliance

– Testing as usual won’t work today

– Early testing is imperative

– Teams look different because testing approaches have to change

IT budgets look better, but must bring value– Companies want greater flexibility for their $$$

– Automation MUST bring value for the money invested

Testing is ripe for SaaS– 21% are piloting SaaS for IT lifecycle components

– 26% are planning to adopt within the next year

© 2011 Forrester Research, Inc. Reproduction Prohibited13

Taking a team approach

BusinessStakeholders

•Setting requirements•Validates design•Approves change

Business analysts

•Gather requirements•Refines requirements•Reviews change•Supports testing

Developers

•Leverages requirements for design and build

•Unit tests•Some integration tests•Writes test harnesses•Supports testing

Testers

•Validates requirements•Helps define quality criteria•Designs test plans, scripts•Executes tests•Regression test•Oversee UAT

© 2011 Forrester Research, Inc. Reproduction Prohibited14

Testing can begin from day one to prevent defects

Business determinesobjectives, sets

requirements

Design and developmentbased upon requirements

Set test planning &Automation strategy

Business stakeholders, BAs & test team set quality and performance requirements for functional and non functional

requirements

Development &testing continues

Continuous build and integrationContinuous testingmanual and automatedStatic and automated performance testing starts

End to end testingUAT

DeploymentPost

Mortem

© 2011 Forrester Research, Inc. Reproduction Prohibited15

Use automation to keep up with change

Capital investments are scrutinized

Infrastructure and environments are challenged

Ramp up time takes too long

Usability is a requirement

Quick time to value is a necessity

Performance testing is cyclical but non negotiable

Testing is an enabler

Environmentsans

constraints

© 2011 Forrester Research, Inc. Reproduction Prohibited16

Agenda

Don’t lose before you get into the game

Why team focused testing can make the difference

Planning performance testing today can help budget planning tomorrow

Developing your test game plan

Recommendations

© 2011 Forrester Research, Inc. Reproduction Prohibited17

Costs: A drill down

Cost of testing• Organizations - Labor costs• Skills - recruiting costs• Tools - capital costs• Environments – capital costs• Support – opex costs

Cost of NOT testing• Customer reported defects• Business liability• Increased maintenance costs• Increased support costs

Poor software quality costs over $60B per year

Finding, repairing defects = approximately 35% of project budget

© 2011 Forrester Research, Inc. Reproduction Prohibited18

Why? Investing in quality can reduce risk, cost and raise value

0

20

40

60

80

100

120

Requirements Code Operations

Cost to Repair

© 2011 Forrester Research, Inc. Reproduction Prohibited19

Plan Today For Tomorrow’s Budget

•Performance data•Defects•Customer SatisfactionGather

historical information

•Leverage historical data for peak planning•Understand your tools usage and

requirementsPlan for today and tomorrow

•Consider on demand to cover what you need when you need it

•Look for flexible options – try to avoid getting trapped into expensive, no win situations

Allocate Funds

© 2011 Forrester Research, Inc. Reproduction Prohibited20

Don’t lose before you get into the game

Why team focused testing can make the difference

Planning performance testing today can help budget planning tomorrow

Developing your test game plan

Recommendations

Agenda

© 2011 Forrester Research, Inc. Reproduction Prohibited21

To win the game, the business needs to be involved

User stories

“Situational realities”

Requirem

ents

Risks/opportunities

© 2011 Forrester Research, Inc. Reproduction Prohibited22

Bringing the team together needs a repeatable, measurable process

Set expectationsCollaborate

ReviewDiscuss

PlanTest

Review

ExecuteReview Refine

Measure

© 2011 Forrester Research, Inc. Reproduction Prohibited23

Don’t lose before you get into the game

Why team focused testing can make the difference

Planning performance testing today can help budget planning tomorrow

Developing your test game plan

Recommendations

Agenda

© 2011 Forrester Research, Inc. Reproduction Prohibited24

Recommendations

Don’t leave performance testing to the end. – Problems will grow as business options expand

Anticipate the unexpected– Life is what happens while you’re busy making other plans

Leverage automation and timing– Flexible tools to performance-test, “just in time”

© 2009 Forrester Research, Inc. Reproduction Prohibited

Thank you

Margo Visitacion+1 609.314.6745mvisitacion@forrester.comwww.forrester.com

Testing Tools Landscape- Load Testing

Matt Poepsel, VP of Performance Strategies, Compuware Gomez

Customer Expectations Are Rising For Web & Mobile Applications

47% of consumers expect a Web page to load in 2 seconds or less

58% of mobile phone users expect websites to load almost as quickly or faster on their mobile phone than their PC

MajorISP

Local ISP

Mobile Carrier

Internet

Content DeliveryNetworks

3rd Party/Cloud Services

Browsers and devices UsersUsers

Storage

Web Servers

App Servers

DB Servers

Mainframe

Load Balancers

Mobile Components

Network

The Web Application Delivery Chain

The Challenge Of Delivering Quality Web Experiences

• Inconsistent geo performance• Bad performance under load• Blocking content delivery• Incorrect geo-targeted content

• Network peering problems

• Outages

• Configuration issues• Oversubscribed POP• Poor routing optimization• Low cache hit rate

• Network peering problems

• Bandwidth throttling

• Inconsistent connectivity• Configuration

errors• Application

design issues• Code defects• Insufficient

infrastructure

• Network resource shortage

• Faulty content transcoding

• SMS routing / latency issues

• Poorly performing JavaScript

• Inconsistent CSS rendering

• Browser/device incompatibility

• Page size too big

• Conflicting HTML tag support

• Too many objects

• Content not optimized for device

• Low cache hit rate

Systems management

tools: “OK”…user is NOT happy

“Outside-in” customer point of view

Test & monitor your site the way your customers use it:• What they do (key pages and transactions)• Where they do it (geographic locations)• How they do it (browsers and mobile devices)• When they do it (normal and peak usage)

Determine the impact on their behavior and your business

Applications are no longer what you build inside your firewall

Number of hosts accessed directly by the browser, per user transaction, averaged across 3,000 companies 10.5

Who Owns End-User Quality?

Line of Business Managers

Product Managers

Marketers

Interface Designers

Application Developers

Architects

Quality Assurance Teams

Infrastructure/Operations Teams

Help Desk StaffAvoid at all costs:

Individual Success, Collective Failure

Quality is a Collective Function.Everybody owns a piece of the puzzle.

How can we work together?

Create Shared Goals

Identify Shared Metrics

Use a Communications Plan

Solve Problems as a Team

Celebrate as a Team

Situational Assessment and Improvement

1. Provide this 5-minute assessment to each of the End-User Quality contributors for your application:

2. The results provide a baseline. Share the results across the team.

3. Identify areas of improvement and execute a plan to get better.

4. Re-administer the assessment after 90 days.

Shared Goals

Shared Metrics

Communications

Solving problems as a team

Celebrating as a team

On a scale of 1 (poor) to 10 (excellent), how well are we are doing in these areas:

Questions

Reduced downtime 45%

Reduced seven-step transaction time by 50%

Increased conversions 10%

Validated decision to consolidate three data centers

Achieved under 3 second response time and 99%+ availability

Reduced homepage load time from 11.3 seconds to 3.4 seconds

Improved page load times 23%

Saved 50%+ in staff and fees

Gomez Customers Enjoy Measurable Benefits

For more information visit Gomez.com or contact us at +1 781.778.2700