reality shock j aracil

Madrid, 26th to 28th May 2014

Reality Shock:

Performance Testing

Jose Aracil - Operations Manager

jose.aracil@globetesting.com

@pep0 - @globetesting

Madrid, 26th to 28th May 2014

A radically different kind of user

System-centric User-centric

Users born after 1980

Mainframe DevicesWebClient/Server

Time

Madrid, 26th to 28th May 2014

2014

A tipping point for app. delivery

“New application

projects surge in 2014

and beyond.”

“Social, mobile, analytics,

cloud have reached the

tipping point in 2014.”

201536 per app

2020120 per app

2010 4 per app

Number of releases

per app per year

• Apps everywhere

• Accessed through multiple devices

• 30x increase in application releases

• Closed feedback loop with customers

• Focus on time-to-value and ROI

What we expect:

Madrid, 26th to 28th May 2014

Changes are coming faster

Cloud and composite applications are increasingly complex

Customers demand mobile access services

The pace of change is only increasing

Project and delivery teams are both agile and global

Businesses need applications to connect with:

• Customers

• Partners

• Suppliers…

Madrid, 26th to 28th May 2014

Dev/test lab

Developer

And the need for speed changes everything

Tester

Agile Development

“The new QA is extensively

automated and executed

by a continuous

integration process”--

Forrester Research

Provision

environments at

“agile speed”

Shift left, test

continuously

Agile

Development,

weekly drops

Madrid, 26th to 28th May 2014

Continuous Testing for

faster app delivery“Shift-Left”

With Confidence

Now Continuous Testing is more critical than ever

• Improve development

efficiency

• Reduce the test burden

• Accelerate application

delivery

Design Develop

Provision

Env.

Deploy

Test App.

Run test

Test results

Madrid, 26th to 28th May 2014

Madrid, 26th to 28th May 2014

Application usage has changed

Performance testing must change

Application development has changed

Madrid, 26th to 28th May 2014

You must

leave the black

box

Madrid, 26th to 28th May 2014

Sorry… I changed my mind!

Madrid, 26th to 28th May 2014

Sorry… I changed my mind!updated

Madrid, 26th to 28th May 2014

Planning Test Strategy

Reporting Final report

Preparation

Tool setupTest Env. + Data Test case design

Monitor config. Scripting

Smoke Test

Execution

Baseline

Soak Test Load Testing Other tests

Stress

Performance Testing

An

ális

is +

Cha

nge

s+

Rep

ort

Madrid, 26th to 28th May 2014

Planning

Preparation

Execution

Reporting

Test Strategy

Tool setupTest Env. + Data Test case design

Monitor config. Scripting

Smoke Test

Baseline

Soak Test Load Testing Other tests

Stress

Final report

Performance Testing

An

ális

is +

Cha

nge

s+

Rep

ort

But what get’s in the way?

Dependencies

Madrid, 26th to 28th May 2014

Good for one project

Madrid, 26th to 28th May 2014

Good for one project

Madrid, 26th to 28th May 2014

New set of problems…

Multiple

projects

Continuous

Integration

Auditing

Multiple people

Deployment of

LoadGenerators

Dependencies

Asset sharing

Reporting

SecurityCollaboration

Best

practices

Network

conditions

Cloud

Low application

maturity

Agile (good and bad)

Madrid, 26th to 28th May 2014

Project based testing challenges

Project AHP LoadRunner 8.1

1500 Vusers

Project BHP LoadRunner 11.0

1500 Vusers

Project COther load test SW

2500 Vusers

Project DNo load test budget

“Hope for the best”

Unrealistic “code freeze”

No sharing of best practices & skills – each project is unique and isolated

Duplicated resources (HW and SW) in separate projects

Some test cases not fully working (missing components)

End-to-end behaviour not fully tester

Madrid, 26th to 28th May 2014

End to end

Search PayBuy Ship Track

EXAMPLE:

The Online Shopping

Process

Product PaymentCustomer Profiles Shipping

Track packagePublic CloudTraditional Managed CloudComposite

APIs enable integration

Each have SLAs and performance

dependencies

Components anywhere– in-house or cloud

Madrid, 26th to 28th May 2014

Virtualisation Four-legged table

User

Network

Service

Data

User

• Virtualise user conditions to simulate real world conditions

Network

• Virtualise network conditions to simulate production architecture and real user conditions

Service

• Virtualise services and service interactions to simulate E2E business processes

Data

• Virtualise production data and transaction responses to simulate application request and responses

Madrid, 26th to 28th May 2014

We are now able to truly apply a

agile approached to load testing

given the reduction in scripting and

build time.

Our scripting time is reduced by

more than 75% and resource

productivity across projects has

improved by more that 50%.

Source: TechValidate. TVID: B40-ED8-557

User

Madrid, 26th to 28th May 2014

Enable Agile performance DeliveryUser

Test

Build Load Test scripts during Dev

Run Unit Tests in Load Test Scenarios

Add Performance Tests as build steps in continuous integration tools

Madrid, 26th to 28th May 2014

Mobile is different

Analyze Mobile Performance on Network

Backend

monitoring

Mobile

performance

automation

Mobile Network

characteristics

Network

Identify bottlenecks before end users

Test in both good and BAD network connections

Understand impact of mobile on entire system

Madrid, 26th to 28th May 2014

Existing Infrastructure

Service Virtualization Concept

Mainframe

JDBC

SOAP

RFC/IDO

C

MQ/CIC

S

Third Party

Application Under Test

Mobile

App

Web

browser

Composite

Application

API

SAP System

Existing database

Web service and

Legacy application

REST

Pay-per-transaction

Under

construction

Service

Madrid, 26th to 28th May 2014

Existing Infrastructure

Service Virtualization Concept

Virtual services are created to

stand in for the real service when

developers or testers need

access

Mainframe

JDBC

SOAP

RFC/IDO

C

MQ/CIC

S

Third Party

Application Under Test

Mobile

App

Web

browser

Composite

Application

API

SAP System

Existing database

Web service and

Legacy application

REST

Pay-per-transaction

Under

construction

Service

Virtualization

SOAP

REST

JDBC

MQ

RFC

CICS

Simulatio

n

Data

Perf.

Config

Service

Madrid, 26th to 28th May 2014

Service Virtualization is the answer

Functional test

Performance Test

Regression test

Dependencies

Madrid, 26th to 28th May 2014

Service Virtualization is the answer

Functional test

Performance Test

Regression test

DependenciesVirtualization

Allow frequent access to app components, APIs and services

Remove constraints with virtual version

No access to a component? Use virtual component

Enable performance test teams to set up simulated services

Madrid, 26th to 28th May 2014

Bring production data into performance

testing as baseline

• Collect, analyze and compare

performance data from production

• Updated log file analysis of production

data for IIS

• Integrate with Google Analytics

DataPerformance application lifecycle

Value:

Performance testing closer to reality with accurate,

realistic, reliable test: reduce risk and improve test results

Madrid, 26th to 28th May 2014

Data

Virtualise production data and

transaction responses to simulate

application request and responses

Data

Madrid, 26th to 28th May 2014

3 Keys to better

performance

• Test early

• Test often

• Monitor and leverage

data from monitoring

as baseline for test

The NEW Reality

User Network Service Data

Madrid, 26th to 28th May 2014

Q&A

Madrid, 26th to 28th May 2014

IF YOU LIKE IT

PLEASE RATE MEVisit our blog to get all the slides

WWW.GLOBETESTING.COM

jose.aracil@globetesting.com