Test Automation as a Service

Jonathon Wright (jlwright@iee.org) www.linkedin.com/in/automation

STARWest (Thursday @ 1:30 PM, 4th October 2012)

© TestingAsAService.com

© BJSS Limited 2011 Commercial in Confidence 2

My starting point

Twitter @Jonathon_Wright with how many slides in this presentation?

First person to guess or be the closest!

GOTTAWANNAWIN

Quote - David Dubovsky, Boca Raton, 1999

The current economic climate is making companies review their approach to IT even more closely

“Existing technologies of virtualisation, business process modelling, cloud based test automation tools and rapid and easy internet access

allow for the development of approaches that allow companies to order testing as a service and pay only for what they use (PAYU).”

• Why (Benefits)

• What (Cloud Automation)

• When (Timeline)

• Scope (Features)

• Transition (AMMi)

• Dependencies (Cloud Type)

• Implementation (Agnostic)

• Universal Test Language

• Maintenance (Asset Loaders)

• Cloud Virtualisation

• Continuous Build & Integration

• Event Driven Automation

Funnel Virtualisation

Dashboards (Real-time)

Three Decades in Automation

1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

2005 - 2006

QTP 8.x

2004 - 2005

QTP 7.x

2006 - 2008

QTP 9.x

2008 - 2010

QTP 10.x

2010 - 2011

QTP 11

1998 - 2001

Modularity driven testing

2001 - 2005

Data driven testing

2005 - 2008

Keyword Driven

2008 - 2011

Hybrid Keyword/Data Driven

Today

1994 - 2001

1999 - 2001

WinRunner 6.x

2001 - 2004

WinRunner 7.x

1998 - 1999

WR x.x

1994 - 1999

Why? – Bit of Background

Recent Explosion in Automation Tools

No longer one Tool to rule them all?

Automation is STILL prone to failure?

Cost of Failed Implementations 2004 2012 (Estimated)

Industry: Test Automation (net worth) $1 billion $6 billion

Automation Projects (failure cost) $½ billion $3 billion

60% 32%

8%

Failure

Working

ROI

92% FAIL TO MEET TARGET ROI

40%

*Ovum study projects the testing tool market to be worth 56 billion US dollars by 2013 growing at an annual rate of 9.5%

Automation is STILL an expensive business?

Manual (Prep + Execution)

Manual (Pure Execution)

Manual'mation

Framework

TAaaS

Effo

rt

Iterations

“The implementation of a self-maintaining test asset loader/scraper that automatically runs after each build and then self-validates against the universal testing language (DSL) and object models built within the

cloud promote a more maintainable and portable solution.”

Introducing Legacy Cloud (First Generation)

Private Combination

Public

SUT SUT SUT

Community Clouds

Public Clouds

Private Clouds

TAaaS TAaaS

“... the prediction that 60% of the average enterprise will have 60% of its applications in the cloud.”

- Karl Flinders, Computer Weekly (2011)

“IBM are working with other cloud suppliers to establish the standards for cloud service provider compliance.”

- Murat Gunenc, IBM Cloud Representative @ Rhodes W1 (2011)

Testing in the Cloud

New Challenges & Opportunities ? and/or

A match made in heaven?

New Challenges

What happens when these services FAIL?

Alternative Service Provider?

Unique Services

(Such as GPS)

Syn

c

Systemic Failure

(DNS or Network)

New Opportunities

“Don’t assume the ecosystems (community or commercial) and other fulfilment systems to be tested in

anything but isolation.” - Julie Gardiner (2012)

Community Clouds already tested?

Consumer Freedom

“Enable sophisticated modelling of the components of cloud deployments, including servers, storage, databases and data.

For the consumption of cloud resources” - PlanForCloud.com (2012)

Instant scalability

Flexible Test Execution

Flexible Test Assets Environments on-demand

Geo-based execution

Instant Accessibility

99.9% Availability (100% Web)

CompareTesting.com

Instant Portability

Best of Breed (BoB) tools on demand

No upfront investment

Pay As You Use (PAYU)

Technology

Process

People

Where are we going today?

Development (.NET 4.0)

AutomationDatabase

Automation Portal (TaaaS)

Application Under Test

2003/2007/2010

Lab Management

Windows

Service

ObjectHandler.dll

DataHandler.dll

EventHandler.dll

Application Programming Interface

Unified Reporting Model

DashboardReportingTest Management Tool Team Foundation Server

Application Lifecycle Management

CodedUI

“Pure ‘Testing as a Service’ Model - Born in the cloud (solution developed specifically with the cloud in mind).”

Automation: Born in the cloud

Taking the journey ‘to the cloud’

Customer Systems

Intranet

Site

Test Suite CMMI/ISO 9000

Rule StoreSoftware Build

Process

Desktop

Application Web Site

Web services PersistenceVirtual lab

The Cloud (System Infrastructure)

Customer Automation Suite

Business Level

Keyword Definition

United Reporting

Credentials

Flexible Cost

Planning

Support

Test Data

DashboardTest Asset

Management

Automation Lifecycle Management

(Test Automation as a Service Model)

Monitoring/

Operational management

Configuration

Reporting

Billing

Customer Take on Customer support

Development

DeploymentMaintenance

How to: Consume cloud services

Test Automation as a Service (TaaaS)

Cloud Web Endpoint

Test Scheduler Service

End Users

QA Person

Run a test«uses»

Schedule a future

test

«uses»

TaaaS Portal

Silverlight

Dashboard

Test Runner Environment - A Virtual Machine

created by the Framework VM Dispenser

TestRunner

Test Data

Set

Technology

AdapterTest Adapter

Platform

Adapter

Client System Test Repository

Business

Level

Keywords

(BPM/BPT)

Test Data

SetsResults

Object

Repository

View real time

test status

«uses»

Business User

«uses»

View historic test

results

«uses»

Dashboard Service

Test Assets Repository

(MTM / Quality Centre)

TD-API

Adapter

Results Service

Test Management

Unified Reporting (Dashboard)Check Development

Progress

Framework

Results

Adapter

Silverlight

Dashboard

Developer

View Live Defect

Build Client System

Build Automation

(Team City / TFS)

Framework

Object

Adapter

Asset Scanner Service

© BJSS Limited 2011 Commercial in Confidence 17

“Supporting natural language with context sensitive validation.” (ATDD, BDD & TDD Compatible)

Technology: Test Designer

Creating Cloud Tests any device, anytime and anywhere

© BJSS Limited 2011 Commercial in Confidence 18 “Supports the full three tier application deployment.”

Technology: Test Lab Designer

Creating Cloud Environments featuring VM Dispenser Technology

© BJSS Limited 2011 Commercial in Confidence 19

*iPhone5 (iOS6.0 not yet fully tested)

Technology: Test Cloud Runners

“Monitor concurrent execution across a instantly scalable number of cloud clients.”

Executing Cloud Tests across a number of platforms

© BJSS Limited 2011 Commercial in Confidence 20

Technology: Agnostic Solutions

“Platform, technology, client, browser, version, language agonistic solutions supporting cross browser testing

through only a few clicks in the lab designer.”

• The test definition is not implementation specific – Test execution is performed by cloud based technology adapters which

act as interpreters for the implementation agnostic language;

Test Run Generate

Environment Under Test (EUT)

Select Platform Adapter

Select Test

Adapter

Select Technology

Adapter

• Application Server • WebServices

• Database Server • Schema • Data

• End-Client Machine

• Windows 2012 • Windows 8 • Windows Hyper-V • Windows 7 • Windows 2008 R2

• Performance • Load

• Functional • Component • Integration • Unit Level • UI Level

• Coded-UI (Win32) • Selenium (ASP.net) • Ranorex (Flex 3/4)

• 1.2.0 (Flex 2) • 2.3.3 (Flex 3) • 3.0.2 (Flex 4)

• Automation.UI (WPF) • White (SWT(Java))

© BJSS Limited 2011 Commercial in Confidence 21

Technology: Event Driven Automation

• Automated testing based on trigger events (passive & active): – Third party system events such as user story or defect status workflow change

‒ Continuous build and integration or source code gated check-in

Pas

sive

A

ctiv

e

Technology

Process

People

Where are we going next?

© BJSS Limited 2011 Commercial in Confidence 23

Example Automation Projects Automation Level Investment in Effort Business Benefit

Description AMMi Gen Implementation Planning Execution Accel. ROI

Client Specific Implementation - Hybrid Approach 4 5 24 63 17 6.9 112%

Client Specific Implementation - BDD Approach 3 4 30 83 22 5.2 85%

Client Specific Implementation - Traditional 1 1 120 250 67 1.7 26%

Testing As A Service Model – Manual’mation 3 1 28 5 25 4.6 200%

Testing As A Service Model - Hybrid Approach 4 5 12 31 19 6.1 186%

TaaaS - Cloud Based Implementation 5 5 5.6 25 7 16.0 306%

Process: Automation Maturity Model Index (AMMi)

“Sometimes half the battle is just starting companies on their automation journey by taking them one step closer to becoming

ready for automation.”

© BJSS Limited 2011 Commercial in Confidence 24

Solution Lifecycle Management

Process: TestingAsAService.com - Model

TEST AUTOMATION AS A SERVICE

© BJSS Limited 2011 Commercial in Confidence 25

Process: Business Process Modelling v2.2

“Business process modelling allows non-subject matter experts (SMEs) to build and perform manual and automated tests

without any previous domain expertise.”

© BJSS Limited 2011 Commercial in Confidence 26

A1

Login.Process

Request.VM

Access.VM

Manage.VM

Logout.Process

A3

A2

C1C3

D2C2D1

C4

E2D4 E1

E3 E4

D3

Design complex tests scenarios from day one ‘first day’ testing

(Release 1.0 = A1>C3>E1>D4>E3>C4)

Process: Business Process Scenarios

“Business Process Scenarios map dynamically to an instance of the Solution Under Test (SUT).”

B1

Login.Process

Request.VM

Access.VM

Manage.VM

Logout.Process

B3

B2

C1C3

D2C2D1

C4

E2

D4 E1

E3 E4

D3

Accept.TermsA1

(Release 2.0 = A1>B1>C3>E1>D4>E3>C4)

© BJSS Limited 2011 Commercial in Confidence 27

Process: Dynamic Test Assets

“Test type agnostic – Pluggable test types so a smoke, regression, integration & performance can be reused against multiple platforms.”

Emerging Test Assets

Application / Component

(e.g. Virtualisation Platform)

Version (e.g. Build,

Release)

Solution (e.g. Time to Market

Improvement)

Client (e.g. Xerox,

Thomson Reuters)

Work Stream (e.g. Finance,

Marketing)

Language (e.g. en-GB

or en-US)

Location (Geo) (e.g. UK,

France, Asia)

Val

idat

e t

est

s

(aga

inst

mo

men

t in

tim

e)

Valid

ate te

sts (again

st new

bu

ilds via asset lo

ader)

© BJSS Limited 2011 Commercial in Confidence 28

“Fault tolerance is built in to report and continue on different levels of fuzzy matching. Combined with the non-technology specific

test definition language, investment in ‘non-fragile’ test assets is much more durable.”

Enter password here ……

Please enter your password to logon

logon logoff

Enter password here ……

Please enter your password to login

login logoff

Test assets are defined by the self-maintaining test asset loader that uses fuzzy logic matching to identify UI objects

Process: Object Recognition (via Fuzzy Logic Matching)

1. XPATH Check 2. Regular Expressions 3. Descriptive Programming 4. Fuzzy Logic 5. Artificial Intelligence 6. Social Intelligence

n/a

Businesses should focus on business, NOT building automation frameworks

2nd Gen

(Data)

3rd Gen

(Keyword)

5st Gen

????(Scriptless)????

4th Gen

(Hybrid)

1st Gen

(Modular)

Gen Zero

(Record)

Next Generations Collaborating Together in Community Testing Cloud?

© BJSS Limited 2011 Commercial in Confidence 30

Testing as a Service (TaaS) Dashboard (Realtime)

Risk Dashboard(Presentation Layer)

Benefit Analysis(Presentation Layer)

Story Board(Presentation Layer)

Business Risk

Project Risk

Technical Risk

Business Domain Language

Test Domain Language

Business Stories

Business Workflow

Business Rules

Business Process Design (BPD)(Visio Design Documentation)

Non-Functional Requirements (NFR)

(Dynamic Data Source (XML))

Detailed Session Definition (DSD)

(Dynamic Data Source (XAML))

Detailed Profile Definition (DPD)

(Dynamic Data Source (XAML))

Detailed Test Definition (DTD)

(Dynamic Data Source (XAML))

Business Components & Interfaces

(Dynamic Data Source (XAML))

Business Process Modelling (BPM)

(BPMNv2)

Business Process Scenarios (BPS)

(eXensible Process Definition Language)

Business Process Tests (BPT)

(Dynamic Data Source (XAML))

Business/Test Transactions Layer

Solution Under Test (SUT) Layer

Dynamic Data Adapters Dynamic Test Adapters

Business Questions

“Integration into current Solution Lifecycle Management (SLM) to include real-time reporting dashboards powered by WCF based reporting frameworks.”

Actionable Insight (Ambassadors of Data)

Funnel Virtualisation

Insight

Data

Data

Data

Data

Data

Data

Data

Data

Data

Data

Data

Data

Data Data

Data

Data

Data Data

Technology

Process

People

The Final Frontier

Testers are testers, not programmers Developers are developers, not testers

“Many of the best tester’s are technically minded and can program but a tester’s real skill is applying testing knowledge to generate reusable test assets.”

People: Cross-functional teams

Julie the ScrumMaster

© BJSS Limited 2011 Commercial in Confidence 33

People: Social Enterprise Ready D

ash

bo

ard

s

Funnel Virtualisation

Do

cum

enta

tio

n

Traditional TaaS

Living Meta Data “Master the Data” Inditex

“Encourage simplicity” D’O

© BJSS Limited 2011 Commercial in Confidence 34

Team

Wo

rkin

g

Remote collaboration

Act

ivit

y St

ream

s

Real-time information

People: Social Enterprise Ready

“Collaborate inventively” McLaren

“Embrace cloud workforces” Mosaic

People: Where are the testers?

Time Zone 1 Time Zone 2 Time Zone 3

Community Tester - Platform: Hybrid Community Cloud (e.g. Facebook)

Beta Tester - Platform: Public Cloud (e.g. beta.bbc.co.uk)

Alpha Tester - Platform: Public/Private Cloud (e.g. alpha.bbc.co.uk)

Cloud Tester - Platform: Private Cloud (e.g. vpn.internal.bbc.co.uk)

Internal Tester - Platform: Private Cloud (e.g. internal.bbc.co.uk)

Testing Command Center (Domain Specific)

• Supporting Core Testing Activities (Environment Monitoring/Management)

•Defect Monitoring & Risk Assessment (Social Enterprise Activity Feeds)

•Testing Management (Test Assets & Resource Enablement)

Solution Command Center (Solution Specific)

• End Client Dashboard/Reporting (KPIs)

• Solution Management (Programme Level)

•Service Provider Management (SLAs/Who does what when)

Automation should be a Global Marketplace

Smaller, possibly remote firms or individuals can now

have easy PAYU access to world class expertise.

© BJSS Limited 2011 Commercial in Confidence 37

Relevant Clear traceability of the business value of Automation through the visualisation of the tests via Business Process Modelling (BPMNv2.2 compliant)

Effective Self-validating test assets achieved using natural language with context sensitive validation against business and testing rules, workflows and data

Maintainable Self-maintaining test asset loader/scraper

Efficient Reports on SUT health including ratings such as percentage availability since build/release, reported errors over time and traffic to error ratio

Manageable Unified platform which non-domain experts can use a natural language to represent business processes and user story acceptance criteria

Portable Technology agonistic - Platform, client/component, browser, version & language Test type agnostic – smoke, regression, integration & performance

Reliable Fault tolerance is built in to report and continue on different levels of fuzzy matching combined with the non-technology specific test definition language

Diagnosable Actionable defects provided by environment under test (EUT) live pause-playback supported by dynamic data adapters (DDA) for accelerated defect investigation and resolution

Test Automation as a Service - Summary

Source: The Big Picture of Test Automation: Test Trustworthiness – Alan Page, Microsoft (2012)

© BJSS Limited 2011 Commercial in Confidence 38

Testing Automation as a Service: Final Thoughts

“The best aspects of these proven test approaches

demonstrate how they have evolved over the past decade,

and this echoes some of the changes towards more lean and

agile business methodologies. They are in a constant state of

evolution – just as the underpinning technology evolves over

time.”

My section in the ‘Experiences in Test Automation’ book

(Dorothy Graham & Mark Fewster) 2012.

Further Information

User: michaeljackson

Password: moonwalk

Hands on Demo

(Hyper-Test.com)

Jonathon Lee Wright (jonathon.wright@testing.uk.com) @jonathon_wright (twitter) | www.TaaaS.net (blog)

www.automation.org.uk | automation.sharepoint.com www.linkedin.com/in/automation