serena webcast: accelerating application delivery with continuous testing

©2014 CloudBees, Inc. All Rights Reserved

Accelerating Application Delivery with Continuous TestingPresented by Soasta and Serena Software

November 20, 2014

Introduction to Serena

2

• Founded 1980

• Headquartered in San Mateo, CA

• >$200M revenue

• 200+ R&D Engineers

• HGGC Portfolio Company

• Follow the sun Support

• Serena Deployment Automation• Increase Deployment Velocity

• Quality, Predictability, Repeatability

• Visual Process Definition

• Helping Global 2000 enterprises build and release better software

Today’s speakers

3

Mark LevyDevOps Evangelist, Serena

Peter GalvinSVP Marketing, SOASTA

Today’s speakers

© 2014 SOASTA. All rights reserved. 4

Expertise


August

2014COMPLETENESS OF VISION

AB

ILIT

Y T

O E

XE

CU

TE

CHALLENGERS LEADERS

NICHE PLAYERS VISIONARIES

GARTNER Magic Quadrant for Integrated Software Quality

Suites

“SOASTA has

consistently

provided

innovation for

mobile and web

applications”

6 SERENA SOFTWARE INC.

Results Comparison

Jan 2014

Oct 2014


Rapid Delivery

Agile developmentContinuous

deliveryDevOps principles

Mobile and Web

Online everythingMobile commerce

Legacy modernization

Complexity

Composite apps3rd party services

API proliferation

Unprecedented Scale

100’s to millionsLocal to global

100% availability


o Small Batch Sizes


o Small Batch Sizes

o Comprehensive Version Control


o Small Batch Sizes


o Simple Branch Strategies


o Small Batch Sizes



o Automation


o Small Batch Sizes



o Automation

o Continuous Feedback


o Small Batch Sizes



o Automation


o Emphasis on working builds


o Small Batch Sizes



o Automation



o Consistent environments


o Small Batch Sizes



o Automation



o Consistent environments

o Developer – Tester Collaboration


50,000 Apps 1,000,000 Apps 1,500,000 Apps


“We didn’t have enough

testing for a very

complicated project”

US Health and Human Services

Secretary, Kathleen Sebelius


Mobile Test Automation



Real TestDevices



Performance Testing

Real TestDevices



Performance Testing

Real Test

Devices

Solid Integration


Does it work as

designed?

Does it perform

under extreme

circumstances?

Approve materials Real Drivers

What happens

in actual

conditions?

Assembly Line


Functional validation

Performance,

reliability,

scalability tests

Real Users

Build

What happens

in actual

conditions?

Unit-tested code


o Rapid & accurate test development

o Reliable, maintainable test automation

o Stability across mobile OS platforms

o Device performance measurement

o Real-Time feedback via Continuous Integration


o Fully managed device cloud installed

behind the firewall

o Real-time device inventory, status and

availability

o Online access for all your teams

o 24x7, secure, remote access to devices

o Manual & Automated functional test

o Remotely access to selected mobile

device and any mobile app

• Use devices for manual testing and monitoring

• Remote screen-share devices for collaboration

on design or issues


o Dedicated Public Device Cloud

o Access to the latest market leading

devices (iOS/Android)

o Automatic installation, maintenance

and upgrade

o Hosted devices in a secure data

center

o Stock devices, not jail broken/rooted

o Seamless integration with TouchTest

and CI environments

o API available to install apps,

reboot/wake-up devices, launch tests

Perfect for 24x7 Automated Mobile Regression Testing


s

Memory leaks

Test Scale

Sta

ge

/ T

ea

m

Dev

& Test

Release

& Deploy

Network

& OpsCDN file placement

Load Balancer configuration

Network bandwidth

Network configuration

DNS routing

Inadequate server resources

Default configuration settings

Unbalanced web serversAuto-scaling failures

Latency between systems

Slow third-party plug-ins

Garbage collection

Database thread counts

Inefficient database queriesSlow pages

Conflict with other apps

Test

Lab

Staging Production (100% +++)

Search technology limits

Method-level tuning

Max sockets exceeded

Firewall max capacity

Global latency variance

Security bottlenecks

Load Testing at All StagesPerformance validation from development to production

Continuous tuning for full-scale…and well beyond


s

Memory leaks

Test Scale

Sta

ge

/ T

ea

m

Dev

& Test

Release

& Deploy

Network



Network bandwidth


DNS routing






Garbage collection




Test

Lab



Method-level tuning








Analytics

CloudTest

Databas

e

Main

Perf Tester 1

Analytics

Load

GeneratorLoad

GeneratorLoad

Generator

San Francisco (GoGrid)

Analytics

Load

GeneratorLoad

GeneratorLoad

Generator

Tokyo (AWS)

Analytics

Load

GeneratorLoad

Generator

Amsterdam (Azure)

Data sources:

Native (SSH), JMX, PerfMon, CA Introscope,

AppDynamics, New Relic, CloudWatch,….

Web

Server

Web

Server

App ServerWeb

Server

Web

Server

App Server

App Server

App Server

Load

Balancer

Cache

Cache

Databas

e

Analytics

Load

GeneratorLoad

Generator

Virginia (IBM)

Analytics

Load

GeneratorLoad

GeneratorLoad

Generator

SOASTA

Global

Test Cloud

Access

for all

Analytics

Load

GeneratorLoad

GeneratorLoad

Generator

East Coast (AWS)

“Conductor”

CDNCredit

Card

Web

analytics

Other 3rd

pty svcs

Chicago

(Rackspace)


Jenkins runs any

unit tests as usual


Run iterative load

tests via Jenkins

Spin up suitable, consistent

test environments in the

cloud or the lab

Jenkins runs any

unit tests as usual


Run iterative load

tests via Jenkins

Mobile? Run functional

automation on real devices

for functional & performance

validation



cloud or the lab

Jenkins runs any

unit tests as usual


Run iterative load

tests via Jenkins

Daily (or more frequent) updates

keep teams quality focused

Embedded failure

details & bottleneck

data isolate issues




validation



cloud or the lab

Jenkins runs any

unit tests as usual


Run iterative load

tests via Jenkins



Embedded failure


data isolate issues

Manage to a

performance

baseline




validation



cloud or the lab

Jenkins runs any

unit tests as usual


Use real user data to

iterate dev & test plans

Run iterative load

tests via Jenkins



Embedded failure


data isolate issues

Manage to a

performance

baseline




validation



cloud or the lab

Jenkins runs any

unit tests as usual

© 2014 SOASTA. All rights reserved. 44CONFIDENTIAL – Not for Distribution

Ut

Unit Testing

Pe

Performance

Testing

Fu

Functional

Testing

Development Testing Operations

TIME

Test

Dev Ops

Traditional Linear Approach

Continuous Testing Approach


SOASTA Customers

“4 months of manual tests now run in 1 hour, overnight.”

“25% reduction in time to market for production

rollouts.”

Steven Winter, Director of Quality

FIS Mobile – Mobile banking platform to over 1000 banks


o 250 hours of manual tests running in just over 1

hour every night

o Regression Tests not run until feature complete

now run nightly – 4 months to overnight

o Fixing bugs earlier – Detailed test results for

developers

o 25% Reduction time to market

o Reduced field failures


SOASTA Customers

“TouchTest has been critical in improving both the speed to deliver

and the quality of our mobile apps.”

“We decreased our test cycles from three weeks to a couple of days.”

Thierry Chassaing, SVP of Product & Engineering.

ItsOn - the leader in Mobile Smart Services.


o Understand your requirements

o Identify the cloud advantages for you

o Automate the obvious and most critical

o Connect to a continuous process

o Align teams with actionable information

Questions?

53

Free Deployment Automation

Please use the Q&A panel to submit your questions

Friday 1 pm ET/ 10 am PTCD: Integrating the Deployment Pipeline Toolchain through Automation

Upcoming DevOps Drive-In December 18th

Weekly Deployment Automation Demo

Get the Free Version

Want to See More?

54

serena.com/sda

Thank you!

serena webcast: accelerating application delivery with continuous testing

Technology