reduce test automation execution time by 80%

!

!

T2#Session!6/25/2015! !10:15!AM!!!!!

“Reduce#Test#Automation#Execution#Time#by#80%”##

Presented#by:#

Tanay#Nagjee#Electric#Cloud#

#####

Brought#to#you#by:#####

##

340!Corporate!Way,!Suite!300,!Orange!Park,[email protected]!E!www.sqe.com!

!

!!!!

!!!!! !!!!!

A software engineer for more than five years, Tanay Nagjee has developed several core features of a highly scalable orchestration/automation platform. As a solutions architect, Tanay tackles complex software delivery problems every day. He works with software companies of all sizes, each with a unique delivery lifecycle and a myriad of tools. Needs common to these companies are automation, orchestration, and acceleration. Tanay shares his experiences with the product team to incorporate customer feedback into the product, which contributes to the architecture of a product suite that is both flexible and intuitive.

Tanay Nagjee

Electric Cloud

© Electric Cloud | electric-cloud.com

Stop Sacrificing Comprehensive Testing to Save Time Tanay Nagjee Solutions Architect Electric Cloud


Healthcare.gov problems in 2014 “Poorly tested”

But, it’s not a laughing matter…

Software errors cost US economy $59.5B annually

– US National Institute of Standards and Technology, 2002

Mass death in WoW

Knight Capital, 2012


Software tester’s toolbox

Unit test tools

Static Analysis

tools

Code Coverage

Tools


Business Pressures that Impact Testing Get-it-out-the-door mentality Will test it more later New feature commitments More frequent builds (CI) (continuous integration)


As development cycles get shorter and more frequent…

We start limiting how frequently we run:

•  Unit tests •  Static analysis •  Code coverage

Which will let bugs go undetected


Typical fixes

•  Manually parallelize test jobs to do more in the same amount of time

•  Buy faster hardware

•  Accept long CI cycles

•  Cut down on tests and analyses that are run during CI cycle


How do I reach testing nirvana?

Shorter CI cycles without sacrificing tests/coverage/analysis

Comprehensive testing

Scale no matter how large the code base grows or how frequent the CI cycle is

Minimize new hardware investment

Continue to use existing tools

Learn what tests take the longest and their dependencies


How do I start?


What’s next?

fine grain controlled

Parallelism to do more stuff at once


Automation + acceleration

Build

Compile, Package

Test

Functional,

non-functional

Deploy

Provision,

Configure, Deploy

weeks

days + automation

minutes + acceleration

How long does it take to get a software update through a delivery pipeline?


Test acceleration with ElectricAccelerator

ElectricAccelerator dramatically accelerates software builds and tests by safely parallelizing jobs across shared clusters of physical or cloud CPU’s.


ElectricAccelerator

Massive

parallelization and

distribution for fast builds and tests

Actionable

visualization and

reports for optimal parallelization

Automatic

dependency

detection for faster, more accurate builds

Efficient resource

utilization. Create build/test clouds to

lower costs


Use case: static test analysis acceleration

•  Large aerospace customer •  Static analysis tool: Parasoft •  Time to run static analysis: !  Before: 5m25s !  After: 1m25s

4x


Try to speed up the commit build. Continuous Integration on a build of a few hours is better than nothing, but getting down to

that magic ten minute number is much better. This usually requires some pretty serious surgery on your code base to do as

you break dependencies on slow parts of the system.

- Martin Fowler


Use case: unit test & code coverage acceleration

•  A customer evaluated ElectricAccelerator (EA) to reduce their cycle time to 10 minutes to enable gated check-ins and continuous delivery.

•  They are sticklers for test-driven development, and need to keep the cycle time low as their test and coverage suites grow over time.

•  Tools: Boost, MS Code Coverage, Bullseye

•  Time to run tests & coverage: !  Before: 7m30s !  After: 3m11s on 2 machines & 39s best case

2.4x 2 Machines

11.5x Best Case


Demo


“If it hurts, do it more frequently, and bring

the pain forward.” Martin Fowler

Thank You!