planning and building scalable test infrastructure

8/14/2019 Planning and Building Scalable Test Infrastructure

1/22

PLANNING &PLANNING &

BUILDING SCALABLEBUILDING SCALABLE

TESTTESTINFRASTRUCTUREINFRASTRUCTURE

Vijayan Reddy

Engineering Manager

Adobe Systems


2/22

Why Test Infrastructure

Where are we missing? How can we save?

How much of your test project ismanual efforts

How much of your test project istest execution

Projected Time slices


3/22

In Scope

This is no Rocket Science

Approach from a typical testlifecycle End to End tasksfor shipping a project

A blue-print for scalable testinfrastructure

Giving tips on cross-linking ofthe tools


4/22

Test Infrastructure

An interface/glue that connects every single task and providesplug-ability for new tasks

All Material Used Tools,Methods, Scripts

Platform that hosts &integrates tools and scripts

Commercial tools dont offercomplete support of alltasks


5/22

How To Build

Available off-the shelf? Probably not

Buy/Build/Customize is the mantragreat commercial software

great open source frameworksgreat engineers in your team


6/22

Top Opportunities

Test Execution Biggestopportunity

Build SystemsBVTPIT

Defect AnalysisLinked Test Cases creation

& Maintenance

Test EnvironmentsMaintenanceSetup / clean up after

executions

Test Reporting / Archiving /Data Mining


7/22

Scale with Meta Controller


8/22

Meta Controller -Requirements

http://staf.sourceforge.net

Able to work p2p, or client server

Able to work across platforms

Able to invoke, monitor tasks, Remotely Trigger Execution, eg.

Able to transfer files Builds, Installers Test Results, Logs, Thread dumps

Able to parallelize across nodes
http://staf.sourceforge.net/http://staf.sourceforge.net/http://staf.sourceforge.net/


9/22

Optimization @ Test Setup

Time on Test Setup, Test Clean upClean Systems

Imaging / ghosting can helpAcross different platforms?

Build the software stack - Eg: Application Server, Java/.NET run time

Database

Deployment of your applications

Deployment of test suites & Setup

Multiply savings by the number of builds, andMachines


10/22

Meta Controller

Control the test automation framework(s)

Glue the controller and tool(s), Script(s)APIs exposed

Command line options / Batch filesParameterize the launch:

Platform / machine agnostic scripting

specific details of each test machine abstracted

in configuration files

Enable Metrics Collection Code Coverage,Profiling


11/22

Scale with Helpful Interfacing

Test cases managed under a system (Manual &Automated tests) (Eg.http://testlink.sourceforge.net)

Why do we need a system, not DOC, PDF, HTMLTo assure test coverage / traceability

Cross linking with Feature Tracking

Cross linking with Bug tracker

Cross linking with Source Control

Help track estimates as well, against test cases/cycles

Interface SCM and Bug Tracker


12/22

Scale in Defect Analysis:Facilitators

Traceability LinkingFeature -> Test Case ->

Changed Files,versions -> Bug ->dependent bugs


13/22

SCM Bug tracker Linking

One spot update of bugs & SCMWhen fix is checked in

Use a templateFill description of fixFill areas affectedFill Recommended tests

Auto - Update Bug trackerAreas AffectedFiles Affected and versions of these files

Useful in Regression testing & Injections check Search up other bugs which had fix in same files and

retest.


14/22

Scale with Early Warnings

Static Code Analyzers

Catch bugs even before thecompilation is done, letalone built and tested

Standard Code violations,Uncaught exceptions,Un-freed memory etc.

Code Security audits forscript injections


15/22

Automated Builds

Check-in Triggered Builds,Scheduled Builds

Build System to accessbuilds, logs, changedfiles, changed versions

Promotion to QA onSuccess, Auto Deploy

Tagging / Merging onSuccess on SCM


16/22

Scale with BVT

Catch the Bugs early with BVT

Automate triggering of BVT onbuilds

Use your Automated Sanity /

Acceptance TestsLink with Build Setup to Bless

builds for QA promotion Saves QA cycles

Automate failure mails Send

with exact test log, pick thecode changes from the build

Auto-log a bug


17/22

Pre Integration Test Setup

PIT saves pitfalls Huge development teams, central builds

Check-in to a PIT branch (that is Sync with Trunk)

Kick offAuto Build

BVT

If Success, auto-Merge change into Trunk

If not, follow BVT failure handling and auto- revertfrom PIT Branch


18/22

Infrastructure for Reports

Common results repository Different test cases / suites /

tools output differently

Build interfaces to import,export

Define common resultsformat

Preferable formatDatabase

XML


19/22

How to Export

Commercial tools export in XMLs

If that helps, standardize other tools / scripts forthat

Use XSLT to transform to common formatDefine Performance results schema

S l th h i t lli t


20/22

Scale through intelligentreporting Easier access for data mining

Easier reporting, trends along builds, alongfeatures, along platforms, locales

Drill down ability on any individual test case, andsee history along builds to isolate injections

Easier collation of all test results

Charting for Non functional parameters

improvements Cross Link to Feature / Test Case tracker


21/22

Take Away

Identify these bridges thatwork for you No onesize fits All theory

Create an integrated

platformBuildBuyCustomize

Available athttp://[email protected]
http://qualinfra.blogspot.com/mailto:[email protected]:[email protected]://qualinfra.blogspot.com/


22/22

Questions

planning and building scalable test infrastructure

Documents