automated continuous test selection methods for large scale devops

Automated Continuous Test Selection for Large Scale DevOps

Marc HornbeekSenior Solutions Architect

2Spirent Communications

What is DevOps?No standard DevOps definition !

Design Right Shift Left

Development

Wants agility

Operations

Wants stability

Continuously eliminate process friction and waste

http://itrevolution.com/

the-history-of-devops/

GeneKim

8.2 million google hits (up from 4M mid September)

CI CD

2009 Flikr 10 deployments/day

21 million

CM

CT

Patrick Debois2007

“Agile Infrastructure”


Spirent and DevOps

Continuous Deployment

(CD)

Continuous Integration

(CI)

Plug

-ins

Plug

-ins

SUTTest Tools

Lab Management

Physical, Virtual, Hybrid Lab

Analytics ALM

Orchestration

Continuous Test (CT)CCM

Test toolsCT OrchestrationLab Management

Professional Services


Large Scale DevOps

100+ dev, multiple sites

10 MLOC+, multi-language

100+ features 10+ build targets

Complex dependencies

200+ topologies10K+ test cases10+ deployment

packages

Millions of test possibilities


Traditional Test Selection Environment

Separate test phases each with its own fixed test suites

Large # of changes per test phase

Fixed resources and time for running test suites for each phase


Large Scale DevOps CT Test Environment

• Many short cycles• Aggregated

changes• Large pool of tests• Orchestrated pipeline• Elastic infrastructure

CT applies at every phase and in every process


Importance of automatic test selection for Large Scale CT

More test beds and complex topologies drives need for efficient utilization of resources

Find high severity defects early

More people testing drives need for efficiency


Test SelectionMethods Evolution

Small scaleModules Single test phaseSnapshot analysis

Large scaleSystem-widePipeline of cyclesTrend analysis


Common CT Test Selection Methods

Base suite always selectedRe-run last failed test casesPriority assignmentsSpecial cases for compliance

and performance tests “All-tests” policies

Static test suites are suitable but may not be optimal for small to medium scale CT


Advanced Large Scale CT Test Selection

Methods

1) Tag code and tests2) Static code analysis 3) Build artifacts 4) Test coverage 5) Test trends

Test selections are automatically determined by…

… and run-time and resource attributes

Which tests will yield the most

critical failures?


Advanced CT Test Selection Methods Focus

The test selection methods operate at different points of the code-build-test pipeline

Building TestingBuild Result

Coverage

CodeCoding


Automated test selection challenges

Too many testsToo

man

y re

sult

sUncertain test coverage

Analysis time

Wasted resources

Diffi

cult

to

lear

n

Wasted time

Too many bugs

Irrelevant results

Diffi

cult

to

debu

gFalse positivesFalse negatives

Setup timeEx

ecut

ion

tim

e

Maintenance effort

Poor

inte

grat

ionDoesn’t scale

Complex tools

Dependencies


Comparison criterion used forlarge scale CT test selection

Person-hours spent on test activities

Critical as % of total # defects

Test resource utilization

Measures(per deployment)


Method 1: Tag Code and Tests

Descriptive attribute “tags” are manually bound to code and test modules.

Tag list is generated for changed modules.

Select tests with tags that match the changed code module tags.


Method 1 Results: Tag Code and Tests

Heterogeneous systems

x Misses important dependencies

Easy to implementx Tag maintenance

x Too many tests and resources


Method 2: Static code analysis

Test cases are manually associated with code modules.

Static code analysis metrics (memory leaks, uninitialized variables & pointers, etc.) identify risky code areas.

Select tests that are associated with highest risk code.


Method 2 Results: Static code analysis

Good results for modules


Easy to implementx Manual code to

test and results association

More efficient than static selection

x Tool costs at large scale

Note: These ratings assume the system programming language is supported by the tool.


Method 3: Build artifacts

Test cases are manually associated with artifacts that are generated by the build process. (image and data files)

Changes in the artifact files resulting from the build process are identified.

Changes in build artifacts are used to select tests.


Method 3 Results: Build artifacts

Works well for changed modules


Improved over static selection methods

Scales well

Fairly easy to implement

Maintenance of test case associations


Method 4: Test Coverage

Manually associate test cases to code statements.

Test coverage metrics identify risky code areas.

Select tests that are associated with least covered areas.


Method 4 Results: Test Coverage

x Programming language dependent


x Low yield for critical defects

x Tests must be created for missing areas

x Tooling costx Maintenance of test

associations with code statements

x Costly to instrument code at large scale


Method 5: Test Trends

Cumulative code-to-test verdict correlation factors (CF) are saved in a table for all code modules and tests.

CF numbers are adjusted when tests are run for a code module. (Note: Values are neutral for new tests or code set to neutral value.)

Test cases are selected which have the highest correlation to the changed code modules.


Method 5 Results: Test Trends

Applies system wide

Program language agnostic

Identifies important dependencies

Flexible at scalex Tooling, initial setup effort

Easy maintenance Fast and relevant

results analysis


Comparison of Test Selection Methods Applied to Large Scale CT

Method

1. Tag code & tests

2. Static analysis

3. Build artifacts

4. Coverage metrics

5. Test trends


Test Trends - Architecture

Reference white paper:http://www.perforce.com/sites/default/files/continuous-change-driven-build-

verification-wp.pdf

The system is a simple form of predictive analytics that uses a table to keep track of code-to-test correlation (CODTEF) factors which drive test cases selection.

CODTEF values self-adjust.


Test Trends – Results Analysis

Results analysis can be derived by “running the matrix in reverse” because the test results correlate to code modules.

Test Results

CodeOwners

CodeOwners

CodeOwners


The test trend method can be applied to other change factors:TopologiesHardwareEnvironment factors (power,

cooling, humidity, etc.0

Test Trends – Additional Applications


Test Trends - Implementation Approach

Break up current test suite into a small set of groups. Identify code group – test group relationships. Tooling to automatically identify group code changes, launch specific test groups, report results according to group owners

Identify code subgroups and initial test correlations. Identify and design specific tools changes.

Implement remaining tools except the advanced metrics

Determine which advanced metrics will be used by the organization (eg. reliability growth modelling)

Create metric tools using the data collected during the Change-Driven loops.

Phase 1Group Level

Phase 2Sub-Group Level

Phase 3Advanced Metrics


www.spirent.com/clear

1) CT Blueprint white papers2) CT eBook3) CT Assessment Tool

Take-aways

http://www.perforce.com/merge/2014-sessions/automated-testing-continuous-

change-management

http://www.perforce.com/sites/default/files/continuous-change-driven-build-

verification-wp.pdf


Best Practices Assessment Tool simplifies the assessment process.


Summary

Of the five methods considered for large scale CT, Test Trend method was preferred

A drawback of the Test Trend method is the complexity of tooling

Test Trend method has additional benefits for analytics and system applications


Questions anyone?


© Spirent Communications, Inc. All of the company names and/or brand names and/or product names and/or logos referred to in this document, in particular the name “Spirent” and its logo device, are either registered trademarks or trademarks pending registration in accordance with relevant national laws. All rights reserved. Specifications subject to change without notice.

spirent.com

Thank youAutomated Continuous Test Selection for Large Scale DevOps

Marc HornbeekSenior Solutions Architect

[email protected]: mhexcalibur

mailto:[email protected]

automated continuous test selection methods for large scale devops

Technology