Cloud-Based Test Micro-Services

Shelley Lambert

IBM, Canada

Cloud-based Test Microservices

Shelley Lambert

IBM Runtime Technologies Test Lead

Disclaimer• THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.

• WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.

• ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR INFRASTRUCTURE DIFFERENCES.

• ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.

• IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.

• IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.

• NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:

• CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS

Development Transformation

• Drawn from my work in IBM Runtimes and on Eclipse OMR, Eclipse OpenJ9, and AdoptOpenJDK projects

• Hundreds of thousands of tests, on thousands of test machines representing 18+ platform variants, multiple versions of Java (v5 – 10)

• Goal to simplify, redesign and decouple from legacy/proprietary tools and processes and create open, agile, flexible tests and test solutions to empower developers

AdoptOpenJDK

What is a microservice?

• A succinct API to accomplish a particular task (happiness knows no bounds, but microservices do…)

• A set of microservices assembled together can create a whole system (replacing a monolithic approach)

• While often used in concert, each service can be used independently

• Benefits: Modular, Simpler, Portable, Customizable Apps, EasierDevelopment & Deployment, Co-operative

ResultAnalytics

Data

Services

UI Layer

Cores

raw refined

custom dashboardother clients

TestGeneration BenchEngineCoreAnalytics

TestSelection

BugPrediction

InputOptions

ResultSummary

ResultCompare

githubrepos

Jenkins servers

Microservices Architecture

Activities (and Questions) Related to Test

Plan Implement Automate Execute

Triage Exclude

Report

What? How?

How often? How easy?

How few? How fast? What failed? Why?

What next?

Decompose into a set of services by test activity,services to help answer the questions

Planning Phase

What tests do you need?

How are people using your product? What are their user stories?

What additional info do you have (code coverage, code churn information)?

Plan Implement Automate Execute

Triage Exclude

Report

What?

‘Planning Phase’ Microservice Examples

• Core Analytics Service – inspect java core files from service reports, gather information on usage, particularly patterns of input parameters

• Bug Prediction Service – based on research* an inexpensive algorithm to find hot spots (most changed files) in code base, applied to any given repo

*BugCache for Inspections: Hit or Miss?

Core Analytics Service

Demo

Bug Prediction Service

Demo

Implementation Phase

How best to design and write your tests? What tools/frameworks to use?

How to minimize number of tests while maximizing functional coverage?

Plan Automate Execute

Triage Exclude

ReportImplement

How?

‘Implementation Phase’ Microservice Example

• Test Generation Service – assist developers in the onerous task of writing thorough functional tests, standardized test naming and structure, CTD for minimal tests with good functional coverage• Upload header files (function/method signatures)

• Inspect input types (primitive or complex)

• Generate combinations of inputs (pairwise/CTD)

• Generate partial test code… (with standardized naming, parameterized inputs)

Automation Phase

• How do I add my tests into a CI workflow?

• How do I replace old practices with new improved ones?

Plan Execute

Triage Exclude

ReportImplement Automate

How often? How easy?

‘Automation Phase’ Microservice Example

• BenchEngine Service – make it easy for anyone to run performance benchmarks • generate scripts to run benchmarks from definition files

• allow for variations and specific platform settings

• send scripts to Jenkins nodes to run automatically run benchmarks

BenchEngine

Demo

BenchEngine

Developer: • fires up browser• connects to BenchEngine service• select inputs

Jenkins

input selections:• benchmark definition file• action type

generateScript:• local run • insert into automated

builds

submitJob

BenchEngine Service

Execution Phase• What is the minimal set of tests I need to run, given a particular change?

• What input options do I run with?

• What is the execution time? Has it changed over time?

Plan

Triage Exclude

ReportImplement Automate Execute

How few? How fast?

‘Execution Phase’ Microservice Example

• Test Selection Service – change based testing, uses code coverage information to generate a smaller/smarter set of tests to run on a pull request

• Input Options Service – grabs input options defined in tests at start of a build, names and stores them (unique sentences of options), for sharing with other builds/tests

TestSelection

Developer: • fires up browser• connects to Test Selection service• select inputs

Jenkins

input selections:• pull request in github• platform(s)

submitJob

Test Selection Service

• (nearly) headless service to feed information to builds• not all services have (or need) a pretty GUI

InputOptions

Jenkins mongodb

Input Options Service

Triage Phase• Has this failure happened before? Is this failure intermittent? If so, are there

recognizable patterns?

• Is there other information besides test output that I can use to help find patterns?

• Do I need to exclude this test?

• Can I re-include it automatically when the issue is fixed?

Plan Execute ReportImplement Automate

Triage Exclude

What failed? Why?

‘Triage Phase’ Microservice Example

• WIP: Results Analytics Service - automate the “googling of error messages”, look for patterns• search across builds at one server?

• search across builds at multiple servers?

• search the internet?

• Use Watson pattern matching / analytics engine...

• If you didn’t want all of the above, then Jenkins plugin?

Reporting Phase

• Can I summarize a large/diverse set of tests?

• Where can I see the test report?

• Can I compare test results easily? From many data sources?

Plan Execute

Triage Exclude

Implement Automate Report

What next?

‘Report Phase’ Microservice Example

• Results Summary Service – summarize the results of multiple builds running on multiple servers, overlay customizable dashboard

• Results Comparison Service – given more than one result of same test, compare them. Examples of results to compare:• FV tests from Build 1 compared with those of Build 12

• Scala (or other 3rd party app) community tests running on OpenJDK vs OpenJDKwith OpenJ9 (Java8 / Java9)

• Last 4 BenchmarkXYZ runs against OpenJ9 (trend)

• Last several months of GC perf stats from OMR

Results Summary Service

• Model the common elements (status, execution time, failure age, etc) for uniform representation

• Leverage Jenkins API

• Progression of enhancements• Raw data• Simple diff• Graphical representations• Overlay customizable dashboard, simple elements, continuous feedback

Results Comparison Service

Demo

• Gather perf metrics from Eclipse OMR builds

• Store historical data in mongodb• GC perf dashboard allows many

different views onto the data

Review: Examples of microservices

Plan Implement Automate Execute

Triage Exclude

Report

Core analytics serviceBug prediction service

Test Generation Service Test Selection ServiceInput-Options Service

Result Analytics Service Result Summary ServiceBenchEngine

Microservices Examples Review

ResultAnalytics

Data

Services

UI Layer

Cores

raw refined

custom dashboardother clients

TestGeneration BenchEngineCoreAnalytics

TestSelection

BugPrediction

InputOptions

ResultSummary

ResultCompare

githubrepos

Jenkins servers

Plan Execute Triage ReportImplement Automate

Questions Addressed

• What aspects of your testing can be shared across multiple products? Decompose by common activities…

• How to identify, abstract, and convert functionality used by many test systems into shared services? Common patterns (apps take inputs, tests produce results)...

• Why bother with microservices? In a cloud? Modular (super-fast dev), portable, assemble for a ‘system’ or singular use, leverage other services…

• Gotchas... Network reliance, fallbacks needed…

Get Involved

Innovate with us at:

• Eclipse OMR

• Eclipse OpenJ9

• AdoptOpenJDK

AdoptOpenJDK