created by jan medved integration & test strategy for lithium

Created by Jan Medvedwww.opendaylight.org

Integration & Test Strategy for Lithium


OpenDaylight is made of multiple projects -> we have to test them individually and together

Projects are quite heterogeneous: core, tools, plugins, apps, etc… -> they have different test needs

OpenDaylight has a common CI process for all projects -> tests should in general be part of the CI

We leave users freedom to select features -> we need an strategy to test a non-rigid distribution

Facts

2


Forget about verifying projects, focus on features Test single feature enabled to identify issues in a feature itself Test all features enabled to identify feature interferences

For a given project, tests are called after: A change (merge) in an upstream project. This triggers build and

system test A change in the project itself. We miss system test on code submit

Be selective with the tests we run A change in a “stem” project will trigger test on many features A change in a “leaf” project will trigger test on few concerned features

High Level Strategy

3


We define 2 integration features for test purposes:compatible-with-all -> useful for all features system testall = compatible-with-all + incompatible features -> useful to detect feature/bundle interferences

We generate 2 distributions (no feature enabled at startup):Regular distribution with all projects feature indexes (except integration)Test distribution = Regular distribution + integration feature index

Distribution Assembly

4


Build test occurs during project code build. Examples are Junit, PAX-EXAM or Karaf feature install test

Every project in OpenDaylight defines 3 build jobs:Verify -> This job builds the project code and pass build tests upon code submitMerge -> This job builds the project code and pass build tests upon code merge. In addition it will upload generated artifacts to Nexus repository.Integration -> This job will build the project code and pass build tests after a merge in an upstream project

Build Test

5


System test requires a framework that installs controller and executes tests against network tools. Example is the Robot test.

For every project user-facing feature we will create 2 system test jobs:feature-only test -> Install single feature with required dependencies and run feature planned testfeature-all test

If feature is compatible -> Install odl-integration-compatible-with-all and run feature planned test

If feature is not compatible -> Install odl-integration-compatible-with-all + incompatible feature and run feature planned test

System Test

6


Test Triggers

7

Code Merge

Release creation

Merge buildfeature-only test

feature-all test

feature-only test

feature-all test

For all available features

For all features in downstream

Code Submit Verify build

*Optional (requires project distribution or some automation to update project artifact)

Integration build

For all projects in downstream

feature-only test*

created by jan medved integration & test strategy for lithium

Documents