thomas kauders - agile test design and automation of a life-critical medical device - eurostar 2010

Thomas Kauders

Agile Test Design and Automationof a Life-Critical Medical Device

A usable model for Agile Test Automation

1 © PrettyGoodTesting 2010

Agile Test Design and Automation of a Life-Critical Medical Device

Project Summary- about me, the project and our test team

Project Challenge- why we needed to automate test

Our Approach- how we designed the automation process

Automation Implementation- how did it take place in practice

Lessons Learned- our successes and concerns

Agenda


Thomas KaudersM.Sc., Test and test manager since 2002

ISEB/ISTQB Certified

Currently:

Assigned at a large medical company

About


More than

11.000 pages

of specifications

About the Project

“The Device”

Not the actual devicedepicted! Used for

illustration only!

700 Man-years

...as from Guthenberg

...to ”The Internet”


The Test Team

1 Test Manager

1 Test Designer

2 Manual Testers

3 Test Automators

...1 Model-Based Test (MBT) Automator

...2 Functional Test Automators


Large and complex system

Life critical system

Volatile requirements R&D

Agile technical requirements

Formal V&V(Verification & Validation)

U.S. Food and Drug Administration

The Project Challenge

Test

Development

Requirements


Importance of formal test

Huge volume of manual test

Short development sprints

Limited ressources

Strict deadline

Possible solutions:• More ressources • Time extension• Test Automation

The Testing Challenge


Considering test automationfrom the very beginning of test design

Cooperating Manual- and Automated Teams

Reusing Test Design across teams

Modularising the test

Gradually adding more ”Pay As You Go”

A New Approach


SRS MMI

Consider Test Automation from the Start

Modular mindset,during analysis andTestCase design!

Aut. TestCases

Man. TestCases

TestDesign

Requirements

ActionWords

Traceability to Reqs?

Based on SRS and MMI?

TestCases OK for Test Automation?


To keep it simple we developed a concept of:

PRIMITIVES (aka ActionWords, KeyWords etc.)

Primitives is a known term by key project members

Sticking to this term eased our ”selling the idea”

Naming the Principle


Close cooperation btw. Man. & Aut. teams

Re-using Test Design across teams


Use of ”mnemonics” resulted in ”readable” names, e.g.:

ST-PRM-FTUS :System Test, Primitive, First Time Use

ST-PRM-LGBK :System Test Primitive, Log Book

Using “Call to test” to use Primitives

Call to test


PTC:

Set Year ()

PTC:

Set Month ()

PTC:

Set Day ()

Primitives (PTC)

Call to test PTC:

Set Month ()

Call to test PTC:

Set Day ()

Call to test PTC:

Set Time ()

Call to test PTC:

Confirm Setup

PTC:

Confirm Cancel Setup

PTC:

Set Time ()

PTC:

Cancel Setup

Cancel Cancel Setup

PTC:

Confirm Setup

Test Cases (STC)

Call to test PTC:

Set Year ()

TC:Navigate to Setup D&T

Call to test PTC:

Set Month ()

Call to test PTC:

Set Day ()

Call to test PTC:

Cancel Setup

Call to test PTC:

Confirm Cancel Setup

Call to test PTC:

Set Year ()

Call to test PTC:

Set Month ()

Call to test PTC:

Set Day ()

Call to test PTC:

Cancel Setup

Call to test PTC:

Cancel Cancel Setup

Call to test PTC:

Set Year ()


Call to test PTC:

Set Time ()

Call to test PTC:

Confirm Setup

PTC:

Verify Setup Complete

PTC:

Navigate to Setup D&T

Call to test PTC:

Verify Setup Canceled

PTC:


PTC:



Think of Lego Bricks


Manual test not coping with the volume of tests

Automated tests running on unstable test system

The ”Death by test script maintenance” issue

Milestones not reachable if not:

- increasing the extent of the automated regression

- increasing the quality of the automated regression

- creating better manual regression testing

- developing smarter approach to test scripting

Not possible without effective test automation!

The Test Automation Challenge


Navigation

Go from A to B (Start to adjust Time)

Action

Input to system (Set Time)

Verification

Expected result (Verify Time set)

Execution

Practical test implementation (Note Test Start Time)

Use parameters to keep numbers of mode

Primitive Classification


Traditional SequentialTestCases

A pseudo test case broken down into Primitives,each Primitive is unique to the SUT (Software Under Test)

PRIM 6

PRIM 5

PRIM 4

PRIM 3

PRIM 2

PRIM 1

Products of Module-based Test Design


PRIM 7

PRIM 6

PRIM 3

PRIM 2.1

PRIM 1

PRIM 7

PRIM 5.1

PRIM 4

PRIM 3

PRIM 2.2

PRIM 1

PRIM 7

PRIM 5.2

PRIM 4

PRIM 3

PRIM 2.3

PRIM 11

2.1 2.2 2.3

3

6 4

5.1

7Saving above 50% of work for just this example.

5.2

An Advantage


During test analysis try to break down each Test Case in unique and identifiable user action sequences

A unique sequence must not exist elsewhere

(stick to this…)

The sequences are to be re-used and combined in anyorder representing complete user actions

Why does this work?


New test delivered once a week

Tests fixed from day to day

Change identified (By Developer or Tester)

Impact assesed

Affected Primitive(s) identified

Change(s) implemented

Test Scripts tested

Delivered and verified that tests are running

Dealing with Change in Automation


Deliver tests Verify tests

Test Test Scripts

ImplementChange

Identify affectedPrimitive(s)

Assess impact

Changeidentified

Must be ableto respond quicklyto change

Daily Change Management


Tracebility from Requirements -> TC´s involved

Change only the involved Primitives

1:1 manual TC´s -> automated TC’s

* SCRUM = Dynamic Requirements!

When Requirement’s are changing*


Easy indication of what has to be changed

Full traceability of what has been changed

1:1 traceability eases quicker validation*

Use of Primitives made the test suite scalable

Use of Primitives made the test suite flexible

* FDA requires full validation of:

Designs, Implementation, Tools, TestCases etc.

What was reached?


Initial Test Analysis…still to be carried out* (no Quick-fix)

Change Impact Analysis…still required* (no Silver Bullet)

* Exploratory Testing and Model-based Testing...is still to be carried out (not a stand-alone thing)!

What was NOT reached?


Equals 7-14 days of man. regression tests...nightly!

- September 2010, 7 days

- December 2010, 14 days (expectedly)

40% WorkLoad Reduction

Every team member plays an important role- Happy Testing!

And…Milestones are now met!

Outcome


Too many Primitives (incorrect partitioning)

Improper mindset during review

- This seems possible! But is it?

Lack of review of Manual and Automated tests

Lack of review of Test Scripts

Perfecting / ”goldplating” test breakdown

Pitfalls


Cooperation between Test Designer, Manual Test Team and Automation Test Team reduces effort, by:- Thinking Automation First- Reusing of Test Documentation/Primitives

Primitives (aka ActionWords, KeyWords etc.):- Promotes flexible & easy maintainable Test Suites

Exploratory Testing and Model Based Testing...still to be carried out (not a stand-alone thing)!

Summary


Time for Questions and Comments

Thomas Kauders: (+45) 3163 [email protected]

PrettyGoodTesting ApSLautruphøj 1-3DK-2750 BallerupDenmark (+45) 3163 0200

www.PrettyGoodTesting.com


Step 1: Using Primitives by calling with new param. values

Step 2: In making test, needed Primitives are implemented

Step 3: Putting it all together in a test script

Step 4: Making a test suite for Hudson

Capture Replay


Continuous Integration


thomas kauders - agile test design and automation of a life-critical medical device - eurostar 2010

Technology

test ptc

test automationfrom

srsmmiconsider test

system test primitive

test teamproject challenge

time ptc

year ptc

month ptc