acceptance testing in devops

Laurent PYCEO, Smartesting

Laurent.py@smartesting.com @py_laurent

www.smartesting.com

Acceptance testing in DevOps

Our long and painful journeytowards DevOps

Product: CertifyIt, eclipse plug-in in JAVA (Model-Based-Testing)

– Waterfall process

– 1 release every 6 months

– Few tests and no TDD

– 1 month (x5 people) to do release testing before deployment

– Poor quality impacting customers’ feedback and adoption

Overview of our development process 2004/06

3

Product: CertifyIt, eclipse plug-in in JAVA (Model-Based-Testing)

– First XP and then Scrum process

– We do TDD

– Continuous integration (code & unit test)

– 1 customer release every 3 months (1 operation release every 3 weeks)

– 1 man/month to do release testing before deployment

– Good feedback from customers!

Tasting agility end of 2006

4

Product: Zest, testing platform in the cloud

– Still Scrum process

– We do TDD and…

– …Acceptance Testing Driven Development (ATDD), 100% automated

– Acceptance tests are part of the CI process

– We do several deployments a day (≈10)

Now we are experiencing DevOps - 2012

5

Shorter iterations (1 to 4 weeks) lead to massive test automation completed by exploratory testing

Acceptance tests become the specification

Testing starts earlier in the development process: Shift left!

What we’ve learnt

Req Manageme

nt & Definition

Test Planning

Execution

Defect manageme

nt

What we’ve learnt

7

To achieve this level of speed (DevOps context), acceptance tests should be:

– Understandable to both developers and business experts to strengthen alignment and enable the ‘shift left’

– Maintainable to manage efficiently changes in requirements and keep pace with continuous deployment

– Automated to enable rapid execution and feed-back

ATDD with Business Domain Language and refactoring capabilities

ATDD & Refactoring

My view as product owner & tester:

Acceptance tests need to be continually reviewed and refactored

just like code!!!

ATDD & Refactoring

My view as product owner & tester:

Acceptance tests need to be continually reviewed and refactored

just like code!!!

Martin Fowler

Acceptance testing driven development

Shift left

PO developers testers Scrummaster

Scrum team

Acceptance testingAcceptance testingShift left

Product Backlog

Sprint Backlog Sprint 1 to 4 weeks

Product at the end of the iteration

Daily meeting

Acceptance Testing Driven Development (ATDD)

Acceptance test is a powerful artifact to improve communication

Test as the definition of ‘STOP’

Written prior to development by tester

Based on a business DSL (domain specific Language)

Confirmed with stakeholders

Mostly automated

Test in natural language

Test fixture

Code

Acceptance Testing Driven Development (ATDD)

Benefits:– Improve communication and collaboration between

project stakeholders

– Shared understanding of what a successful implementation means

– Better coverage of business expectations

– Faster feed back

Challenges:– New methodology that requires rigor and discipline

– Find the right balance between people/process/tool

Continuous acceptance testing

Key features:

– Define progressively your Action Words and your business domain language for test authoring

– Suggestion to promote the reuse of Action Word and avoid duplication

– Refactoring: when an Action Word is modified, impacts are performed automatically across the tests

– Optimization: inspection features enable to continuously optimize the acceptance tests

– Create scripts and accelerate test automation (Ruby, Java, XML…)

Integrations with:

Zest: create acceptance tests with DSL

15

Watir, Appium…

Collaboration through acceptance tests & DSL

16

Product OwnerValidate acceptance tests

TesterCreate acceptanceTests

DeveloperAutomate acceptancetests

• DSL• Refactoring capabilities

Define new business entities…

Define progressively your business terminology with Action Words. Enable collaboration based on acceptance tests.

…or define business entities right from the tests

Promote test steps into Action Words (what developers call extract function). This is refactoring!

Evils of duplication

A fundamental principle

Reuse, reuse and reuse Action Words!

Create and maintain consistent scenarios for your project

Suggestions

Analyse and optimize continuously your tests

When duplications are identified, refactoring options are suggested!

Add, remove, modify Action Words and Scenarios

Test refactoring enables to perform automatically impact analysis and test maintenance tasks

Add parameters to Action Word

Propagate automatically tothe scenarios

Generate scripts

Use of Action Words significantly decrease the cost of automation and accelerate the overall testing cycle

So your acceptance tests are…

UnderstandableDefinition of business domain tests enable better alignment of the team

MaintainableRefactoring and optimization

features dramatically accelerate maintenance

Can be automatedGood design based on Action Words streamlines the test

automation phase

• DSL• Refactoring capabilities

And remember

Acceptance tests need to be continually reviewed and

refactored just like code!!!

Give a tryZest-testing.com

Laurent PYCEO, Smartesting

Laurent.py@smartesting.com @py_laurent

www.smartesting.com