continuous testing in vegas

Continuous TestingJared Richardson

Agile Artisans

@JaredRichardson

GrowsMethod.com

Who am I?

Cofounder of GROWS Methodology

Author Ship It! and Career 2.0

Screencast editor PragProg.com

2nd public signatory of the Agile Manifesto

Started AgileRTP in 2007

AgileArtisans.com

Jared Richardson

Who are you?

What’s up with the material?

Major rewrite

Introduction

Slides on USB

On Slideshare.com

Provided to conference as well

Who’s a User?

Already using continuous integration?

Continuous deployment?

Which tools?

Test first?

What would you say you do here?

Roles?

Responsibilities?

Why are you here?

What are your tests for this session?

Objectives

Understand Continuous Feedback in context of a healthy SDLC

Understand the Continuous Paradigm

Understand an end-to-end Continuous workflow

Understand how to scale your efforts

Outline

GROWS Methodology

Continuous paradigm

Battleship!

Continuous Integration (benefits & workflow)

Continuous Testing (benefits & workflow)

Miscellaneous tips & tricks

Breaks

Halfway (90 minutes)?

Each hour ?

Speak up!

Verifies

Use or delete Implements

Software Development Life Cycle

wikipedia.org/wiki/Waterfall_model#/media/File:Waterfall_model.svg

Traditional Approach

Requirements

Write Code

The Traditional Approach

Rewrite (60%)

“Discussion”

The architects generate a

document and hand it off

An isolated event, primarily worked by the

customer proxy

Code like mad!

We’re starting late!

Not what we meant!

Wrong archIgnored my docs

etcso on

Write Code

Requirements

A More Agile Approach

Before significant

time and $$ is invested

Discuss

Include the Golden Triad

Dev QA

Customer

Rewrite

Write Code

Architecture

Proof of Concept

Requirements Arch

Why Feedback?

Feedback is good for budgets

Cheaper

Faster

More accurate

Lower risk

flickr.com/photos/alancleaver/4293345631

To Err is HumanWhat’s your recovery

ExperimentationLearn about software by writing software

Can’t study your way into experience

Like riding a bike

Create feedback loops

flickr.com/photos/schwar/6801765768

Requirements

Rewrite

Discuss

Development

Node.js

Bootstrap

Angular.js

Pick one

Monolithic

Peer to peer

Pick one

Developersdisagree on

which architectureis best

Continue until you can demo

Developersdisagree on

which Javascripttoolkit is best

ReviseRewriteAdjust

Frequent demos discover

communication failures early

Discuss

IntentionalExperimentatio

nStudy

Experiment

Decide! Learn

Learn Decide!

Two Paths

GROWS Methodology

Directed Empiricism

Why Bother?

https://www.mountaingoatsoftware.com/blog/agile-succeeds-three-times-more-often-than-waterfall

AcceptanceWe’re not very good

We’re going to mess up

Mistakes are inevitable

How do we recover?

ContinuousParadigm

ExperimentationWe’re not very good

We’re going to mess up

Mistakes are inevitable

How do we recover?

STICK ExperimentsShort

Time boxed

Informative

Concrete

Specific

flickr.com/photos/peterpearson/1062279922

Automated TestsAnswer Questions

Is this done?

Does it still work?

Does it work somewhere besides my machine?

This is what I’m asking you to write.

Everything is checked in.

flickr.com/photos/bobowen/4501875907

Product or system

flickr.com/photos/30691679@N07/3808395213

Module/Sub

System

flickr.com/photos/kevino/1438502858

Feedback Loops are Fractal

flickr.com/photos/dancentury/4579906149

Agility Is...

Tighter Feedback Loops

Which Level?

Iterations

Months

Quarters

Optional

C*/Board

Director Level

Management Level

DeveloperLevel

flickr.com/photos/89228431@N06/11348332905

Minimize Our Dark Times

Code Change

Validation

Fast feedback leads to fast

flickr.com/photos/bogenfreund/556656621

Different Levels

Manual acceptance tests

Unit tests

Continuous Testing

Different Levels

Good at the bottom

Great at the top

Ignore the middle

Continuous Feedback

Continuous Integration

Continuous Delivery

Continuous Testing

Continuous Monitoring

Continuous deployment is controversial.

Most people ... think I’m advocating

low-quality code

or an undisciplined

cowboy-coding

development process.

Eric Ries

radar.oreilly.com/2009/03/continuous-deployment-5-eas.html

Language is a Tool

Use it

ExerciseHow long is your SDLC (idea to verification)?

What are your SDLC feedback bottlenecks?

Write down 3 worst bottlenecks

Gather into groups of 5 to 7

Share and discuss

Keep your list!

Take ten minutes

Watch Code

Build product

Run tests

Publish results

Fast feedbackKnown stateEasily consumed historyShines a light

Slows you downNags you after every code changeExposes problemsShares issuesForces you to deal with problemsSmoke alarm

53flickr.com/photos/117168856@N06/12486837493

Perception of Problem Accumulation

Actual Problem Accumulation

Continuous Problem Resolution

Sidebar: How We Learn

Dreyfus Model of Skills Acquisition

Familiar

Working

Smooth

Teaching

Dreyfus ModelofSkillsAcquisition

Recipes

Intuition

Principles

Key Points

Skills don’t map

59dilbert.com/strip/1990-09-13

Key Points

Beginners need steps

1. Use source code management (SCM)2. Check in ~everything~3. Automate the build (script)4. Acquire a build environment (isolated)5. Install build tools (dev tools)

Java Tooling

Publish results

Continuous Integration ServersManyJenkinsCruise ControlBamboo PulseTFS

Jenkins

Open sourceEstablishedSplit from HudsonJava applicationAny application serverSelf contained

Example

Publish results

javac (Oracle & IBM)message brokeretc

ws ejb deploymessage broker toolssshexec (to scripts)

JUnitHP's UFTSeleniumCucumber

Web pageRSSEmail

Common Scenarios

Dependent Builds

My build depends on another team’s buildNo problem

We’ll trigger your build based on their results

Dependent Builds

Services

Client/GUI

Data Layer

Complicated Dependencies

Our software is too complicated for CI69

Dependent Builds

Data Layer

Data Services

Rich Client

Common Libs

Web Portal

Biz Services

What’s Big?

3,000 developers300 major projects

All relatedPlatform to UI

Multiple branches

Common Integration Build

Chained Continuous Integration

Team 1 Team 2 Team 3

Integrationbuild

Multiple Platforms

Platform Testing

Team 1 Team 3Team 2 Team 4 Team 5

Integration Build

Platform1 Plat 3Plat 2 Plat 4 Plat 5

Battleship!flickr.com

/photos/puuikibeach/4121540002

Each player has

a board

flickr.com/photos/spookyam

d/6333317567

flickr.com/photos/jking89/1363983490

Place your fleet

Drop your bombs

Two Waysto Play

flickr.com/photos/compacflt/22797421725

flickr.com/photos/bz3rk/2436165683

Use all your ammo quickly

Use ammo one shot at a time

Round 1Start a timer

Place your fleet

Use all your “ammo”

Check with your opponent

Record hits/misses/time

Play twice

Round 2Start a timer

Place your fleet

Fire 1 round

Check with your opponent

Continue until out of ammo

Record hits/misses/time

Ships & Ammo

1 Sub (2 dot ship)

2 Destroyers (3 dots ships)

1 Battleship (4 dot ship)

1 Aircraft carrier (5 dot ship)

35 rounds to fire

ResultsWhich way is faster?

Which spends less “budget”?

Which way is more effective?

Which way do you prefer?

Feedback

Faster with feedback?

Slower with feedback?

Is slower bad?

Efficient or Effective

A B C D E F G H I J

Opponent’s Fleet

A B C D E F G H I J

My Fleet

A B C D E F G H I J

Opponent’s Fleet

A B C D E F G H I J

My Fleet

A B C D E F G H I

Time Shots Misses Hits Sinks % hits % sinks1st All at once

2nd All at once

3rd Interactive

4th Interactive

Battleship! An exercise by Tony Brill1 2 3 4

Continuous Testing

Watch Code

Build product

Run tests

Publish results

Continuous Testing

Watch CI

Deploy product

Integration tests

Publish results

Continuous Testing

Compile & unit test

Deploy &integration

Unit Tests

PublishContinuousIntegration

AgileArtisans.com

Unit Tests

Deploy to Dev

IntegrationTests

FirstLook

ContinuousTesting

AgileArtisans.com

SCMDevelopers

Unit Tests

Deploy to Dev

IntegrationTests

FirstLook

ContinuousTesting

On Demand Deployto QA

ManualTesting

AgileArtisans.com

Continuous Testing

Build in CI (compile & unit tests)

Deploy to dev (staging)

Run integration tests

What does this tell us?

If it works

By keeping fast feedback fastProviding slow feedback slowly

Don’t tie fast feedback to slow feedback

Fast vs Slow

“Fast”Integratio

nTests

Compiles& Unit Tests

SlowIntegratio

nTests

5-10 minutes

30 to 60

4 to 8 hours

What's this look like?

Continuous Testing

Compile & unit test

Deploy &integration

Multi Platform Deploys

CICD Win7

CD Linux

CD OS X

Exercise

Remember your SDLC bottlenecks? What could continuous integration solve?What could continuous testing solve?Regroup with your team and discussBring your top 3 solutions

As We Explore a Continuous World...

What Concepts Emerge?

Nuke and PaveReset (completely)Start overCompletely Scripted

BinariesNuke & PaveDeleteRecompileDeployUse

flickr.com/photos/x-ray_delta_one/4152356464

DataNuke & PaveDelete schemaDrop tablesRecreateRestockRailsLiquibase

Rails ActiveRecord

CREATE TABLE products (

id int(11) NOT NULL auto_increment,

name varchar(255),

PRIMARY KEY (id)

Rails ActiveRecord

change_table :card_deck do |table| tablet.remove :description, :name tablet.string :first_name tablet.index :part_number tablet.rename :list_name, :last_nameend

Rails ActiveRecordrake db:migraterake db:migrate VERSION=0rake db:reset

ActiveRecordrake db:resetDrops schemaRecreates schema

Operating System

Nuke & PaveVirtualizeRecreate

Steps for a New OS

Install VirtualBox

Install Vagrant

vagrant init hashicorp/precise32

vagrant up

Benefits

Scriptable

Reproducible environment

Solid results

Fast results

Repeatable results

Day one

Easiest

“If you don’t do it on Day One,you probably won’t”

Andy Hunt

andy.pragprog.com

Difficult

Necessary

Costs Move to Zero

Once scripted…

What changes?

How do We Improve Our Feedback Loops?

Developers

Devs write code

normally

Code is shared via a source code

management tool

ContinuousTesting

Providing immediate

feedback and a shared

environment for a QA

“First Look”

TriggeredDeployto QA

QA team members

trigger this deploy

ManualTesting

A stable environment for

manual and exploratory

testing

SCMDevelopers

Unit Tests

Deploy to Dev

IntegrationTests

FirstLook

ContinuousTesting

On Demand Deployto QA

ManualTesting

Rapid feedbackPair Dev & QA

First Look• QA can see what’s been coded immediately

• Differences in opinions can be discussed quickly

• Fewer problems during the larger testing cycles

• Keeps QA & Dev working together as partners

• Minimizes feedback for the previous iteration

• Faster feedback for QA

Continuous Testing

• Includes QA in the continuous feedback cycle

• Retains a traditional QA stable testing environment

• Dev & QA should both review all tests (unit & int.)

• QA requests trigger the QA env. deployment

Alternatives

flickr.com/photos/lightsinthedark/5810724330/

Work in a vacuum

Builds

Deploys

flickr.com/photos/pmillera4/9317924474

Ruthlessly Automate

Be Lazy!

You have builds scripted

CI servers running

Tests written

Things deploy

What else can you do?

flickr.com/photos/ofernandezberrios/2720567252

Environments

Staging

5 minutes

IntegrationTests

8:15am

8:20am

8:50am

8:25am

8:30am

8:35am

8:40am

8:45am

Traditionally…

Someone needs the appNotices it’s down

URGENT!FIX!

Fire drill

flickr.com/photos/kaibara/5332191131

Detect Problems

Before it’s urgentDetectNotify

Wait5 minutes

IntegrationTests

Continuous

MonitoringNotify

Every Platform

Feedback Loops

IdentifyReduceMinutesNot days

Never weeks

Months

flickr.com/photos/parkstreetparrot/9764446493

Stepsflickr.com

/photos/jalalspages/842122500

Continuous Integration (aka compiles) flickr.com/photos/jalalspages/842122500

Continuous Integration (aka compiles)

Test coverage numbers (aka unit tests)

flickr.com/photos/jalalspages/842122500

Continuous deployment (aka deploys to dev/test)

Continuous deployment (aka deploys to dev/test)Continuous testing (aka integration tests)

Continuous monitoring (test against environs)

Culture Change

Fundamental change in development

StepsAutomate builds

Add unit tests

Automate deployments

Continuous Delivery

Add integration testsflickr.com/photos/untitlism/

2547423465

Your Move

ExerciseAt lunch

Continuous TestingTwo 3x5 cards3 reasons for

3 barriers to adoptionGroups of 5 to 7

Brainstorm on solutions

flickr.com/photos/curtisperry/6119618990

ExerciseUseful or not...

Will you pursue or not...

Continuous Testing

Continuous Monitoring

Test automation

How will these ideas help or hurt your org?

Write down your answers and pass them up

GrowsMethod.com

AgileArtisans.com

jared@GrowsMethod.com

An Addendum

Test Automation

Unit tests

Acceptance Tests

Unit Tests

One class

One method

Illuminating

Integration Tests

High level

Lots of coverage

No detail

Continuous Feedback

Large amounts of code

Small amounts of change

Tests run after every change

Requires frequent code checkins

What's Better?

Unit or Integration?

Best Is...

AutomatedReliable

Frequently executed

Test LifecycleSet up stateUse product Verify results(Clean up)

Set Your Teams Up

Builds work

Functionality works

Biggest risk?

Requirements

Developers

Testers

Customers

The Team

Grid Driven Development

Remove technical barriers

Everyone speaks “Excel”

Testing Strategies

Test First is best

Defect Driven Testing

Blitzkrieg Testing

Open standards...

Junit or Junit compatible

Avoid vendor lock in

Glue tools

Time matters....

Fast servers

Incremental change

Script Tips

Save scripts in source code management

Modular

Basic encapsulation

One target => One function

Script Targetsall depends_on clean, compile, test-unit

clean # deletes build artifacts

compile # builds the product (Ruby?)

test-unit # run your tests

deploy_locally # to your dev environment

deploy_dev (?) # to a shared dev environment

test-integration # run your integration tests

Resources

JUnit JUnit.org

Jenkins Jenkins-ci.org

BDD Dannorth.net/introducing-bdd

Cucumber Cukes.info

Continuous Deployment en.wikipedia.org/wiki/Continuous_delivery

Devops Resources

VagrantUp.com

GetChef.com/chef

PuppetLabs.com

Ansible.com

VirtualBox.org

Data Resources

Guides.RubyOnRails.org/active_record_basics.html

Liquibase.org

FlywayDB.org

Additional Content

Material for you to review later…

Release Planning

Releases contain featuresHow many are ready to ship?

Releases contain features

Team ReleaseDate

ReleaseTheme

# Feature

StoriesStories

Complete%

Releases contain features

Team Release Features Stories StoriesComplete % Done

A 7/1 25 392 17 0.23

B 7/15 13 208 104 0.5

C 7/12 96 2,400 150 0.16

D 12/30 476 ?? 234 ?

Release dateBurn Up Chart

167Time

Features

Features in the release

Requiredrate of work

Will we hit our date?

168Time

Features

ACTUALrate of work

Burn Up Chart

169Time

Features

ACTUALrate of work

Feature Creep

170Time

FeaturesACTUAL

rate of work

Observe Progress

On track

Behind

Unrealistic

The Solution

Reduce features

DO NOT focus on increased work

Good or Not?

173Time

Features

ACTUALrate of work

What’s the Fix?

174Time

Features

How do I make the teamwork harder?

LongerHours

Work Faster!EvenLater

More Mistakes

Lots of Debugger

What’s the Fix?

176Time

Features

What’s the Fix?

177Time

Features

What’s the Fix?

178Time

Features

What Happens to Overloaded Planes?

179flickr.com/photos/jameski/5322886899

continuous testing in vegas

Software

improve security through continuous testing

continuous testing

continuous testing vs. test automation: three key...

white paper continuous testing - fisglobal.com ·...

continuous load testing

continuous performance testing

baseline testing provides opportunities for continuous...

continuous friction testing

enable continuous delivery with continuous testing-parasoft

continuous testing for continuous delivery

continuous performance testing - mhi

white paper continuous testing - fis · continuous testing...

continuous security testing - devseccon

enable continuous delivery with continuous testing -...

continuous testing: contents · this is where continuous...

ontap continuous integration/testing

unit testing & continuous integrat

acceptance testing for continuous...

exploratory testing for agile continuous testing

continuous testing 2016