Continuous Testing

C o r y F o yC o r y F o y , L L C

@ C o r y _ f o y

Who am I?

3

Cofounder of GROWS Methodology

Author Ship It! and Career 2.0

Screencast editor PragProg.com

2nd public signatory of the Agile Manifesto

Started AgileRTP in 2007!

Coach

AgileArtisans.com

Jared Richardson

4

CTO of Pretty Kool Apps, Founder of Cory Foy, LLC

Helped write the Software Craftsmanship Manifesto

Former Microsoft PFE

Development Clients include MIT, Beats By Dre

Global Agile + Lean Coach

Former Assistant Fire Chief

blog.coryfoy.com

Cory Foy

Who are you?

Who’s A User?

Already using continuous integration?

Continuous deployment?

Which tools?

What would you say you do here?

Roles?

Responsibilities?

Why are you here?

What are your tests for this session?

Objectives

Understand Continuous Feedback in context of SDLC

Understand the Continuous Paradigm

Understand the basics of Jenkins

Understand an end-to-end Continuous workflow

Miscellaneous practical tips and tricks

OutlineSDLC

Continuous paradigm

Continuous Integration (benefits & workflow)

Introduction to Jenkins

Continuous Testing (benefits & workflow)

Jenkins as a continuous deployment/testing tool

Miscellaneous tips & tricks

Breaks

Half-way (90 minutes)?

Each hour ?

SDLC Feedback

Requirements

Write Code

The Traditional Approach

Demo

Rewrite

“Discussion”

The architects generate a

document and hand it off

An isolated event, primarily worked by the

customer proxy

Arch

Code like mad! We’re starting late!

Not what we meant! Wrong arch

Ignored my docs etc

so on

Demo

Write Code

Requirements

The Agile Approach

Rewrite

Before significant time and $$ is

invested

Discuss

Include the Golden Triad

Dev QA

Customer

Demo

Write Code

Architecture

Proof of Concept

Requirements Arch

Technical Vision StatementDe

fine

Your

Goal

s

DB

Services

Front End

Design

System Vision StatementDe

fine

Your

Goal

s

Team

1

Team

2

Team

3

Feat A

Feat B

Feat C

Technical Vision

Busines

s Visio

n

Technical Product OwnerTe

st a

nd E

xper

imen

t Cont

inuo

usly

Team

1

Team

2

Team

3

Feat A

Feat B

Feat C

Technical Vision

Busines

s Visio

n

Experimentation

Learn about software by writing software

Can’t study your way into experience

Like riding a bike

3+ experiments

Create feedback loops

flickr.com/photos/schwar/6801765768

Exercise - Dot Game

Groups of 7

Roles: BA, PM, Developers, Tester

5 Minute Rounds

Requirements

Rewrite

Discuss

Development

Developers disagree on

which Javascript toolkit is best

Node.js

Bootstrap

Angular.js

Pick one

SOA

Monolithic

Peer to peer

Pick one

Developers disagree on

which architecture is best

Demo

Revise Rewrite Adjust

Frequent demos discover

communication failures early

Continue until you can demo

Discuss

Intentional Experimentation Study

Experiment

Learn

Decide! Learn

Learn Decide!

Two Paths

Expect Well Less Well Change

What we wanted to happen What went

well

What didn’t go so well

What we want to

try

ContinuousParadigm

What Is Software?

Feedback Loops

Feedback Loops

Discover What We Need

Discover How to Build ItBuild It

What Is Agile?

29

AgileManifesto.org

30 AgileManifesto.org

Individuals and interactions over processes and tools Working software over comprehensive documentation

Customer collaboration over contract negotiation Responding to change over following a plan

Feedback Loops

flickr.com/photos/bobowen/4501875907

Product or system

flickr.com/photos/30691679@N07/3808395213

Module/Sub System

flickr.com/photos/kevino/1438502858

Class

Feedback Loops are Fractal

flickr.com/photos/dancentury/4579906149

Pairing

Test FirstDemos

???

Feedback Loops Inside Feedback Loops

???

Systems are closed

Change is slow

Interdependencies are low

Certainty is high

Variability is low

Newtonian Management Methods Work:

Facilitating Organizational Change: Lessons from Complexity Science

System Agents

Emergent Patterns

Com

plex Adaptive System

Facilitating Organizational Change: Lessons from Complexity Science

Agile Is...

Tighter Feedback Loops

Which Level?

38

Initiative

Themes

Epics

Features

Stories

39

Initiative

Themes

Epics

Features

Stories Days

Iterations

Months

Quarters

Years

Optional

Optional

Initiative

Themes

Epics

Features

Stories

Management Level

40

Initiative

Themes

Epics

Features

Stories

DeveloperLevel

41

Minimize Dark Times

Code Change

Validation http://flickr.com/photos/bogenfreund/556656621/

Fast feedback leads to fast fixes

ExerciseCreate a Value Stream Map

Start with where work comes in to your org

Continue until it is in a customer’s hands

Each step gets a box

Map Time Map Anchors

One Vision One Voice One Team

Outcomesover

Output

Velocityover

Capacity

Throughputover

Utilization

Leadershipover

Management

286.5 / 1592.5

7 weeks of work40 weeks to do the 7 weeks

Different Levels

Manual acceptance tests

Unit tests

Continuous Testing

Continuous Feedback

Continuous Integration

Continuous Delivery

Continuous Testing

Continuous deployment is controversial.

Most people ... think I’m advocating low-quality code

or an undisciplined cowboy-coding

development process.

ERIC RIES radar.oreilly.com/2009/03/continuous-deployment-5-eas.html

Language is a Tool

Use it

Continuous Integration

Continuous Integration

Watch Code

Build product

Run tests

Publish results

Pros

Fast feedbackKnown stateEasily consumed historyShines a light

52

Cons

Slows you downNags you after every code changeExposes problemsShares issuesForces you to deal with problemsSmoke alarm 53flickr.com/photos/117168856@N06/12486837493

54

Work

Time

Perception of Problem Accumulation

55

Work

Time

Actual Problem Accumulation

56

Work

Time

Continuous Problem Resolution

DirectionsAtlasMapsPrinted maps!Navigation softwareWhat’s better?

57

Feedback Loops

Sidebar: How We Learn

Dreyfus Model of Skills Acquisition

59

60

Start

Familiar

Working

Smooth

Teaching

Dreyfus ModelofSkillsAcquisition

en.wikipedia.org/wiki/Dreyfus_model_of_skill_acquisition

61

Start

Familiar

Working

Smooth

Teaching

Dreyfus ModelofSkillsAcquisition

Steps

Recipes

Intuition

Stuck

Principles

Key Points

Skills don’t map

62dilbert.com/strip/1990-09-13

Key Points

Beginners need steps

63

StepsUse source code management (SCM)Automate the build (script)Acquire a build box (isolated)Install a build environment (dev tools)

64

Skunkworkable

Java Tooling

Publish results

Continuous Integration Servers

ManyJenkinsCruise ControlBamboo PulseTFSTravis CI

66

Jenkins

Open sourceEstablishedSplit from HudsonJava applicationAny application serverSelf contained

67

Example

Publish results

javac (Oracle & IBM)message brokeretc

ws ejb deploymessage broker toolssshexec (to scripts)

JUnitHP's UFTSelenium

Web pageRSSEmail

Dependent Builds

Services

Client/GUI

Data Layer

Dependent Builds

Data Layer

Data Services

Rich Client

Common Libs

Web Portal

Biz Services

Chained Continuous Integration

Team 1 Team 2 Team 3

Integrationbuild

Platform Testing

Team 1 Team 3Team 2 Team 4 Team 5

Integration Build

Platform1 Plat 3Plat 2 Plat 4 Plat 5

See JenkinsSample project Add to JenkinsExplore the GUI

73

Travis CI

RSpec / Guard

Continuous Testing

Continuous Integration

Watch Code

Build product

Run tests

Publish results

Continuous Testing

Watch CI

Deploy product

Integration tests

Publish results

Continuous Testing

CI CT

Compile & unit test

Deploy &integration

test

Continuous Testing

Build in CI (compile & unit tests)

Deploy to dev (staging)

Run integration tests

What does this tell us?

If it works

What's this look like?

Continuous Testing

CI CT

Compile & unit test

Deploy &integration

test

Multiplatform Deploys

CICD Win7

CD Linux

CD OS X

ExerciseRemember your SDLC bottlenecks? What could continuous integration solve? What could continuous testing solve?Regroup with your team and discussBring your top 3 solutions

83

Cut Anchors

As We Explore a Continuous World...

84

What Concepts Emerge?

85

Not Just CodeDatabasesServersHardware

86

A Continuous Paradigm affects all portions of our

ecosystem

Nuke and PaveReset (completely)Start overCompletely Scripted

87

BinariesNuke & PaveDeleteRecompileDeployUse

88flickr.com/photos/x-ray_delta_one/4152356464

DataNuke & PaveDelete schemaDrop tablesRecreateRestockRailsLiquibase

89flickr.com/photos/x-ray_delta_one/4152356464

Operating System

Nuke & PaveVirtualizeRecreate

90flickr.com/photos/x-ray_delta_one/4152356464

DatabasesDataSchemaBusiness Logic (Stored Procs)

91

Data UpdatesETL ProcessesData MappingMigrating Data

92

How can we apply continuous

paradigms here?Script Changes

Script Deploy

Write Tests

Schema / Stored ProcsTable StructureIndexesTriggersStored Procs

93

How can we apply continuous

paradigms here?Script Changes

Script Deploy

Write Tests

Raw SQL

CREATE TABLE products (   id int(11) NOT NULL auto_increment,   name varchar(255),   PRIMARY KEY  (id));

94

SQL Server Data Tools

95

SQL Server Data Tools

http://www.infoq.com/articles/tdd-dbpro-Foy

Rails ActiveRecord

change_table :card_deck do |table| table.remove :description, :name table.string :first_name table.index :part_number table.rename :list_name, :last_name end

96

Rails ActiveRecordrake db:migrate

Determines the best way to migratePreserves the dataIs reversible (rake db:rollback)

97

ActiveRecordrake db:reset

Drops schemaRecreates schema

98

Steps for a New OS

Install VirtualBoxInstall Vagrantvagrant init hashicorp/precise32vagrant up

99

Steps for a New OS

100

Benefits

Scriptable

Reproducible environment

Solid results

Fast results

Repeatable results

Immutable Infrastructure

“IT operations teams realized that it is far more expensive to debug and fix a faulty virtual server than to replace it with a new server”

Hidden Jobs One Off Processes

Production Hacks

When?

The best time is day one

The second best time is today

“If you don’t do it on Day One,you probably won’t”

Andy Hunt

104andy.pragprog.com

Easy?

No

Difficult

Necessary

Use agility in your approach!

Costs Move to Zero

Once scripted…

What changes?

How do We Improve Our Feedback Loops?

Developers

Devs write code normally

SCM

Code is shared via a source code management

tool

Continuous Testing

Providing immediate

feedback and a shared

environment for a QA “First Look”

Triggered Deploy to QA

QA team members trigger

this deploy

Manual Testing

A stable environment for

manual and exploratory testing

SCMDevelopers

Watch

Build

Unit Tests

Deploy to Dev

Integration Tests

First Look

Continuous Testing

On Demand Deploy to QA

Manual Testing

Rapid feedbackPair Dev & QA

First Look• QA can see what’s been coded immediately

• Differences in opinions can be discussed quickly

• Fewer problems during the larger testing cycles

• Keeps QA & Dev working together as partners

• Minimizes feedback for the previous iteration

• Faster feedback for QA

Continuous Testing

• Includes QA in the continuous feedback cycle

• Retains a traditional QA stable testing environment

• Dev & QA should both review all tests (unit & int.)

• QA requests trigger the QA env. deployment

Write Code

Requirements

The Traditional Approach

DemoDiscuss

Rewrite

Typically a confrontation &

argument

The demo forces a

discussion of the requirements

An isolated event, primarily worked by the customer proxy

Demo

Write Code

Requirements

The Agile Approach

Rewrite

Discuss before significant time is

invested

Include the Golden Triad

Discuss

Alternatives

flickr.com/photos/lightsinthedark/5810724330/

Work in a vacuum

Ruthlessly Automate!

Builds

Deploys

Tests

flickr.com/photos/pmillera4/9317924474

IN Summary...

Culture Change

Fundamental change in development

StepsAutomate builds

Continuous Integration

Add unit tests

Automate deployments

Continuous Delivery

Add integration testsflickr.com/photos/untitlism/2547423465

Your move

Thank you!

Cory Foy

foyc@coryfoy.com

@cory_foy