continuous testing in vegas
TRANSCRIPT
Continuous TestingJared Richardson
Agile Artisans
@JaredRichardson
GrowsMethod.com
Who am I?
3
Cofounder of GROWS Methodology
Author Ship It! and Career 2.0
Screencast editor PragProg.com
2nd public signatory of the Agile Manifesto
Started AgileRTP in 2007
Coach
AgileArtisans.com
Jared Richardson
Who are you?
What’s up with the material?
Major rewrite
Introduction
Slides on USB
On Slideshare.com
Provided to conference as well
Who’s a User?
Already using continuous integration?
Continuous deployment?
Which tools?
Test first?
What would you say you do here?
Roles?
Responsibilities?
Why are you here?
What are your tests for this session?
Objectives
Understand Continuous Feedback in context of a healthy SDLC
Understand the Continuous Paradigm
Understand an end-to-end Continuous workflow
Understand how to scale your efforts
Outline
GROWS Methodology
Continuous paradigm
Battleship!
Continuous Integration (benefits & workflow)
Continuous Testing (benefits & workflow)
Miscellaneous tips & tricks
Breaks
Halfway (90 minutes)?
Each hour ?
Speak up!
SDLC
Idea
Verifies
Use or delete Implements
Software Development Life Cycle
wikipedia.org/wiki/Waterfall_model#/media/File:Waterfall_model.svg
Traditional Approach
Requirements
Write Code
The Traditional Approach
Demo
Rewrite (60%)
“Discussion”
The architects generate a
document and hand it off
An isolated event, primarily worked by the
customer proxy
Arch
Code like mad!
We’re starting late!
Not what we meant!
Wrong archIgnored my docs
etcso on
Demo
Write Code
Requirements
A More Agile Approach
Before significant
time and $$ is invested
Discuss
Include the Golden Triad
Dev QA
Customer
Rewrite
Demo
Write Code
Architecture
Proof of Concept
Requirements Arch
Why Feedback?
Feedback is good for budgets
Cheaper
Faster
More accurate
Lower risk
flickr.com/photos/alancleaver/4293345631
To Err is HumanWhat’s your recovery
plan?
ExperimentationLearn about software by writing software
Can’t study your way into experience
Like riding a bike
Create feedback loops
flickr.com/photos/schwar/6801765768
Requirements
Rewrite
Discuss
Development
Node.js
Bootstrap
Angular.js
Pick one
SOA
Monolithic
Peer to peer
Pick one
Developersdisagree on
which architectureis best
Continue until you can demo
Developersdisagree on
which Javascripttoolkit is best
Demo
ReviseRewriteAdjust
Frequent demos discover
communication failures early
Discuss
IntentionalExperimentatio
nStudy
Experiment
Learn
Decide! Learn
Learn Decide!
Two Paths
GROWS Methodology
Directed Empiricism
Why Bother?
https://www.mountaingoatsoftware.com/blog/agile-succeeds-three-times-more-often-than-waterfall
AcceptanceWe’re not very good
We’re going to mess up
Mistakes are inevitable
How do we recover?
ContinuousParadigm
ExperimentationWe’re not very good
We’re going to mess up
Mistakes are inevitable
How do we recover?
STICK ExperimentsShort
Time boxed
Informative
Concrete
Key
Specific
flickr.com/photos/peterpearson/1062279922
Automated TestsAnswer Questions
Is this done?
Does it still work?
Does it work somewhere besides my machine?
This is what I’m asking you to write.
Everything is checked in.
flickr.com/photos/bobowen/4501875907
Product or system
flickr.com/photos/30691679@N07/3808395213
Module/Sub
System
flickr.com/photos/kevino/1438502858
Class
Feedback Loops are Fractal
flickr.com/photos/dancentury/4579906149
Agility Is...
Tighter Feedback Loops
Which Level?
35
36
Days
Iterations
Months
Quarters
Years
Optional
Optional
C*/Board
37
VP
38
Director Level
39
Management Level
40
DeveloperLevel
41
flickr.com/photos/89228431@N06/11348332905
Minimize Our Dark Times
Code Change
Validation
Fast feedback leads to fast
fixes
flickr.com/photos/bogenfreund/556656621
Different Levels
Manual acceptance tests
Unit tests
Continuous Testing
Different Levels
Good at the bottom
Great at the top
Ignore the middle
Continuous Feedback
Continuous Integration
Continuous Delivery
Continuous Testing
Continuous Monitoring
Continuous deployment is controversial.
Most people ... think I’m advocating
low-quality code
or an undisciplined
cowboy-coding
development process.
Eric Ries
radar.oreilly.com/2009/03/continuous-deployment-5-eas.html
Language is a Tool
Use it
ExerciseHow long is your SDLC (idea to verification)?
What are your SDLC feedback bottlenecks?
Write down 3 worst bottlenecks
Gather into groups of 5 to 7
Share and discuss
Keep your list!
Take ten minutes
Continuous Integration
Continuous Integration
Watch Code
Build product
Run tests
Publish results
Pros
Fast feedbackKnown stateEasily consumed historyShines a light
52
Cons
Slows you downNags you after every code changeExposes problemsShares issuesForces you to deal with problemsSmoke alarm
53flickr.com/photos/117168856@N06/12486837493
54
Work
Time
Perception of Problem Accumulation
55
Work
Time
Actual Problem Accumulation
56
Work
Time
Continuous Problem Resolution
Sidebar: How We Learn
Dreyfus Model of Skills Acquisition
57
58
Start
Familiar
Working
Smooth
Teaching
Dreyfus ModelofSkillsAcquisition
Steps
Recipes
Intuition
Stuck
Principles
Key Points
Skills don’t map
59dilbert.com/strip/1990-09-13
Key Points
Beginners need steps
60
Steps
1. Use source code management (SCM)2. Check in ~everything~3. Automate the build (script)4. Acquire a build environment (isolated)5. Install build tools (dev tools)
61
Java Tooling
Publish results
Continuous Integration ServersManyJenkinsCruise ControlBamboo PulseTFS
63
Jenkins
Open sourceEstablishedSplit from HudsonJava applicationAny application serverSelf contained
64
Example
Publish results
javac (Oracle & IBM)message brokeretc
ws ejb deploymessage broker toolssshexec (to scripts)
JUnitHP's UFTSeleniumCucumber
Web pageRSSEmail
Common Scenarios
66
Dependent Builds
My build depends on another team’s buildNo problem
We’ll trigger your build based on their results
67
Dependent Builds
Services
Client/GUI
Data Layer
Complicated Dependencies
Our software is too complicated for CI69
flick
r.com
/pho
tos/
chao
ticm
ind7
5/16
1697
2227
0/
Dependent Builds
Data Layer
Data Services
Rich Client
Common Libs
Web Portal
Biz Services
What’s Big?
3,000 developers300 major projects
All relatedPlatform to UI
Multiple branches
71
Common Integration Build
72
Chained Continuous Integration
Team 1 Team 2 Team 3
Integrationbuild
Multiple Platforms
74
Platform Testing
Team 1 Team 3Team 2 Team 4 Team 5
Integration Build
Platform1 Plat 3Plat 2 Plat 4 Plat 5
Battleship!flickr.com
/photos/puuikibeach/4121540002
Each player has
a board
flickr.com/photos/spookyam
d/6333317567
flickr.com/photos/jking89/1363983490
Place your fleet
Drop your bombs
Two Waysto Play
flickr.com/photos/compacflt/22797421725
flickr.com/photos/bz3rk/2436165683
Use all your ammo quickly
Use ammo one shot at a time
Round 1Start a timer
Place your fleet
Use all your “ammo”
Check with your opponent
Record hits/misses/time
Play twice
Round 2Start a timer
Place your fleet
Fire 1 round
Check with your opponent
Continue until out of ammo
Record hits/misses/time
Ships & Ammo
1 Sub (2 dot ship)
2 Destroyers (3 dots ships)
1 Battleship (4 dot ship)
1 Aircraft carrier (5 dot ship)
35 rounds to fire
ResultsWhich way is faster?
Which spends less “budget”?
Which way is more effective?
Which way do you prefer?
Feedback
Faster with feedback?
Slower with feedback?
Is slower bad?
Efficient or Effective
A B C D E F G H I J
1
2
3
4
5
6
7
8
9
10
Opponent’s Fleet
A B C D E F G H I J
1
2
3
4
5
6
7
8
9
10
My Fleet
A B C D E F G H I J
1
2
3
4
5
6
7
8
9
10
Opponent’s Fleet
A B C D E F G H I J
1
2
3
4
5
6
7
8
9
10
My Fleet
A B C D E F G H I
1
2
3
4
5
6
7
8
9
A B C D E F G H I
1
2
3
4
5
6
7
8
9
A B C D E F G H I
1
2
3
4
5
6
7
8
9
A B C D E F G H I
1
2
3
4
5
6
7
8
9
A B C D E F G H I
1
2
3
4
5
6
7
8
9
A B C D E F G H I
1
2
3
4
5
6
7
8
9
A B C D E F G H I
1
2
3
4
5
6
7
8
9
A B C D E F G H I
1
2
3
4
5
6
7
8
9
Time Shots Misses Hits Sinks % hits % sinks1st All at once
2nd All at once
3rd Interactive
4th Interactive
Battleship! An exercise by Tony Brill1 2 3 4
Continuous Testing
Continuous Integration
Watch Code
Build product
Run tests
Publish results
Continuous Testing
Watch CI
Deploy product
Integration tests
Publish results
Continuous Testing
CI CT
Compile & unit test
Deploy &integration
test
Watch
Build
Unit Tests
PublishContinuousIntegration
AgileArtisans.com
Watch
Build
Unit Tests
Deploy to Dev
IntegrationTests
FirstLook
ContinuousTesting
AgileArtisans.com
SCMDevelopers
Watch
Build
Unit Tests
Deploy to Dev
IntegrationTests
FirstLook
ContinuousTesting
On Demand Deployto QA
ManualTesting
AgileArtisans.com
Continuous Testing
Build in CI (compile & unit tests)
Deploy to dev (staging)
Run integration tests
What does this tell us?
If it works
How?
By keeping fast feedback fastProviding slow feedback slowly
Don’t tie fast feedback to slow feedback
Fast vs Slow
“Fast”Integratio
nTests
Compiles& Unit Tests
SlowIntegratio
nTests
5-10 minutes
30 to 60
4 to 8 hours
What's this look like?
Continuous Testing
CI CT
Compile & unit test
Deploy &integration
test
Multi Platform Deploys
CICD Win7
CD Linux
CD OS X
Exercise
Remember your SDLC bottlenecks? What could continuous integration solve?What could continuous testing solve?Regroup with your team and discussBring your top 3 solutions
99
As We Explore a Continuous World...
100
What Concepts Emerge?
101
Nuke and PaveReset (completely)Start overCompletely Scripted
102
BinariesNuke & PaveDeleteRecompileDeployUse
103
flickr.com/photos/x-ray_delta_one/4152356464
DataNuke & PaveDelete schemaDrop tablesRecreateRestockRailsLiquibase
104
flickr.com/photos/x-ray_delta_one/4152356464
Rails ActiveRecord
CREATE TABLE products (
id int(11) NOT NULL auto_increment,
name varchar(255),
PRIMARY KEY (id)
);
105
Rails ActiveRecord
change_table :card_deck do |table| tablet.remove :description, :name tablet.string :first_name tablet.index :part_number tablet.rename :list_name, :last_nameend
106
Rails ActiveRecordrake db:migraterake db:migrate VERSION=0rake db:reset
107
ActiveRecordrake db:resetDrops schemaRecreates schema
108
Operating System
Nuke & PaveVirtualizeRecreate
109
flickr.com/photos/x-ray_delta_one/4152356464
Steps for a New OS
Install VirtualBox
Install Vagrant
vagrant init hashicorp/precise32
vagrant up
110
Benefits
Scriptable
Reproducible environment
Solid results
Fast results
Repeatable results
When?
Day one
Easiest
“If you don’t do it on Day One,you probably won’t”
Andy Hunt
113
andy.pragprog.com
Easy?
No
Difficult
Necessary
Costs Move to Zero
Once scripted…
What changes?
How do We Improve Our Feedback Loops?
Developers
Devs write code
normally
SCM
Code is shared via a source code
management tool
ContinuousTesting
Providing immediate
feedback and a shared
environment for a QA
“First Look”
TriggeredDeployto QA
QA team members
trigger this deploy
ManualTesting
A stable environment for
manual and exploratory
testing
SCMDevelopers
Watch
Build
Unit Tests
Deploy to Dev
IntegrationTests
FirstLook
ContinuousTesting
On Demand Deployto QA
ManualTesting
Rapid feedbackPair Dev & QA
First Look• QA can see what’s been coded immediately
• Differences in opinions can be discussed quickly
• Fewer problems during the larger testing cycles
• Keeps QA & Dev working together as partners
• Minimizes feedback for the previous iteration
• Faster feedback for QA
Continuous Testing
• Includes QA in the continuous feedback cycle
• Retains a traditional QA stable testing environment
• Dev & QA should both review all tests (unit & int.)
• QA requests trigger the QA env. deployment
Alternatives
flickr.com/photos/lightsinthedark/5810724330/
Work in a vacuum
Builds
Deploys
Tests
flickr.com/photos/pmillera4/9317924474
Ruthlessly Automate
Be Lazy!
You have builds scripted
CI servers running
Tests written
Things deploy
What else can you do?
flickr.com/photos/ofernandezberrios/2720567252
Dev
Environments
QA
Staging
Prod
5 minutes
IntegrationTests
8:15am
8:20am
8:50am
8:25am
8:30am
8:35am
8:40am
8:45am
Traditionally…
Someone needs the appNotices it’s down
URGENT!FIX!
Fire drill
flickr.com/photos/kaibara/5332191131
Detect Problems
Before it’s urgentDetectNotify
Fix
Wait5 minutes
IntegrationTests
Continuous
MonitoringNotify
Every Platform
Feedback Loops
IdentifyReduceMinutesNot days
Never weeks
Months
flickr.com/photos/parkstreetparrot/9764446493
Stepsflickr.com
/photos/jalalspages/842122500
Steps
Continuous Integration (aka compiles) flickr.com/photos/jalalspages/842122500
Steps
Continuous Integration (aka compiles)
Test coverage numbers (aka unit tests)
flickr.com/photos/jalalspages/842122500
Steps
Continuous Integration (aka compiles)
Test coverage numbers (aka unit tests)
Continuous deployment (aka deploys to dev/test)
flickr.com/photos/jalalspages/842122500
Steps
Continuous Integration (aka compiles)
Test coverage numbers (aka unit tests)
Continuous deployment (aka deploys to dev/test)Continuous testing (aka integration tests)
flickr.com/photos/jalalspages/842122500
Steps
Continuous Integration (aka compiles)
Test coverage numbers (aka unit tests)
Continuous deployment (aka deploys to dev/test)Continuous testing (aka integration tests)
Continuous monitoring (test against environs)
flickr.com/photos/jalalspages/842122500
Culture Change
Fundamental change in development
StepsAutomate builds
Continuous Integration
Add unit tests
Automate deployments
Continuous Delivery
Add integration testsflickr.com/photos/untitlism/
2547423465
Your Move
ExerciseAt lunch
Continuous TestingTwo 3x5 cards3 reasons for
3 barriers to adoptionGroups of 5 to 7
Brainstorm on solutions
flickr.com/photos/curtisperry/6119618990
ExerciseUseful or not...
Will you pursue or not...
Continuous Integration
Continuous Testing
Continuous Monitoring
Test automation
How will these ideas help or hurt your org?
Write down your answers and pass them up
An Addendum
Test Automation
Unit tests
Acceptance Tests
Unit Tests
One class
One method
Fast
Illuminating
Integration Tests
High level
Lots of coverage
No detail
Continuous Feedback
Large amounts of code
Small amounts of change
Tests run after every change
Requires frequent code checkins
What's Better?
Unit or Integration?
150
Best Is...
AutomatedReliable
Frequently executed
151
Test LifecycleSet up stateUse product Verify results(Clean up)
152
Set Your Teams Up
Builds work
Functionality works
Biggest risk?
Requirements
Developers
Testers
Customers
The Team
Grid Driven Development
Remove technical barriers
Everyone speaks “Excel”
Testing Strategies
Test First is best
Defect Driven Testing
Blitzkrieg Testing
Tips
Open standards...
Junit or Junit compatible
Avoid vendor lock in
Glue tools
Time matters....
Fast servers
Incremental change
Script Tips
Save scripts in source code management
Modular
Basic encapsulation
One target => One function
Script Targetsall depends_on clean, compile, test-unit
clean # deletes build artifacts
compile # builds the product (Ruby?)
test-unit # run your tests
deploy_locally # to your dev environment
deploy_dev (?) # to a shared dev environment
test-integration # run your integration tests
Resources
JUnit JUnit.org
Jenkins Jenkins-ci.org
BDD Dannorth.net/introducing-bdd
Cucumber Cukes.info
Continuous Deployment en.wikipedia.org/wiki/Continuous_delivery
Devops Resources
VagrantUp.com
GetChef.com/chef
PuppetLabs.com
Ansible.com
VirtualBox.org
Data Resources
Guides.RubyOnRails.org/active_record_basics.html
Liquibase.org
FlywayDB.org
Additional Content
Material for you to review later…
Release Planning
Releases contain featuresHow many are ready to ship?
164
Releases contain features
165
Team ReleaseDate
ReleaseTheme
# Feature
s#
StoriesStories
Complete%
Done
1
2
3
Releases contain features
Team Release Features Stories StoriesComplete % Done
A 7/1 25 392 17 0.23
B 7/15 13 208 104 0.5
C 7/12 96 2,400 150 0.16
D 12/30 476 ?? 234 ?
Release dateBurn Up Chart
167Time
Features
Features in the release
Requiredrate of work
Will we hit our date?
168Time
Features
ACTUALrate of work
Burn Up Chart
169Time
Features
ACTUALrate of work
Feature Creep
170Time
FeaturesACTUAL
rate of work
Observe Progress
On track
Behind
Unrealistic
171
The Solution
Reduce features
DO NOT focus on increased work
172
Good or Not?
173Time
Features
ACTUALrate of work
What’s the Fix?
174Time
Features
How do I make the teamwork harder?
175
LongerHours
Work Faster!EvenLater
More Mistakes
Lots of Debugger
Time
What’s the Fix?
176Time
Features
What’s the Fix?
177Time
Features
What’s the Fix?
178Time
Features
What Happens to Overloaded Planes?
179flickr.com/photos/jameski/5322886899