Extending Continuous Integration

About the speaker

• Johannes Brodwall

• Lame duck Lead software architect at BBS

• Organizer Oslo XP meetup

• Organizer Smidig 2008

• Blog: http://brodwall.com/johannes/blog

.meetup.com/13

What’s the point of Continuous Integration?

•Pay now to insure against defects

•Pay less to deploy to production

•Release whenever you want

What is Continuous Integration?

•Unit test

•Functional tests

System run in test harness

Result:

Every check in is is now tested

Will this find all our defects?

No!

• Limited to our test harness

• Limited to our imagination

• And then there’s performance

So what should we do?

(Hint: CI is about paying now to avoid paying later)

Pay more now!

Our approach: Automated system test

•Realistic setup

•Realistic load

•Realistic variation

How to automate system tests

1.Automated build (and test)

2.Scrap old data

3.Install latest snapshot

4.Download production data

5.Replay production data

6.Verify result

7.Send results and logs via email

”Did you meet any problems?”

No

No,Only challenges!

The hard bits

1.Installation

2.Integration

3.Simulation

4.Verification

Installation

How to automate installation

1.Easy, scripted, always working install

2.Simplify(Replace WebSphere with Jetty)

(Combine components)

3.Reduce integration

4.All nodes look the same

(Side effects)

•Easy installation

•Simpler design

•Simpler architecture

Integration

IntegrationDealing with dependencies

Dealing with integration

1.Don’t integrate - Do it yourself!

2.Simulate other system1.Fake responses

2.Keep canned data (data centric)

3.Integrate with test version

Simulation

SimulationPut realistic load on the system

SimulationDepends on what you system does

How to simulate production

• In our case: Files

• Crawler (Dyrkorn & Watne)

• Load generator (D&W)

• Record HTTP requests

Verification

VerificationFinding out if it worked

How to verify results

• Compare with production

• Verify integrity

• Check logs

Comparing

1.Store test result in database

2.Store production result in database

3.Full outer join on key fields

4.Find missing or mismatched status

5.Filter out know deviations

Date

Number of files

Okay

Missing

Extra

ExtraMismatched

ExtraExtraExtraKnown deviations

Consistency checks

SQL expressions that pick out things that are weird

Logging

It matters!

Error logs should be empty if everything is okay

Result:

Every build is now system tested

Will this find all our defects?

No!

• Limited integration

• Limited stability

No!Automated staging

No!Automated staging

Automated staging

• ”Next” version

• Lock-step with production

• Promote after a week

• Monitor 9:00-16:00

Only when you can think as an operator, can you master your system

Result:

Every release is hardened

Will this find all our defects?

No!

• Wrong requirements

• Poor solutions

• New user behavior exposes bugs

• Bugs we didn’t care enough about

The goal:Release after every iteration

The sad reality:Pilot release after every

iteration

Pilotproduction

Why releases every iteration?

• Find more bugs

• Try the easy solution first

• Find new requirement faster

Exploit opportunity

Result:

Compiler Unit test Systemtest Staging Prod

Defect cost

RealismFind all the bugs cheaply!

Make sure it always works

Pay more now to pay less later

The goal:Release after every iteration

(And throw away the bugtracker)

Thank you!