puppet camp tokyo 2014: fireballs, ice bats and 1,000,000 plugins: a story of continuous delivery

Otto Jongerius Build Engineering @ Atlassian

@xzu

Fireballs, Icebats and 1,000,000 Plugins

PuppetCamp Tokyo - May 2014

Continuous Delivery (a refresher)

Develop => Build => Test => Release!

Continuous Delivery (a refresher)

Develop => Build => Test => Release!

smaller change delta

more feedbackfaster feedback

automated upgrade smaller magnitude of failure

Our happy path

0

200

400

600

800

2011 2012 2013 2014 2015

Deployments/year

quarterly fortnightly weekly

daily

many/day

A little bit of history…

A little bit of history…

0

12500

25000

37500

50000

October 2011 June 2012 January 2013 June 2013 April 2014

50.000

32.000

24.000

15.000

500

Approximate number of OnDemand instances

A little bit of history…

Atlassian Application

Application Core

Application Bundled Plugins

plugin1 plugin2

plugin3 plugin4

pluginN

A little bit of history…

Atlassian Application

Application Core

Application Bundled Plugins

plugin1 plugin2

plugin3 plugin4

pluginN

Alice

Bob

Charlie

DaveEd

FrankGerry

A little bit of history…bash$ cat jira/pom.xml … <dependency> <groupId>com.atlassian.jira</groupId> <artifactId>jira-api</artifactId> <version>${project.version}</version> </dependency> …

A little bit of history…

Atlassian Application

Application Core

Application Bundled Plugins

plugin1 plugin2

plugin3 plugin4

pluginN

Atlassian Application

Application Core

Application Bundled Plugins

plugin1 plugin2

plugin3 plugin4

pluginN

Atlassian Application

Application Core

Application Bundled Plugins

plugin1 plugin2

plugin3 plugin4

pluginN

Enter the fireball

Enter the fireballbash$ cat fireball/pom.xml … <dependency> <GroupId>${jira.group.id}</GroupId> <artifactId>${jira.war.artifact.id}</artifactId> <version>${jira.war.version}</version> … </dependency> …

Launching the fireballOpenVZ Container

Postgres

ApacheConfluence

JIRA

Bamboo

CrowdCus

tom

er In

stan

ce

NFS

Launching the fireball

Datacenter

Rack Rack Rack RackRack

Rack Rack Rack RackRack

Rack Rack Rack RackRack

Rack Rack Rack RackRack

Rack

Launching the fireball

bash$ sudo /path/to/new.version/install.sh

INSTALLED_SERVICES=“postgres jira bamboo confluence crowd apache” for service in ${INSTALLED_SERVICES}; do /path/to/new.version/$service/install.sh done

…

OR

http://www.flickr.com/photos/64577732@N00/1414883135http://www.flickr.com/photos/44124473769@N01/292642699

Launching the fireball

Right?

A little bit of history…

jira/install.sh

postgres/install.sh

confluence/install.sh

bamboo/install.sh

crowd/install.sh

* (over) simplified model

A little bit of history…

jira/install.sh

postgres/install.sh

confluence/install.sh

bamboo/install.sh

crowd/install.sh

* (over) simplified model

A little bit of history…

jira/install.sh

postgres/install.sh

confluence/install.sh

bamboo/install.sh

crowd/install.sh

* (over) simplified modelhttp://www.flickr.com/photos/64577732@N00/1414883135

A little bit of history…

http://www.flickr.com/photos/7337467@N04/3051244623

Our happy path

0

200

400

600

800

2011 2012 2013 2014 2015

Deployments/year Progress

quarterly fortnightly weekly

daily

many/day

You are here

A little bit of history…

Atlassian Application

Application Core

Application Bundled Plugins

plugin1 plugin2

plugin3 plugin4

pluginN

Alice

Bob

Charlie

DaveEd

FrankGerry

Enter the icebat

Fire + Ball

Ice + Bat

Enter the icebat

j2ee-appjira

postgres

crowd

confluence

bamboo

apache

j2ee-plugin

vendor plugin

Enter the icebat

j2ee-appjira

postgres

crowd

confluence

bamboo

apache

j2ee-plugin

vendor plugin

Enter the icebat

j2ee-appjira

postgres

crowd

confluence

bamboo

apache

j2ee-plugin

vendor plugin

Enter the icebat

j2ee-appjira

postgres

crowd

confluence

bamboo

apache

j2ee-plugin

vendor plugin

Enter the icebat

j2ee-appjira

postgres

crowd

confluence

bamboo

apache

j2ee-plugin

vendor plugin

postgres

Enter the icebat

j2ee-appjira

postgres

crowd

confluence

bamboo

apache

j2ee-plugin

vendor plugin

postgres5.4

6.1

5.0

2.7.5

1.1

hier

a - v

ersi

ons.

yam

l

Enter the icebat

Development5.4m3

6.2-snapshot

5.1-m1

2.7.5-m6

1.2.1-snapshot

hier

a - v

ersi

ons.

yam

lDogfooding Production

5.4m2.1

6.1m7

5.1

2.7.5-m5

1.2hi

era

- ver

sion

s.ya

ml 5.4m2

6.1m6

5.0-m1

2.7.5-m2

1.1-m1

hier

a - v

ersi

ons.

yam

l

Local overridesglobal configuration

datacenter configurationrack configurationlocal configuration

http://www.flickr.com/photos/streamishmc/4936051/

Hard Labour

Jason ‘Mechanical Turk’ Birch

The Great Leap Forward

The Great Leap Forward

The Great Leap Forward

Applications

The Great Leap Forward

Plugins

The Great Leap Forward

Environments

The Great Leap ForwardVersions

The great leap forward

The great leap forward

“client”

The great leap forward

“client”

facterenvironment

licensed products

The great leap forward

“client”

facterenvironment

licensed products

manifesto

The great leap forward

“client”

facterenvironment

licensed products

puppet apply

manifesto

Our happy path

0

200

400

600

800

2011 2012 2013 2014-01 2014-12 2015

Deployments/year Progress

quarterly fortnightly weekly

many/week

many/day

You are here

twice daily

Next Steps

• Onboard everything into Manifesto, such that development teams control their own destinies • Automated acceptance testing • Fault tolerance for manifesto • Dark feature (feature flags) control within Manifesto/icebat

Special thanks to…

Ryan ThomasJason BirchMatt Moor

Thanks!

Dev OpsDevs who can OpOps who can Dev

We’re hiring! atlassian.com/jobs (or speak to me)