yahoo release management

19
Release Management From Packages to Production Philip J. Hollenback [email protected] April 2011

Upload: phil-hollenback

Post on 19-Jun-2015

836 views

Category:

Technology


0 download

DESCRIPTION

An overview of how we do software release management in Yahoo Mail.

TRANSCRIPT

Page 1: Yahoo Release Management

Release ManagementFrom Packages to Production

Philip J. [email protected]

April 2011

Page 2: Yahoo Release Management

Hi There

Yahoo! Mailmaybe you’ve heard of it?

Release Managementwe install the softwaressupported by many others in Service Engineering

Page 3: Yahoo Release Management

Environment

10k freebsd machinesOther teams handle the hardwareWe focus on just the software installs

Page 4: Yahoo Release Management

Houston?

Core mail backend servers means lots of legacy codeAbout 1500 software packages

Mail must be 100% reliablePeople expect email to always be available

It’s hard to load test mail service

Page 5: Yahoo Release Management

Solution

Distributed, cached software repositories with distKeep system state in IgorAssemble releases with SRMMassively parallel pushes with PogoRigorous release cycleDO IT FAST

(3 week release timeline)

Page 6: Yahoo Release Management

Release Philosophy

All Packages All EnvironmentsPackages must be installed on all servers of a given typeOnly way we can properly test releasesAny other solution leads to many separate ’releases’

Page 7: Yahoo Release Management

Which brings us to. . .

Page 8: Yahoo Release Management
Page 9: Yahoo Release Management

Release Trains

Start a new train every 3 weeksRelease Management and QA hand release back and forthRelease assembly with SRMQA does integrationQA final test & signoffPush to Prod in about a week (with Pogo)

That’s 10k hosts worldwide, folks

Page 10: Yahoo Release Management

Delays?

Train delays suckSee the part about DOING IT FAST

too much backup means we cancelroll changes into next train

QA is limiting factor

Page 11: Yahoo Release Management

Massively Parallel Pushes

Launch software push on lots of hosts at oncebut with constraints & healthcheck

Starting a push to about 3000 hosts

$ pogo run -R Train_34 -I @mail.farm.xset.deploy-stage4-usPassword:p0000031421; http://pogo.corp.yahoo.com/p0000031421status: running: constraints computed

Page 12: Yahoo Release Management
Page 13: Yahoo Release Management

Job Complete

10 hours to push to just over 3000 servers:

End of Push

CMR completed successfullyStart Wed Apr 13 2011 22:24:45 GMT-0700 (PDT)End Thu Apr 14 2011 08:24:44 GMT-0700 (PDT) (+10h0s)Total Hosts : 3046 (Skipped 36 invalid hosts)Finished : 3038Completion % : 99.73%

ps - open source pogo can be found athttps://github.com/nrh/pogo/

Page 14: Yahoo Release Management

Yes, We Do Rollbacks

Rigorous Release Testingforward and backwardbare metal

Full rollback roughly every 6 months

Page 15: Yahoo Release Management

What Works

massively parallel pushesrollbackrigorous release testing

Page 16: Yahoo Release Management

Challenges

getting developers to care about old codepackage quality, install/remove scriptsseparating bugfix and feature trainsmoving settings to central config servers

Page 17: Yahoo Release Management

To Sum It Up

DON’T BE CLEVER!

Install the same packages everywhereTest releases thoroughly before rolloutThe simple approach is most likely to succeed

Page 18: Yahoo Release Management

Thanks for Listening!

Phil [email protected]

@philiphhttp://www.hollenback.net

This Is A Team EffortJen DraperJerrod KensilBrian McNeffJoon Kim

Shajeeb MuhammadPrem RamnathTisha EmmanuelPradipta Ghosh

Page 19: Yahoo Release Management

Any Questions?