from 12 to 3500 deployments per year in production
DESCRIPTION
Practices that enabled ITHAKA's engineering team to increase its change velocity in production from 12 releases per year to over 70 per week on average.TRANSCRIPT
![Page 1: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/1.jpg)
FROM 12 TO 3500DEPLOYMENTS PER YEARHOW ITHAKA INCREASED ITS VELOCITY TO 70 DEPLOYMENTS
PER WEEK IN PRODUCTIONby @archiecowan
![Page 2: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/2.jpg)
ABOUT JSTOR ANDITHAKA
JSTOR is a digital library of more than 2,000academic journals, books, and primary sources.
JSTOR helps people discover, use, and buildupon a wide range of content through a powerful
research and teaching platform, and preservesthis content for future generations. JSTOR is part
of ITHAKA, a not-for-profit organization thatalso includes Ithaka S+R and Portico.
![Page 3: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/3.jpg)
ABOUT ARCHIEWorking in tech since 2002Held titles like
DeveloperQuality Assurance EngineerSoftware EngineerUI DeveloperLead Software EngineerOperationsTechnical ArchitectEnterprise Architect
Enjoy working on ...Distributed SystemsHigh AvailbilityAnalyticsWebsitesCloudy CloudsGetting stuff done
![Page 4: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/4.jpg)
WHEN DOES YOUR USERREALIZE VALUE FROM
YOUR PRODUCT?
![Page 5: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/5.jpg)
WHEN THEY CAN USE IT?
![Page 6: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/6.jpg)
WHEN IT'S INPRODUCTION!
1. Users use production2. The sooner you get changes to production, the sooner you
know you did the right thing for your user.3. The more agile and safe you are with changes in production,
the better you can serve your user.
![Page 7: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/7.jpg)
ITHAKA CARES DEEPLYABOUT SOFTWARE
QUALITY1. All production releases must be rigorously tested2. Releases must not require outages3. User/Publisher impacts must be minimized
This requires a lot of time
![Page 8: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/8.jpg)
OUR OLD RELEASEPROCESS
1 week, product team plans lots of changes for a release (scopeis frozen after this)1-2 months, build lots of changes into a release2 weeks, code freeze and manual regression testing2-3 days, code fixes, rollback testing1-2 days, lots of meetings (not the whole time, but schedulesare complicated)4-8 hours, tell publishers they can't do anything for 4-8 hourson release day1-4 hours, ops people get up really early and begin restarts, 1-4hours2-4 hours, qa tests, 2 - 4 hours
![Page 9: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/9.jpg)
3 MONTHS TO DELIVERCHANGES
Additional requests are deferred to the next release periodThat could be as long as a 6 month wait depending on timingEven tiny changes are expensive with this process, they alsohave to wait
![Page 10: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/10.jpg)
WE WANTED THIS TO BEBETTER
WITHOUT GIVING UP OUR QUALITYASPIRATIONS
![Page 11: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/11.jpg)
ITHAKA CARES DEEPLYABOUT SOFTWARE
QUALITY1. All production releases must be rigorously tested2. Releases must not require outages3. User/Publisher impacts must be minimized4. Process changes must increase iteration speed5. Flexibility is more important than features
![Page 12: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/12.jpg)
INFRASTRUCTUREPrivately managed, artisanal machines in colosInstances AWS Public Cloud, configuration managementunder source control
![Page 13: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/13.jpg)
AGGRESSIVEAUTOMATION OF THE
APPLICATIONDEPLOYMENT PIPELINE
Developers open Tickets for OperationsAn "Operations" team creates resuable services fordevelopment - A private PaaS
![Page 14: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/14.jpg)
$ # multi-datacenter release, $ # load balancer update, $ # provision new instances,$ # instance configuration,$ # cdn configuration$ git push prod master
Because this is so easy, peopleoutside engineering are beginningto use the deployment platform to
deploy apps
![Page 15: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/15.jpg)
RELEASE PLANNINGA week of planning for a large batch of stories released toproduction quarterlyEvery story is planned with the expectation that it will beindividually deployed to production
![Page 16: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/16.jpg)
TEST AUTOMATIONManual TestingEvery story increases the size of our unit and integration testsuites
![Page 17: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/17.jpg)
APPLICATIONARCHITECTURE
Monolithic ArchitectureMicroservices Architecture
![Page 18: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/18.jpg)
LOAD TESTINGSynthetic Load TestingSynthetic Load Testing, Log Replay, Gradual Rollout
![Page 19: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/19.jpg)
DEPLOYMENT PROCESSChanges impact all users at the same timeGradual Rollout to small user groups, 5-10% at a time
![Page 20: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/20.jpg)
INTEGRATIONENVIRONMENTS
Multiple developer, test and integration, environmentsA single TEST environment and app developers use FeatureFlags to mask work in progress
![Page 21: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/21.jpg)
ONCALL DUTY2-3 people on level 1 pager duty for others software50+ people on level 1 pager duty for the software they write
![Page 22: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/22.jpg)
LARGE CHANGES TOPRODUCTION
Lots of coordination between teams, complex changesequences to change productionAll SLAs require backward compatibility processes, decoupledteams, coordination not required to change production
![Page 23: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/23.jpg)
LEAD TIME FOR CHANGES3-6 months for small or large changesas little as 1-7 days for changes
![Page 24: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/24.jpg)
DEPLOYMENT RATE12 deployments per yearover 70 deployments per week
![Page 25: From 12 to 3500 deployments per year in production](https://reader036.vdocuments.us/reader036/viewer/2022062406/55895189d8b42a12638b46be/html5/thumbnails/25.jpg)
TAKEAWAYSIntroduce changes to your team as you would to production: 1change at a timeOur experience is not the result of blind application of bestpractices - it's about retrospecting and iteration