state of mobile continuous delivery at spotify

State of mobile CD at SpotifyKristian Lindwall@klindwall

@ixhd@klindwall

30

5 years ago

40050

300

Now

30

100+ teams in 5 sites

Cross functional and full stack

Self-organizing

“Feel like a mini-startup”

End to end responsibilitySystems runs, works, scales

Section name 10

Section name 11

IOS Android

Feature squads(1-2 devs per platform in each team)

Client infrastructure Squads(tooling, support, Releases)

Backend Infrastructure squads

Daily commits

Quick history of mobile CD

Section name 13

• Desktop first• Quality process completely detached from dev

process, quality unknown• Little or no automated testing• A lot of work on feature branches, painful

merges• Irregular releases, occured because of company

requirements or external events

State of mobile CD

Section name 14

• Mobile equally important to desktop• Feature set determined release date• Low predictability leading to scope creep and

even more risk (“this has to go out with the release”)

State of mobile CD

Section name 15

• Introduced release manager role• Introduced three-week schedule.• Started taking test automation seriously• Had a hard time getting engineering and feature

development squads to “conform” to time-bound releases

• Missed releases. Unpredictably still because of low quality

State of mobile CD

Section name 16

• Specialised infra teams• Put tests together with source code for apps and

library• Started running and enforcing automated tests

with code changes• Nightly dogfooding

State of mobile CD

Section name 17

• Buy-in and mindset in the tech and product organizations caught up with our intention

• Bi-weekly releases• Almost all releases hit the market within 14 days of branching

date (almost predictable!)• Client branches and dependencies are cut automatically - no

judgement call

State of mobile CD

Section name 18

• Current challenge: Make releases predictable to within a few days of branch cut

• Moving towards shorter cycles (1 week probably)

State of mobile CD

Development of the Spotify clients today

CI System(Teamcity)

• 1 project per client• 80-120 commits

per day• ~1000 commits

per release

• 30-50 active developers per client and day

• Avg. time for pull request verification: <10m

• ~70% pull requests merged in <4hrs

• ~10000 unit tests/client (as well as a number of UI-based tests and integration tests)

Testing strategy

Snapshot:Releasing mobile

at Spotify

Current release cycle

Release branch only accept bugfixes

Current release cycleDistributionAndroid: Private Google Play channeliOS: Crashlytics

Current release cycleDistributionAndroid: Google Play Store, Beta channeliOS: App Store, Testflight programFeedback in beta forum

Current release cycle

DistributionAndroid: Google Play Store, gradual rolloutiOS: Beta has sufficient usage and quality -> release to 100% on App Store

Current painpoints

Developer pain points

Flakiness

What is “flaky tests”?• Unstable tests – timing issues, logical errors, hard to write good tests• Infrastructure – network, new Xcode versions, build framework,

unreliable test data• Backend – latency, changes, regional

Why is it bad?• Broken window syndrome• No trust in test results

Flaky Tests

Elastic

CI System

Logs

Test

resu

ltsBuil

d res

ults

Smart Alerting KibanaOdeneye

Dealing with flakiness

Dealing with flakiness

Time

Build times

Releasability

Releases Android Feb 2015-Mar 2016

Lessons learnt Getting people dedicated to target mobile CD

was a game changer

Weed out bottlenecks one at a time. Slowness, flakiness etc will cause people to do the worng thing.

Continuous delivery is mostly about changing behavior, not technology

Thank you!

Kristian Lindwall@klindwall