from a small club to a giant stadium - sc2 · spotify project title 00.00.2015 proprietary &...

Project Title 00.00.2015 Proprietary & Confidential 1Spotify

From a Small Club to a Giant StadiumScaling Engineering at Spotify Wouter de Bie - Director of Engineering [email protected]

mailto:[email protected]

2Project Title 00.00.2015 Proprietary & ConfidentialSpotify

How did we go from this..


.. to this?

00.00.2015 4Spotify

What is our scale?

00.00.2015 5Spotify

Spotify was founded in 2006. First launched in Sweden in 2008

Spotify went from 1 to 60+ countries in 9 years

Spotify went from 0 to 150 Million users in 9 years

00.00.2015 6Spotify

Currently we’re at about 3500 employees, of which 1700 are in R&D

Spotify consists of 400+ micro services

We have 60+ PB of data under management

00.00.2015 7Spotify

Scaling Spotify • The organization • Our technology platform

I’ll be talking about the two parts that we scaled over the years: our org and tech platform.

00.00.2015 8Spotify

Tech organization

We started with an oldskool tech org, as seen in many organizations.

9

Project Title 00.00.2015 Proprietary & ConfidentialSpotify

Backend Frontend Data QA

Different team doing different things. The Backend Team does backend, the frontend team does frontend, etc

10



But all these teams are dependent on each other. And as the org grows, those dependencies become bigger and bigger. Also, these teams don’t move in lock-step and have different priorities and eventually there is the chance of a dead-lock since they are all waiting for each other. Since we need to stay ahead in the game, we need to move fast.

00.00.2015 11Spotify

The Matrix organization

Because of this we designed the Matrix Org

00.00.2015 12Spotify

Autonomy is the word…

It’s all about autonomy. We believe that through autonomy (aka cutting down on dependencies), we achieve speed.

13



14



Instead of a team per competence, we create a team that can take on a task or mission from A to Z with all the competence needed.

15



Squad

We call this a squad

16



Squad

Agile Coach

Product Manager

And we add a product manager and an Agile Coach


Who’s the Boss?

Problem: who’s the boss of the squad? PM? AC?

18



Chapter

We organize hierarchically in Chapters

19



Chapter

Chapter Lead

And a Chapter Lead is a first line engineering manager


VP Engineering

And those chapter leads reported to the VP of Engineering, who reported to the CTO


“What if we have multiple $competence chapters?”


Say that we have a bunch of chapters


And some of them have the same competence (e.g. backend)


Guild

We create a Guild where engineers with that competence come together

00.00.2015 25Spotify

Tribes


VP Engineering

Initially our org looked like this. All CLs reporting to the VP of Eng


VP Engineering

But this group grew too big, so we had to do something


We split up different chapters and let them report to multiple VPs.


Tribe Tribe

And the sub organization we call a tribe.


“What if we have multiple $competence chapters in different

tribes?”

00.00.2015 31Spotify

Still one guild for one competence. Just a little bigger..


CTO Head of Product

At the time we did this, our product org was separate and had PMs “embedded” in squads. Product reported to a Head of Product

00.00.2015 33Spotify

Missions


CTO

After a while, the amount of tribes became to big for the CTO, so we had to come up with something else.


Head of R&D

We split the tribes into groups


Head of R&D

Mission

And called it a mission


Creator User Engagement

Core Experience Revenue Growth

Opportunities

Technology Platform

Data

Design / Product Insights

Now we have 8 mission in total and we make a distinction between vertical an horizontal missions. Here are the vertical missions




Opportunities

Technology Platform

Data


And the horizontal missions that support the vertical ones




Opportunities

Technology Platform

Data


00.00.2015 40Spotify

A Typical Mision • One or more Engineering Tribes • Product organization • People Operations • (sometimes other mission specific sub-orgs)


VP Data

Engineering Operations Research Solutions Design Product

My mission


“What if we have multiple $competence chapters in different tribes

in different missions?”

00.00.2015 43Spotify

Still one guild for one competence. Just a little bigger..

00.00.2015 44Spotify

Organizational Learnings

Some learnings that we have had over the years.

00.00.2015 45Spotify


No Org Model is Perfect Spotify is optimized for speed

through autonomy, Other things don’t come easy and

have to be mitigated for

00.00.2015 46Spotify


Our org model seems complex

On-boarding is sometimes a little hard because of terms

and internal roles used

00.00.2015 47Spotify


Dependencies move up the stack

First there were team dependencies, then tribe dependencies and now Mission dependencies

00.00.2015 48Spotify

Scaling Technology

The only stereotypical stock photo image. Apart from the org there is tech

00.00.2015 49Spotify

Micro Services vs

Monolith And have a standard protocol

What does this accomplish?


Autonomy

00.00.2015 51Spotify

Ops in squads a.k.a. DevOps



Autonomy

00.00.2015 53Spotify

Transition to the Cloud Google Cloud Platform in our case



Autonomy … and also because other companies are better at running a data center

00.00.2015 55Spotify

Trying to prevent multi-tenant solutions … think about a 3500 node cluster with 1500 users



Autonomy

00.00.2015 57Spotify

Trying to prevent multi-tenant solutions … think about a 3500 node cluster with 1500 users

00.00.2015 58Spotify

The downside No organizational model is perfect.

• Autonomy requires a lot of coordination.

• Some things need centralized management like data governance or quality

Obviously there is a downside too..


That’s it! Questions?

from a small club to a giant stadium - sc2 · spotify project title 00.00.2015 proprietary &...

Documents