evolution of spotify's ad architecture (qcon 2016 shanghai)

Evolution of Spotify’s Ads ArchitectureA case study on evolving complex systems

Email : kinshuk@spotify.comTwitter : @_kinshukmishra

Ad

Why did Spotify evolve Ads architecture?

Our new needs●

● Change in scale

● Emergence of new client platforms

● Cheap computing

● New ways to create demand

● New features to meet business objectives

Context of Spotify’s Ads scale and challenges

● 70+ million MAU free users

● 60 countries

● 10+ billion ad impressions per month

● 10 ad formats over 3 media types (audio, video, display)

● Multi-platform (mobile, desktop, TVs, web)

2013 2014 2015 2016 2017

Ad server replacementTargeting & User

ModelingDMP audience augmentation

Self-serve & Creative generation

Ad optimizationsMove to GCPAd ExchangesMulti platform client

Evolution timeline

Circa 2013

Proxy ServiceDesktop

LogDelivery HDFS

User Profile

Batch

Basic Ad Server

CampaignManagement

Billing/Reporting

Ad server design flaws Proxy Service

Desktop Routerhash(userid)

Ad server ring with partitions

Ad server instance

During the ad server transition

Proxy ServiceDesktop

LogDelivery HDFS

User Profile

Batch

Smart Ad Server

CampaignManagement

Billing/Reporting

Ad Server Proxy

(routing) Basic Ad Server

Gradual transition from basic to smart ad serving

After the ad server transition

Proxy ServiceDesktop

LogDelivery HDFS

User Profile

Batch

CampaignManagement

Billing/Reporting

Smart Ad Server

Division of responsibilities

Desktop iOS

Android

Ads SDK

Desktop

Web

Rendering Ad trigger decisioning

Ads Ranking

Ads Caching

Ad batching & fetch communication

Ad decisioning

Ad fetch orchestration

Client context

Ad Trigger & Render

Before After

Multi platform client design

iOS

Proxy Service

Android

Ads SDK

Desktop

Web

LogDelivery HDFS

User Profile

Batch

CampaignManagement

Billing/Reporting

Smart Ad Server

Targeting & User Modeling

iOS

Android

Ads SDK

Desktop

Web

Proxy Service

LogDelivery HDFS

User Profile

Targeting Service

Stream Batch

Smart Ad Server

CampaignManagement

Billing/Reporting

Ad Exchange Integration

iOS

Android

Ads SDK

Desktop

Web

Proxy Service

LogDelivery HDFS

User Profile

Targeting Service

Stream Batch

CampaignManagement

Billing/Reporting

Ad Server

Decision Delivery

Ad Server

Ad Exchanges

DMP audience augmentation

Modeling

DMPiOS

Android

Ads SDK

Desktop

Web

Proxy Service

LogDelivery HDFS

User Profile

Targeting Service

Stream Batch

CampaignManagement

Billing/Reporting

Ad Server

Decision Delivery

Ad Server

Ad Exchanges

Move to Google Cloud Platform

Modeling

DMPiOS

Android

Ads SDK

Desktop

Web

Proxy Service

LogDelivery GCS

User Profile

Targeting Service

Stream Batch

CampaignManagement

Billing/Reporting

Ad Server

Decision Delivery

Ad Server

Ad Exchanges

Self-serve and Creative generation

iOS

Proxy Service

Android

Ads SDK

Desktop

Web

Chromecast/Playstation/

FireTV

Ad Aggregation

Service

LogDelivery GCS

User Profile

Targeting Service

DMP

Stream Batch

Ad Server

Decision Delivery Ad Exchanges

CampaignManagement

Modeling Self-Serve Service

Creative Generation Payments

Billing/Reporting

Client got even thinner

iOS

Proxy Service

Android

Ads SDK

Desktop

Web

Chromecast/Playstation/

FireTV

Ad Aggregation

Service

LogDelivery GCS

User Profile

Targeting Service

DMP

Stream Batch

Ad Server

Decision Delivery Ad Exchanges

CampaignManagement

ModelingSelf-Serve Service

Creative Generation Payments

Billing/Reporting

Ad Optimizations

iOS

Proxy Service

Android

Ads SDK

Desktop

Web

Chromecast/Playstation/

FireTV

Ad Aggregation

Service

LogDelivery GCS

User Profile

Targeting Service

DMP

Stream Batch

Ad Server

Decision Delivery Ad Exchanges

CampaignManagement

Optimization

ModelingSelf-Serve Service

Creative Generation Payments

Billing/Reporting

Connecting all the dots ...

Then

Proxy ServiceDesktop

LogDelivery HDFS

User Profile

Batch

Ad Server

Decision Delivery

CampaignManagement

Billing/Reporting

Now

iOS

Proxy Service

Android

Ads SDK

Desktop

Web

Chromecast/Playstation/

FireTV

Ad Aggregation

Service

LogDelivery GCS

User Profile

Targeting Service

DMP

Stream Batch

Ad Server

Decision Delivery Ad Exchanges

CampaignManagement

Optimization

ModelingSelf-Serve Service

Creative Generation Payments

Billing/Reporting

War stories and lessons learnt

Design your systems to be master of one thing

Remember division of responsibilities?

Desktop iOS

Android

Ads SDK

Desktop

Web

Rendering Ad trigger decisioning

Ads Ranking

Ads Caching

Ad batching & fetch communication

Ad decisioning

Ad fetch orchestration

Client context

Ad Trigger & Render

BAD GOOD

Appends are easier than updates

Remember that time?

iOS

Android

Ads SDK

Desktop

Web

Proxy Service

LogDelivery HDFS

User Profile

Targeting Service

Stream Batch

Smart Ad Server

CampaignManagement

Billing/Reporting

The 100:10:1 rule

Agile is your friend

Test with minimal blast radius

Things will always go wrong!

Thank You!

Email : kinshuk@spotify.comTwitter : @_kinshukmishra