Transforming Mobile Push Notifications with Big DataDennis Waldron, Data EngineeringPablo Varela, Systems Engineering

Who is Plumbee?

● 12.8M Installs● 209K Daily Active Users● 818K Monthly Active Users

● Social Games Studio● Mirrorball Slots & Bingo● Facebook Canvas, iOS

Data Providers

Inhouse data = 99.9% of all data

In Total:

● 98TB (907 days of data)● All stored in Amazon S3

Daily:

● 78GB compressed● ~450M events/day● 4,800 events/second (peak)

Aggregates

Application/Game Servers

End Users (Desktop & Mobile)

Log Aggregators

Amazon S3 (Simple Storage Service)DataPipeline

Amazon EMR(Elastic MapReduce)

Amazon Redshift

Daily Batch Processing

Plumbee Employees

Analytics (SQL Queries)

Architecture - Overview

Events (JSON)

SQS Analytics Queue

Events (JSON)

Amazon Web Service

End Users (Desktop & Mobile)

Application/Game Servers

● Collect everything!

● RPC events intercepted by annotated endpoints. (Requests)

● All mutating state changes recorded:○ DynamoDB, MySQL, Memcache

(Blobs Updates)● Custom Telemetry (Other):

○ Client: click tracking, loading time statistics, GPU data...

○ Server: promotions, transactions, Facebook user data...

Game Data

RPC

OTHER 15%

77%

9%

MySQL

MemCache

GENERATES

DynamoDB

Game Data - Example RPC Endpoint Annotation

/** * Example annotation */@SQSRequestLog(requestMessage = SpinRequest.class)@RequestMapping(“/spin”)public SpinResponse spin(SpinRequest spinRequest) {

…

}

Example Event - userStats● All events are recorded in JSON.● Structure:

○ Headers○ Categorization Data (metadata)○ Payload (message)

● Important Headers:○ timestamp○ testVariant○ plumbeeUid

Analytics (SQL Queries)

Aggregates

Application/Game Servers

End Users (Desktop & Mobile)

Amazon S3 (Simple Storage Service)DataPipeline

Amazon EMR(Elastic MapReduce)

Amazon Redshift

Daily Batch Processing

Plumbee Employees

Architecture - Collection

Log Aggregators

Events (JSON)

SQS Analytics Queue

Events (JSON)

Data Collection (I) - PUT

Application/Game Servers

Events (JSON)

SQS Queue

Log Aggregators

Producers Consumers

What is SQS (Simple Queue Service)?

A cloud-based message queue for transmitting messages between producers and consumers

SQS Provides:

● ACK/FAIL semantics● Unlimited number of messages● Scales transparently● Buffer zone

What is Apache Flume?

A distributed, reliable, and available service for efficiently collecting, aggregating, and

moving large amounts of log data

SQS Queue

Apache Flume

Consumers

Data Collection (II) - GET

Amazon S3 (Simple Storage Service)

S3 Data:

● Partitioned by: date / type / sub_type● Compressed with: Snappy● Aggregated in 512MB chunks

● Pluggable component architecture● Durability via transactions● File channel use Elastic Book Store (EBS) volumes (network attached storage)

○ Protects against Hardware failure

● SQS Flume Plugin: https://github.com/plumbee/flume-sqs-source

Data Collection (III) - Flume

Flume Agent

Source(Custom)

Sink(HDFS)

SQS Queue

Channel(File Based)

S3 Bucket

Transactions

A + B + C = Flow

A B C

Aggregates

Application/Game Servers

End Users (Desktop & Mobile)

Amazon S3 (Simple Storage Service)DataPipeline

Amazon EMR(Elastic MapReduce)

Amazon Redshift

Daily Batch Processing

Plumbee Employees

Analytics (SQL Queries)

Architecture - Processing

Events (JSON)

SQS Analytics Queue

Events (JSON)

● Daily activity● Orchestrated by Amazon DataPipeline● Includes generation of reports● Configured with JSON

What is DataPipeline?

A cloud-based data workflow service that helps you process and move data between

different AWS services

Extract, Transform, Load

SCH

EDU

LEC

OM

MA

ND

RES

OU

RC

E

What is Elastic Map Reduce?

Cloud-based MapReduce implementation to process vast amounts of data built on top of

the open-sourced Hadoop framework.

Two phases:

● Map() Procedure -> Filtering & Sorting● Reduce() -> Summary operation

Extract & Transform (I)Penguin

HorseCake

Cake

Penguin

Penguin

Penguin

Horse

Horse

CakeCake

HorseHorseHorse

PenguinPenguinPenguinPenguin

Cake: 2 Horse: 3

REDUCE()

MAP()

RA

W D

ATA

SOR

TED

QU

EUES

RES

ULT

Penguin:4

What is Hive?

An open-sourced Apache project with provides a SQL-Like interface to summarize, query and

analysis large datasets by leveraging Hadoop’s MapReduce infrastructure.

● Not really SQL, HQL -> HiveQL● No transactions, materialized views,

limited subquery support, ...

Extract & Transform (II)

SELECT plumbeeuid, COUNT(*) AS spins FROM eventlog

-- Partitioned data access WHERE event_date = '2014-11-18' AND event_type = 'rpc' AND event_sub_type = 'rpc-spin' -- Aggregation GROUP BY plumbeeuid;

Table: Eventlog● Mounted on top of raw data● SerDe provides JSON parsing● Target data via partition filters

● Hive has limitations! ○ Speed, JSON

● Most of our transformations use:

Streaming MapReduce Jobs

What is Streaming?

“A Hadoop utility that allows you to create and run MapReduce jobs using any

executable script as a mapper or reducer”

Extract & Transform (III)

for line in sys.stdin: data = json.loads(line) print data['plumbeeUid'] + '\t' + 1

results = defaultdict(int)

for line in sys.stdin: plumbee_uid, count = line.split('\t') results[plumbee_uid] += int(count)

print results

Emits, Key value Pairs 466264 => 1, 376166 => 1 983131 => 1, 466264 => 1

Hadoop sorts and shuffles the data making sure matching keys are processed by a single reducer!

JSON rpc-spin Data

Result:{ 466264: 2, 376166: 1, 983131: 1 }

map()

reduce()

Results

EMR Transformed data:

● Referred to as aggregates● Stored in S3● Accessible via EMR cluster

Load (I) - Problem

Raw S3 JSON Data Aggregated Data

EMR Transformation(Hive & Streaming Jobs)

Problem

● We don’t run long-lived EMR clusters.

EMR requires:

● Specialists knowledge● Is slow, processing and booting “offline”.

5.4TB

Use Amazon Redshift for fast “online” data access

What is Redshift?

A column-oriented database which uses Massive Parallel Processing (MPP) techniques

to support analytics style SQL based workloads across large datasets.

Power comes from:

● Query parallelization● Column-oriented design

Redshift Provides:

● Low latency JDBC and ODBC access● Fault Tolerance● Automated Backups

Load (II) - Redshift

Redshift (x3 nodes): 0.33sEMR (x20 nodes): 135.46s

Load (II) - Column-Oriented Databases

ID First Name Last Name Country

1 Penguin Situation GB

2 Cheese Labs US

3 Horse Barracks GB

ID First Name Last Name Country

1 Penguin Situation GB

2 Cheese Labs US

3 Horse Barracks GB

Row-oriented Database - MySQL

Column-oriented Database - Redshift

● East to add/modify records● Could read irrelevant data.● Great for fast lookups (OLTP)

● Only read in relevant data● Adding rows requires multiple

updates to column data.● Great for aggregation queries

(OLAP)

Aggregates

Application/Game Servers

End Users (Desktop & Mobile)

Amazon S3 (Simple Storage Service)DataPipeline

Amazon EMR(Elastic MapReduce)

Amazon Redshift

Daily Batch Processing

Plumbee Employees

Analytics (SQL Queries)

Architecture - Revisit

Log Aggregators

Events (JSON)

SQS Analytics Queue

Events (JSON)

Q&A

Targeted Push Notifications

Mirrorball Slots: Kingdom of Riches

Mirrorball Slots: Challenges

● recurring timed event● collect symbols from non-winning

spins● get free coins if enough symbols are

collected

Some players ask for notifications

Use Cases

Building blocks

Data Collection

Players

Data Collection

Amazon Redshift

Architecture - Overview

Amazon S3

Amazon Redshift

Batch ProcessorsAmazon SNS

PublisherTrigger Segmentation Workers

Players

Targeting

Mobile Push

User Targeting

User targeting

Run SQL queries directly against Redshift

Amazon Redshift User Segment

SQL Query

User targeting: Query example

-- Target all mobile users SELECT plumbee_uid, arnFROM mobile_user

User targeting: Query example (II)-- Target lapsed users (1 week lapse)

SELECT plumbee_uid, arnFROM mobile_userWHERE last_play_time < (now - 7 days)

Demo (I)Mobile MBS Notifications

Architecture - Mobile Push

Amazon S3

Amazon Redshift

Batch ProcessorsAmazon SNS

PublisherTrigger Segmentation Workers

Players

Targeting

Mobile Push

Amazon Simple Notification Service

What is SNS?

“Amazon Simple Notification Service (Amazon SNS) is a fast, flexible, fully managed push messaging service”

Amazon SNS

Amazon SNS

Amazon SNS: Device Registration

Game Servers SQS Analytics Queue Amazon RedshiftPlayers

Amazon SNS

register deviceevent

register

Amazon SNS: ARN Retrievalprivate String getArnForDeviceEndpoint(String platformApplicationArn, String deviceToken) {

CreatePlatformEndpointRequest request =

new CreatePlatformEndpointRequest()

.withPlatformApplicationArn(platformApplicationArn)

.withToken(deviceToken);

CreatePlatformEndpointResult result = snsClient.createPlatformEndpoint(request);

return result.getEndpointArn();

}

Amazon SNS: Analytics Eventprivate String registerEndpointForApplicationAndPlatform( final long plumbeeUid,

String platformARN, String platformToken) {

final String deviceEndpointARN = getArnForDeviceEndpoint( platformARN, platformToken);

sqsLogger.queueMessage( new HashMap<String, Object>() {{

put("notification", "register");

put("plumbeeUid", plumbeeUid);

put("provider", platformName);

put("endpoint", deviceEndpointARN );

}}, null);

return deviceEndpointARN;

}

Amazon SNS: Mobile Pushprivate void publishMessage(UserData userData, String jsonPayload) { amazonSNS.publish(new PublishRequest() .withTargetArn( userData.getEndpoint()) .withMessageStructure( "json") .withMessage( jsonPayload));}

{"default": "The 5 day Halloween Challenge has started today! Touch to play NOW!"}

Payload example

Architecture - Orchestration

Amazon S3

Amazon Redshift

Batch ProcessorsAmazon SNS

PublisherTrigger Segmentation Workers

Players

Targeting

Mobile Push

Amazon Simple Workflow

What is Amazon SWF?

“Amazon Simple Workflow (Amazon SWF) is a task coordination and state management service for cloud applications.”

What Amazon SWF provides● consistent execution state management

● workflow executions and tasks tracking

● non-duplicated dispatch of tasks

● task routing and queuing

● the AWS Flow Framework

Architecture - Orchestration

Amazon S3

Amazon Redshift

Batch ProcessorsAmazon SNS

PublisherTrigger Segmentation Workers

Players

Targeting

Mobile Push

Mobile Push: Scheduling

Trigger Publish Service Amazon Simple Workflow

Mobile Push: Targeting

query querytarget users

Amazon SWF Amazon EC2

Worker(Segmentation)

Amazon Redshift

Amazon S3

Mobile Push: Processing

publish pushbatch 1-N

Read data + push End UserAmazon SWFWorkers

(Processing)

Mobile Push: Reporting

send send

Amazon SWF Amazon EC2

Worker(Reporting)

Amazon SES

Demo (II)

Q&A