high availability, real-time and scalable architectures
TRANSCRIPT
![Page 1: High availability, real-time and scalable architectures](https://reader031.vdocuments.us/reader031/viewer/2022030306/586fe2ab1a28ab18428b7c7d/html5/thumbnails/1.jpg)
Patricio Rocca - Agosto 2016@patriciorocca
Arquitecturas de tiempo real y
escalables
![Page 2: High availability, real-time and scalable architectures](https://reader031.vdocuments.us/reader031/viewer/2022030306/586fe2ab1a28ab18428b7c7d/html5/thumbnails/2.jpg)
About Jampp
We are a tech company that helps companies grow their mobile business by driving engaged users to their appsWe are a team of 75 people, 30% in the engineering team.
Located in 6 cities across the US, Latin America, Europe and Africa
Machine learning
Post-install event optimisation
Dynamic Product Ads and Segments
Data Science
Programmatic Buying
![Page 3: High availability, real-time and scalable architectures](https://reader031.vdocuments.us/reader031/viewer/2022030306/586fe2ab1a28ab18428b7c7d/html5/thumbnails/3.jpg)
We process 220,000 bid requests per second
We process each bid request in less than 100ms
We manage 40Tb of data everyday
We do real time machine learning
Jampp Architecture Impressive Facts
And… we are just a team of 22 nerds :) or :(
![Page 4: High availability, real-time and scalable architectures](https://reader031.vdocuments.us/reader031/viewer/2022030306/586fe2ab1a28ab18428b7c7d/html5/thumbnails/4.jpg)
Bid
Real-Time Bidding Workflow
Auction Win
Exchange Exchange
Publisher Publisher
Jampp Bidder
Jampp Machine Learning
Jampp Engageme
nt Segments
Builder
IMPRESSION ;-)
PLACEHOLDER IMPRESSION
![Page 5: High availability, real-time and scalable architectures](https://reader031.vdocuments.us/reader031/viewer/2022030306/586fe2ab1a28ab18428b7c7d/html5/thumbnails/5.jpg)
Real-Time Tracking Workflow
In-App Event
IMPRESSION ;-)
Click
Jampp NodeJS
TrackingPlatform
Client TrackingPlatform
Publisher Client
IMPRESSION
![Page 6: High availability, real-time and scalable architectures](https://reader031.vdocuments.us/reader031/viewer/2022030306/586fe2ab1a28ab18428b7c7d/html5/thumbnails/6.jpg)
Let’s talk about architecture!
![Page 7: High availability, real-time and scalable architectures](https://reader031.vdocuments.us/reader031/viewer/2022030306/586fe2ab1a28ab18428b7c7d/html5/thumbnails/7.jpg)
Real Time Bidding Architecture
![Page 8: High availability, real-time and scalable architectures](https://reader031.vdocuments.us/reader031/viewer/2022030306/586fe2ab1a28ab18428b7c7d/html5/thumbnails/8.jpg)
Bid Price = CPI * eCTR * eCVR * (1-margin) * 1000
Python + Tornado + Cython + nginx (+ antigravity)
Caching, layers upon layers upon layers
Leaky bucket-ish feedback loop for pacing
With predictive local projections to account for imperfect and laggy inter-server communication
Selective, aggregate logging
Circa-25TB of data generated per day makes naïve logging… unwise
Online Machine Learning (stochastic gradient descent model)
Real Time Bidding Architecture (details)
![Page 9: High availability, real-time and scalable architectures](https://reader031.vdocuments.us/reader031/viewer/2022030306/586fe2ab1a28ab18428b7c7d/html5/thumbnails/9.jpg)
In-process L1 serves all requests
µs latency access a lifesaver for real-time, latency-constrained workloads
Local L2 in each server
Buffers responses from the L3
Saves bandwidth to-from the L3(3 MB/s x 230 servers x 8 procs = death)
Decreases promotion latency to L1
Remote L3 provides main distributed cache storage
Remote S3 precomputed slow-changing bundles
Speeds up load of massive near-static data
Caching
![Page 10: High availability, real-time and scalable architectures](https://reader031.vdocuments.us/reader031/viewer/2022030306/586fe2ab1a28ab18428b7c7d/html5/thumbnails/10.jpg)
Uses logistic regression to predict P(click | impression) or P(install | click) using context features
Online solution that incrementally learns from the Real Time Bidding events just in time
Uses regularization and hashing trick to explore a huge feature space and keep only the statistically most informative ones
Machine Learning
![Page 11: High availability, real-time and scalable architectures](https://reader031.vdocuments.us/reader031/viewer/2022030306/586fe2ab1a28ab18428b7c7d/html5/thumbnails/11.jpg)
Stream Processing Architecture
![Page 12: High availability, real-time and scalable architectures](https://reader031.vdocuments.us/reader031/viewer/2022030306/586fe2ab1a28ab18428b7c7d/html5/thumbnails/12.jpg)
Stream Processing Architecture (details)
Uses Amazon Kinesis for durable streaming data and Amazon Lambda for data processing
DynamoDB as temporal data storage for enrichment and analytics
S3 provides a Single Source of Truth for batch data applications
Decouples data from processing to enable multiple Big Data engines running on different clusters/infrastructure
Easy on demand scaling by AWS™
![Page 13: High availability, real-time and scalable architectures](https://reader031.vdocuments.us/reader031/viewer/2022030306/586fe2ab1a28ab18428b7c7d/html5/thumbnails/13.jpg)
Data Push
Pick your partition key for evenly distributing data across shards
Encoding protocol matters! MessagePack offered the best trade off between compression and serialization speed factor
![Page 14: High availability, real-time and scalable architectures](https://reader031.vdocuments.us/reader031/viewer/2022030306/586fe2ab1a28ab18428b7c7d/html5/thumbnails/14.jpg)
Data Processing and Enrichment
Write/Read Batching to reduce the HTTPS protocol overhead and costs
Exponential backoff + Jitter to reduce the impact of in-app events bursts sent by the tracking platforms
Increased Data Retention Period from 1 day (default) to 3 days on the raw data streams
![Page 15: High availability, real-time and scalable architectures](https://reader031.vdocuments.us/reader031/viewer/2022030306/586fe2ab1a28ab18428b7c7d/html5/thumbnails/15.jpg)
Spark + Hadoop + PrestoDB = <3
Firehose real time data-ingestion to S3 and auto scaling capabilities
EMR Cluster simplifies our data processing
Spark ETLs are executed by Airflow, to enrich data, de-normalize and convert JSON to Parquet.
Spark Streaming for real-time anomaly detection and fraud prevention
![Page 16: High availability, real-time and scalable architectures](https://reader031.vdocuments.us/reader031/viewer/2022030306/586fe2ab1a28ab18428b7c7d/html5/thumbnails/16.jpg)
Dunno fuck with real time! (caching and cython to the rescue)Rent first, build laterDevelopment and staging for Big Data projects should involve
production traffic or be prepared for troublePrestoDB is really amazing in regards to performance, maturity and
feature setKinesis, Dynamo and Firehose use HTTPS as transport protocol
which is slow, requires aggressive batching and exponential back-off + jitter
Monitoring, logs and alerts managed by AWS Cloudwatch oversimplifies production support
Lessons Learned
![Page 17: High availability, real-time and scalable architectures](https://reader031.vdocuments.us/reader031/viewer/2022030306/586fe2ab1a28ab18428b7c7d/html5/thumbnails/17.jpg)
Gracias ;-)
geeks.jampp.com