![Page 1: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/1.jpg)
MINUTES TO SECONDSHow Careem uses volatile storage to power their dispatching algorithms
![Page 2: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/2.jpg)
WHAT TO EXPECT?
Ingest fast moving data with in-memory storage
Storing volatile data for efficient lookups
Real-time decision making with sub millisecond lookups
Recommended practices for maximum utilization of resources
![Page 3: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/3.jpg)
It comes from the Arabic word Kareem – meaning generosity
![Page 4: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/4.jpg)
To simplify and improve the lives of people,and build an awesome organization that inspires
LET’S BE CAREEM (GENEROUS)
The reason we exist
![Page 5: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/5.jpg)
BASICS – GROUND CONDITIONS
• Infrastructure
• High speeds in some cities
• Far away exits
• Social norms
![Page 6: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/6.jpg)
BASICS - CAPTAINS
• Central to our brand• Where did the name come from?
• Person in command• Leader of a team
• Entrepreneurs
![Page 7: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/7.jpg)
MARKETPLACE
Reliability
Match quality
Tracking
![Page 8: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/8.jpg)
BASICS - CONNECTIVITY
• Low-end Android devices
• Bad GPS sensors
• Extreme temperatures – 46� to 56�
• Limited bandwidth
![Page 9: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/9.jpg)
METRICS
ETA
Time needed to make the best match (time to match)
Age of captain location [ping]
Ratio of requests matched
![Page 10: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/10.jpg)
REQUIREMENTS
Find the best captain in the minimum amount of time
Ability to provide an upfront ETA (promised ETA)
Lowest possible delta between promised and actual ETAs
Ability to look up any captain’s location and status for tracking
Ping as fast as possible
![Page 11: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/11.jpg)
TAKE 1
Captain app
Pings (json) over REST api
Stored in a MySQL table
![Page 12: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/12.jpg)
SCALE
19CITIES
450,000CUSTOMERS
9,000CAPTAINS
![Page 13: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/13.jpg)
SCALE
945,000ETA Requests
13 MPINGS
16 MLOOKUPS
![Page 14: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/14.jpg)
ISSUES: TAKE 1
Continuous deadlocks and performance issues
MySQL 5.6 - no geo-spatial support
Finding nearby cars was not trivial
For stability, we had to reduce the ping frequency to 60s
Affected our ETAs and customer experience
Cost impact – had to use over provisioned resources
https://bugs.mysql.com/bug.php?id=48652
![Page 15: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/15.jpg)
PERFORMANCE
0
100
200
300
400
500
600
700
800
Promised ETA Actual ETA Time to match Captain ping age
![Page 16: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/16.jpg)
PERFORMANCE
40%RELIABILITY
95%UPTIME
![Page 17: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/17.jpg)
LEARNINGS: TAKE 1
Data is volatile – only the most recent version is relevant and the continuous frequency rebuilds the dataLocksNeed a mechanism to support µs lookupsEasy, efficient and super fast nearby-car lookupsAbility to alter object schema at willNeed a buffer to deal with traffic spikesNeed historical time series data against only 2 anchors - captains and bookingNeed a mechanism of representing coordinates as a scalar value Speed is king
![Page 18: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/18.jpg)
TOOLS
Managed In-Memory NosqlService
Queue for Captain Apps to Send Pings to
NoSQL persistence data store for storing historical data
against captains and bookings
ElastiCache for Redis SQS DynamoDB
![Page 19: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/19.jpg)
WHY REDIS?
Single threaded lock free architecture
Rich data structure set
Data structures provide built-in operations that process data optimally at the database level rather than the application level
Pipelining
Primary – replica configuration – for scaling out reads and fail-over support
StringSorted Sets
![Page 20: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/20.jpg)
TAKE 2
Used STS to control access to SQS
Pushed pings (json) to a queue
Worker wrote data to DynamoDB
AZ-2
Big country 1
row
primary nodes
AZ-1 AZ-3
replica nodes
![Page 21: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/21.jpg)
DATA STRUCTURES AT WORK: CAPTAIN INDEXING
Key Valuecaptain:234 {”captainId”: 234, ….. ”lastUpdated”:1506349247000}
captain:89 {”captainId”: 89, ….. ”lastUpdated”:1506349202000}
captain:236 {”captainId”: 236, ….. ”lastUpdated”:1506349247000}
captain:78 {”captainId”: 78, ….. ”lastUpdated”:1506349143000}
![Page 22: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/22.jpg)
DATA STRUCTURES AT WORK: GEO INDEXING
zValue Scorecaptain:234 1506349247000
captain:89 1506349202000
captain:78 1506349143000
Key: geohash:aabbc:product:12
zValue Scorecaptain:234 1506349247000
captain:89 1506349202000
captain:78 1506349143000
Key: geohash:aabbc:product:12
zValue Scorecaptain:236 1506349247000
captain:77 1506349143000
Key: geohash:aabbd:product:12
![Page 23: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/23.jpg)
DATA STRUCTURES AT WORK: GEO INDEXING
zValue Scorecaptain:234 1506349247000
captain:89 1506349202000
captain:78 1506349143000
Key: geohash:aabbc:product:12
zValue Scorecaptain:234 1506349247000
captain:78 1506349143000
Key: geohash:aabbc:product:12
zValue Scorecaptain:89 1506349414000
captain:236 1506349247000
captain:77 1506349143000
Key: geohash:aabbd:product:12
![Page 24: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/24.jpg)
GEO LOOKUPS – NEARBY CAPTAINS
![Page 25: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/25.jpg)
SCALE
47CITIES
6MCUSTOMERS
250,000CAPTAINS
9,000450,00019
PREVIOUS VALUES
![Page 26: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/26.jpg)
SCALE
4.1 META Requests
80 MPINGS
26 MLOOKUPS
16 M13 M945,000
PREVIOUS VALUES
![Page 27: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/27.jpg)
PERFORMANCE
0
50
100
150
200
250
300
350
400
450
Promised ETA Actual ETA Time to match Captain ping age
![Page 28: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/28.jpg)
PERFORMANCE
86%RELIABILITY
99.99%UPTIME
![Page 29: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/29.jpg)
PERFORMANCE - LOOKUPS
![Page 30: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/30.jpg)
SUMMARY
We were able to increase the frequency of pings to 4 per minute (every 15s)
Improved our ETAs and customer experience as more pings meant that we were dispatching very close to captain’s locations
Side benefit: granular level captain tracking for customers (in ride)
Average time to match reduced from 2 minutes to 15 seconds
![Page 31: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/31.jpg)
UNDER THE HOOD
![Page 32: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/32.jpg)
OUR TECH STACK
![Page 33: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/33.jpg)
REDIS – DIFFERENT MODES
Standalone
Primary / replica
Clustered
![Page 34: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/34.jpg)
HYGIENE
Always have multiple slaves in your cluster
Configure back ups with a replica
For writes never connect directly to the writable node directly, use the
primary endpoint
Set aside 30% memory
![Page 35: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/35.jpg)
SCALING
Primary ReplicaReplica
Replica
application reads
application reads
application reads
• Read scale out
• Choose the right client library
![Page 36: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/36.jpg)
FURTHER SCALING
![Page 37: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/37.jpg)
FASTER PINGS BY GREATER NUMBER OF CAPTAINS
We wanted to increase the frequency of pings further (every 5s)
Even more captains on the network
Even more customers reading assigned captain’s data
Same performance demand
More balanced read and writes
![Page 38: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/38.jpg)
SCALE
80CITIES
15 MCUSTOMERS
450,000CAPTAINS
250,0006 M48
PREVIOUS VALUES
![Page 39: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/39.jpg)
SCALE
15 META Requests
140 MPINGS
48 MLOOKUPS
26 M80 M4.1 M
PREVIOUS VALUES
![Page 40: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/40.jpg)
SHARDING
• Cluster mode with Redis 3.0• Multiple primaries with each having
its own set of replicas• Hash slots• CRC• Application agnostic Replica
Replica
Replica
Primary
Replica
Replica
Replica
Primary
Replica
Replica
Replica
Primary
![Page 41: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/41.jpg)
TAKE 3
AZ-1
Lastseen clustercountry1
AZ-2 AZ-3AZ-1
Geoindexingclustercountry1
AZ-2 AZ-3AZ-1 AZ-2 AZ-3
AZ-1
Geoindexingclustercountry2
AZ-2 AZ-3
Lastseen clustercountry2
![Page 42: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/42.jpg)
PERFORMANCE
0
50
100
150
200
250
300
350
400
450
Promised ETA Actual ETA Time to match Captain ping age
![Page 43: MINUTES TO SECONDS · Recommended practices for maximum utilization of ... Stored in a MySQL table. SCALE 19 CITIES 450,000 CUSTOMERS 9,000 CAPTAINS. SCALE 945,000 ETA Requests 13](https://reader034.vdocuments.us/reader034/viewer/2022042810/5f979fbc5052a7163318c2db/html5/thumbnails/43.jpg)