charlotte software engineering meetup - jan.2015
TRANSCRIPT
PolyglotPersistence
Majid Fatemian
R E D V E N T U R E S
@majidfn
Relational Non-Relational Key-Value Big Data
Application
Relational
Application
Relational Relational Non-Relational
Relational
DATA
BIGDATA
Capture
SearchStorage
Q u e r y
Privacy Visualize
Analysis
Non-Relational
1 2 3
Non-Relational Cluster Schemaless
I. Document
II. Key-Value
III. Graph
IV. Column
MongoDB, CouchBase
Memcache, Redis
Neo4J, OrientDB
Cassandra, HBase
P
C AAvailabilityConsistency
Partition Tolerance
P
AAvailabilityConsistency
Partition Tolerance
C
CP
CA APMySQL
Postgres
MongoDBCouchBase
Cassandra
Riak
2 x
1 x
1 x
2011
2012
2013
2014
2011
2012
2013
201
Application
Normalized
ACID {ConsistencyRelational
Replication
Fault ToleranceAvailability{Relational
OPs / DBAs Tools Devs{DevOps
Relational
2011
2012
2013
201
ScalabilityRelational
SchemaRelational
{ player_id: 1234, * level: 40, * duration: 75, * activity: activity, game_id: 4001, platform: WiiU, difficulty_level: medium, repetition: 178, score: 450, endurance:3}
{ player_id: 1234, level: 12, duration: 120, activity: activity}
Aggregated
Detailed
stats
statsPlayers’ Profiles
Individual Activities
DistributionRelational
Data CData B
Data A
Data Access Layer
ACID Transaction Partitioning Replication
Tools OPs
Data Volume Fixed Schema Scale Up Manual Sharding
+ -
NoSQL...
mongoDB
1 2 3
OPs Devs Community
Schema-lessNon-Relational
player_id: 1234, * level: 40, * duration: 75, * activity: activity, game_id: 4001, platform: WiiU, difficulty_level: 2, repetition: 178, score: 450, endurance:3
player_id: 1234, level: 75, duration: 120, activity: activity
ScalabilityNon-Relational
ScalabilityNon-Relational
+
ShardingNon-Relational
ShardingNon-Relational
Data A
{key}
ShardingNon-Relational
Data A
A
Data B
B
Data C
C
{key}
Map/ReduceNon-Relational
K, V
K,V1 K,V2 K,V3
Map/ReduceNon-Relational
QueryK, V
K,V1 K,V2 K,V3
Map/ReduceNon-Relational
Query
Map
K, V
K,V1 K,V2 K,V3
Map/ReduceNon-Relational M
ap
K,V1 K,V2 K,V3
Map/ReduceNon-Relational
ResultReduce
Map
K,V1 K,V2 K,V3
DenormalizedNon-Relationalread / }write
Disk Space
Data ModelNon-RelationalQuery
Aggregation}
Eventual- Consistency
Non-Relational
X1
X1 X1
Eventual- Consistency
Non-Relational
X2
W
X1 X1
X2
Eventual- Consistency
Non-Relational
X2
W
X1 X1
R
X1
X2
Eventual- Consistency
Non-Relational
X2
W
X2
X2
R
X2
X2
BalancerNon-Relational
BalancerNon-Relational+
BalancerNon-Relational
Non-RelationalResilience
A B C
Non-RelationalResilience
A C
LockingNon-RelationalDB Level (2.2)}Document Level (3.x)
BackupNon-Relational
Point-in-time
Denormalized data Disk Space Expertise Complex QueryingEventual Consistency Resource Usage DB-Level Locking
-
Schema-Less Aggregated Data Large Scale Data
Sharding Map/Reduce
Memory Storage Journaling
+
Random ProfileSearch(1M)
Full History Retrieval(6M)
Insert(6M)
AggregationMap/Reduce
(6M)
0
0.003
0.005
0.008
0.01
MySQLMongoDB
PersistencePolyglot
Players’ Profile Leaderboards
Activity Details
Total Sum
Activity 1
Activity n
Activity m
……
Shard 1
Shard 2
Config
Data Access Layer
Application
TheChallenge
Inconsistency1
Data Access Layer
Commit2-Phase
Data Access Layer
W
Pending
Commit2-Phase
Data Access Layer
Pending
W Commit2-Phase
Data Access Layer
Done
Commit2-Phase
2 Latency
2 Latencyvs. Consistency
Operations3
Operations3Exponential complexity
Availability4
Availability4 Plan for failure
Data Access Layer
Data Access Layer
Data Access Layer
Data Integrity5
Data Integrity5Backward compatibility
Unit{TEST IntegrationFunctional
Data Migration6
Profile
History }{ profile: { GUID:…, platform: { name:’xbox’, xuid:… } }, activity:…, duration:…, datetime:… }
3rd Party
1 One-Shot Migration
1 One-Shot Migration4 hours !!
1 One-Shot Migration4 hours !!
Live !!2 Profile versioning
+1
V+11
V+11
V
+1
2
FinalResults
Scalability
2011 2012 2013 2014
9x response time
3 years uptime
Go Hybrid!>
Go Hybrid!Thoughtfully
>
ThankYOU!
@majidfn