nosql like there is no tomorrow
DESCRIPTION
The AWS NoSQL team shares the design philosophy behind DynamoDB and lessons learned in building for massive scale.TRANSCRIPT
![Page 1: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/1.jpg)
@ksshams@swami_79
NoSQL like there is NoTomorrow
KhawajaEngineering Lead for NoSQL,
AWS
![Page 2: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/2.jpg)
@ksshams
NASA JPL has visited every planet in the solar system ... except Pluto
![Page 3: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/3.jpg)
25 Gbps!
![Page 4: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/4.jpg)
![Page 5: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/5.jpg)
![Page 6: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/6.jpg)
![Page 7: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/7.jpg)
@ksshams
![Page 8: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/8.jpg)
@ksshams@swami_79
let’s start with a trilogy …
![Page 9: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/9.jpg)
@ksshams@swami_79
once upon a time...
(in 2000)
episode 1
![Page 10: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/10.jpg)
@ksshams@swami_79
a half mile away... (Seattle)
![Page 11: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/11.jpg)
@ksshams@swami_79
amazon.com - a rapidly growing Internet based retail business relied on relational databases
![Page 12: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/12.jpg)
@ksshams@swami_79
we had 1000s of independent services
![Page 13: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/13.jpg)
@ksshams@swami_79
each service managed its own state in
Relational Databases
![Page 14: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/14.jpg)
@ksshams@swami_79
Relational Databases are pretty seductive
![Page 15: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/15.jpg)
@ksshams@swami_79
first of all... SQL!!
![Page 16: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/16.jpg)
@ksshams@swami_79
so it is easier to query..
![Page 17: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/17.jpg)
@ksshams@swami_79
easier to learn
![Page 18: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/18.jpg)
@ksshams@swami_79
They are as versatile as a swiss army knife
complex queries key-value access
transactionsanalytics
![Page 19: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/19.jpg)
@ksshams@swami_79
Relational Databases are very similar toSwiss Army Knives
![Page 20: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/20.jpg)
@ksshams@swami_79
sometimes.. swiss army knifes.. can be more than what you bargained for
![Page 21: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/21.jpg)
@ksshams@swami_79
partitioningeasy
re-partitioning
HARD..
![Page 22: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/22.jpg)
@ksshams@swami_79
so we bought
bigger boxes...
![Page 23: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/23.jpg)
@ksshams@swami_79
Q4 was hard-work at Amazon
benchmark new hardware
migrate to new hardware
repartition databases
pray ...
![Page 24: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/24.jpg)
@ksshams@swami_79
Relational Databases have availability
challenges..
![Page 25: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/25.jpg)
@ksshams@swami_79
then.. (in 2005)
episode 2
![Page 26: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/26.jpg)
@ksshams@swami_79
amazon dynamopredecessor to dynamoDB
specialist tool : •limited querying capabilities•simpler consistency
replicated DHT with consistent hashingoptimistic replication“sloppy quorum”anti-entropy mechanismobject versioning
![Page 27: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/27.jpg)
@ksshams@swami_79
dynamo had many benefits• higher availability• we traded it off for consistency
• incremental scalability• no more repartitioning • no need to architect apps for peak• just add boxes
• simpler querying model ==>> predictable performance
![Page 28: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/28.jpg)
@ksshams@swami_79
but dynamo was not perfect...
lacked strong consistency
![Page 29: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/29.jpg)
@ksshams@swami_79
but dynamo was not perfect...
scaling was easier, but...
![Page 30: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/30.jpg)
@ksshams@swami_79
but dynamo was not perfect...
steep learning curve
![Page 31: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/31.jpg)
@ksshams@swami_79
but dynamo was not perfect...
dynamo was a library ==>> not a service...
![Page 32: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/32.jpg)
@ksshams@swami_79
then.. (in 2012)
episode 3
![Page 33: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/33.jpg)
@ksshams@swami_79
ADMIN
DynamoDB
Managed NoSQL Database
Fast & Predictable Performance
Built for Scale
![Page 34: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/34.jpg)
@ksshams@swami_79
“Even though we have years of experience with large, complex NoSQL architectures, we are
happy to be finally out of the business of managing it ourselves.” - Don MacAskill, CEO
DynamoDB
![Page 35: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/35.jpg)
@ksshams@swami_79
DynamoDB Goals and Philosophies
durability and availability
scale is our problem
easy to use
scale in rps
consistent and low latencies
![Page 36: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/36.jpg)
@ksshams@swami_79
durability is key…
![Page 37: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/37.jpg)
@ksshams@swami_79
availability is key…
![Page 38: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/38.jpg)
@ksshams@swami_79
scale is our problem, not yours..
![Page 39: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/39.jpg)
@ksshams@swami_79
Fault Tolerant Design
Infrastructure Fails - deal with it!
Planning for failures is not easy
How do you ensure your recovery strategies work correctly?
![Page 40: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/40.jpg)
@ksshams@swami_79
Byzantine General Problem
![Page 41: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/41.jpg)
@ksshams@swami_79
A simple 2-way replication system of a traditional database…
Primary Standby
Writes
![Page 42: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/42.jpg)
@ksshams@swami_79
P S
S is dead, need to
trigger new replica
P is dead, need to promote
myself
![Page 43: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/43.jpg)
@ksshams@swami_79
Improved Replication: Quorum
Writes
Replica
Quorum: Successful write on a majority
Replica
Replica
![Page 44: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/44.jpg)
@ksshams@swami_79
Easy?
Replica B
Replica C
Writes from client XReplica A
Replica D
New member in the group
Should I continue to serve reads? Should I start a new quorum?
Replica E Replica F
Reads and Writes from
client Y
Classic Split Brain Issue in Replicated systems leading to lost writes!
![Page 45: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/45.jpg)
@ksshams@swami_79
Building correct distributed systems is not straight forward..
Handle partial failures of replicas
Handle replica failures
Ensure there isn’t a parallel quorum
Handle concurrent failures
![Page 46: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/46.jpg)
@ksshams@swami_79
Trends in the World Of Databases
![Page 47: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/47.jpg)
@ksshams@swami_79
A decade ago, it was all about the DBAs
![Page 48: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/48.jpg)
@ksshams@swami_79
Last 5 years have been about self service.
![Page 49: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/49.jpg)
@ksshams@swami_79
Today is about managed services.
![Page 50: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/50.jpg)
@ksshams@swami_79
Plan for Success … Plan for Scale
![Page 51: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/51.jpg)
@ksshams@swami_79
![Page 52: NoSQL like there is No Tomorrow](https://reader038.vdocuments.us/reader038/viewer/2022102900/54b3a7e14a795958198b45f3/html5/thumbnails/52.jpg)
@ksshams@swami_79