1

NoSQL, CAP, and relativity2013-09-18Lars Marius Garshol, larsga@bouvet.no, http://twitter.com/larsga

2

Agenda

• CAP theorem mostly• A bit about NoSQL databases• Are triple stores NoSQL?• Connection with Einstein’s theory

of relativity• And, finally, a surprise

3

NoSQL databases

4

What makes a NoSQL database?• Doesn’t use SQL as query

language– usually more primitive query language– sometimes key/value only

• BASE rather than ACID– that is, sacrifices consistency for

availability– much more about this later

• Schemaless– that is, data need not conform to a

predefined schema

5

BASE vs ACID

• ACID– Atomicity– Consistency– Isolation– Durability

• BASE– Basically Available– Soft-state– Eventual consistency

6

Eventual consistency• A key property of non-ACID systems• Means

– if no further changes made,– eventually all nodes will be consistent

• In itself eventual consistency is a very weak guarantee– when is “eventually”? it doesn’t say– in practice it means the system can be inconsistent

at any time• Stronger guarantees are sometimes made

– with prediction and measuring, actual behaviour can be quantified

– in practice, systems often appear strongly consistent

7

Implementing ev. consistency• Nodes must exchange information

about writes– basically, after performing a write, node must

inform all other replicas of the changed objects– signal OK to reader before or during replication– for example, by broadcast to all nodes

• Must have some way to deal with conflicts– all nodes must agree on conflict resolution– common solution: embed clock value in write

message, then let last writer win– clock need not be in sync for all nodes

8

What’s wrong with ACID?

• Semantics are easier for developers– applications can lean back and just trust

the db• However, doesn’t scale as well– that is, doesn’t scale with number of

nodes– requires too much communication and

agreement between nodes• Bigger web sites have therefore

gone BASE– Facebook, Flickr, Twitter, ...

9

Other benefits of NoSQL

• Schemaless– possible to write much more flexible code– schema evolution vastly easier

• Avoid joins– document databases allow hierarchical

non-normalized objects to be retrieved directly

10

Downsides to NoSQL• Everyone knows SQL

– few people know your specific NoSQL database• Lack of validation

– code will typically do anything the database lets it get away with (especially over time)

• No standards– you can’t easily switch databases– (well, except with SPARQL)

• Lack of maturity– lack of supporting tools, unpleasant

surprises, ...• Weak query languages

– means you have to do more in code– may hurt performance

11

Triple stores (SPARQL)

• Non-SQL? Yes• BASE? No• Schemaless? Yes

Only two out of three, so whethertriple stores are NoSQL databasesis debatable.

At the very least, they differ substantiallyfrom the core examples of NoSQL databases.

12

Can triple stores be BASE?

• In theory, yes– nothing inherent in graph structure that

prevents it• But,– how do you shard graph data?– no known way to do it that’s efficient

• So, in practice this is hard

13

When should you use NoSQL?• If scalability is a concern– however, don’t forget that sites like Flickr

and Wikipedia used RDBMSs for years– relational databases scale a long way

• If schemalessness is important– sometimes it really is– seriously consider RDF for this use case

• If fashion is a concern– for a surprising number of people, using

something new and shiny is the main thing

14

The CAP Theorem

15

CAP

• Consistency– all nodes always give the same answer

• Availability– nodes always answer queries and accept

updates• Partition-tolerance– system continues working even if one or

more nodes go quietCAP Theorem: You can only have two of these.Partition tolerance: Without this, the cluster dies the moment one node goes silent. Can’t really drop this one.

16

C ≠ C

• C in ACID– means all data obeys constraints in

schema• C in CAP– means all servers agree on the data

• However,– ACID implementations also follow the C

in CAP

17

History

• First formulated by Eric Brewer in 2000– based on experience with Inktomi search

engine– described the SQL/NoSQL divide very

well– coined the BASE acronym

• Formalized and proven in 2002– by Seth Gilbert and Nancy Lynch

• Today CAP is better understood– widely considered a key tradeoff in

designing distributed systems– and particularly databases– in some ways gave rise to NoSQL

databases

18

Consistent or Available?

request request request

19

Consistency

DB node 1

DB node 2

Client 1

Client 2

read account Xbalance -> 100

set account X balance = 0 set account X

balance = 0set account X balance = 0

set account X balance = 0

20

Availability

DB node 1

DB node 2

Client 1

Client 2

read account Xbalance -> 100

set account X balance = 0 set account X

balance = 0

set account X balance = 0

read account Xbalance -> 100

set account X balance = 0

Happy customer walksaway, richer by 200.

Servers eventually agreebalance is 0.

21

What exactly is the problem?

• The ordering of events affects the outcome

• The different nodes do not necessarily observe the same order

• However, it is possible to impose a consistent ordering of events

• The trouble is, that involves communication with all nodes

• Waiting for all of them takes time

22

Math digression

23

Time, Clocks and the Ordering of Events in a Distributed SystemThe origin of this paper was a note titled The Maintenance of Duplicate Databases by Paul Johnson and Bob Thomas. I believe their note introduced the idea of using message time-stamps in a distributed algorithm. I happen to have a solid, visceral understanding of special relativity (see [5]). This enabled me to grasp immediately the essence of what they were trying to do. ... I realized that the essence of Johnson and Thomas's algorithm was the use of timestamps to provide a total ordering of events that was consistent with the causal order. ...

It didn't take me long to realize that an algorithm for totally ordering events could be used to implement any distributed system.

http://research.microsoft.com/en-us/um/people/lamport/pubs/pubs.html#time-clocks

24

Order theory

• An ordering relation is any relation ≤ such that– a ≤ a (reflexivity)– if a ≤ b and b ≤ a then a = b

(antisymmetry)– if a ≤ b and b ≤ c then a ≤ c (transitivity)

• A total order is an order such that– a ≤ b or b ≤ a (totality)

• A partial order is any order which is not total– that is, for some pairs a and b, neither a ≤

b nor b ≤ a

25

Examples

• Total orders– normal ordering of numbers and letters

• Partial orders– ordering sets by the subset relation (see

figure)– “In general, the ·order-relation· on

duration is a partial order since there is no determinate relationship between certain durations such as one month (P1M) and 30 days (P30D)” XML Schema, pt2

26

Relativity

27

History

• 1687– Isaac Newton publishes

Philosophiæ Naturalis Principia Mathematica

– physics begins– no changes over next two

centuries• 1905– Albert Einstein publishes special

relativity– abandons notions of fixed time

and space• 1916– Einstein’s general relativity– takes into account gravity– no changes since

28

...(we skip 270 slides)

29

The barn is too small

• Three people (M, F, and B) own a board (5m wide) and a barn (4m wide)

• The board doesn’t fit inside the barn!

• What to do?

4

30

31

We have a solution!

As seen by F & B: board is 4m long (relativistic shortening), barn continues to be 4m wide.

When the board is exactly inside the barn, F and B will close their doors simultaneously,and the problem will be solved.

(As seen by M: board is 5m long (at rest relative to him), barn shortened to 3.2m wide. Pay no attention to this.)

32

What they observeF and B• When the board is

just inside, both close their doors simultaneously

• Right after, the board crashes through back door

M• B shuts his door just

as the front of the board reaches him

• 0.6 seconds later, F closes his door

• Board crashes through back door

33

The key point• They don’t agree on the order of events!

– and this is not a paradox– it is in fact how the universe works

• Change the story slightly and the three people could have three different orders of events

• No total order of events exists on which all observers can agree– the ordering of events in the universe is a

partial order• What then of causality?

– if A causes B, but some people think B happened before A, then what?

34

Resolution

• A, B, and C are events• The cone is the “light cone”

from A– that is, the spread of light from A

• C is outside the cone– therefore A cannot influence C– observers may disagree on order

of A&C• B is inside the cone– therefore A can influence B– observers may not disagree on

order

35

...(we skip 532 slides)

36

Back to CAP

37

Relevance to the CAP theorem• Distributed nodes will never agree

on the order of events– unless a communication delay is

introduced• Basically, only events inside the

“light cone” can be totally ordered– communications delay can never be less

than time taken by light to traverse physical distance

– in practice, it will be quite a bit bigger– how big depends on hardware and

design constraints

38

One solution: Paxos

• Protocol created by Leslie Lamport• Can be used to introduce logical clock– all nodes agree to always increase the

number• Which again can order events• Allowing all nodes to agree on order of

eventshttp://research.microsoft.com/en-us/um/people/lamport/pubs/pubs.html#paxos-simple

39

Another solution: ev. consistency• Essentially, this solution says some

% of errors is acceptable– Amazon may have to compensate

purchasers with a gift card once every 100,000 transactions

– business value of remaining available is higher than cost of errors

• Even in banking this may apply– ATMs often continue working even if

contact with bank is lost– allow withdrawals up to some limit– accept it if customers overcharge– (they’ll have to pay fees and interest,

anyway)

40

CALM• Client code may have to compensate

for database inconsistencies– this can quickly become complex– complex means error-prone

• However, there is a way around that– CALM = consistency as logical monotonicity– means facts used by clients to make decisions

never change• A database that never deletes or

overwrites is CALM– for example because it logs trades or other

events– non-CALM databases may require ACID

41

A CALM example

• Client 1 reads A = 10• Client 1 uses this information to

write B = 5• If Client 2 now reads B = 5, that

client cannot read a value of A older than A = 10

• Doing so would violate what’s known as “causal consistency”

42

ACID 2.0

• A bit misleading, because it’s not really ACID at all

• Basically requires update operations to have these properties– associativity a + (b + c) = (a

+ b) + c– commutativity a + b = b + a– idempotence f(x) = f(f(x))– distributed

• One approach is to use datatypes which guarantee these properties

43

SDShare updates

• These are actually– associative– commutative– idempotent

• So SDShare is already ACID 2.0...

44

Another solution: datatypes

• CRDTs– commutative, replicated data types

• Basically,– data types designed so that the order of

operations doesn’t matter– the end result is always the same,

regardless of the order of operations

45

An example

• A problem with our cash withdrawal example is the overwrite operation

• What if the operation were “increment(account, -100)” instead?– this operation is associative and

commutative– (not inherently idempotent, however)

• Nodes can now apply incoming updates as they get them– the ordering of updates can be ignored– once all updates are applied, all nodes

will agree (which is eventual consistency)

46

Thus far, all is well, and everyone agrees

47

“To go wildly faster, one

must remove all four

sources of the overhead

discussed above. This is

possible in either a SQL

context or some other

context.”

48

What on earth is he talking about?

Everyone knows SQL and ACID don’t scale!

49

Meanwhile, at Google...

50

AdWords database difficulties

This backend was originally based on a MySQL database

that was manually sharded many ways. The uncompressed dataset is tens of terabytes, which is small compared to many NoSQL instances, but was large enough to cause difficulties with sharded MySQL. The MySQL sharding scheme assigned each customer and all related data to a fixed shard. This layout enabled the use of indexes and complex query processing on a per-customer basis, but required some knowledge of the sharding in

application business logic. Resharding this revenue-critical database as it grew in the number of customers and their data was extremely costly. The last resharding took

over two years of intense effort, and involved coordination and testing across dozens of teams to minimize risk.

51

More background

We store financial data and have hard requirements on data integrity and consistency. We also have a lot of experience with eventual consistency systems at Google. In all such systems, we find developers spend a significant fraction of their time building extremely complex and error-prone mechanisms to cope with eventual consistency and handle data that may be out of date. We think this is an unacceptable burden to place on developers and that consistency problems should be solved at the database level.

52

Yet more background

At least 300 applications within Google use Megastore (despite its relatively low per- formance) because its data model is simpler to manage than Bigtable’s, and because of its support for synchronous replication across datacenters. (Bigtable only supports eventually-consistent replication across data- centers.) Examples of well-known Google applications that use Megastore are Gmail, Picasa, Calendar, Android Market, and AppEngine.

53

Requirements

• Scalability– scale simply by adding hardware– no manual sharding

• Availability– no downtime, for any reason

• Consistency– full ACID transactions

• Usability– full SQL with indexes

Uh, didn’t we just learnthat this is impossible?

54

Spanner

• Globally-distributed semi-relational db– SQL as query language– versioned data with non-locking read-

only transactions• Externally consistent reads/writes• Atomic schema updates– even while transactions are running

• High availability– experiment: killing 25 out of 125 servers

has no effect (except on throughput)

55

Transaction model

• Fairly close to traditional MVCC– every row has a timestamp– reads have associated timestamp, see

database as of that point in time• The key is a consistent order of

timestamps across nodes

56

Spanner architecture

57

Spanner architecture #2

58

TrueTime• Enables consistency in Spanner by giving

transactions timestamps– that is, imposes a consistent ordering on

transactions• Represents time with uncertainty interval

– the bigger the uncertainty, the more careful nodes must be

– bigger uncertainty leads to slower transactions• Uses two kinds of time servers to reduce

uncertainty– GPS-based servers– atomic clocks

59

Use of Paxos is key

• Combines Paxos with TrueTime to ensure timestamps are monotonically increasing

• Paxos requires majority votes to agree– implies less than half of data centers can

fail at any one time• AdWords therefore runs with 5 data

centers– allows two simultaneous failures without

effect– three on East Coast, two on West Coast– (in Google East Coast + West Goast =

globally)

60

Data model

61

F1 – the next layer up

• Builds on Spanner, adds– distributed SQL queries– including joins from external sources– transactionally consistent indexes– asynchronous schema changes– optimistic transactions– automatic change history

62

Why built-in change history?“Many database users build mechanisms to log changes, either from

application code or using database features like triggers. In the MySQL

system that AdWords used before F1, our Java application libraries

added change history records into all transactions. This was nice, but it

was inefficient and never 100% reliable. Some classes of changes would

not get history records, including changes written from Python

scripts and manual SQL data changes.”

Application code is not enough to enforce business rules,because many important changes are made behind theapplication code. For example, data conversion.

Look at any database that’s a few years old, and you’llfind data disallowed by the application code, but allowedby the schema.

63

Distributed queries

64

Two interfaces

• NoSQL interface– basically a simple key->row lookup– simpler in code for object lookup– faster because no SQL parsing

• Full SQL interface– good for analytics and more complex

interactions

65

Status

• >100 terabyte of uncompressed data– distributed across 5 data centers– Five nines (99.999%) uptime

• Serves up to hundreds of thousands of requests/second

• SQL queries scan trillions of rows/day

• No observable increase of latency compared to MySQL-based backend– but change tracking and sharding now

invisible to application

66

Winding up

67

Conclusion

• NoSQL is mostly about BASE– to some degree also schemalessness

• The CAP Theorem is key to understanding distributed systems– NoSQL is BASE because of CAP

• The CAP Theorem is a consequence of the theory of relativity

• New systems seem to indicate that ACID may scale, after all– basically, the speed of light is greater

than we thought

68

Further reading• NoSQL eMag, InfoQ, pilot issue May 2013– http://www.infoq.com/minibooks/emag-NoSQL

• Brewer’s original presentation– http://www.cs.berkeley.edu/~brewer/cs262b-

2004/PODC-keynote.pdf• Proof by Lynch & Gilbert– http://lpd.epfl.ch/sgilbert/pubs/BrewersConjecture-

SigAct.pdf• Why E=mc2?, Cox & Forshaw• Eventual Consistency Today: Limitations,

Extensions, and Beyond, ACM Queue– http://queue.acm.org/detail.cfm?id=2462076

69

Further reading

• Spanner paper– http://research.google.com/archive/

spanner.html• F1 papers– http://research.google.com/pubs/

pub38125.html– http://research.google.com/pubs/

pub41376.html