martin slides

8/3/2019 Martin Slides

1/27

Martin Kaufmann ETH ZrichMay 2nd, 2009 1

Building aDatabase on S3

Martin Kaufmann

Hot Topics in Data Management SystemsSeminar 2009


2/27

May 2nd, 2009 Martin Kaufmann ETH Zrich 2

Motivation

Database


3/27


Motivation

Database

How can we do better?


4/27


Motivation

Database


5/27


Goal

Build a database system in the cloud Preserve advantages of utility computing service Availability

Scalability

Same characteristics as general-purpose database system Support for small objects and frequent updates

ACID properties

Is that possible?


6/27


Overview

Motivation What is Amazon AWS? Amazon S3

Amazon SQS

Building a database on S3 CAP Theorem

Using S3 as a disk

Basic commit protocols (example why needed)

Evaluation of the paper

Related work Amazon SimpleDB

Conclusion


7/27May 2nd, 2009 Martin Kaufmann ETH Zrich 7

Amazon S3 (Simple Storage Service)

S3 is a distributed storage based on (key-value) pairs Possible to save objects between 1 byte and 5 GB

Advantages Infinite storage

Perfect scalability

~ 100% availability

Drawbacks Slow compared to local disk

Eventual consistency only

Can be quite expensive

Client

S3

(key, value)



Amazon S3 in more detail

Objects are organized in buckets Objects and buckets are identified by a URI

Access by means of a simple Web Service API

Most important operationsput(uri, bytestream)

get(uri)

get-if-modified-since(uri, timestamp)

Object 1

Object n

Bucket

k

S3



Amazon SQS (Simple Queueing Service)

SQS provides queues for storing messages

Message size to 256 KB, infinite number, always available

Each queue is identified by a URI, each message by an ID

Most important operationscreateQueue(uri)

sendMessage(uri, msg)

receiveMessage(uri, number-of-msg, timeout)

deleteMessage(uri, msg-id)

Queue k

Message 1

Message 2

...

Message nSQS

Client

send()

receive()



Overview

Motivation

What is Amazon AWS? Amazon S3

Amazon SQS


Using S3 as a disk




Conclusion



CAP Theorem

Three goals: Availability, Consistency, Partition Tolerance

Eric Brewer`s CAP Theorem (2000)

Can only have 2 of them at a time in a distributed system!

Availability

ConsistencyPartition

Tolerance

RelationalDB

AmazonAWS



Using S3 as a disk

Observation: S3 has similary properties like a disk

Can`t we just exchange the disk with S3?

Can we store each tuple directly as an object in S3?

Database Database

S3



Let`s do some experiments

Store objects of different size on S3

Result: S3 very slow compared to disk

S3 needs a minimum of 100 ms

per request (for every size)

Problems This is too slow!

No consistency guarantee!0,001 s

0,01 s

0,1 s

1 s

10 s

10 KB 100 KB 1000 KB

Amazon S3

Harddisk

Conclusion S3 superior to local disk in terms of throughput

Need to read data in chunks larger than 100 KB,

cluster several records (= tuples) into pages

Need to take care of concurrent access

Comparison: S3 and harddisk with 7200rpm, 10ms



Architecture

Application Layer

Database client

Management Layer

Record Manager

Page Manager

Storage Layer

Amazon S3

Pages

Application Layer

Database client

Management Layer

Record Manager

Page Manager

Client 1: Client n:

Records Records

ManagementLayer



Record Manager

Record Relational tuple Record contains key and payload

Key is unique in one collection

Collection Relational table: Each record belongs to a collection

A collection is stored as an S3 bucket

Functions of the Record Manager Create new objects, read and objects, scan collections, commit/abort

Management Layer

Record Manager

Page Manager

Page i

Record a

S3

Record b

Bucketk

Page j

Record c



Page Manager

Page Single object in S3

Buffer pool for S3 pages

After commit All updates propagated to S3

Page is kept in buffer

After abort Delete modified pages from buffer pool

Management Layer

Record Manager

Page Manager

Page i

Record a

S3

Record b

Bucket

kPage j

Record c



Are we finished now?

Looks nice, BUT what happens if

two clients update different records in the same page?

Page i

Record a

S3

Record b

Bucketk

Page iRecord c

Client 1

Client 2



Basic commit protocols

Pending update (PU) queue for every page

Queues are implemented on Amazon SQS

Clients submit committed log records to PU queue

Checkpointing: PU queue written to S3 regularly

PU Queue iUpdate 1

Update 2

...

Update n

Client a

updateS3

Page i

checkpoint

Client b

update



Are we finished NOW ?

Checkpointing is done by clients!

What happens if two clients carry out a checkpoint on thesame page?

We have to introduce a lock queue for every PU queue.

Lock Queue i

Token

PU Queue i

Update 1

Update 2

...

Update n

Client S3

checkpoint

1. get token (i)

2. grant token (i)

3. execute checkpoint



Overview

Motivation


Amazon SQS


Using S3 as a disk




Conclusion



Evaluation

Strong points

Interesting idea to build a database in the cloud

Try to use textbook methods where it is possible

Working prototype

Provides database with infinite amount of storage Preserve adantages of utility computing

Weak points

Security problems not solved

Did not explore the possibility to use EC2

Did not consider aggregation functions



Overview

Motivation


Amazon SQS


Using S3 as a disk




Conclusion



Amazon SimpleDB

Architecture Domain elastic table, no support / need for schema

Item has one or many attributes, different attributes per item!

Attribute (name, {value}) pair, a name can have multiple values

Properties Indexing done atomatically

Simple, Flexible, Scalable

Similarities to DB on S3 paper

Store data on S3, no strict consistency

Differences Each tuple can have different attributes

An attribute name can have multiple values

Name

= Harry

Hobby= Quidditch

Name

= Ron

Friend

= Hermine,

Harry

Domain

ItemAttribute



Overview

Motivation


Amazon SQS


Using S3 as a disk




Conclusion


25/27


Conclusion

Databases in the cloud have lots of advantages Infinity amount of storage

Perfect availability

Scalability

High throughput

There are also disadvantages High latency

Only eventual consistency (all solutions so far!)

If a DB in the cloud matches the needs dependson the application! Performance requirements, storage capacity,

number of concurrent users, cost,


26/27


References

M. Brantner et al. Building a Database on S3.SIGMOD, 2008.

Eric Brewer. Towards rebust distributed systems.Principles of Distributed Systems 2000.

Amazon Web Services

http://aws.amazon.com/


27/27

The end

Any questions ?

martin slides

Documents