provable semantics for storage stacks - usenix · beyond storage apis: provable semantics for...

BEYOND STORAGE APIS: PROVABLE SEMANTICS FOR STORAGE STACKS

UNIVERSITY OF WISCONSIN-MADISON

RAMNATTHAN ALAGAPPAN

VIJAY CHIDAMBARAM

THANUMALAYAN PILLAI

REMZI ARPACI-DUSSEAU

ANDREA ARPACI-DUSSEAU

AWS ALBARGHOUTHI

2

Application

Laptops

2

Application

Laptops Desktops

2

Application

Laptops Desktops

Mobile Devices

2

Application

Laptops Desktops

Mobile Devices

Private and Public Clouds

2

Application

Laptops Desktops

Mobile Devices

Private and Public Clouds

Heterogeneity of environments is increasing

STORAGE STACKS: DEEP AND DIVERSE

3

Windows IO stack has 18 layers! [ThereskaSOSP13]

STORAGE STACKS: DEEP AND DIVERSE

3

Application

SSTF

Disk

ext4

Workstation

Application

CFQ

SSD

btrfs

Laptop

Application

CFQ

SSD

F2FS

Mobile

Windows IO stack has 18 layers! [ThereskaSOSP13]

APPLICATION PORTABILITY

Applications should be portable between environments

- Reduce development effort and bugs

- Avoid vendor lock-in

4

Amazon EC2 OpenStack Nova

Application 1 Application 2

Common API

API Compatibility is not enough!


Application correctness depends upon unspecified properties

of the storage stack


Application correctness depends upon unspecified properties

of the storage stack

Results: data corruption, data loss, unavailability [PillaiOSDI14]

THE VISION

6

Application

Disk

ext4

Node 1

…

SSD

btrfs

Node 1

…

CFQ

SSD

F2FS

Mobile

Dat

acen

ter

THE VISION

6

Application

Disk

ext4

Node 1

…

SSD

btrfs

Node 1

…

CFQ

SSD

F2FS

Mobile

Dat

acen

ter

?

??

Quick, automated check at deployment

THE VISION

6

Application

Disk

ext4

Node 1

…

SSD

btrfs

Node 1

…

CFQ

SSD

F2FS

Mobile

Dat

acen

ter

?

??

Quick, automated check at deployment

✅

❌

✅

THE VISION

7

Application

Disk

ext4

Node 1

…

SSD

btrfs

Node 1

…

Dat

acen

ter

THE VISION

7

Application

Disk

ext4

Node 1

…

SSD

btrfs

Node 1

…

Dat

acen

ter

Which is the best node to deploy this application to?

THE VISION

7

Application

Disk

ext4

Node 1

…

SSD

btrfs

Node 1

…

Dat

acen

ter

??


THE VISION

7

Application

Disk

ext4

Node 1

…

SSD

btrfs

Node 1

…

Dat

acen

ter

??


✅

✅

THE VISION

7

Application

Disk

ext4

Node 1

…

SSD

btrfs

Node 1

…

Dat

acen

ter

??


✅

Best: least # of stack layers, least utilized, etc.

✅

OUTLINE

Introduction

Portability Bug Study

First Steps Toward The Vision

The Road Ahead

8

1

3

2

4

PORTABILITY BUG STUDY

Portability bug: bug that occurs when an application is moved to a different environment

Studied public bug databases

- Android deployed on different mobile devices

- Applications run on cloud platforms and on NFS

Performed our own experiments based on previous work [PillaiOSDI14]

9

OPERATIONS NOT SUPPORTED

10

SQLite creates temporary files by opening a file and unlinking them

Not supported by the daemon emulating FAT32

on the sd card

FAT32 File System

UNEXPECTED ERROR CODES

11

fsync() on FreeBSD returns ENOLCK

even on success

MySQL restarts when it sees that error

NFS

ORDERING REQUIREMENTS NOT MET

12

Cloud

Partial File!log

filepart2

filepart 1

filepart1

File System

Inotify

13

ext4

Workstation

Guarantee: Committed data can always be read back after a crash

POSIX

btrfs

Laptop

POSIX

All file systems are not created equal: On the complexity of crafting crash-consistent applications, OSDI 2014

13

ext4

Workstation



POSIX

btrfs

Laptop

POSIX


13

ext4

Workstation

POSIX

btrfs

Laptop

POSIX

Write A Write B Write C

A

A B A B C

B


13

ext4

Workstation

We term this an Application Crash Vulnerability

POSIX

btrfs

Laptop

POSIX


13

ext4

Workstation

We term this an Application Crash Vulnerability

POSIX

btrfs

Laptop

POSIX

API compatibility is not enough!


OUTLINE

Introduction



The Road Ahead

14

1

3

2

4

Formally verify that an application

will run correctly on a given storage stack

15

What application requires

<=What

storage stack provides

WHY IS THIS HARD?

16

Application requirements can be complex

- e.g., append(“AB”) should result in file containing A or AB

- if then else form

Binary or numerical checks are not sufficient

<= What storage stack provides


WHY IS THIS HARD?

16

Application requirements can be complex

- e.g., append(“AB”) should result in file containing A or AB

- if then else form

Binary or numerical checks are not sufficient

Need expressive language for specifying application requirements



WHY IS THIS HARD?

17

Disk provides atomic reads and writes

File system provides atomic metadata operations

Disk

ext3

PostgresPostgres provides ACID transactions



WHY IS THIS HARD?

17

Disk provides atomic reads and writes

File system provides atomic metadata operations

Disk

ext3

PostgresPostgres provides ACID transactionsNeed to dynamically compute

guarantees provided by the stack



OVERVIEW

18

Application

CFQ

Disk

ext4

Model guarantees the application requires from storage as a theorem

Model guarantees provided by each layer of the storage stack

as axioms

Ex: application will be crash-consistent if all writes are ordered and atomic

Ex: disk guarantees sector-level reads and writes are atomic even with crash

OVERVIEW

18

Application

CFQ

Disk

ext4

Model guarantees the application requires from storage as a theorem

Model guarantees provided by each layer of the storage stack

as axioms

Ex: application will be crash-consistent if all writes are ordered and atomic

Ex: disk guarantees sector-level reads and writes are atomic even with crash

Prove application theorem using axioms from storage stack

SYSTEM

19

Application Requirements

Stack Configuration Library of

stack-layer specifications

Result

PROVER

E.g., put() must be atomic

E.g., put() is atomic on given stack

E.g., Key-Value Store on top of disk

SYSTEM

20




Result




SYSTEM

20




Result



Use the proof assistant to manually write machine-checked proofs


EXPERIENCE WITH ISABELLE

Modelled a simple 2-layer stack

- Key-value store on top of a disk

Proved put() is atomic

About 160 lines of code (lots of trial and error)

Code available at: https://github.com/ramanala/StorageStackSemantics

21

OUTLINE

Introduction



The Road Ahead

22

1

3

2

4

CHALLENGES

Obtaining specifications

- Developer provides/written by grad students

- How to figure out automatically?

Automatic proofs

- Use Z3 instead of Isabelle?

Proofs without specifications

- Know a layer provides guarantees, without knowing how

Verifying implementations

23

CONCLUSION

The promise of software-defined storage

- Increases in performance, flexibility, and utilization

- Unspoken aspect: application correctness!

Simply ensuring API compatibility is not enough

- Storage semantics are complex and nuanced

PL tools like SMT solvers/proof assistants can help match application to diverse storage stacks

Interesting, significant challenges on path ahead

24

THANK YOU! QUESTIONS?

SOURCE CODE AT:HTTP://CS.WISC.EDU/~VIJAYC

VIJAY CHIDAMBARAM UNIVERSITY OF WISCONSIN MADISON [email protected] | HTTP://CS.WISC.EDU/~VIJAYC

provable semantics for storage stacks - usenix · beyond storage apis: provable semantics for...

Documents