IDO: Intelligent Data Outsourcing with Improved RAID Reconstruction Performance

in Large-Scale Data Centers

Suzhen Wu§*, Hong Jiang*, Bo Mao* §Xiamen University

*University of Nebraska–Lincoln

Data Deluge

2

Social Network

Business Intelligence

Scientific Simulation

Mobile Apps

2,300 tweets per

second

275 EB data flowing per day in 2020

How to safely store such a huge data volume proposes a big challenge to

the system administrators!

Where Are We?

3

Laptop and Desktop Data Center

Interruptible Event

Common Case

Disk Failure in the Real World

4

•  Higher  error  rates  than  expected  – Complete  disk  failures,  2%~4%  on  average;  – Latent  sector  errors,  3.45%;  

•  CorrelaBon  in  drive  failures  – e.g.,  aCer  one  disk  fails,  another  disk  failure  will  likely  occur  soon.  

•  RAID  reconstrucBon  becomes  an  operaBonal  state  in  data  centers  –  Increasing  disk  capacity  and  number  of  drives  

More Observations

•  Linux software RAID (MD) mailing list: too many complains about the slow recovery speed.

•  Storage at Exascale: Some thoughts from Panasas CTO Garth Gibson. Disk failure is a normal case in exascale storage systems.

•  ……

5

RAID Reconstruction Challenges

6

•  Online RAID Reconstruction:

     •  Two challenges:

– Real-time user performance;

– Window of vulnerability.

User I/O Requests

Reconstruction I/O Requests

How many user I/O requests can be eliminated from degraded RAID directly affects the reconstruction performance.

The State of the arts

7

•  Optimizing the reconstruction workflow: – DOR (CMU PDL) – Live-block recovery (USENIX FAST’04)

– PRO (USENIX FAST’07)

•  Optimizing the user I/O requests: – MICRO (IEEE TC’08)

– WorkOut (USENIX FAST’09) – VDF (USENIX ATC’11)

Compare with State of the arts

8

Characteris*cs PRO  (FAST’07)

WorkOut  (FAST’09)

VDF  (USENIX’11)

IDO    (LISA’12)

ProacBve √

Temporal  Locality √ √ √ √

SpaBal  Locality √ √

User  I/O √ √ √

ReconstrucBon  I/O √ √

Observation 1

9

•  RAID reconstruction is an operational state in large-scale data centers which means reactive scheme is inefficient. – Reactive vs. Proactive?

•  Existing studies are all reactive schemes.

Example 1: Reactive vs. Proactive

10

Example 1: Reactive vs. Proactive

11

Observation 2

12

•  With large RAM and SSDs, the temporary locality is poor at HDD level. However, the spatial locality is good due to the sequential accesses of HDDs. – Temporal locality vs. Spatial locality?

•  Existing studies mostly focus on temporal locality and ignore spatial locality.

Example 2: Temporal vs. Spatial

13

a b c d(a) Request-based approach

Migrate requested “a” to Surrogate Set

a

(b) Zone-based approach a b c d

a b c d

Migrate hot zone to Surrogate Set

The Motivation

14

0%

20%

40%

60%

80%

100%

WebSearch2 Financial2 Microsoft Project

Use

r I/O

traf

fic re

mov

ed fr

om

degr

aded

RA

IDReactive-requestReactive-zone

Proactive-requestProactive-zone

IDO: Intelligent Data Outsourcing

15

•  The main idea: –  Proactively identify the hot data zones; –  Upon disk failure,

•  Recovery the hot data zones first;

•  Migrate the hot data zones to surrogate set;

•  Redirect the user I/O requests.

•  The design objectives –  Reducing reconstruction time; –  Improving the user I/O performance;

–  Applicable to other background tasks.

System Overview

16

Failed Disk

New Disk

Software RAID Controller

Network

Stor

age

Nod

e

Stor

age

Nod

e Data Migration

Working / Degraded RAID Surrogate RAID Working / Surrogate RAID

RAID Reconstruction

IDO

RAID Reconstruction

Hot Zone Identifier

Data Migrator

Request Distributor

Data Reclaimer

Software RAID Controller

IDO

Request Distributor

Hot Zone Identifier

Data Migrator

Data Reclaimer

Performance Evaluation

17

•  IDO prototype is a built-in module in Linux MD, compared with WorkOut and VDF.

•  Intel Xeon 3440 processor, 8GB DDR memory, WDC WD1600AAJS SATA disks.

•  Trace-driven evaluations

RAID5 Results

18

(a) Average Response Time during Recovery

(b) Reconstruction Time

0

10

20

30

40

50

60

Fin1 Fin2 Web2 Proj

Ave

rage

Res

pons

e T

ime

(ms) WorkOut

VDFIDO

0

500

1000

1500

2000

2500

3000

3500

Fin1 Fin2 Web2 Proj

Rec

onst

ruct

ion

Tim

e (s

)

WorkOutVDFIDO

RAID6 Results

19

0

300

600

900

1200

1500

1800

Fin1 Fin2 Web2 Proj

Rec

onst

ruct

ion

Tim

e (s

)

WorkOutVDFIDO

0

10

20

30

40

Fin1 Fin2 Web2 Proj

Ave

rage

Res

pons

e T

ime

(ms) WorkOut

VDFIDO

(a) Average Response Time during Recovery

(b) Reconstruction Time

Detailed Real-time Results

20

(a) WebSearch2.spc

0.1

1

10

100

1000

0 500 1000 1500 2000 2500Use

r R

espo

nse

Tim

e (m

s)

Reconstruction Time (s)

WorkOut VDF IDO

(b) Microsoft Project

VDF ends

WorkOut ends

IDO ends

1

10

100

1000

0 100 200 300 400 500

Use

r R

espo

nse

Tim

e (m

s)Reconstruction Time (s)

WorkOut VDF IDO

VDF ends

WorkOut ends

IDO ends

Shorter Reconstruction Time Shorter Reconstruction Times

Shorter Reconstruction Time Lower user response times

Reduce I/Os and Sensitivity Study

21

•  Sensitivity & overhead analysis (in the paper).

0

20

40

60

80

100

Fin1 Fin2 Web2 Proj

Perc

enta

ge (%

of T

otal

)WorkOutVDF

3.4 1.3

IDO

•  Reduced I/Os:

Extendibility Evaluation

22

(a) Re-synchronization Time (b) Average Response Time

0

500

1000

1500

2000

2500

3000

Fin1 Fin2 Web2 Proj

DefaultWorkOutIDO

Re-

sync

hron

izat

ion

Tim

e (s

)

0510152025303540

Fin1 Fin2 Web2 Proj

Ave

rage

Res

pons

e T

ime

(ms) Default

WorkOutIDO

Summary of IDO

23

•  RAID reconstruction is an operational state in large-scale data centers!

•  Salient features of IDO: – Proactive; – Exploit both temporal and spatial localities; – Optimize both user and reconstruction IOs;

– Portability and extendibility.

Thanks!

24