1
PARAID: A PARAID: A Gear-ShiftingGear-ShiftingPower-Aware RAIDPower-Aware RAID
Charles Weddle, Mathew Oldham, Jin Qian, An-I Andy Wang – Florida St. UniversityCharles Weddle, Mathew Oldham, Jin Qian, An-I Andy Wang – Florida St. University
Peter Reiher – University of California, Los AngelesPeter Reiher – University of California, Los Angeles
Geoff Kuenning – Harvey Mudd CollegeGeoff Kuenning – Harvey Mudd College
2
MotivationMotivation
Energy costs are risingEnergy costs are rising An increasing concern for serversAn increasing concern for servers
No longer limited to laptopsNo longer limited to laptops
Energy consumption of disk drivesEnergy consumption of disk drives 24% of the power usage in web servers24% of the power usage in web servers 27% of electricity cost for data centers27% of electricity cost for data centers More energy More energy more heat more heat more cooling more cooling lower lower
computational density computational density more space more space more costs more costs
Is it possible to reduce energy consumption Is it possible to reduce energy consumption without degrading performance while without degrading performance while maintaining reliability?maintaining reliability?
PARAID: A Gear-Shifting Power-Aware RAID
3
ChallengesChallenges
EnergyEnergy Not enough opportunities to spin down RAIDsNot enough opportunities to spin down RAIDs
PerformancePerformance Essential for peak loadsEssential for peak loads
ReliabilityReliability Server-class drives are not designed for Server-class drives are not designed for
frequent power switchingfrequent power switching
PARAID: A Gear-Shifting Power-Aware RAID
4
Existing WorkExisting Work
Most trade performance for energy Most trade performance for energy savings directly savings directly e.g. vary speed of diskse.g. vary speed of disks
Most are simulated resultsMost are simulated results
PARAID: A Gear-Shifting Power-Aware RAID
5
ObservationsObservations
RAID is configured for peak performanceRAID is configured for peak performance RAID keeps all drives spinning for light loadsRAID keeps all drives spinning for light loads
Unused storage capacityUnused storage capacity Over-provision of storage capacityOver-provision of storage capacity Unused storage can be traded for energy savingsUnused storage can be traded for energy savings
Fluctuating loadFluctuating load Cyclic fluctuation of loadsCyclic fluctuation of loads Infrequent on-off power transitions can be effectiveInfrequent on-off power transitions can be effective
PARAID: A Gear-Shifting Power-Aware RAID
6
Performance vs. Energy Performance vs. Energy OptimizationsOptimizations
Performance benefitsPerformance benefits Realized under heavy loadsRealized under heavy loads
Energy benefitsEnergy benefits Realized instantaneously Realized instantaneously
7
Power-Aware RAIDPower-Aware RAID
Skewed striping for energy savingsSkewed striping for energy savings
Preserving peak performancePreserving peak performance
Maintaining reliabilityMaintaining reliability
EvaluationEvaluation
ConclusionConclusion
PARAID: A Gear-Shifting Power-Aware RAID
8
Skewed StripingSkewed Striping for Energy Saving for Energy Saving
Use over-provisioned spare storageUse over-provisioned spare storage Organized into hierarchical overlapping subsetsOrganized into hierarchical overlapping subsets
PARAID: A Gear-Shifting Power-Aware RAID
RAID
1 2 3 4 5
9
Skewed StripingSkewed Striping for Energy Saving for Energy Saving
Each set analogous to gears in automobilesEach set analogous to gears in automobiles
PARAID: A Gear-Shifting Power-Aware RAID
RAID
Gears123
1 2 3 4 5
10
Skewed StripingSkewed Striping for Energy Saving for Energy Saving
Soft states can be reclaimed for spaceSoft states can be reclaimed for space Persist across rebootsPersist across reboots
PARAID: A Gear-Shifting Power-Aware RAID
RAID
SoftStates
Gears123
1 2 3 4 5
11
Skewed StripingSkewed Striping for Energy Saving for Energy Saving
Operate in gear 1Operate in gear 1 Disks 4 and 5 are powered offDisks 4 and 5 are powered off
PARAID: A Gear-Shifting Power-Aware RAID
RAID
Gears123
1 2 3 4 5
SoftStates
12
Skewed StripingSkewed Striping for Energy Saving for Energy Saving
Approximate the workloadApproximate the workload Gear shift into most appropriate gearGear shift into most appropriate gear
Minimize the opportunity lost to save powerMinimize the opportunity lost to save power
Energy( PoweredOn Disks )
Workload( Disk Parallelism )
Conventional RAID PARAID
workload
PARAID: A Gear-Shifting Power-Aware RAID
13
Skewed StripingSkewed Striping for Energy Saving for Energy Saving
Adapt to cyclic fluctuating workloadAdapt to cyclic fluctuating workload Gear shift when gear utilization threshold is metGear shift when gear utilization threshold is met
time
load
utilization threshold
gear shift
PARAID: A Gear-Shifting Power-Aware RAID
14
Preserving Peak PerformancePreserving Peak Performance
Operate in the highest gear Operate in the highest gear When the system demands peak performanceWhen the system demands peak performance Uses the same disk layoutUses the same disk layout
Maximize parallelism within each gearMaximize parallelism within each gear Load is balancedLoad is balanced Uniform striping patternUniform striping pattern
Delay block replication until gear shiftsDelay block replication until gear shifts Capture block writesCapture block writes
PARAID: A Gear-Shifting Power-Aware RAID
15
Maintaining ReliabilityMaintaining Reliability
Reuse existing RAID levels (RAID-5)Reuse existing RAID levels (RAID-5) Also used in various gearsAlso used in various gears
Drives have a limited number of power Drives have a limited number of power cyclescycles Ration number of power cyclesRation number of power cycles
PARAID: A Gear-Shifting Power-Aware RAID
16
Maintaining ReliabilityMaintaining Reliability
Busy disk stay powered on, idle disks stay powered offBusy disk stay powered on, idle disks stay powered off Outside disks are role exchanged with middle disksOutside disks are role exchanged with middle disks
busydisks
powercycleddisks
idledisks
role exchange
Disk 1
Gear 1Gear 2
Gear 3
Disk 2 Disk 3 Disk 4 Disk 5 Disk 6
PARAID: A Gear-Shifting Power-Aware RAID
17
File system
RAID
PARAID block mappingPARAID block mapping
Disk device driver
User space
Linux kernel
Soft RAIDSoft RAID
Reliability managerReliability manager
Load monitorLoad monitor
Gear managerGear manager
Admin toolAdmin tool
Logical Component DesignLogical Component Design
PARAID: A Gear-Shifting Power-Aware RAID
18
Asymmetric Gear-Shifting PoliciesAsymmetric Gear-Shifting Policies
Up-shift (aggressive)Up-shift (aggressive) Moving utilization average + moving standard Moving utilization average + moving standard
deviation > utilization thresholddeviation > utilization threshold
Downshift (conservative)Downshift (conservative) Modified utilization moving average + moving Modified utilization moving average + moving
standard deviation < utilization thresholdstandard deviation < utilization threshold Moving average modified to account for fewer drives Moving average modified to account for fewer drives
and extra parity updatesand extra parity updates
PARAID: A Gear-Shifting Power-Aware RAID
19
ImplementationImplementation
Prototyped in Linux 2.6.5Prototyped in Linux 2.6.5 Open source, software RAIDOpen source, software RAID
Implemented block I/O handler, monitor, Implemented block I/O handler, monitor, disk managerdisk manager
Implemented user admin tool to configure Implemented user admin tool to configure devicedevice
Updated Raid Tools to recognize PARAID Updated Raid Tools to recognize PARAID levellevel
PARAID: A Gear-Shifting Power-Aware RAID
20
EvaluationEvaluation
ChallengesChallenges Prototyping PARAIDPrototyping PARAID Commercial machinesCommercial machines Conceptual barriersConceptual barriers Benchmarks designed to measure peak Benchmarks designed to measure peak
performanceperformance Trace replayTrace replay Time consumingTime consuming
PARAID: A Gear-Shifting Power-Aware RAID
21
EvaluationEvaluation
multimeter
USB cableclient
server
powersupply
12v & 5vpower lines
powermeasurement
probes
SCSIcable
crossover cable
Xeon 2.8 Ghz, 512 MB RAM36.7 GB 15k RPM SCSI
P4 2.8 Ghz, 1 GB RAM160 GB 7200 RPM SATA
RAID
RAID
RAID
RAID
RAID
BOOT
PARAID: A Gear-Shifting Power-Aware RAID
Measurement frameworkMeasurement framework
22
EvaluationEvaluation
Three different workloads using two different Three different workloads using two different RAID settingsRAID settings Web trace - RAID level 0 (2-disk gear 1, 5-disk gear 2)Web trace - RAID level 0 (2-disk gear 1, 5-disk gear 2)
Mostly read activityMostly read activity Cello99 - RAID level 5 (3-disk gear 1, 5-disk gear 2)Cello99 - RAID level 5 (3-disk gear 1, 5-disk gear 2)
I/O-intensive workload with writesI/O-intensive workload with writes PostMark - RAID level 5PostMark - RAID level 5
Measure peak performance and gear shifting overheadMeasure peak performance and gear shifting overhead
Speed up trace playbackSpeed up trace playback To match hardwareTo match hardware Explore range of speed up factors and power savingsExplore range of speed up factors and power savings
PARAID: A Gear-Shifting Power-Aware RAID
23
Web TraceWeb Trace
UCLA CS Dept Web Servers (8/11/2006 – 8/14/2006)UCLA CS Dept Web Servers (8/11/2006 – 8/14/2006) File system: ~32 GB (~500k files)File system: ~32 GB (~500k files) Trace replay: ~95k requests with ~4 GB data (~260 MB unique)Trace replay: ~95k requests with ~4 GB data (~260 MB unique)
PARAID: A Gear-Shifting Power-Aware RAID
0
0.1
0.2
0.3
0.4
0.5
0.6
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96
hours
GB/hour
24
Web Trace Power SavingsWeb Trace Power Savings
0
10
20
30
40
50
60
0 5 10 15 20 25 30
hours
wattsRAID-0
PARAID-0
0
10
20
30
40
50
60
0 5 10 15 20 25 30
hours
wattsRAID-0
PARAID-0
PARAID: A Gear-Shifting Power-Aware RAID
64x – 60 requests/sec
128x – 120 requests/sec 256x – 240 requests/sec
0
10
20
30
40
50
60
0 5 10 15 20 25 30
hours
wattsRAID-0
PARAID-0
64x - 34%
128x - 28%
256x - 10%
Energy Savings
25
Web Trace LatencyWeb Trace Latency
0
0.2
0.4
0.6
0.8
1
1 10 100 1000 10000 100000
msec
RAID-0
PARAID-0
0
0.2
0.4
0.6
0.8
1
1 10 100 1000 10000 100000
msec
RAID-0
PARAID-0
PARAID: A Gear-Shifting Power-Aware RAID
256x
128x 64x
0
0.2
0.4
0.6
0.8
1
1 10 100 1000 10000 100000
msec
RAID-0
PARAID-0256x - within 2.7%
64x - 240% 80ms vs. 33ms
Overhead
26
Web Trace BandwidthWeb Trace Bandwidth
-20
30
80
130
180
0 5 10 15 20 25 30
hours
MB/secRAID-0
PARAID-0
-20
30
80
130
180
0 5 10 15 20 25 30
hours
MB/secRAID-0
PARAID-0
PARAID: A Gear-Shifting Power-Aware RAID
256x
128x 64x
0
20
40
60
80
100
120
140
160
180
0 5 10 15 20 25 30
hours
MB/secRAID-0
PARAID-0
256x - within 1.3% in high gear
Overhead
27
Cello99 TraceCello99 Trace
Cello99 WorkloadCello99 Workload HP Storage Research LabsHP Storage Research Labs 50 hours beginning on 9/12/199950 hours beginning on 9/12/1999 1.5 million requests (12 GB) to 440MB of unique blocks1.5 million requests (12 GB) to 440MB of unique blocks I/O-intensive with 42% writesI/O-intensive with 42% writes
PARAID: A Gear-Shifting Power-Aware RAID
28
Cello99 Power SavingsCello99 Power Savings
0
10
20
30
40
50
0 10 20 30 40 50
hours
wattsRAID-5
PARAID-5
05
101520253035404550
0 10 20 30 40 50
hours
wattsRAID-5
PARAID-5
PARAID: A Gear-Shifting Power-Aware RAID
128x – 1000 requests/sec
32x – 270 requests/sec
64x – 550 requests/sec
0
10
20
30
40
50
0 10 20 30 40 50
hours
wattsRAID-5
PARAID-532x - 13%
64x - 8.2%
128x - 3.5%
Energy Savings
29
Cello99 Completion TimeCello99 Completion Time
0.9
0.92
0.94
0.96
0.98
1
1 10 100 1000 10000 100000
msec
RAID-5
PARAID-5
0.9
0.92
0.94
0.96
0.98
1
1 10 100 1000 10000 100000
msec
RAID-5
PARAID-5
PARAID: A Gear-Shifting Power-Aware RAID
128x
64x 32x
0.9
0.92
0.94
0.96
0.98
1
1 10 100 1000 10000 100000
msec
RAID-5
PARAID-5
32x - 1.8ms,26% slowerdue to time spent in lowgear
Overhead
30
Cello99 BandwidthCello99 Bandwidth
1
10
100
1000
0 500000 1000000 1500000
request number
MB/secRAID-5
PARAID-5
1
10
100
1000
0 500000 1000000 1500000
requests
MB/secRAID-5
PARAID-5
PARAID: A Gear-Shifting Power-Aware RAID
1
10
100
1000
0 500000 1000000 1500000
request number
MB/secRAID-5
PARAID
64x 32x
128x
Overhead
< 1% degra-dation duringpeak hours
31
PostMark BenchmarkPostMark Benchmark
Popular synthetic benchmarkPopular synthetic benchmark Generates ISP-style workloadsGenerates ISP-style workloads Stresses peak read/write performance of storage Stresses peak read/write performance of storage
devicedevice
PARAID: A Gear-Shifting Power-Aware RAID
32
Postmark PerformancePostmark Performance
PARAID: A Gear-Shifting Power-Aware RAID
0
50
100
150
200
1K files, 50K trans 20K files, 50K trans 20K files, 100Ktrans
seconds
RAID-5 PARAID-5 high gear PARAID-5 low-gear
33
Postmark Power SavingsPostmark Power Savings
PARAID: A Gear-Shifting Power-Aware RAID
0
10
20
30
40
50
60
70
80
1 11 21 31 41 51 61 71 81 91 10 111 12 13 14 151 16 171
seconds
wat
ts RAID5
PARAID
34
Related WorkRelated Work
PergamumPergamum EERAIDEERAID RIMACRIMAC Hibernator Hibernator MAIDMAID PDCPDC BlueFSBlueFS
PARAID: A Gear-Shifting Power-Aware RAID
35
Lessons LearnedLessons Learned
Third version of design, early design too Third version of design, early design too complicatedcomplicated
Data alignment problemsData alignment problems Difficult to measure system under normal loadDifficult to measure system under normal load Hard to predict workload transformations due to Hard to predict workload transformations due to
complex system optimizationscomplex system optimizations Challenging to match trace environmentsChallenging to match trace environments
PARAID: A Gear-Shifting Power-Aware RAID
36
ConclusionConclusion
PARAID reuses standard RAID-levels without PARAID reuses standard RAID-levels without special hardware while decreasing their energy special hardware while decreasing their energy use by 34%.use by 34%. Optimized version can save even more energyOptimized version can save even more energy
Empirical evaluation importantEmpirical evaluation important
PARAID: A Gear-Shifting Power-Aware RAID
37
QuestionsQuestions
PARAID: A PARAID: A Gear-ShiftingGear-Shifting
Power-Aware RAIDPower-Aware RAID
ContactContact Andy Wang – Andy Wang – [email protected]