deconstructing storage arrays timothy e. denehy, john bent, florentina i. popovici, andrea c....
Post on 04-Jan-2016
215 Views
Preview:
TRANSCRIPT
Deconstructing Storage Arrays
Timothy E. Denehy, John Bent, Florentina I. Popovici,
Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau
University of Wisconsin, Madison
Gray-box Research
• Computer systems becoming more complex• Transistors• Lines of code
• Each component is becoming more complex• Interactions between subsystems can affect
• Performance• Reliability• Power• Security
Gray-box Research
• Interfaces remain the same• Changes can be difficult and impractical• Support multiple platforms or legacy systems• Commercial acceptance for wide-spread
adoption• Hardware and software phenomenon
• IA-32 instruction set, POSIX OS, SCSI storage
• Problem: lack of information
Gray-box Solution
• Treat target system as a gray-box• General characteristics are known
• Extract information from an existing interface• e.g. determine cache contents
• Exploit information to control system behavior• e.g. access cached data first
Gray-box Information Techniques
• Make assumptions about target system
• Observe system inputs and outputs
• Statistical methods• Draw inferences about internal structure
• Microbenchmarks and probes• Parameterize system components• Observe system under controlled input
Gray-box Applications
• Gray-box techniques have been used to identify• Memory hierarchy parameters [Saavedra and Smith]• Processor cycle time [Staelin and McVoy]• Low-level disk characteristics [Worthington et al.]• Buffer cache replacement algorithms [Burnett et al.]• File system data structures [Sivathanu et al.]
• storage array characteristics: Shear
Shear
• Software tool that automatically determines the important properties of a storage array
• Enables file system performance tuning with knowledge of storage array characteristics
• Acts as a management tool to help configure, monitor, and maintain storage arrays
Outline
• Introduction• Shear
• Background• Algorithm
• Case Studies• Performance: Stripe-aligned Writes• Management: Detecting Misconfiguration,
Failure• Conclusion
Shear Goals
• Determine storage array characteristics
0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031 SCSISCSI
Shear Goals
• Determine storage array characteristics• Number of disks
0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031 SCSISCSI
Shear Goals
• Determine storage array characteristics• Number of disks• Chunk size
0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031 SCSISCSI
Shear Goals
• Determine storage array characteristics• Number of disks• Chunk size• Layout and redundancy scheme
0 1 2 316171819
4 5 6 7 8 9 1011 1213141520212223 24252627 28293031
0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031
RAID-0
SCSISCSI
Shear Goals
• Determine storage array characteristics• Number of disks• Chunk size• Layout and redundancy scheme
0 1 2 31617181924252627
4 5 6 720212223
4 5 6 720212223
RAID-1
SCSISCSI
28293031 28293031
0 1 2 31617181924252627
0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031
Shear Goals
• Determine storage array characteristics• Number of disks• Chunk size• Layout and redundancy scheme
0 1 2 320212223
P P P P
4 5 6 7 8 9 101112131415
P P P P
P P P P16171819 P P P P32333435 24252627 28293031
36373839 40414243 44454647
0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031
RAID-5
SCSISCSI
Shear Motivation
• Performance• Tune file systems to array characteristics
• Management• Verify configuration• Detect failure
Shear Techniques
• Microbenchmarks and probes• Controlled, random access read and write
patterns• Measure response time of access patterns• Measure steady-state performance
• Statistical clustering• Automatically classify fast and slow regimes• Identify patterns that utilize only a single disk
Shear Assumptions
• Storage array• Layout follows a repeatable pattern• Composed of homogeneous disks
• System• Able to bypass the file system and buffer
cache• Little traffic from other processes
Outline
• Introduction• Shear
• Background• Algorithm
• Case Studies• Performance: Stripe-aligned Writes• Management: Detecting Misconfiguration,
Failure• Conclusion
Shear Algorithm
• Pattern size
• Chunk size
• Layout of chunks to disks
• Level of redundancy
Determining the Pattern Size
• Find the size of the layout's repeating pattern• Not always the stripe size
• Choose a hypothetical pattern size• Perform random reads at multiples of that
distance
• Repeat for a range of pattern sizes
• Cluster results and identify actual pattern size
Pattern Size Example
RAID-04 Disks
8 KBChunks
Pattern Size Example
Testing2 KB
RAID-04 Disks
8 KBChunks
Pattern Size Example
Testing4 KB
RAID-04 Disks
8 KBChunks
Pattern Size Example
Testing6 KB
RAID-04 Disks
8 KBChunks
Pattern Size Example
Testing8 KB
RAID-04 Disks
8 KBChunks
Pattern Size Example
Testing10 KB
RAID-04 Disks
8 KBChunks
Pattern Size Example
Testing12 KB
RAID-04 Disks
8 KBChunks
Pattern Size Example
Testing14 KB
RAID-04 Disks
8 KBChunks
Pattern Size Example
Testing16 KB
RAID-04 Disks
8 KBChunks
Pattern Size Example
Testing18 KB
RAID-04 Disks
8 KBChunks
Pattern Size Example
Testing20 KB
RAID-04 Disks
8 KBChunks
Pattern Size Example
Testing22 KB
RAID-04 Disks
8 KBChunks
Pattern Size Example
Testing24 KB
RAID-04 Disks
8 KBChunks
Pattern Size Example
Testing26 KB
RAID-04 Disks
8 KBChunks
Pattern Size Example
Testing28 KB
RAID-04 Disks
8 KBChunks
Pattern Size Example
Testing30 KB
RAID-04 Disks
8 KBChunks
Pattern Size Example
Testing32 KB
RAID-04 Disks
8 KBChunks
Pattern Size Example
RAID-04 Disks
8 KBChunks
Pattern Size Example
RAID-04 Disks
8 KBChunks
Actual32 KB
cluster
cluster
cluster
Shear Algorithm
• Pattern size
• Chunk size
• Layout of chunks to disks
• Level of redundancy
Determining the Chunk Size
• Chunk size• amount of data contiguously allocated to one disk
• Find the boundaries between disks
• Choose a hypothetical boundary offset• Perform random reads on both sides of that offset
• Repeat for all offsets in the pattern size
• Cluster results and identify actual chunk size
Chunk Size Example
RAID-04 Disks
8 KBChunks
Chunk Size Example
Testing0 KB
RAID-04 Disks
8 KBChunks
Chunk Size Example
Testing2 KB
RAID-04 Disks
8 KBChunks
Chunk Size Example
Testing4 KB
RAID-04 Disks
8 KBChunks
Chunk Size Example
Testing6 KB
RAID-04 Disks
8 KBChunks
Chunk Size Example
Testing8 KB
RAID-04 Disks
8 KBChunks
Chunk Size Example
Testing10 KB
RAID-04 Disks
8 KBChunks
Chunk Size Example
Testing12 KB
RAID-04 Disks
8 KBChunks
Chunk Size Example
Testing14 KB
RAID-04 Disks
8 KBChunks
Chunk Size Example
Testing16 KB
RAID-04 Disks
8 KBChunks
Chunk Size Example
RAID-04 Disks
8 KBChunks
Chunk Size Example
RAID-04 Disks
8 KBChunks
Actual8 KB
cluster
cluster
Shear Algorithm
• Pattern size
• Chunk size
• Layout of chunks to disks
• Level of redundancy
Determining the Read Layout
• Find mapping of chunks to disks
• Choose a pair of chunks in the pattern• Perform random reads to both chunks
• Repeat for all pairs of chunks
• Cluster results and identify chunks on same disk
Read Layout Example
07
RAID-0 ZIG-ZAG 4 Disks
16
25
34
Read Layout Example
07
RAID-0 ZIG-ZAG 4 Disks
16
25
34
Testing { 0, 0 }
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
16
25
34
Testing { 0, 1 }
07
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
16
25
34
Testing { 0, 2 }
07
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
16
25
34
Testing { 0, 3 }
07
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
16
25
34
Testing { 0, 4 }
07
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
16
25
34
Testing { 0, 5 }
07
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
16
25
34
Testing { 0, 6 }
07
Read Layout Example
07
RAID-0 ZIG-ZAG 4 Disks
16
25
34
Testing { 0, 7 }
Read Layout Example
07
RAID-0 ZIG-ZAG 4 Disks
16
25
34
Testing { 1, 1 }
Read Layout Example
07
RAID-0 ZIG-ZAG 4 Disks
25
34
Testing { 1, 2 }
16
Read Layout Example
07
RAID-0 ZIG-ZAG 4 Disks
25
34
Testing { 1, 3 }
16
Read Layout Example
07
RAID-0 ZIG-ZAG 4 Disks
25
34
Testing { 1, 4 }
16
Read Layout Example
07
RAID-0 ZIG-ZAG 4 Disks
25
34
Testing { 1, 5 }
16
Read Layout Example
07
RAID-0 ZIG-ZAG 4 Disks
25
34
Testing { 1, 6 }
16
Read Layout Example
07
RAID-0 ZIG-ZAG 4 Disks
25
34
Testing { 1, 7 }
16
Read Layout Example
07
RAID-0 ZIG-ZAG 4 Disks
25
34
16
Read Layout Example
07
RAID-0 ZIG-ZAG 4 Disks
25
34
Actual{ 0, 7 } { 1, 6 } { 2, 5 } { 3, 4}
16
cluster cluster
Shear Algorithm
• Pattern size
• Chunk size
• Layout of chunks to disks
• Level of redundancy
Determining Level of Redundancy
• Ratio of read to write bandwidth reveals the type of redundancy in the array
• Expected R/W ratios:• RAID-0: 1 (no redundancy)• RAID-1: 2 (mirroring)• RAID-4: varies (examine write layout)• RAID-5: 4 (parity)
Shear Experience
• Shear has been applied to
• Linux software RAID• Poor RAID-5 parity updates
• Adaptec hardware RAID controller• Implements RAID-5 left-asymmetric layout
– RAID-0– RAID-1– Chained
Declustering
– RAID-4– RAID-5– P+Q
Outline
• Introduction• Shear
• Background• Algorithm
• Case Studies• Performance: Stripe-aligned Writes• Management: Detecting Misconfiguration,
Failure• Conclusion
RAID-5 Performance
• Small writes on RAID-5 are problematic• Require two reads, parity calculation, two
writes
• Writing in full stripes is more efficient
0 1 2 320212223
P P P P
4 5 6 7 8 9 101112131415
P P P P
P P P P16171819 P P P P32333435 24252627 28293031
36373839 40414243 44454647RAID-5
Stripe-aligned Writes
• Overcome RAID-5 small write problem
• Modified Linux disk scheduler• Groups writes into full stripes• Aligns writes along stripe boundaries• Approximately 20 lines of code
• Experiment• Hardware RAID-5, 4 disks, 16 KB chunks• Create 100 files of varying sizes
Stripe-aligned Writes Experiment
• Simple modification has a large impact
Detecting Misconfigurations
Corr
ect
RAID 5-LS RAID 5-LA RAID 5-RS RAID 5-RA
• Software RAID, 4 Disks, 8 KB Chunks
• What if one disk is accidentally used twice?
Detecting Misconfigurations
Corr
ect
Mis
config
RAID 5-LS RAID 5-LA RAID 5-RS RAID 5-RA
Detecting Failures
• Software RAID• RAID-5 LS• 10 disks• 8 KB chunks
Detecting Failures
• Software RAID• RAID-5 LS• 10 disks• 8 KB chunks
• Disk 5 fails
Outline
• Introduction• Shear
• Background• Algorithm
• Case Studies• Performance: Stripe-aligned Writes• Management: Detecting Misconfiguration,
Failure• Conclusion
Conclusion
• Gray-box research• Extract / exploit information from existing
interfaces
• Shear• Extracts information
• Automatically determines storage array properties• Exploits information
• File system performance tuning• Storage management
Questions?
http://www.cs.wisc.edu/adsl/
top related