Erasure Coding in Windows Azure Storage

Cheng Huang, Huseyin Simitci, Yikang Xu, Aaron Ogus, Brad Calder, Parikshit Gopalan, Jin Li, and Sergey Yekhanin

Microsoft Corporation

AbstractWindows Azure Storage (WAS) is a cloud storage sys-

tem that provides customers the ability to store seem-ingly limitless amounts of data for any duration of time.WAS customers have access to their data from anywhere,at any time, and only pay for what they use and store. Toprovide durability for that data and to keep the cost ofstorage low, WAS uses erasure coding.

In this paper we introduce a new set of codes for era-sure coding called Local Reconstruction Codes (LRC).LRC reduces the number of erasure coding fragmentsthat need to be read when reconstructing data fragmentsthat are offline, while still keeping the storage overheadlow. The important benefits of LRC are that it reduces thebandwidth and I/Os required for repair reads over priorcodes, while still allowing a significant reduction in stor-age overhead. We describe how LRC is used in WAS toprovide low overhead durable storage with consistentlylow read latencies.

1 IntroductionWindows Azure Storage (WAS) [1] is a scalable cloud

storage system that has been in production since Novem-ber 2008. It is used inside Microsoft for applicationssuch as social networking search, serving video, musicand game content, managing medical records, and more.In addition, there are thousands of customers outside Mi-crosoft using WAS, and anyone can sign up over the In-ternet to use the system. WAS provides cloud storagein the form of Blobs (user files), Tables (structured stor-age), Queues (message delivery), and Drives (networkmounted VHDs). These data abstractions provide theoverall storage and work flow for applications runningin the cloud.

WAS stores all of its data into an append-only dis-tributed file system called the stream layer [1]. Data isappended to the end of activeextents, which are repli-cated three times by the underlying stream layer. Thedata is originally written to 3 full copies to keep the datadurable. Once reaching a certain size (e.g., 1 GB), ex-tents are sealed. These sealed extents can no longer bemodified and thus make perfect candidates for erasurecoding. WAS then erasure codes a sealed extent lazily inthe background, and once the extent is erasure-coded theoriginal 3 full copies of the extent are deleted.

The motivation for using erasure coding in WAScomes from the need to reduce the cost of storage. Era-sure coding can reduce the cost of storage over 50%,

which is a tremendous cost saving as we will soon sur-pass an Exabyte of storage. There are the obvious costsavings from purchasing less hardware to store that muchdata, but there are significant savings from the fact thatthis also reduces our data center footprint by 1/2, thepower savings from running 1/2 the hardware, along withother savings.

The trade-off for using erasure coding instead of keep-ing 3 full copies is performance. The performance hitcomes when dealing withi) a lost or offline data frag-ment andii) hot storage nodes. When an extent iserasure-coded, it is broken up intok data fragments, anda set of parity fragments. In WAS, a data fragment maybe lost due to a disk, node or rack failure. In addition,cloud services areperpetually in beta [2] due to frequentupgrades. A data fragment may be offline for seconds toa few minutes due to an upgrade where the storage nodeprocess may be restarted or the OS for the storage nodemay be rebooted. During this time, if there is an on-demand read from a client to a fragment on the storagenode being upgraded, WAS reads from enough fragmentsin order to dynamically reconstruct the data being askedfor to return the data to the client. This reconstructionneeds to be optimized to be as fast as possible and useas little networking bandwidth and I/Os as possible, withthe goal to have the reconstruction time consistently lowto meet customer SLAs.

When using erasure coding, the data fragment theclient’s request is asking for is stored on a specific stor-age node, which can greatly increase the risk of a storagenode becoming hot, which could affect latency. One waythat WAS can deal with hot storage nodes is to recog-nize the fragments that are hot and then replicate them tocooler storage nodes to balance out the load, or cache thedata and serve it directly from DRAM or SSDs. But, theread performance can suffer for the potential set of readsgoing to that storage node as it gets hot, until the datais cached or load balanced. Therefore, one optimizationWAS has is if it looks like the read to a data fragment isgoing to take too long, WAS in parallel tries to perform areconstruction of the data fragment (effectively treatingthe storage node with the original data fragment as if itwas offline) and return to the client whichever of the tworesults is faster.

For both of the above cases the time to reconstruct adata fragment for on-demand client requests is crucial.The problem is that the reconstruction operation is onlyas fast as the slowest storage node to respond to reading

of the data fragments. In addition, we want to reducestorage costs down to 1.33x of the original data usingerasure coding. This could be accomplished using thestandard approach of Reed-Solomon codes [13] wherewe would have (12, 4), which is 12 data fragments and4 code fragments. This means that to do the reconstruc-tion we would need to read from a set of 12 fragments.This i) greatly increases the chance of hitting a hot stor-age node, andii) increases the network costs and I/Osand adds latency to read that many fragments to do thereconstruction. Therefore, we want to design a new fam-ily of codes to use for WAS that provides the followingcharacteristics:

1. Reduce the minimal number of fragments that needto be read from to reconstruct a data fragment. Thisprovides the following benefits:i) reduces the net-work overhead and number of I/Os to perform a re-construction;ii) reduces the time it takes to performthe reconstruction since fewer fragments need to beread. We have found the time to perform the re-construction is often dominated by the slowest frag-ments (the stragglers) to be read from.

2. Provide significant reduction in storage overheadto 1.33x while maintaining higher durability than asystem that keeps 3 replicas for the data.

In this paper, we introduce Local Reconstruction Codes(LRC) that provide the above properties. In addition, wedescribe our erasure coding implementation and impor-tant design decisions.

2 Local Reconstruction CodesIn this section, we illustrate LRC and its properties

through small examples, which are shorter codes (thushigher overhead) than what we use in production, in or-der to simplify the description of LRC .

2.1 DefinitionWe start with a Reed-Solomon code example to ex-

plain the concept ofreconstruction cost. A (6, 3) Reed-Solomon code contains 6 data fragments and 3 parityfragments, where each parity is computed from all the6 data fragments. When any data fragment becomes un-available, no matter which data and parity fragments areused for reconstruction, 6 fragments are always required.We definereconstruction cost as the number of frag-ments required to reconstruct an unavailabledata frag-ment. Here, the reconstruction cost equals to 6.

The goal of LRC is to reduce the reconstruction cost.It achieves this by computing some of the parities froma subset of the data fragments. Continuing the examplewith 6 data fragments, LRC generates 4 (instead of 3)parities. The first two parities (denoted asp0 andp1) areglobal parities and are computed fromall the data frag-ments. But, for the other two parities, LRC divides the

x0 y0x1 x2 y2y1

p0

p1

pypx

Figure 1:A (6, 2, 2) LRC Example. (k = 6 data frag-ments,l = 2 local parities andr = 2 global parities.)

data fragments into two equal size groups and computesone local parity for each group. For convenience, wename the 6 data fragments (x0, x1 andx2) and (y0, y1andy2). Then, local paritypx is computed from the 3data fragments in one group (x0, x1 andx2), and localparitypy from the 3 data fragments in another group (y0,y1 andy2).

Now, let’s walk through reconstructingx0. Instead ofreadingp0 (or p1) and the other 5 data fragments (x1,x2, y0, y1 andy2), it is more efficient to readpx and twodata fragments (x1 andx2) to computex0. It is easy toverify that the reconstruction ofany single data fragmentrequires only 3 fragments, half the number required bythe Reed-Solomon code.

This LRC example adds one more parity than theReed-Solomon one, so it might appear that LRC reducesreconstruction cost at the expense of higher storage over-head. In practice, however, these two examples achievecompletely different trade-off points in the design space,as described in Section 3.3. In addition, LRC providesmore options than Reed-Solomon code, in terms of trad-ing off storage overhead and reconstruction cost.

We now formally define Local Reconstruction Codes.A (k, l, r) LRC dividesk data fragments intol groups,with k/l data fragments in each group. It computes onelocal parity within each group. In addition, it computesr global parities from all the data fragments. Letn bethe total number of fragments (data + parity). Thenn =k + l + r. Hence, the normalized storage overhead isn/k = 1 + (l + r)/k. The LRC in our example is a(6, 2, 2) LRC with storage cost of1 + 4/6 = 1.67x, asillustrated in Figure 1.

2.2 Fault ToleranceThus far, we have only defined which data fragments

are used to compute each parity in LRC. To completethe code definition, we also need to determinecodingequations, that is, how the parities are computed fromthe data fragments. We choose the coding equations suchthat LRC can achieve theMaximally Recoverable (MR)property [14], which means it can decode any failure pat-tern which is information-theoretically decodable.

Let’s first explain the Maximally Recoverable prop-erty. Given the (6, 2, 2) LRC example, it contains 4 par-ity fragments and can tolerateup to 4 failures. However,

2

x0 y0x1 x2 y2y1

p0

p1

pypx

(a) 3 Failures

x0 y0x1 x2 y2y1

p0

p1

pypx

(b) 4 Failures

Figure 2: Decoding 3 and 4 Failures in LRC.

LRC is not Maximum Distance Separable [12] and there-fore cannot toleratearbitrary 4 failures. For instance, saythe 4 failures arex1, x2, x3 andpx. This failure patternis non-decodable because there are only two parities -the global parities - that can help to decode the 3 miss-ing data fragments. The other local paritypy is uselessin this example. It isimpossible to decode 3 data frag-ments from merely 2 parity fragments, regardless of cod-ing equations. These types of failure patterns are calledinformation-theoretically non-decodable.

Failure patterns that are possible to reconstruct arecalledinformation-theoretically decodable. For instance,the 3-failure pattern in Figure 2(a) and the 4-failure pat-tern in Figure 2(b) are both information-theoretically de-codable. For these two failure patterns, it is possible toconstruct coding equations such that it is equivalent tosolving 3 unknowns using 3 linearly independent equa-tions in Figure 2(a) and 4 unknowns using 4 linearly in-dependent equations in Figure 2(b).

Conceivably, it is not difficult to construct a set of cod-ing equations that can decode a specific failure pattern.However, the real challenge is to construct asingle set ofcoding equations that achieves theMaximally Recover-able (MR) property [14], or being able to decode all theinformation-theoretically decodable failure patterns – theexact goal of LRC.

2.2.1 Constructing Coding EquationsIt turns out that the LRC can tolerate arbitrary 3 fail-

ures by choosing the following two sets of coding coeffi-cients (α’s andβ’s) for groupx and groupy, respectively.We skip the proofs due to space limitation. Let

qx,0 = α0x0 + α1x1 + α2x2 (1)

qx,1 = α2

0x0 + α2

1x1 + α2

2x2 (2)

qx,2 = x0 + x1 + x2 (3)

and

qy,0 = β0y0 + β1y1 + β2y2 (4)

qy,1 = β2

0y0 + β2

1y1 + β2

2y2 (5)

qy,2 = y0 + y1 + y2. (6)

Then, the LRC coding equations are as follows:

p0 = qx,0 + qy,0, p1 = qx,1 + qy,1, (7)

px = qx,2, py = qy,2. (8)

Next, we determine the values ofα’s andβ’s so thatthe LRC can decode all information-theoretically decod-able 4 failures. We focus on non-trivial cases as follows:

1. None of the four parities fails. The four failuresare equally divided between groupx and groupy.Hence, we have four equations whose coefficientsare given by the matrix, which result in the follow-ing determinant:

G =

1 1 0 00 0 1 1αi αj βs βt

α2

i α2

j β2

s β2

t

Det(G) = (αj − αi)(βt − βs)(αi + αj − βs − βt).

2. Only one of px and py fails. Assumepy fails. Forthe remaining three failures, two are in groupx andthe third one in groupy. We now have three equa-tions with coefficients given by

G′ =

1 1 0αi αj βs

α2

i α2

j β2

s

Det(G′) = βs(αj − αi)(βs − αj − αi).

3. Both px and py fail. In addition, the remaining twofailures are divided between groupx and groupy.We have two equations with coefficients given by

G′′ =

(

αi βs

α2

i β2

s

)

Det(G′′) = αiβs(βs − αi).

To ensure all the cases are decodable, all the matricesG, G′ andG′′ should be non-singular, which leads to thefollowing conditions:

αi, αj , βs, βt 6= 0 (9)

αi, αj 6= βs, βt (10)

αi + αj 6= βs + βt (11)

One way to fulfill these conditions is to assign toα’sand β’s the elements from a finite field GF(24) [12],where every element in the field is represented by 4 bits.α’s are chosen among the elements whose lower order 2bits are zero. Similarly,β’s are chosen among the ele-ments whose higher order 2 bits are zero. That way, thelower order 2 bits ofα’s (and the sum ofα’s) are alwayszero, and the higher order 2 bits ofβ’s (and the sum ofβ’s) are always zero. Hence, they will never be equal andall the above conditions are satisfied.

This way of constructing coding equations requires avery small finite field and makes implementation practi-cal. It is a critical improvement over our own Pyramid

3

codes [14]. In Pyramid codes, coding equation coeffi-cients are discovered through a search algorithm, whosecomplexity grows exponentially with the length of thecode. For parameters considered in Windows AzureStorage, the search algorithm approach would have re-sulted in a large finite field, which makes encoding anddecoding complexity high.

2.2.2 Putting Things TogetherTo summarize, the(6, 2, 2) LRC is capable of decod-

ing arbitrary three failures. It can also decode all theinformation-theoretically decodable four failure patterns,which accounts for 86% of all the four failures. In short,the (6, 2, 2) LRC achieves the Maximally Recoverableproperty [14].

2.2.3 Checking DecodabilityGiven a failure pattern, how can we easily check

whether it is information-theoretically decodable? Hereis an efficient algorithm. For each local group, if the lo-cal parity is available, while at least one data fragmentis erased, weswap the parity with one erased data frag-ment. The swap operation marks the data fragment asavailable and the parity as erased. Once we completeall the local groups, we examine the data fragments andthe global parities. If the total number of erased frag-ments (data and parity) is no more than the number ofglobal parities, the algorithm declares the failure pat-tern information-theoretically decodable. Otherwise, itis non-decodable. This algorithm can be used to verifythat the examples in Figure 2 are indeed decodable.

2.3 Optimizing Storage Cost, Reliabilityand Performance

Throughout the entire section, we have been focusingon the (6, 2, 2) LRC example. It is important to note thatall the properties demonstrated by the example general-ize to arbitrary coding parameters.

In general, the key properties of a(k, l, r) LRC are:i) single data fragment failure can be decoded fromk/lfragments;ii) arbitrary failures up tor + 1 can be de-coded. Based on the following theorem, these propertiesimpose a lower bound on the number of parities [21].

Theorem 1. For any (n, k) linear code (with k data sym-bols and n− k parity symbols) to have the property:

1. arbitrary r + 1 symbol failures can be decoded;2. single data symbol failure can be recovered from

dk/le symbols,

the following condition is necessary:

n− k ≥ l + r. (12)

Since the number of parities of LRC meets the lowerboundexactly, LRC achieves its properties with the min-imal number of parities.

2.4 SummaryNow, we summarize Local Reconstruction Codes. A

(k, l, r) LRC dividesk data fragments intol local groups.It encodesl local parities, one for each local group, andr global parities. Any single data fragment failure can bedecoded fromk/l fragments within its local group.

In addition, LRC achieves Maximally Recoverableproperty. It tolerates up tor + 1 arbitrary fragment fail-ures. It also tolerates failures more thanr+1 (up tol+r),provided those are information-theoretically decodable.

Finally, LRC provides low storage overhead. Amongall the codes that can decode single data fragment fail-ure fromk/l fragments and tolerater + 1 failures, LRCrequires the minimum number of parities.

3 Reliability Model and Code SelectionThere are many choices of parametersk, l and r

for LRC. The question is: what parameters should wechoose in practice? To answer this, we first need to un-derstand the reliability achieved by each set of param-eters. Since 3-replication is an accepted industry stan-dard, we use the reliability of 3-replication as a refer-ence. Only those sets of parameters that achieve equal orhigher reliability than 3-replication are considered.

3.1 Reliability ModelReliability has long been a key focus in distributed

storage systems [28, 29, 30, 31]. Markov models arecommonly used to capture the reliability of distributedstorage systems. The model is flexible to consider bothindependent or correlated failures [10, 32, 33, 34]. Weadd a simple extension to generalize the Markov model,in order to capture unique state transitions in LRC. Thosetransitions are introduced because the failure mode de-pends on not only the size of failure, but also which sub-set of nodes fails. In our study, we focus on independentfailures, but the study can be readily generalized to cor-related failures [10, 34].

3.1.1 Modeling (6, 2, 2) LRC

We start with the standard Markov model to analyzereliability. Each state in the Markov process representsthe number of available fragments (data and parity). Forexample, Figure 3 plots the Markov model diagram forthe(6, 2, 2) LRC.

Let λ denote the failure rate of a single fragment.Then, the transition rate from all fragments healthyState 10 to one fragment failureState 9 is 10λ.The extension from the standard Markov model lies inState 7, which can transition into two states with 6healthy fragments.State 6 represents a state wherethere are four decodable failures. On the other hand,State 6F represents a state with four non-decodablefailures. Letpd denote the percentage of decodable fourfailure cases. Then the transition rate fromState 7 to

4

State 6 is 7λpd and the transition toState 6F is7λ(1− pd).�� � � � � ������ �� �� ���� ��� � �������� �Figure 3: Markov Reliability Model for the (6, 2,2) LRC. (State represents the number of healthy frag-ments.)

In the reverse direction, letρ9 denote the transition ratefrom one fragment failurestate 9 back to all frag-ments healthystate 10, which equals to the averagerepair rate of single-fragment failures.

Assume there areM storage nodes in the system, eachwith S storage space andB network bandwidth. Whensingle storage node fails, assume the repair load is evenlydistributed among the remaining(M−1) nodes. Further,assume repair traffic is throttled so that it only usesε ofthe network bandwidth on each machine. When erasurecoding is applied, repairing one failure fragment requiresmore than one fragment, denoted as repair costC. Then,the average repair rate of single-fragment failures isρ9 =ε(M − 1)B/(SC).

Other transition ratesρ8 throughρ6 can be calculatedsimilarly. In practice, however, the repair rates beyondsingle failure are dominated by the time taken to detectfailure and trigger repair.1 LetT denote the detection andtriggering time. It suffices to setρ8 = ρ7 = ρ6 = 1/T .

Any single storage node stores both data and parityfragments from different extents. So, when it fails, bothdata and parity fragments need to be repaired. The repaircost can be calculated by averaging across all the frag-ments.

For the(6, 2, 2) LRC, it takes 3 fragments to repair anyof the 6 data fragments and the 2 local parities. Further,it takes 6 fragments to repair the 2 global parities. Hence,the average repair costC = (3 × 8 + 6 × 2)/10 = 3.6.Also, enumerating all four failure patterns, we obtain thedecodability ratio aspd = 86%.

3.1.2 Reliability of (6, 2, 2) LRC

Now, we use a set of typical parameters (M = 400,S = 16TB, B = 1Gbps,ε = 0.1 andT = 30 min-utes) to calculate the reliability of the LRC, which isalso compared to 3-replication and Reed-Solomon codein Table 1. Since the Reed-Solomon code tolerates three

1When multiple failures happen, most affected coding groups onlyhave a single fragment loss.Unlucky coding groups with two or morefragment losses are relatively few. Therefore, not many fragments entermulti-failure repair stages. In addition, multi-failure repairs are priori-tized over single-failure ones. As a result, multi-failure repairs are fastand they take very little time, compared to detecting the failures andtriggering the repairs.

MTTF (years)

3-replication 3.5×109

(6, 3) Reed-Solomon 6.1×1011

(6, 2, 2) LRC 2.6×1012

Table 1: Reliability of 3-Replication, RS and LRC.

failures, while 3-replication tolerates only two failures,it should be no surprise that the Reed-Solomon code of-fers higher reliability than 3-replication. Similarly, theLRC tolerates not only three failures, but also 86% ofthe four-failure cases, so it naturally achieves the highestreliability.

3.2 Cost and Performance Trade-offsEach set of LRC parameters (k, l andr) yields one set

of values of reliability, reconstruction cost and storageoverhead. For the (6, 2, 2) LRC, the reliability (MTTF inyears) is2.6 × 1012, the reconstruction cost is 3 and thestorage overhead is 1.67x.

We obtain many sets of values by varying the param-eters. Since each fragment has to place on a differentfault domain, the number of fault domains in a clusterlimits the total number of fragments in the code. Weuse 20 as the limit here, since our storage stamps (clus-ters) have up to 20 fault domains. Using the reliability(MTTF) of 3-replication as the threshold, we keep thosesets of parameters that yield equal or higher reliabilitythan 3-replication. We then plot the storage overheadand the reconstruction cost of the remaining sets in Fig-ure 4. Again, each individual point represents one set ofcoding parameters. Each parameter set represents cer-tain trade-offs between storage cost and reconstructionperformance.

Different coding parameters can result in the samestorage overhead (such as 1.5x), but vastly different re-construction cost. In practice, it only makes sense tochoose the one with the lower reconstruction cost. There-fore, we outline the lower bound of all the trade-offpoints. The lower bound curve characterizes the funda-mental trade-off between storage cost and reconstructionperformance for LRC.

3.3 Code Parameter SelectionSimilarly, for (k, r) Reed-Solomon code, we vary the

parametersk andr (so long ask + r ≤ 20) and also ob-tain a lower bound cost and performance trade-off curve.We compare Reed-Solomon to LRC in Figure 5. On theReed-Solomon curve, we mark two special points, whichare specific parameters chosen by existing planet-scalecloud storage systems. In particular, RS (10, 4) is usedin HDFS-RAID in Facebook [8] and RS (6, 3) in GFS IIin Google [9, 10]. Again, note that all the trade-off pointsachieve higher reliability than 3-replication, so they are

5

0

2

4

6

8

10

12

1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2

Reco

nstru

ctio

n Re

ad C

ost

Storage Overhead

Trade-off Points

Lower Bound

Figure 4: Overhead vs. Recon. Cost.

0

2

4

6

8

10

12

1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2

Reco

nstru

ctio

n Re

ad C

ost

Storage Overhead

Reed-Solomon LRC

same read cost1.5x to 1.33x

same overheadhalf read cost

RS(10,4)

RS(6,3)

LRC(12,2,2)

LRC(12,4,2)

Figure 5: LRC vs. RS Code.

0

2

4

6

8

10

12

1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2

Reco

nstru

ctio

n Re

ad C

ost

Storage Overhead

Reed-Solomon HoVer Stepped Combination LRC Weaver

Figure 6: LRC vs. Modern Codes.

candidates for WAS.The comparison shows that LRC achieves better cost

and performance trade-off than Reed-Solomon across therange of parameters. If we keep the storage overhead thesame, reconstruction in LRC is much more efficient thanthat in Reed-Solomon. On the other hand, if we keep re-construction cost the same, LRC can greatly reduce stor-age overhead, compared to Reed-Solomon. In general,we can choose a point along the cost and performancetrade-off curve, which can reduce storage overhead andreconstruction cost at the same time.

Compared to (6, 3) Reed-Solomon, we could keepstorage overhead the same (at 1.5x) and replace theReed-Solomon code with a (12, 4, 2) LRC. Now, thereconstruction cost is reduced from 6 to 3 for single-fragment failures, a reduction of 50%. This is shown bythe vertical move in Figure 5.

Alternatively, as shown by the horizontal move in Fig-ure 5, we could keep reconstruction cost the same (at6) and replace the Reed-Solomon code with a (12, 2, 2)LRC. Now, the storage overhead is reduced from 1.5x to1.33x. For the scale of WAS, such reduction translatesinto significant savings.

3.4 Comparison - Modern Storage CodesWe apply the same reliability analysis to state-of-the-

art erasure codes designed specifically for storage sys-tems, such as Weaver codes [18], HoVer codes [19] andStepped Combination codes [20]. We examine the trade-off between reconstruction cost and storage overheadwhen these codes are at least as reliable as 3-replication.

The results are plotted in Figure 6. The storage over-head of Weaver codes is 2x or higher, so only a singlepoint (storage overhead = 2x, reconstruction cost = 3) isshown in the Figure. We observe that the trade-off curveof Stepped Combination codes is strictly below that ofReed-Solomon. Therefore, both codes can achieve moreefficient reconstruction than Reed-Solomon when stor-age overhead is fixed. Similarly, they require less storageoverhead when reconstruction cost is fixed. HoVer codesoffer many trade-off points better than Reed-Solomon.

Compared to these modern storage codes, LRC issuperior in terms of the trade-off between reconstruc-tion cost and storage overhead. The primary reason isthat LRC separates parity fragments into local ones andglobal ones. In this way, local parities only involve min-imum data fragments and can thus be most efficient forproviding reconstruction reads when there is a hot spot,for reconstructing single failures, and for reconstructinga single fragment that is offline due to upgrade. On theother hand, global parities involve all the data fragmentsand can thus be most useful to provide fault tolerancewhen there are multiple failures. In contrast, in Weavercodes, HoVer codes and Stepped Combination codes, allparities carry both the duty of reconstruction and faulttolerance. Therefore, they cannot achieve the same trade-off as LRC.

LRC is optimized for reconstructing data fragments,but not parities, in order to quickly reconstruct on-demand based reads from clients. In terms of parity re-construction, Weaver codes, HoVer codes and SteppedCombination codes can be more efficient. For instance,let’s compare Stepped Combination code to LRC at thestorage overhead of 1.5x, where both codes consist of12 data fragments and 6 parities. For the Stepped Com-bination code, every parity can be reconstructed from 3fragments, while for LRC the reconstruction of globalparities requires as many as 12 fragments. It turns outthat there is fundamental contention between parity re-construction and data fragment reconstruction, which westudied in detail separately [21]. In WAS, since parityreconstruction happens only when a parity is lost (e.g.,disk, node or rack failure), it is off the critical path ofserving client reads. Therefore, it is desirable to trade theefficiency of parity reconstruction in order to improve theperformance of data fragment reconstruction, as is donein LRC.

3.5 Correlated FailuresThe Markov reliability model described in Section 3.1

assumes failures are independent. In practice, correlatedfailures do happen [10, 34]. One common correlated fail-

6

ure source is all of the servers under the same fault do-main. WAS avoids these correlated failures by alwaysplacing fragments belonging to the same coding groupin different fault domains.

To account for additional correlated failures, theMarkov reliability model can be readily extended byadding transition arcs between non-adjacent states [10].

4 Erasure Coding Implementation in WASThe WAS architecture has three layers within a storage

stamp (cluster) - front-end layer, partitioned object layerand stream replication layer [1].

Erasure coding in WAS is implemented in the streamlayer as a complementary technique to full data repli-cation. It is also possible to implement erasure cod-ing across multiple storage stamps on several data cen-ters [38]. Our choice to implement erasure coding insidethe stream layer is based on the fact that it fits the over-all WAS architecture where the stream layer is responsi-ble for keeping the data durable within a stamp and thepartition layer is responsible for geo-replicating the databetween data centers (see [1] for more details).

4.1 Stream Layer ArchitectureThe main components of the stream layer are the

Stream Managers (SM), which is a Paxos [37] replicatedserver, and Extent Nodes (EN) (see Figure 7).

Streams used by the partition layer are saved as a list ofextents in the stream layer. Each extent consists of a listof append blocks. Each block is CRC’d and this block isthe level of granularity the partitioned object layer usesfor appending data to the stream, as well as reading data(the whole block is read to get any bytes out of the block,since the CRC is checked on every read). Each extent isreplicated on multiple (usually three) ENs. Each writeoperation is committed to all nodes in a replica set in adaisy chain, before an acknowledgment is sent back tothe client. Write operations for a stream keep appendingto an extent until the extent reaches its maximum size (inthe range of 1GB-3GB) or until there is a failure in thereplica set. In either case, a new extent on a new replicaset is created and the previous extent is sealed. Whenan extent becomes sealed, its data is immutable, and itbecomes a candidate for erasure coding.

4.2 Erasure Coding in the Stream LayerThe erasure coding process is completely asyn-

chronous and off the critical path of client writes. TheSM periodically scans all sealed extents and schedules asubset of them for erasure coding based on stream poli-cies and system load. We configure the system to auto-matically erasure code extents storing Blob data, but alsohave the option to erasure code Table extents too.

As a first step of erasure coding of an extent, the SMcreates fragments on a set of ENs whose number depends

EN EN EN

EN EN EN

SM

EN

EN

SMSM

paxos

A. Erasure code an extent

B. Ack with Metadata

Stream Layer

…

…

2

1

3 2

2

2

3

33

Figure 7: Erasure Coding of an Extent (not all targetENs are shown).

on the erasure coding parameters, for example 16 frag-ments for LRC (12, 2, 2).

The SM designates one of the ENs in the extent’sreplica set as the coordinator of erasure coding (see Fig-ure 7) and sends it the metadata for the replica set. Fromthen on, the coordinator EN has the responsibility ofcompleting the erasure coding. The coordinator EN has,locally on its node, the full extent that is to be erasure-coded. The EN prepares the extent for erasure coding bydeciding where the boundaries for all of the fragmentswill be in the extent. It chooses to break the extent intofragments at append block boundaries and not at arbi-trary offsets. This ensures that reading a block will notcross multiple fragments.

After the coordinator EN decides what the fragmentoffsets are, it communicates those to the target ENs thatwill hold each of the data and parity fragments. Thenthe coordinator EN starts the encoding process and keepssending the encoded fragments to their designated ENs.The coordinator EN, as well as each target EN, keepstrack of the progress made and persists that informationinto each new fragment. If a failure occurs at any mo-ment in this process, the rest of the work can be pickedup by another EN based on the progress information per-sisted in each fragment. After an entire extent is coded,the coordinator EN notifies the SM, which updates themetadata of the extent with fragment boundaries andcompletion flags. Finally, the SM schedules full replicasof the extent for deletion as they are no longer needed.

The fragments of an extent can be read directly by aclient (i.e., the partition or front-end layer in WAS) bycontacting the EN that has the fragment. However, ifthat target EN is not available or is a hot spot, the clientcan contact any of the ENs that has any of the fragmentsof the extent, and perform a reconstruction read (see Fig-

7

EN EN EN

EN EN EN

SM

EN

EN

SMSM

paxos

Partition Layer/Client

Data

A. Open Stream

B. Stream Metadata

Stream Layer

…

…

1

234

2

233

Reconstru

ction

Read

× Figure 8: Reconstruction for On-Demand Read(notall target ENs are shown).

ure 8). That EN would read the other needed fragmentsfrom other ENs, then reconstruct that fragment, cachethe reconstructed fragment locally in case there are otherreads to it, and return the results to the client. LRC re-duces the cost of this reconstruction operation consider-ably by reducing the number of source ENs that need tobe accessed.

If the EN or the disk drive that hosts the extent frag-ment is unavailable for an extended period of time, theSM initiates the reconstruction of the fragment on a dif-ferent EN. This operation is almost identical to the recon-struction steps shown in Figure 8 except for the fact thatthe operation is initiated by the SM instead of the client,and the data is written to disk rather than being sent backto the client.

4.3 Using Local Reconstruction Codes inWindows Azure Storage

When LRC is used as an erasure coding algorithm,each extent is divided intok equal-sizedata fragments.Thenl local andr globalparity fragments are created.

The placement of the fragments takes into account twofactors: i) load, which favors less occupied and lessloaded extent nodes;ii) reliability, which avoids plac-ing two fragments (belonging to the same erasure cod-ing group) into the same correlated domain. There aretwo primary correlated domains: fault domain and up-grade domain. A fault domain, such as rack, categorizesa group of nodes which can fail together due to com-mon hardware failure. An upgrade domain categorizesa group of nodes which are taken offline and upgradedat the same time during each upgrade cycle. Upgradedomains are typically orthogonal to fault domains.

Let’s now use LRC (12, 2, 2) as an example and il-lustrate an actual fragment placement in WAS. A WASstamp consists of 20 racks. For maximum reliability,each of the total 16 fragments for an extent is placed in

a different rack. If each fragment were similarly placedon a different upgrade domain, then at least 16 upgradedomains are required. In practice, however, too manyupgrade domains can slow down upgrades and it is de-sirable to keep the number of upgrade domains low. InWAS, we currently use 10 upgrade domains. This meansthat we have at most 10% of the storage stamps (clus-ter) resources offline at any point in time during a rollingupgrade.

Given our desire to have fewer upgrade domains thanfragments, we need an approach for placing the frag-ments across the upgrade domains to still allow LRC toperform its fast reconstruction for the fragments that areoffline. To that end, we exploit the local group propertyof LRC and group fragments belonging to different lo-cal groups into same upgrade domains. In particular, weplace the two local groupsx andy so that their data frag-mentsxi andyi are in the same upgrade domaini (i.e.,x0 andy0 are placed in the same upgrade domain, butdifferent fault domains). Similarly, we place the localparitiespx andpy in one upgrade domain as well. Thetwo global parities are placed in two separate upgradedomains from all other fragments.

Take for example LRC (12, 2, 2). In total, we use 9upgrade domains for placing the fragments for an extent.There are two local groups, each with 6 data fragments,plus 2 local parities (1 per group), and then 2 global pari-ties. When using 9 upgrade domains, we put the 2 globalparities into two upgrade domains with no other frag-ments in them, we then put the 2 local parities into thesame upgrade domain with no other fragments in them,and then the remaining 6 upgrade domains hold the 12data fragments for the 2 local groups.

During an upgrade period, when one upgrade domainis taken offline, every single data fragment can still be ac-cessed efficiently - either reading directly from the frag-ment or reconstructing from other fragments within itslocal group.

4.4 Designing for Erasure CodingScheduling of Various I/O Types. The stream layer

handles a large mix of I/O types at a given time: on-demand open/close, read, and append operations fromclients, create, delete, replicate, reconstruct, scrub, andmove operations generated by the system itself, andmore. Letting all these I/Os happen at their own pace canquickly render the system unusable. To make the systemfair and responsive, operations are subject to throttlingand scheduling at all levels of the storage system. EveryEN keeps track of its load at the network ports and onindividual disks to decide to accept, reject, or delay I/Orequests. Similarly, the SM keeps track of data replica-tion load on individual ENs and the system as a wholeto make decisions on when to initiate replication, erasure

8

coding, deletion, and various other system maintenanceoperations. Because both erasure coding and decodingrequires accessing multiple ENs, efficiently schedulingand throttling these operations is crucial to have fair per-formance for other I/O types. In addition, it is also im-portant to make sure erasure coding is keeping up withthe incoming data rate from customers as well as inter-nal system functions such as garbage collection. We havea Petabyte of new data being stored to WAS every cou-ple of days, and the built out capacity expects a certainfraction of this data to be erasure-coded. Therefore, theerasure coding needs to be scheduled such that it keepsup with the incoming rate of data, even when there arecritical re-replications that also need to be scheduled dueto a lost disk, node or rack.

Reconstruction Read-ahead and Caching.Recon-struction of unavailable fragments is done in unit sizesgreater than the individual append blocks (up to 5MB) toreduce the number of disk and network I/Os. This read-ahead data is cached in memory (up to 256MB) of theEN that has done the reconstruction. Further sequentialreads are satisfied directly from memory.

Consistency of Coded Data. Data corruption canhappen throughout the storage stack for numerous rea-sons [36]. In a large-scale distributed storage system,data can become corrupted while at rest, while being reador written in memory, and while passing through severaldata paths. Therefore, it is essential to check the con-sistency of the data in every step of the storage systemoperations in addition to periodically scrubbing the dataat rest.

Checksum and parity are the two primary mechanismsto protect against data corruption [35]. In WAS, we em-ploy various CRC (Cyclic Redundancy Check) fields todetect data and metadata corruptions. For example, eachappend block contains a header with CRC of the datablock, which is checked when the data is written and ev-ery time data is read. When a particular data read orreconstruction operation fails due to CRC checks, theoperation is retried using other combinations of erasure-coded fragments. Also, the fragment with the corruptedblock is scheduled for regeneration on the next availableEN.

After each erasure encoding operation, several decod-ing combinations are tried from memory on the coordi-nator EN to check for successful restorations. This stepis to ensure that the erasure coding algorithm itself doesnot introduce data inconsistency. For LRC (12, 2, 2),we perform the following decoding validations beforeallowing the EC to complete:i) randomly choose onedata fragment in each local group and reconstruct it us-ing its local group;ii) randomly choose one data frag-ment and reconstruct it using one global parity and theremaining data fragments;iii) randomly choose one data

fragment and reconstruct it using the other global parityand the remaining data fragments;iv) randomly choosetwo data fragments and reconstruct them;v) randomlychoose three data fragments and reconstruct them; andvi) randomly choose four data fragments (at least one ineach group) and reconstruct them. After each decodingcombination above, the CRC of the decoded fragment ischecked against the CRC of the data fragment for suc-cessful reconstruction of data.

Finally, the coordinator EN performs a CRC of all ofthe final data fragments and checks that CRC against theoriginal CRC of the full extent that needed to be erasure-coded. This last step ensures we have not used data thatmight become corrupted in memory during coding op-erations. If all these checks pass, the resulting codedfragments are persisted on storage disks. If any failureis detected during this process, the erasure coding oper-ation is aborted, leaving the full extent copies intact, andthe SM schedules erasure coding again on another ENlater.

Arithmetic for Erasure Coding. Directly using Ga-lois Field arithmetic for the erasure coding implementa-tion is expensive because of all the emulation operationsrequired on top of the integer arithmetic. Therefore, it isgeneral practice to optimize Galois Field arithmetic op-erations by pre-computing and using addition and mul-tiplication tables, which improves coding speed. In ad-dition, Reed-Solomon codes can be further optimized bya transformation that enables the use of XOR operationsexclusively [22]. To get the best possible performancefor this transformation, the XOR operations can be or-dered specifically based on the patterns in the codingand decoding matrices [23]. This scheduling removesmost of the redundant operations and eliminates check-ing the coding matrix again and again during the actualcoding pass. WAS uses all of the above optimizationsto streamline in-memory encode and decode operations.Since modern CPUs perform XOR operations extremelyfast, in-memory encode and decode can be performed atthe speed which the input and output buffers can be ac-cessed.

5 PerformanceWAS provides cloud storage in the form of Blobs (user

files), Tables (structured storage), Queues (message de-livery), and Drives (network mounted VHDs). Applica-tions have different workloads, which access each typeof storage differently. The size of I/O can be polarized:small I/O is typically in the 4KB to 64KB range, predom-inantly assessing Tables and Queues; large I/Os (mostly4MB), primarily accessing Blobs; and Drives can see amixture of both. In this section, we characterize the per-formance of LRC and compare it to Reed-Solomon forsmall and large I/Os, respectively.

9

�� �������

�� ����� ���� ���� ���0

50

100

150

200

250

300

350

Light Heavy

Laten

cy (m

s)

Cluster Load

Direct RS (read k) RS (read k+1) RS (read k+2) RS (read k+3) LRC

(a) Latency

0

5

10

15

Direct RS-(k) RS-(k+1) RS-(k+2) RS-(k+3) LRC

I/O (n

ormali

zed)

(b) Reconstruction I/O

Figure 9: Small (4KB) I/O Reconstruction - (12, 4)Reed-Solomon vs. (12, 2, 2) LRC.

We compare LRC (12, 2, 2) to Reed-Solomon (12,4), both of which yield storage cost at 1.33x. Note that(12, 3) Reed-Solomon isnot an option, because its reli-ability is lower than 3-replication. Results are obtainedon our production cluster with significant load variationover time. We separate the results based on the clusterload and contrast the gain of LRC when the cluster islightly loaded to when it is heavily loaded. The produc-tion cluster, where these results were gathered, has one1Gbps NIC for each storage node.

5.1 Small I/OsThe key metric for small I/O is latency and the num-

ber of I/Os taken by the requests. We run experimentswith a mixture of direct read (reading a single fragment),reconstruction read with Reed-Solomon and LRC. Theaverage latency results are summarized in Figure 9.

When the cluster load is light, all the latencies ap-pear very low and comparable. There is not much differ-ence between direct read and reconstruction with eitherReed-Solomon or LRC. However, when the cluster loadis heavy, there are definitely differences.

When a data fragment is unavailable, the reconstruc-tion read with Reed-Solomon can be served by randomlyselectingk = 12 fragments out of the remaining 15 frag-ments to perform erasure decoding. Unfortunately, thelatency turns out much larger than that of direct read –305ms vs. 91ms – because it is determined by the slowestfragment among the entire selection. Given the high la-

�� � ! �" ��# �## ��! �$� ��%"�#&� �&#'

$� ' "0

200

400

600

800

1000

1200

1400

Light Heavy

Laten

cy (m

s)

Cluster Load

Direct RS (read k) RS (read k+1) RS (read k+2) RS (read k+3) LRC

(a) Latency

0

5

10

15

Direct RS-(k) RS-(k+1) RS-(k+2) RS-(k+3) LRCBa

ndwi

dth

(norm

alized

)(b) Reconstruction Bandwidth

Figure 10:Large (4MB) I/O Reconstruction - (12, 4)Reed-Solomon vs. (12, 2, 2) LRC.

tency, we exploit a simple technique - selecting more (de-noted ask′) fragments and decoding from the firstk ar-rivals (represented asRS (read k’) or RS+(k’)in Figure 9). This technique appears very effective inweeding out slow fragments and reduces the latency dra-matically.

In comparison, reconstruction with LRC is fast. Itrequires only 6 fragments to be read and achieves thelatency of 166ms, which is comparable to 151ms withReed-Solomon reading 13 fragments. Note that ex-tremely aggressive reads (reading all 15) with Reed-Solomon achieves even lower latency, but it comes at acost of many more I/Os. The relative number of I/Os,normalized by that of direct read, is shown in Figure 9(b).The fast reconstruction and the I/O cost savings are thereasons why we chose LRC over Reed-Solomon for Win-dows Azure Storage’s erasure coding.

5.2 Large I/OsWe now examine the reconstruction performance of

4MB large I/Os. The key metric is latency and bandwidthconsumption. We compare direct read to reconstructionwith Reed-Solomon and LRC. The results are presentedin Figure 10.

The results are very different from the small I/O case.Even when the cluster load is light, the reconstructionwith erasure coding is already much slower than directread, given the amount of bandwidth it consumes. Com-pared to direct read taking 99ms, Reed-Solomon with 12

10

fragments takes 893ms – 9 times slower.In large I/Os, the latency is mostly bottlenecked by

network and disk bandwidth, and the bottleneck for theseresults was the 1Gbps network card on the storage nodes.Since LRC reduces the number of fragments by half, itslatency is 418ms and significantly reduced from that ofReed-Solomon. Note that, different from the small I/Ocase, aggressive reads with Reed-Solomon using morefragments does not help, but rather hurts latency. Theobservation is similar when the cluster load is heavy.

Because of the significantly reduced latency and thebandwidth savings, which are particularly importantwhen the system is under heavy load or has to recoverfrom a rack failure, we chose LRC for Windows AzureStorage.

5.3 Decoding LatencyTo conclude the results, we also wanted to compare

the latency spent on decoding fragments between Reed-Solomon and LRC. The average latency of decoding4KB fragments is 13.2us for Reed-Solomon and 7.12usfor LRC. Decoding is faster in LRC than Reed-Solomonbecause only half the number of fragments are involved.Even so, the decoding latencies are typically in microsec-onds andseveral orders of magnitude smaller than theoverall latency to transfer the fragments to perform thereconstruction. Therefore, from the latency of decod-ing standpoint, LRC and Reed-Solomon are compara-ble. Note that, pure XOR-based codes, such as Weavercodes [18], HoVer codes [19] and Stepped Combinationcodes [20], can be decoded even faster. The gain of fasterdecoding, however, would not matter in WAS, as thedecoding time is orders of magnitude smaller than thetransfer time.

6 Related WorkErasure Coding in Storage Systems:Erasure cod-

ing has been applied in many large-scale distributedstorage systems, including storage systems at Facebookand Google [3, 4, 5, 8, 9, 10]. The advantage oferasure coding over simple replication is that it canachieve much higher reliability with the same storage,or it requires much lower storage for the same relia-bility [7]. The existing systems, however, do not ex-plore alternative erasure coding designs other than Reed-Solomon codes [13]. In this work, we show that, un-der the same reliability requirement, LRC allows a muchmore efficient cost and performance trade-off than Reed-Solomon.

Performance: In erasure-coded storage systems, nodefailures trigger rebuilding process, which in turn re-sults in degraded latency performance on reconstructionreads [6]. Moreover, experience shows that transient er-rors in which no data are lost account for more than 90%of data center failures [10]. During these periods as well

as upgrades, even though there is no background data re-building, reads trying to access unavailable nodes are stillserved through reconstruction.

Complementary to system techniques, such as loadbalancing and prioritization [11], LRC explores whetherthe erasure coding scheme itself can be optimized to re-duce repair traffic and improve user I/Os.

Erasure Code Design:LRC is a critical improvementover our own Pyramid codes [14]. LRC exploits non-uniform parity degrees, where some parities connect tofewer data nodes than others. Intuitively, the parities withfewer degrees facilitate efficient reconstruction. This di-rection was originally pioneered for communication bylandmark papers on Low Density Parity Check (LDPC)codes [15, 16]. LDPC were recently explored in the areaof storage [17, 20]. In particular, Plank et al. [17] ap-plied enumeration and heuristic methods to search forparity-check erasure codes of small length. Due to expo-nential search space, the exploration was limited to 3, 4and 5 parities. The codes discovered cannot tolerate arbi-trary three failures, which is the minimum requirement inWAS. Stepped Combination codes [20] are LDPC codeswith very small length, offering fault tolerance guaranteeand efficient reconstruction, but do not provide the sametrade-offs that LRC can achieve..

Reed-Solomon Codes are Maximum Distance Separa-ble (MDS) codes [12], which require minimum storageoverhead for given fault tolerance. LRC is not MDS andthus requires higher storage overhead for the same faulttolerance. The additional storage overhead from the localparities are exploited for efficient reconstruction. This di-rection of trading storage overhead for reconstruction ef-ficiency is also explored by other state-of-the-art erasurecodes designed specifically for storage systems, such asWeaver codes [18], HoVer codes [19] and Stepped Com-bination codes [20]. We show that LRC achieves bettertrade-offs than these modern storage codes for WAS’ era-sure coding design goals .

To improve reconstruction performance, instead ofreading from fewer fragments as in LRC, a promisingalternative is to read instead from more fragments, butless data from each [24, 25, 26, 27]. However, practi-cal solutions known so far [26, 27] achieve only around20%-30% savings in terms of I/O and bandwidth, muchless than LRC.

7 SummaryErasure coding is critical to reduce the cost of cloud

storage, where our target storage overhead is 1.33x ofthe original data. When using erasure coding, fast recon-struction of offline data fragments is important for perfor-mance. In Windows Azure Storage, these data fragmentscan be offline due to disk, node, rack and switch failures,as well as during upgrades.

11

We introduced Local Reconstruction Codes as a wayto reduce the number of fragments that need to be readfrom to perform this reconstruction, and compared LRCto Reed-Solomon. We showed that LRC (12, 2, 2), whichhas a storage overhead of 1.33x, saves significant I/Osand bandwidth during reconstruction when compared toReed-Solomon (12, 4). In terms of latency, LRC hascomparable latency for small I/Os and better latency forlarge I/Os.

We chose LRC (12, 2, 2) since it achieves our 1.33xstorage overhead target and has the above latency, I/Oand bandwidth advantages over Reed-Solomon. In ad-dition, we needed to maintain durability at the same orhigher level than traditional 3 replicas, and LRC (12, 2,2) provides better durability than the traditional approachof keeping 3 copies. Finally, we explained how erasurecoding is implemented, some of the design considera-tions, and how we can efficiently lay out LRC (12, 2,2) across the 20 fault domains and 10 upgrade domainsused in Windows Azure Storage.

8 AcknowledgementsWe would like to thank Andreas Haeberlen, Geoff

Voelker, and anonymous reviewers for providing valu-able feedback on this paper. We would also like to thankall of the members of the Windows Azure Storage team.

References[1] B. Calder et al., “Windows Azure Storage: A Highly Available

Cloud Storage Service with Strong Consistency,”ACM SOSP,2011.

[2] D. T. Meyer et al., “Fast and Cautious Evolution of Cloud Storage,”HotStorage, 2010.

[3] J. Kubiatowicz et al., “OceanStore: An Architecture forGlobal-Scale Persistent Storage,”ACM ASPLOS, Nov. 2000.

[4] A. Haeberlen, A. Mislove, and P. Druschel, “Glacier: Highlydurable, decentralized storage despite massive correlatedfailures,”USENIX NSDI, 2005.

[5] M. Abd-El-Malek et al., “Ursa Minor: Versatile Cluster-basedStorage,”USENIX FAST, 2005.

[6] C. Ungureanu et al., “HydraFS: A High-Throughput File Sys-tem for the HYDRAstor Content-Addressable Storage System,”USENIX FAST, 2010.

[7] H. Weatherspoon, and J. Kubiatowicz, “Erasure coding vs. repli-cation: A quantitative comparison,”In Proc. IPTPS, 2001.

[8] D. Borthakur et al., “HDFS RAID,”Hadoop User Group Meeting,Nov. 2010.

[9] A. Fikes, “Storage Architecture and Challenges,”Google FacultySummit, 2010.

[10] D. Ford et al., “Availability in Globally Distributed Storage Sys-tems,”,USENIX OSDI, 2010.

[11] L. Tian et al., “PRO: A Popularity-based Multi-threaded Re-construction Optimization for RAID-Structured Storage Systems,”USENIX FAST, 2007

[12] F. J. MacWilliams and N. J. A. Sloane, “The Theory of ErrorCorrecting Codes,”Amsterdam: North-Holland, 1977.

[13] I. S. Reed and G. Solomon, “Polynomial Codes over Certain Fi-nite Fields”,J. SIAM, 8(10), 300-304, 1960.

[14] C. Huang, M. Chen, and J. Li, “Pyramid Codes: FlexibleSchemes to Trade Space for Access Efficiency in Reliable DataStorage Systems,”Proc. of IEEE NCA, Cambridge, MA, Jul. 2007.

[15] R. G. Gallager, “Low-Density Parity-Check Codes,”MIT Press,Cambridge, MA, 1963.

[16] M. G. Luby et al., “Efficient Erasure Correcting Codes,”IEEETransactions on Information Theory, 2011.

[17] J. S. Plank, R. L. Collins, A. L. Buchsbaum, and M. G. Thoma-son, “Small Parity-Check Erasure Codes - Exploration and Obser-vations,“Proc. DSN, 2005.

[18] J. L. Hafner, “Weaver codes: Highly fault tolerant erasure codesfor storage systems,”USENIX FAST, 2005.

[19] J. L. Hafner, “HoVer Erasure Codes for Disk Arrays,”Proc. ofDSN, 2006.

[20] K. M. Greenan, X. Li, and J. J. Wylie, “Flat XOR-based erasurecodes in storage systems: Constructions, efficient recovery, andtradeoffs,”IEEE Mass Storage Systems and Technologies, 2010.

[21] P. Gopalan, C. Huang, H. Simitci, and S. Yekhanin, “On theLo-cality of Codeword Symbols,”Allerton, 2011.

[22] J. Blomer et al., “An XOR-Based Erasure-Resilient CodingScheme,” Technical Report No. TR-95-048, ICSI, Berkeley, Cali-fornia, Aug. 1995.

[23] J. Luo, L. Xu, and J. S. Plank, “An Efficient XOR-SchedulingAlgorithm for Erasure Codes Encoding,”Proc. DSN, Lisbon, Por-tugal, June, 2009.

[24] A. G. Dimakis et al., “Network Coding for Distributed StorageSystems,”IEEE Transactions on Information Theory, Vol. 56, Is-sue 9, Sept. 2010.

[25] L. Xiang et al., “Optimal recovery of single disk failurein RDPcode storage systems,”ACM SIGMETRICS, 2010.

[26] O. Khan et al., “Rethinking Erasure Codes for Cloud FileSystems: Minimizing I/O for Recovery and Degraded Reads,”USENIX FAST, San Jose, Feb. 2012.

[27] Y. Hu et al., “NCCloud: Applying Network Coding for the Stor-age Repair in a Cloud-of-Clouds,”USENIX FAST, San Jose, 2012.

[28] J. H. Howard et al., “Scale and Performance in a Distributed FileSystem,”ACM ToCS, Feb. 1988.

[29] M. Satyanarayanan et al., “CODA: A Highly Available File Sys-tem for a Distributed Workstation Environment,”IEEE Transac-tions on Computers, Apr. 1990.

[30] B. Liskov et al., “Replication in the Harp File System,”ACMSOSP, Pacific Grove, CA, Oct. 1991.

[31] F. Dabek et al., “Wide-Area Cooperative Storage with CFS,”ACM SOSP, 2001.

[32] B. G. Chun et al., “Efficient Replica Maintenance for DistributedStorage Systems,”USENIX NSDI, 2006.

[33] Q. Xin et al., “Reliability mechanisms for very large storage sys-tems,” Proc. of IEEE Conference on Mass Storage Systems andTechnologies, 2003.

[34] S. Nath et al., “Subtleties in Tolerating Correlated Failures inWide-Area Storage Systems,”USENIX NSDI, 2006.

[35] A. Krioukov et al., “Parity Lost and Parity Regained,”USENIXFAST, Feb. 2008.

[36] L. N. Bairavasundaram et al., “An Analysis of Data Corruption inthe Storage Stack,”USENIX FAST, Feb. 2008.

[37] L. Lamport, “The Part-Time Parliament,”ACM Transactions onComputer Systems, vol. 16, no. 2, pp. 133-169, May 1998.

[38] J. K. Resch, and J. S. Plank, “AONT-RS: Blending Security andPerformance in Dispersed Storage Systems,”USENIX FAST, Feb.2011.

12