Design Tradeoffs forSSD Reliability

Bryan S. Kim, Jongmoo Choi, Sang Lyul Min

Seoul National University, Dankook University

High-level objectives

Understand the SSD-internal mechanisms behind fail-slow symptoms

• H. Gunawi et al, “Fail-slow at scale: evidence of hardware performance faults in large production systems”, FAST 2018

High-level objectives

Examine SSD-internal reliability enhancement techniques

• Images from Google searches

High-level objectives

Think about system- and device-level approachesfor handling errors

• Images from Google searches

How bad is it?

5

• L. Grupp et al, “Characterizing flash memory: anomalies, observations, and applications”, Micro 2009

1E-8

1E-7

1E-6

1E-5

1E-4

1E-3

1E-2

1E-1

2008 2010 2012 2014 2016 2018 2020

Err

or

rate

mea

sure

ment

Year published

How bad is it?

6

• H. Sun et al, “Quantifying reliability of solid-state storage from multiple aspects”, SNAPI 2011• Y. Cai et al, “Error patterns in MLC NAND flash memory: measurement, characterization, and analysis”, DATE 2012

1E-8

1E-7

1E-6

1E-5

1E-4

1E-3

1E-2

1E-1

2008 2010 2012 2014 2016 2018 2020

Err

or

rate

mea

sure

ment

Year published

How bad is it?

7

• Y. Cai et al, “Data retention in MLC NAND flash memory: characterization, optimization, and recovery”, HPCA 2015• Data from an industry partner, 2018

1E-8

1E-7

1E-6

1E-5

1E-4

1E-3

1E-2

1E-1

2008 2010 2012 2014 2016 2018 2020

Err

or

rate

mea

sure

ment

Year published

SSD’s reliability issue

Error-prone

memory

ReliableSSD

RBER: 10-4~10-2 UBER: <10-15

• How to make SSD reliable?

• Performance overhead?

• Across different chips and wear states?

8

Flash memory errors

Wear-out

CG

FG

Vprog

9

Flash memory errors

Wear-out Retention loss

CG

FG

Vprog

CG

FG

0V

10

-- --

Flash memory errors

Wear-out Retention loss Disturbance

CG

FG

Vprog

CG

FG

0V

CG

FG

Vpass

11

-- -- --

Flash memory error modeling

RBER (cycles, time, reads)= ε+ α ∙ cyclesk

+ β ∙ cyclesm ∙ timen

+ γ ∙ cyclesp ∙ readsq

• N. Mielke et al, “Reliability of solid-state drives based on NAND flash memory”, Proceedings of the IEEE, 2017

12

From measurements to model

• H. Sun et al, “Quantifying reliability of solid-state storage from multiple aspects”, SNAPI 2011• Y. Cai et al, “Data retention in MLC NAND flash memory: characterization, optimization, and recovery”, HPCA 2015• Y. Cai et al, “Read disturb errors in MLC NAND flash memory: characterization, mitigation, and recovery”, DSN 2015• Data from an industry partner, 2018

Measurement (data) Model

• 3x-nm MLC (2011)

• 2y-nm MLC (2015)

• 3D TLC (2018)

13

1E-7

1E-6

1E-5

1E-4

1E-3

1E-2

1E-1

1E0

Raw

bit

err

or

rate

Wear Retention Disturbance

Error model: 3x-nm MLC (2011)

Wear up to10K P/E cycles

10K P/E cycles +up to 10K readsor up to 1 year

14

Error model: 2y-nm MLC (2015)

15

1E-7

1E-6

1E-5

1E-4

1E-3

1E-2

1E-1

1E0

Raw

bit

err

or

rate

Wear Retention Disturbance

Wear up to10K P/E cycles

10K P/E cycles +up to 10K readsor up to 1 year

Error model: 3D TLC (2018)

16

1E-7

1E-6

1E-5

1E-4

1E-3

1E-2

1E-1

1E0

Raw

bit

err

or

rate

Wear Retention Disturbance

Wear up to10K P/E cycles

10K P/E cycles +up to 10K readsor up to 1 year

SSD reliability enhancements

• Error correction code

• Data re-reads

• Intra-SSD redundancy

• Background relocation

17

Error correction code

ECCencoder

ECC decoder

Flash memory

18

Error correction code

Data DataP

ECCencoder

ECC decoder

Flash memory

19

Error correction code

ECCencoder

ECC decoderData Data P

Flash memory

20

Flash memory

Data re-reads

ECCencoder

ECC decoder Data P

21

Flash memory

Data re-reads

ECCencoder

ECC decoder Data P

1 2

22

Data re-reads

ECCencoder

ECC decoder

Data P

Flash memory

Data

23

Summary: ECC and data re-reads

• Error correction code

– Predictable performance

– Is fixed at design-time

• Data re-read

– Is much more powerful than ECC

– Increases latency for correcting errors

24

Evaluation: data re-readFor the 3D TLC (2018)

25

0

0.5

1

1.5

2

2.5

3

3.5

25-bit 50-bit 75-bit 100-bit

No

rm.

av

g.

RT

ECC correction strength

1K cycles 3K cycles 5K cycles

Why is data re-read bad?For the 3D TLC (2018)

26

0.5

0.6

0.7

0.8

0.9

1

0 25 50 75 100 125 150 175

Cu

mu

lati

ve

pro

bab

ilit

y

Number of raw bit errors

inf-bit ECC

100-bit ECC

75-bit ECC

50-bit ECC

25-bit ECC

Observations

• Repeated data re-reads make it worse

– 75-bit: ~30% increased latency at end-of-life

27

Intra-SSD redundancy

D0D1D2D3D4D5D6

D0

D1

D2

D3

D4

D5

D6

P

P

28

Flash memory chips

Intra-SSD redundancy

D0

D1

D2

D3

D4

D5

D6

PD0D1PD3D4D5D6D2

29

Flash memory chips

Summary: intra-SSD redundancy

• Error correction code– Is fixed at design-time

• Data re-read– Increases latency for correcting errors

• Intra-SSD redundancy– Protects against random and sporadic errors– Increases write amplification– Increases read amplification on errors

30

Evaluation: redundancy

0

0.5

1

1.5

2

s = 15 s = 7

No

rm.

av

g.

RT

Stripe size

1K cycles 3K cycles 5K cycles

0

0.5

1

1.5

2

2.5

3

3.5

4

s = 15 s = 7

No

rm.

3 m

ines

Qo

S

Stripe size

1K cycles 3K cycles 5K cycles

For the 3D TLC (2018) with 75-bit ECC

31

Observations

• Repeated data re-reads make it worse

• Overheads of redundancy outweigh its benefits

– +56% latency at end-of-life

32

Observations

• Repeated data re-reads make it worse

• Overheads of redundancy outweigh its benefits

• Scrubbing reduces error-induced latency,but increases internal traffic– +25% latency at end-of-life

– Highly dependent on accuracy of error prediction

33

Observations

• Repeated data re-reads make it worse

• Overheads of redundancy outweigh its benefits

• Scrubbing reduces error-induced latency,but increases internal traffic

• We need to consider data characteristicsand compositionally combine reliability enhancements

34

Holistic reliability management

• Cold data

– Need protection against retention errors

– Least write amplification with redundancy

– Likely to be identified by GC

Holistic reliability management

• Cold data

– Selective redundancy for GC-ed data

• Read-hot data

– Need protection against disturbance errors

– # of data re-reads can be used as proxy

– Likely to be identified by scrubber

Holistic reliability management

• Cold data

– Selective redundancy for GC-ed data

• Read-hot data

– Cost-benefit scrubbing

• Write-hot data

– No special attention required

Evaluation

38

0

1

2

3

4

No

rm.

av

g. R

TECC + re-read Oracle scrub HRM

9.0 5.4

ECC + re-read : Rely on ECC and data re-readsOracle scrub : Scrub based on oracle knowledgeHRM : Holistic reliability management

For the 3D TLC (2018)with 75-bit ECC@ end-of-life state

The bright side of flash memory

• S. Lee, “Emerging Challenges in NAND Flash Technology”, Flash Summit 2011

39

1E-8

1E-7

1E-6

1E-5

1E-4

1E-3

1E-2

1E-1

2008 2010 2012 2014 2016 2018 2020

Err

or

rate

mea

sure

ment

Year published

The dark side of flash memory

40