““Storage is Cheap” and Other LiesStorage is Cheap” and Other Lies

Lance Stuchell, University of Michigan LibraryCurating and Managing Research Data for Re-UseICPSR, July 31, 2013

“Hard Drives” by Michael Muni CC-BY-NC-ND

Overview

• Instability of storage hardware costs • Check out David Rosenthal’s blog at

http://blog.dshr.org/

• Costs of storing digitized moving images

• Effects of storage costs on digital preservation, formats, etc.

“Storage is Cheap”

• Perception is based on “Kryder's Law”• A 2005 Scientific American article about Mark

Kryder, Seagate's Senior VP of Research • Magnetic disk density increases quickly• Disk density closely tied to pricing • 30-year history of disk prices dropping about

40% per year

• Disk costs were affordable & predictable• 30% of total storage costs

The Party's Over

• Mid 2011: Kryder’s Law slows• Latest projections ~ 20% density growth

• Late 2011: Flooding in Taiwan causes shortfall of 70 million disk drives• Prices remain over 50% higher• Not expected to return to pre-flood levels

until 2014

• Changes in technology David S. H. Rosenthal, “Storage Will Be A Lot Less Free Than It Used To Be,” 2012.http://blog.dshr.org/2012/10/storage-will-be-lot-less-free-than-it.html

“Optimistically, for the rest of this decade the rapid decrease in cost per bit of storage that

has been a constant of the last three decades will be much slower; it might even stop.”

David S. H. Rosenthal, et al., “The Economics of Long-Term Digital Storage” 2012.http://www.lockss.org/locksswp/wp-content/uploads/2012/09/unesco2012.pdf

A/V Digitization at MLibrary

Digitization of Special Collections material

Audio:• Uncompressed BWF master files• 1 original audio object ≈ 5 GB storage

Moving Image:• Uncompressed preservation master• Compressed production master• 1 original video object ≈ 40.4 GB Storage

A/V Storage Estimates

A/V Storage EstimatesTB/Year

Initial Costs Estimates

Value Storage: $250/TB per year

Initial Costs Estimates

Value Storage:Tape Backups:

$250/TB per year$1,825/TB per year

Initial Costs Estimates

Value Storage:Tape Backups:

$250/TB per year$1,825/TB per year

2018: $245,3252023: $485,200

2013: $32,450

Reaction

Current Costs Estimates

Value Storage:Tape Backups

(MLib):

$250/TB per yearEquipment costsCheaper most years

Current Cost Estimates

Current Cost Estimates

Current Cost Estimates

Total 11 Year Storage Cost

$569,368

Current Cost Estimates

Current Cost Estimates

Looking Ahead

• Assuming steady pricing• Instability in per-bit storage

• Will not fall under 20% growth until 2018

• ITS storage costs may not fall at same rate

• Long-term costs of cloud are not known• Amazon costs haven't decreased at HD rate1

• Tape storage for preservation copies?

• Economies of scale• Digital Preservation Network

1Rebecca Pool, “Is cloud storage the answer to preservation?” Research Information, 2/14/2013

Ramifications

• Preservation• Some moving image material is not

retained in uncompressed formats• Still image formats are balance of

preservation and size

• Appraisal and re-appraisal • What are we keeping and for how long?

• Costs of starting up

Storage Takeaways

• HD costs are no longer predictable • Budgets often neglect storage

• Ignoring significant and ongoing costs

• Storage costs are community wide problem• Question of scale• Best practice may not be possible

• Backups can cost more than primary storage

• Archival storage ain’t cheap!

Thanks!!