spectra logic blackpearl developer summit 2015
TRANSCRIPT
BlackPearl Developer Summit
October 20, 2015
Into Tomorrow with Disk & Tape Technology: Investing in the Future
Nathan ThompsonCEO & FounderSpectra Logic Corporation
World Data Storage and Population in 2020• IDC projects the “Digital Universe” to reach 40 Zettabytes by 2020
• Spectra has been studying this and are writing a whitepaper based on storage demand & production capacity, and we believes the actual “storage” will be 15ZB-25ZB by 2020• By 2030 Spectra projects that “stored portion” of Digital Universe to grow to between 150ZB and
250ZB
• World population is expected to reach 7.7B by 2020 (UN 2012)
• All these predictions are tightly connected as we move into a full digital era.
Society is Dependent on the “Digital Universe”• Healthcare• Drug development• Property definitions and rights• Agriculture genetics• Food processing and
distribution• Energy development,
production & distribution• Communications• Transportation
• Semiconductors design & production• Software design• All levels of government• Education• Social safety nets• Law enforcement• Security• Agriculture production
Society’s Genome?• Digital Universe should be considered to be society’s DNA—it carries us forward• Organizations are like organisms—they must preserve their DNA—their most
important asset. How much DNA is there? And just with our DNA we never know what is needed, so we have to keep it all
• Maintaining and moving this information forward, “Digital Preservation” may be our most important mission• Geographic Dispersal • Genetic Diversity • Accuracy of Replication—we have this one in T10, S3, Hash Codes (SHA256, MD5)
• At this time there is no high-capacity permanent storage medium. All data must be made “moveable” to another medium. Same with organisms. Same with Ptolemy
• Again, society may only make multiple copies if those copies are cost effective
Wikipedia Article On Digital Preservation
The Digital Universe: Risks and Threats
• Internal Data Loss• Cyber Attack• Scarcity of Media
- Hardware failure or system malfunction- Human error- Accidental deletion- Software corruption error- Simple computer viruses- Environmental lost (fire, flood, earthquake)
CyberCaliphate targets U.S. military spouses BREAKING NEWS
The New Hacktivist…The New Criminal Approach: RansomwareThe New Nation-State Attack… Sony- Large quantities of its data were destroyedMalware infecting hard disk firmware remained hidden for 15 years
The New Nation-State Attack… Saudi Aramco- 30,000 hard drives were scrapped and replaced- Shift from cyber spying to destruction of operations and dataIf not 40 ZB, 15-25 Zettabytes
by 2020?
How Does 15-25ZB Digital Universe Survive? • 15ZB is 15,000,000,000,000,000,000,000 bytes• Geographic Disbursal: Preferably removable
media, not susceptible to common mode failures• Genetic Diversity: Two or more copies of
information on genetically different media to reject common failure modes. • Accuracy Of Replication: Additional copies must
be identical• Additional copies must be cost effective
Spectra’s area of focus & expertise
Cost
/TB
$$$
$
Very high performance media (flash)
Online medium performance (disk)
Nearline w/ latency performance (tape, optical MAID?)
Perf
orm
ance
Low
High
Spectra’s Market is the Bottom 80% Of Storage
Most of the storage world is interested in the high performance, segments of storage. At Spectra we are interested in the “bottom 80%”—whether is it delivered via partner apps, NAS or Private Cloud. You will hear us refer to this as “deep storage.”
Let’s Look At Archive Disk Optical Media
Derivative of BluRay & DVDDeveloped by Sony & PanasonicWrite once, 50 year media lifeRoadmap: 300GB (2016), 500GB (2019), 1TB (unknown)
UnknownsMedia format interchange between drivesOptical drive longevity (may be a consumer grade design)Media costs (needs to be $2.50 per disk to compete)Market acceptance
Currently on hold at Spectra due technical and market uncertainty
Spectra 15,000 Disk Robot DesignStrengths / Weaknesses
Let’s Look At Shingled Magnetic Recording (SMR) Disk• Spectra has launched SMR disk product called Spectra Verde DPE (Digital
Protection / Enterprise)• As low at 9c/Gigabyte at 2PB. Built in data compression• NAS (CIFS, NFS, ftp) Bulk Storage for Archive, Backup, Unstructured Data, Large
Files. Not for databases
• Spectra has invented “Very Wide RAID” of 20 + 3 for efficiency of use• Massive error detection/correction using Galois Field mathematics & continuous
checksums• Tape-like, but allows typically 1 GB/s random read/write in bulk storage
applications
• Good growth potential.
7.4 PB Verde DPE NAS in Rack
Spectra Continues to Develop the World’s Largest Capacity Tape Libraries
• Tape Is A Growth Market For Spectra – Continued library innovation and
commitment to R&D– Increasing reliability and performance– Significant new product in 2016
2017 Robotic Tape/Optical & Disk At List Price
$10
$60
$110
$160
$210
$260
12 TB Shingled Magnetic Disk8TB Shingled Magnetic Disk300GB Archive Disk OpticalLTO-7 Tape
Capacity (PBytes)
Price
Per
Tby
te
Investing to Make These Technologies Useful…• Allow customers to seamlessly use Disk, Flash, Tape and possibly in the
future, Archive Disk (optical)• Enable Digital Preservation including Geographic Dispersion, Genetic
Diversity, Accuracy Of Replication• Lower the cost of storage, such that multiple copies can be created on
different media or cloud storage• Dramatically reduce the cost of developing tape storage applications.
Ease application development by use of cloud and web methods• Support our partner infrastructure across multiple industry segments
Data Flow - Formats - Media Migration
• Data Flow – Single Copy to Tape
• Data Flow – One Copy Disk and One Copy
Tape
• Data Flow – Replication with two copies
• All media is written in LTFS Open Format
• Tape media migration is seamless†
• Future data migration to new
technologies is equally seamless †
Spectra S3 EnabledApplications
What we are excited about
• IBM and Fujifilm 220TB tape demonstration
What we are excited aboutArcticBlue – A Breakthrough in
nearline disk storage• Leverages BlackPearl’s hybrid storage
architecture to maximize longevity, efficiency, and cost effectiveness with online disk performance
• Delivering everything you expect from disk performance at the cost and longevity of tape
• Instant access to all your data, across all bands, across all nodes.
• As low as 10₵ per Gigabyte
Spectra Storage Forecast White Paper will be published in October 2015
Spectra Created Book “Society’s Genome” will be published around the start of 2016
Developing for BlackPearls future
Steven EasterProduct ManagerSpectra Logic Corporation
Exciting Changes Coming to BlackPearl• BlackPearl code 1.1 and 1.2 supports Tape Eject• Can through UI can eject with API, or end user can eject through BlackPearl GUI• Can display location (in library or offline/ejected) to UI with Get Physical Placement API
• BlackPearl - next code is 2.4 (GA Dec 2015) • ACLs – Flexibility of permissions, resource-based policies and user based policies• More API calls supported in SDK• Advanced Bucket Management (ABM)• One Policy per Bucket – • e.g. two copies on tape, • or one copy tape + once copy on disk minimum of 60 days
• Can support ABM by providing multiple ‘targets’ with a 1:1 mapping to a bucket
Verde and Verde DPE: NAS Disk
Online Disk (SAS)
What if… Deep Storage were Simple?
Nearline Disk
Deep Storage Tape
Introducing – Advanced Bucket Management
Storage Domains
Data Policy
User BucketsBucket 3
Temporary Persistence
Rule
Online Disk100TB usable
Permanent Persistence
Rule
Nearline Disk
2PB usable
Permanent Persistence
Rule
T950 TS1150
1500 Slots
Permanent Persistence
Rule
T200 LTO7
50 Slots
One copy is ejected - vaulted
Data
DataData
…
4 copy policy User or Application: PUT
Data Data
User or Application: GETData
ABM - design• A Bucket can have multiple storage types – multiple copies• Using Get Physical Placement (or similar command TBD) to tell objects location(s)
• Provide information about number of copies and type of storage to end user
• Allows for a Disk only bucket for highly active content, or small files
• And a second bucket for colder data, large files, - copy on tape + redundant copy
S3 or Spectra S3• Spectra S3 (formally DS3)• Uses the BULK commands – higher performance• Supports the latency of getting data from tape
• S3 (Pure S3)• Single file PUT, single file GET, • Using other (non-spectra) SDKs or API• Works with disk only BlackPearl – with one Permanent copy is on Disk • Used as quick deployment while Spectra S3 integration is in development • As a limited solution, until glacier model command in 2016 (POST Object Restore)
Client Development Tools and Client Building Example
Jeff BraunsteinDeveloper Evangelist
Tools and Client Example
• View at:http
://www.youtube.com/watch?v=GYoSwrvhVM0&t=52m38s
BlackPearl Integration and Client Development
• Empress Media, Inc – Started in 1998
• eMAM- Media Asset Management System• Storage and Archive Management• Integrated Workflow management• Distributed Hybrid Architecture• Scalable
Partnering with Spectra• BlackPearl Provides:• Plug and Play Model• Scales at a flat to
decreasing cost• No application level
dependencies• On premise or
geographically dispersed archives
• Streamlined Workflow• Empress provides
• eMAM Software
• Spectra provides• Storage (Verde)• Archive middleware (BlackPearl)• Archive (LTO TSeries Tape Libraries)
• Simplified Business• Support and integration services
from two vendors.
eMAM and BlackPearl Together
• Workflow
• Private Cloud Infrastructure
• Easy Implementation with Fully Developed Client
• Hybrid Cloud Distribution
eMAM and BlackPearl Together
• Object Storage – Files are stored as object with metadata• Archive – PUT operation to archive the assets• Restore – GET operation to restore the assets• BulkPUT – for batch Archive operation• BulkGET – for batch Restore operation
Features
Black Pearl configurationArchive Profile configuration
UI - Archive options
Screenshots
Developer view
Live Demo
Interface
Asset Archive/Restore Project Archive/Restore Category Archive/Restore
Asset profile selection
Storage locations
Browse options
Category Archive/Restore
Social Media publishing Asset delivery with auto restore option
Anywhere Production Management (Ingest/Store/Archive/Restore)
Q&A Panelist
Steven EasterProduct ManagerSpectra Logic Corporation
Mark PaulsonEngineering Project ManagerBlackPearl and Verde ProductsSpectra Logic Corporation
Ryan MooreEngineering Project ManagerSDK and Client teamSpectra Logic Corporation
Jeff BraunsteinDeveloper EvangelistSpectra Logic Corporation
Anoop ThomasDirector of TechnologyEmpress Media, Inc.
developer.spectralogic.com