storage archive manager what’s new - stanford...

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Storage Archive Manager – What’s New

Dan Deppen Senior Product Manager StorageTek Tape September 16, 2014


Storage Abstraction Layer

Tape Archive

Primary Disk

Capacity Disk

Sto

rage

Ab

stra

ctio

n L

ayer

(S

AM

-QFS

)

Ap

plic

atio

ns

Stan

dar

d P

OSI

X F

ile S

yste

m A

PI


New Capabilities for SAM-QFS 5.4

•Horizontal scalability

•Automated data/media integrity validation

•LTFS

•Cloud

• Improvements for core SAM-QFS functionality

Archive Virtualization /

Automation

DIV

Horizontal Scalability

Automated DIV

LTFS

Up to

5.2

5.3

5.4

Cloud


New Challenges

How do you move data fast enough?


Tape I/O Horizontal Scalability

QFS Clients Metadata

Server (MDS)

Primary

Disk Tape Storage

Tier

Up to SAM-QFS 5.3 - Without Tape I/O Horizontal Scalability

MDS and QFS Clients

can write to disk

Only MDS can write to

tape

Disk IO

Tape IO



QFS Clients

Metadata

Server

(MDS)

Primary

Disk

Tape Storage

Tier

SAM-QFS 5.4 - With Tape I/O Horizontal Scalability

Now IO from MDS

and QFS Clients

can push data to

tape

No MDS

bottleneck



QFS Clients

Metadata

Server

(MDS)

Primary

Disk

Tape Storage

Tier

Round Robin I/O Allocation

•IO assigned by round

robin process

•MDS updates metadata



• Ingest >1 PB of data per day into the archive

•Use 10 servers to archive data instead of 1

•Faster migration of data to new tape drive technology

•Enable use of commodity servers

Increase Data Ingest Rates By 10X


New Challenges

How do you verify archive data?


Automated Data Integrity Validation

Verify data is still accessible without sending it back to the host

Audits are automated Ex: check files every 6 months

Self-healing Can be configured to automatically re-archive damaged files

Utilize DIV in entire environment

Support for LTO5 and LTO6

Benefits


Automated Data Integrity Validation 5 Types of Media Validation

– Basic verify (~1 minute)

Validate MIR region

– Standard verify (~10-20 minutes)

Validate 2 upper and 2 lower data tracks on tape

– Complete verify (~6/10 hours)

Validate entire contents of tape cartridge

– Complete verify plus (~6/10 hours)

Validate entire contents of tape cartridge including

any DIV CRC’s

– Legacy verify (4/6 hours)

Validate entire contents of tape

T10000 C/D LTO 5/6


New Challenges

How do you centralize data?


• Generate video files in LTFS format at remote locations, then store in a central archive – Video surveillance

– Media and entertainment

– Oil & gas

• Export files in open format

LTFS Import / Export


LTFS Import / Export

1. Import LTFS tapes into the archive

2. Export archive data onto LTFS tapes

3. Read / write LTFS tapes which are not part of SAM archive

– Similar functionality to LTFS-LE

New Functionality

FILE

FILE INDEX

LTFS Format

SAM-QFS Archive

FILE

FILE INDEX

LTFS Format

Export Import

LTFS

Access

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 16

Cloud Archiving


Cloud

•2 cases: –Building public or private cloud infrastructures

–Archiving to a public or private cloud

17


Building Cloud Infrastructures Support for Cloud API’s

• Use SAM as a OpenStack Swift cloud server

• Store and retrieve data using OpenStack Swift


Swift Integration With SAM-QFS Automation

Application Archive Manager QFS File System

Disk Cache OpenStack Swift Interface

GET / PUT / DELETE


Storage Network

Fire

wall

SAM-QFS – SWIFT Deployment Diagram (Logical)

REST

REST Swift Proxy External

SWIFT Client

Storage Node QFS Client



SAM/QFS 5.4

REST

REST

REST Storage Node QFS Client


REST

Up to 512 Nodes


Where to Store Data? It Depends

On Premise

• Disk and/or tape

• High performance

• Secure

Remote Site / Private Cloud

• Disk and/or tape

• Network bottleneck

• Secure

• Higher initial cost than cloud / lower total cost

• Data is offline

Public Cloud

• Lower initial cost, no staff / infrastructure

• Less control over data

• Network bottleneck

• Data is offline


Archive to Any Tier

Cloud Tier

QFS Clients

Metadata Server

Primary Disk Tape Storage Tier

(control path)

Data path

• Can archive to any public cloud with an NFS gateway

• Manage on / off premise data from a single interface

Remote Site /

Private Cloud


Other SAM-QFS 5.4 Updates

• Easier initial setup and installation

• Wizard will provide functionality for: – Disk device selection

– Modify existing filesystems

– New filesystem creation

– Archive device detection (libraries / drives)

– Simple archive configuration

– Disk formatting

– Filesystem mkfs and mount

Configuration Wizard


Other SAM-QFS 5.4 Updates

• Oracle Enterprise Linux / RedHat Enterprise Linux –5.x (tested 5.9, 5.10)

–6.x (tested 6.3, 6.4, 6.5)

• SuSE Linux 11 Enterprise Server 11 –Server pack 1

Linux Client Support


Stage Queue Prioritization

– Enables admin to take better control of the stage queue

– Option to stage file(s) immediately • Can be new request or a file already in the

stage queue

Stage

Queue

Files

High Priority File

Application


SAM-QFS 5.4 additional features

– Image Package System (IPS) installation

– Support for CIFS ephemeral user ID’s • Maps Windows identities to QFS

• Any authenticated Active Directory user can use QFS share

– Extended file system attributes • Support for Solaris server and Solaris / Linux clients

• Add custom metadata – Checksums / hashes

– Tag the priority of data

storage archive manager what’s new - stanford...

Documents