aws webcast - library systems on the aws cloud

Library Workloads on Amazon

Web Services

May 27th, 2015

Sri Elaprolu – Manager, Solutions Architecture

Worldwide Public Sector

AWS Overview

Library Workloads and Use Cases

Getting Data into AWS

Storage Services

Transcoding Service

Content Delivery Service

Q & A

Agenda

What is AWS?

Application Services

Compute Storage Databases

Networking

AWS Global Infrastructure

Deployment & Administration

Amazon Web Services

11 Regions

29 Availability Zones

53 Edge locations

AWS Global Infrastructure

Customer Decides Where Applications and Data Reside

AWS Availability Zone (AZ) View

- Multiple Isolated locations within a Region

- Availability Zone = 1 or more “data center”

- Independent Failure Zone

- Physically separated

- On separate Low Risk Flood Plains

- Discrete UPS

- Onsite backup generation facilities

- Fed from different segments of utility provider

- Redundantly connected to multiple tier-1 ISP’s

- No “Disaster Recovery Datacenter”

- Built for Continuous Availability

- Customer decides Availability Zone for Compute

Availability

Zone AAvailability

Zone B

Availability

Zone C

Sample US Region

~ Data Center

Architected for Government Security Requirements

http://aws.amazon.com/security

Certifications and accreditations for workloads that matter

AWS CloudTrail and AWS Config - Call logging and configuration management for governance & compliance

• Log, review, alarm on all user actions

• Browse and query database of current and previous state of cloud resources

Experience

Since 2006 supporting

large numbers of

customers across 190

countries

Innovation

Rapid delivery of new

services and features based

on customer feedback

Robust Platform

Number of services and

features, virtually to

support every use case

imaginable

Simple Pricing

Philosophy

48 Price reductions

Expect more reductions

in the future

Global Footprint

11 Regions

29 Availability Zones

53 Edge Locations

Eco system

Thousands of partners

(ISV, SI, consulting)

23 categories; 2100 apps

in Marketplace

AWS Differentiators

Library Workloads

Library Use Cases

Online Public

Access Catalogs

Library Catalogs

Online databases

Institutional

Repositories

Online Archive

Intellectual Output

Digital Asset

Storage

Protect from Loss and

Degradation

Offsite Storage

Redundancy and

Durability

Backups

Offsite

Redundant

Development Space

Disposable Environments

Start and Stop Frequently

Dspace

Open Journal Systems

Open Conference Systems

Thesis and Dissertation Systems

Web Properties – WordPress

DuraCloud Preservation System

• Consortium of higher education institutions in

Texas that has provided shared digital library

services since 2005

• The mission of the Texas Digital Library (TDL) is

to enable each of its member libraries to advance

a program of digital initiatives in support of

research, scholarship, and learning.

Getting Data into AWS

Data Ingestion Options

AWS Direct ConnectDedicated bandwidth between

your site and AWS

InternetTransfer data in a secure SSL tunnel over the

public Internet

AWS Import/ExportPhysical transfer of media into and

out of AWS

AWS Ingestion Options - Internet

1. Multipart upload

2. Request rate optimization

3. TCP window scaling

4. TCP selective

acknowledgement

AWS has customers that ingest roughly 1 PB per day

AWS Ingestion Options - AWS Direct Connect

• Private connectivity to AWS– Physical connection – 1 Gbps or 10 Gbps port

• Consistent network performance

• Consider burst models on ingest

• Reduces costs for bandwidth-heavy outbound workloads

• US Locations

• CoreSite 32 Avenue of the Americas, NY

• CoreSite One Wilshire & 900 North Alameda, LA

• Equinix DC1 – DC6 & DC10 - DC11, Ashburn, VA

• Equinix SV1 & SV5, San Jose, CA

• Equinix SE2 & SE3, Seattle, WA

AWS Ingestion Options - AWS Import/Export

• Rapidly move data into

and out of AWS

• Portable storage device

shipment to AWS

• Supports– Amazon EBS

– Amazon S3

– Amazon Glacier

• Use cases– Initial data migration

– Content distribution via portable

devices

– Disaster recovery

Amazon Storage Services

Amazon Simple Storage Service (S3)Highly scalable object storage

1 byte to 5 TB in size

99.999999999% durability

Amazon Elastic Block Store (EBS)High-performance block storage device

1 GB to 16 TB in size

Mount as drives to instances with

snapshot/cloning functionalities

Magnetic and General Purpose SSD

Amazon GlacierLong-term object archive

Extremely low cost per gigabyte

99.999999999% durability

AWS Storage and Archive Options

Amazon Elastic Block Store (EBS)

• High I/O block storage for Amazon

EC2

• Point-in-time snapshots to Amazon S3• 99.999999999% Durability

• Snapshot software is FREE

• Point-in-time snapshots across

regions

Amazon Simple Storage Service (S3)

• Durable and low cost

• Unlimited number of objects and volume

• Back up to Amazon S3 buckets via

HTTP/HTTPS

– Create scripts using PowerShell,

Perl, Python…

– Numerous solutions for data backup

• Authentication mechanisms ensure data

is kept secure

• Reduced redundancy storage (RRS)

option

Amazon Glacier

• $0.01 per GB/mo, $120 per TB/yr

• 3-5 hour data retrieval latency

• Archives: single file or zipped files

• Vaults: collection of archives

• Infinite archival storage

• 99.999999999% durability

• Immutable, encrypted by default

Object Life Cycle Management Amazon S3 → Amazon Glacier

• Seamlessly move data from Amazon S3 → Amazon Glacier

• 3-5 hour asynchronous retrieval

• Data lifecycle policies

• $0.01 per GB for Amazon Glacier costs

Why AWS for Storage and Archiving?

• Protect digital content from fragility

• Protect digital assets from loss and degradation

• Promote learning

• Share research

TCO: On-Premise Cost Considerations

1. Primary storage hardware (primary / remote site)

2. Storage growth (cost of upgrades)

3. Storage management software and 3rd party tools

4. Professional services

5. Hardware maintenance

6. Software maintenance

7. Backup software

8. Backup hardware (primary / remote site)

9. Offsite tape storage / vault

10. Archive software

11. Archive hardware

12. Power

13. Cooling

14. Space

15. Labor

16. Cost of capital

17. Training

18. Asset depreciation

19. Migration

20. Decommission / remove

21. Recycle

22. …

Storage on AWS

10 TB S3 = $ 3,631.20 per YEAR

5 TB S3 | 5 TB Glacier = $ 2,433.12 per YEAR

10 TB Glacier = $ 1,228.80 per YEAR

Price based on US-EAST-1 region; correct as of May 22nd, 2015

Amazon Elastic

Transcoder

• Managed Transcoding Service built on EC2

– No software to buy or manage

– No need to manage capacity

– Seamless integration with AWS S3 and Amazon CloudFront CDN

• Transcode Content for Any Device

– Select from over 30 transcode presets

– Define up to 50 presets per AWS Account!

• Process jobs in parallel and on demand

Amazon Elastic Transcoder: Overview

• Self-Service Control

• Un-Matched On Demand Capacity

• Industry Leading Reliability

• Lowest Cost Transcoding Service

• Highly Secure

• Global Availability

• Rapidly Releasing New Features

Amazon Elastic Transcoder: Why Customers prefer the service

Pipeline 1 Pipeline 2

Input Bucket 1 Input Bucket 2

Output BucketPipelines, jobs, & outputs

ALL run in parallel

:

:

:

:

:

:

:

Job N (Progressing)

Job N+1 (Complete)

Job N+2 (Progressing)

:

:

:

:

:

:

:

:

:

:

Job M (Progressing)

Job M+1 (Progressing)

Job N+3 (Progressing)

Job N+4 (Submitted)

SNS Topic

Amazon Elastic Transcoder: How it works?

Amazon Content Delivery

Service

• Full Feature Caching Network

• Global Infrastructure

• Tuned for Optimal Performance

• Massively Scalable

• Highly Secure

• Robust Analytics

• Self Service

• Priced to Minimize Cost

Amazon CloudFront: Content Delivery Network

• Media and Entertainment

• Gaming

• Digital Catalogs

• Digital Advertising

• Software Downloads

• Dynamic Websites and Applications

Amazon CloudFront: For Any Market Segment

Video StreamingOn-demand & Live Streaming

RTMP (Flash) and HTTP(S)

Adaptive Bitrate Live Streaming

Microsoft Smooth Streaming

Whole Site DeliveryStatic & Dynamic Content

Mobile Detect, CORS Support

Multiple Cache Behaviors

Multiple Origin Servers

SecurityPrivate Content (Signed URLs)

Custom SSL (Dedicated IP & SNI)

Geo Restriction

HTTP to HTTPS Redirect

High Availability99.9% SLA

Automatic Origin Failover

Custom Error Pages

Serve Stale Content when Origin unavailable

High PerformanceLatency Based Routing

TCP Optimization

Persistent Connections

EDNS Client Subnet

Low TCOPay for use

Commit-Based lower pricing

Price Classes

Preferential Pricing for AWS origins

Amazon CloudFront: Popular Features

Dynamic

StaticVideo

User

Input

SSL

Amazon CloudFront: Deliver all your site

Thank You!