4K Media Workflows on AWS

Usman Shakeel Principal Solutions Architect Lead (M&E)

Amazon Web Services

Content has gravity – and getting heavier…

4K at pro-res 4444 over a TB

HD at 422HQ ~ 80 GB (A 60 min length)

What does this mean for … Near term storage

Archive storage

Content

Content has gravity – and getting heavier…

Affiliate Delivery

Consumer Delivery 10X of normal bandwidth usage today !

Can networks handle the peak usage 4K streaming?

Media Average Bit Rate

1080P 4K

Blu-Ray 24 Mbps

iTunes 5 Mbps

YouTube 3 Mbps 6 Mbps

Netflix 6 Mbps 16 Mbps

Content

and carries weight over to processing …

Limits of physical Infrastructure

How many 4K titles per server? • Max Storage per server

• Max Bandwidth per server

HVEC, VP9 Encoding • Require more processing cores Content

Where is the problem?

More Bandwidth $$$$$

More Powerful Compute $$$$$

Way more Storage $$$$$

Some Progress (ABR, HVEC, VP10)

Does it matter? Screen Size

Limited number of 4K mobile devices

Viewing Distance

HDMI 2.0

and there is more …

Where is the sliding scale on my Infrastructure?

A Concept - The Content Lake Inspired from Data Lake (Coined by James Dixon in 2010)

A single store of all of digital content that you create and acquire in any form or factor •Don’t assume any resolutions/formats (for now or future)

•It is up to the consumer (application consuming the content) to use the appropriate infrastructure for processing

Amazon S3 : the Content Lake • Durable, cost-effective and fast

• Highly scalable front-end – Multi-part uploads (parallel writes)

– Range-gets (parallel reads)

• Use Amazon S3 with on-premises storage in a hybrid model

• No Storage capacity Upper-bounds

• Secure

Hydrating the Content Lake

Amazon S3

AWS Import/ Export Amazon S3

(multi-part Upload)

AWS Storage Gateway

Direct Connect

N x 1G | 10G

Massively Scalable Front-end

Edge Locations

Availability Zone

Region

Dallas (2)

St.Louis

Miami

Jacksonville Los Angeles (2)

Seattle

Ashburn (3)

Newark

New York (3)

Dublin

London (2)

Amsterdam (2)

Stockholm

Frankfurt (2) Paris (2)

Singapore(2)

Hong Kong (2)

Tokyo (2)

Sao Paulo

South Bend

San Jose Palo Alto Hayward

Osaka Milan

Sydney

Madrid Seoul

Mumbai

Chennai

Regional Lakes …

Content Life-Cycle

Amazon Glacier (Life Cycle Policies)

Amazon S3

Near-term Storage

Long Term Archive

Both tiers accessible through a single API

Oldest content trickles down to glacier automatically to save cost

Consuming the Content Lake

Amazon S3

Amazon S3 (range-gets)

Direct Connect

N x 1G | 10G

Massively Scalable S3 Front-end

EBS

Instance Store

c Massively Scalable Compute on AWS Cloud

On-Prem Apps

Consuming the Content Lake

Compute Intensive Intel ES-2666 v3 (Haswell) optimized specifically for EC2

Memory Intensive Lowest price point per GiB of RAM

GPUs 1,536 CUDA cores 4GB of video memory

Enhanced Networking Higher PPS, Lower network jitter, low latency

IO Intensive SSD Storage, EBS Optimized

High Storage 24 x 2000 GiB per instance

Massively Scalable Compute Layer

Amazon S3

c4 g2

m3

HVEC Encode

GPU Transcode

HD i2

Package

Queues (orchestrated by Amazon SWF)

Content Lake

Task Specific

• Spot Instances (no charge if spot is terminated)

• Reserved Instances

• Pay for only what you use

• Buy for your need – Short term/Long term

– Real-time/Batch workloads

• AWS Marketplace – Pay for what licenses you use

Cost Effective Compute

VS. $2.80/hr

Launch a CloudFormation stack

with all the infrastructure

resources for a specific project

Autoscale the stack as

appropriate

AMI

CloudFormation

Template CloudFormation

Terminate

Template

Automated Infrastructure Layers based on Project Scope

AWS Ecosystem (License included in hourly* pricing)

INGEST STORE MANAGE SECURE PROCESS

CREATE MONETIZE

INTEGRATE DELIVER

Scalable Affiliate delivery on AWS

Traditional multi-hop satellite distribution

First Hop Second Hop

Field Source / Encoder Headend / Processing Affiliate Spoke / Decoder

Multi-hop Affiliate distribution with AWS

First Hop

Field Source / Encoder Headend / Processing Affiliate Spoke

Ingest

Fan Out

Egress

Scale Out

Multi-Region, Multi-AZ

Cellular

Internet

Amazon Direct Connect

Secure VPN

Internet

Amazon S3

Amazon Glacier

Amazon Route53

c4 g2 m3

High Capacity Egress

GPU Transcode

Ingest

Multi-path distribution

Encoder

Broadcast Decode

Low Bitrate Proxy

Fan out / fan in

Size workflow to compute

Flexible multi-format

HLS w/ Cloudfront CDN

MPEG-UDP w/FEC

Amazon Direct Connect

Content Lake

AWS Import/ Export

S3 multi-part API

For Source, Renditions, Metadata Sidecar Files

Ingest

AWS Elastic Beanstalk

DAM Web Service

CloudFront Distribution

Amazon DynamoDB

Amazon CloudSearch

DAM Catalog

Rendition Workers

EC2 ASG

Amazon SNS Topic

Amazon SQS Queue Rendition Jobs

Amazon SQS Queue Metadata Processing Jobs

S3 Notification

Elastic Transcoder Proxy / Thumbnail

Generation

Processing

Metadata Workers

EC2 ASG

A Simple DAM at AWS

AWS Import/ Export

S3 multi-part API

For Source, Renditions, Metadata Sidecar Files

Ingest

AWS Elastic Beanstalk

DAM Web Service

CloudFront Distribution

Amazon DynamoDB

Amazon CloudSearch

DAM Catalog

Rendition Workers

EC2 ASG

Amazon SNS Topic

Amazon SQS Queue Rendition Jobs

Amazon SQS Queue Metadata Processing Jobs

S3 Notification

Elastic Transcoder Proxy / Thumbnail

Generation

Processing

Metadata Workers

EC2 ASG

A Simple DAM at AWS (Cost)

$

$

AWS Import/ Export

S3 multi-part API

For Source, Renditions, Metadata Sidecar Files

Ingest

AWS Elastic Beanstalk

DAM Web Service

CloudFront Distribution

Amazon DynamoDB

Amazon CloudSearch

DAM Catalog

Rendition Workers

EC2 ASG

Amazon SNS Topic

Amazon SQS Queue Rendition Jobs

Amazon SQS Queue Metadata Processing Jobs

S3 Notification

Elastic Transcoder Proxy / Thumbnail

Generation

Processing

Metadata Workers

EC2 ASG

A Simple DAM at AWS (Using Spot)

0.$

0.$

AWS Import/ Export

S3 multi-part API

For Source, Renditions, Metadata Sidecar Files

Ingest

AWS Elastic

Beanstalk

DAM

Web Service

CloudFront

Distribution

S3 Notification

Elastic Transcoder

Proxy / Thumbnail

Generation

Processing

Lambda

Metadata

Extract & Persist

Amazon

DynamoDB

Amazon

CloudSearch

DAM Catalog

Server-less Applications using AWS Lambda

0.0$

yeah more science…

Remote application visualization for production tools, VFX modeling Content stays in the Cloud

Powerful workstations in the cloud when needed

Amazon S3

GPU

Content Lake

Amazon Appstream

Remote Dumb Terminal

Amazon EC2

The Content Lake on AWS

Process

User Delivery/Consumption VFX/Production

On-Prem Apps

Archive

Content Lake Partner/Affiliate/ Service Provider

The Content Lake on AWS

EBS

Instance Store

Amazon Glacier (Life Cycle Policies)

Amazon S3

Amazon EBS/EC2 Instance Store

Process

User Delivery/Consumption VFX/Production

On-Prem Apps

Archive

Partner/Affiliate/ Service Provider

Content Access Transfer

The Content Lake on AWS

EBS

Instance Store

Amazon Glacier (Life Cycle Policies) Amazon EBS/EC2 Instance Store

Process

Partner/Affiliate/ Service Provider

User Delivery/Consumption VFX/Production

On-Prem Apps

Archive

Amazon Glacier (Life Cycle Policies) Amazon EBS/EC2 Instance Store

c

c

Direct Connect

Content Access Transfer

Disposable Infrastructure

Autoscaling Workload specific

Amazon S3

South Lower Hall: SL9016 http://aws.amazon.com/digital-media/

Come check out…

Presentation Theater, Meet the AWS experts, cool demos and Solutions Ecosystem

Thank you