la m and e symposium nov 2017 all decks.pdf

© 2017, Amazon Web Services, Inc. or its Affiliates.

November 7th 2017

AWS Media & EntertainmentLA Media Symposium - Loews Hollywood


Today’s Agenda: AM

Time General Track

10:00 AM Registration

10:30 AM Welcome - AWS Update

Bhavik Vyas, AWS Global Partner Segment Lead for M&E

10:45 AM Hulu Live TV on AWS

Rafael Soltanovich, VP of Software Development for Hulu

11:00 AM Broadcast Media Management in the Cloud

Chris Blandy, EVP Technology Solutions for Fox

11:30 AM Securing Content torage and Workflows to Meet Studio Requirements

Eli Mezie, Lead Architect Executive at ISE

Dan Meacham, CISO for Legendary Pictures

12:00 PM Networking Lunch


Today’s Agenda: PMTime Track #1: Production & Content Creation Track #2: Broadcast & OTT Track

1:00 PM

Track Session #1

Rendering in Cloud with Deadline 10

Chris Bond, AWS Thinkbox CEO

Jason Fotter, Co-Founder and CTO, FuseFX

Track Session #1

AI Distribution - C-SPAN

Dan Mbanga, AWS BD Artificial Intelligence

1:45 PM

Track Session #2

Panel Discussion: Remote Creatives - Storytelling In a Distributed Age

Igor Boshoer, Founder/CEO for Linumio;

David Benson, CTO for Bebop

David Andrade, Cofounder for Theory Studios

Andy Stack, COO for Arcturus Studio

Track Session #2

Real-time Media Analysis with AI in Media

Kon Wilms, AWS Specialist Media Solutions Architect

2:30 PM Break (15mins)

2:45 PM

Track Session #3

Enhanced Content Workflows using Amazon AI


Track Session #3

Delivery Side Supply Chain and OTT

Ken Kanagaraj, Principal Solutions Architect,

AWS Elemental

3:15 PM

Track Session #4

attn: Scaling to 500 million monthly views

Jake McGraw CTO Attn

Peter Agelasto, Founder Digital ReLab;

Luke Adams, CTO Digital ReLab;

Track Session #4

Low Latency Streaming via AWS and Wowza

Mike Talvensaari, Wowza VP of Product;

Chris Gruiz, WW Business Dev & Ops, AWS Marketplace

4:00 PM Closing Comments

4:15 PM Networking Reception


L e a r n m o r e a b o u t t h e i n n o v a t i v e M & E s o l u t i o n s b u i l t o n A W S b y v i s i t i n g

o u r p a r t n e r s p o n s o r s i n t h e r e c e p t i o n a r e a d u r i n g t h e i n t e r m i s s i o n .

T H A N K Y O U


Content Acquisition

Production/Post

Playout & Distribution

DAM/MAM/ Archive

OTT

Digital Supply Chain

Publishing

Analytics

AWS Enabled Media Workloads


Ingest/Move Store Transform Process

LIVE OR FILE

WAN OR LAN

BLOCK OR OBJECT POST PRODUCTION

CONTENT FINISHING

LIVE OF FILE

PACKAGE & DELIVER

Basic Media Pipeline

© 2017, Amazon Web Services, Inc. or its Affiliates. © 2017, Amazon Web Services, Inc. or its Affiliates.

Supply Chain Is the Media Dynamo

Playout &

DistributionLive TV

B2B/Cinematic

DeliveryDPP, IMF, NABA

Publishing

OTTLive-To-VOD

AVOD, SVOD, TVOD

Catchup TV

Cloud PVR

Normalization

Digital Supply Chain

Ingestion

Manual QC

Auto-QC

Metadata Extraction

Transcoding

Delivery

Packaging

Clips/Edits

CMS/DAM

Production/Post

ProductionEditorial

Dailies

Post

Social Media

Marketing/Promo

DAM & ArchiveActive Archive

Golden Copy

Preservation

Content AcquisitionLive

File


Media Challenges and Cloud Benefits

Content Growth

More content, higher

quality (4K, HDR…).Scalable Storage

Peak DemandUnpredictable,

bursty workloadsElastic Compute, Pay

as you go

Competitive PressuresChanging consumption

patternsAgility and Innovation

Core CompetencyDatacenters vs.

core bizOutsource


Approaches to Running Media Workloads on AWS

Custom APIs & code base

3rd party/ Open Source

BYO Interface(s)

CUSTOM:

BUILD YOUR OWN

Multiple

Inputs

AWS Elemental APIs

For Live, File, SSAI, OTT

3rd party

Svcs

Appliance VM

On-Premises

AWS ELEMENTAL

MEDIA SOLUTIONS

KEY AWS

PLATFORM

SERVICES

S3/

Glacier/

EFS

EC2Cloud

Front

RDS,

Aurora, Redshift,

DynamoDB

Lambda

SWF

AML/

KinesisRekognition

Polly/ Lex

Deadline

Pipeline Scheduler

Render/VFX Workloads

(Onprem, Hybrid, Cloud)

THINKBOX

Thinkbox MarketplacePAYG 3rd Party Software

BYOL, S/PaaS, MSP,

Media, Security,

Devops, Monitoring,

Data Analysis,

Automation

PARTNER-BASED

AWS MarketPlace


Common Digital Media Workflows on AWS

File transfer (DX, S3

transfer

accelerator,

Aspera,

Signiant…)

Ingest

Snowball,

Snowmobile

Live stream (e.g. AWS

Elemental)

Manage

Rights, Tag,

Admin

Analyze/ AI/ Monetize

Collect, Process, Analyze (Kinesis,

Redshift, AML, Rekognition…)

Amazon S3, IA, Glacier, EBS, EFS

Store

Process

AWS Elemental

Transform, QC,

Protect, Package

Stream (OTT),

Export, Playout,

Publish

Deliver

Edit,

Review,

Model…

Visualize


Performance Efficiency: Go Serverless

AWS Lambda Amazon DynamoDBAmazon S3

Compute Storage Database

Amazon Kinesis

Orchestration & State Management Monitoring & Debugging

AWS X-RayAWS Step Functions

Analytics

Amazon SNS

Amazon API GatewayAmazon SQS

API Proxy Messaging & Queues AI/ML

Amazon Rekogniton

Amazon Polly


AWS Well-Architected Best Practices

Reliability

Build for

failure

Performance

Efficiency

More with

less

Cost

Optimization

Margin &

Bottom-line

Operational

Excellence

Build, Run,

Monitor,

Scale

Security

Job #1.


AWS Migration and Transition Methodology

HOW?

STAGES OF ADOPTION

What and Why?

AWS Cloud Adoption Framework (CAF) v2

Bu

sin

ess

Ca

pa

bil

ity

Fo

cuse

d

BusinessValue Realization

PeopleRoles and Readiness

GovernancePrioritization and Control

Te

chn

ica

l C

ap

ab

ilit

y

Fo

cuse

d

PlatformApplications and

Infrastructure

SecurityRisk and Compliance

OperationsHybrid and Dynamic

How and Who?


NASA/JPL

AWS Media Customers



Discovery

Communications

Post Production

Scripps, Sony MCS,

FuzeFX, Bebop

Analytics/ AI

C-Span, MLB,

Channel 4

DAM & Archive

PBS, Charlie Rose

(Wazee), Discovery,

Sony DADC, Turner

Media Supply Chain

Turner (CNN), Sony DADC,

Discovery, Fox

Publishing

The Guardian,

Financial Times,

Gannett, Time Inc.,

Newscorp

OTT

Hulu,

Netflix,

MLB, BBC,

PBS

Acquisition

Scripps

AWS Customer Workload Adoption


Hulu Live TV on AWS

Rafael Soltanovich, VP of Software Development

S o W h a t I s H u l u ?

A N e w H u l u E x p e r i e n c e W i t h

L i v e T V

We Solved A Ton Of Challenges In Less Than A Year

Greater than

1,000 streamsPersonalization and

recommendation

MPEG - DASH

for live

Commercial DRM

encryption

Metadata

reconciliation

H u l u ’ s F i r s t L a r g e -S c a l e D e p l o y m e n t

I n t o T h e C l o u d

H u l u L i v e O n A W S

Internet gateway

Multi-AZ

ALB

Amazon CloudFront(and others)

Manifest

RDS Aurora EC2

AWS

VPC

Amazon S3

Packaging

SegmentData

EC2

Live DB

ElastiCache

Users

Internet Gateway

Encoding

Vendors

ManifestPackaging

DirectConnect

RDS Aurora EC2

SegmentData

EC2

Live DB

ElastiCache

A W S P a r t n e r s h i p B e y o n d T h e T o o l s

Operational readiness

Performance tuning and optimization

A W S P a r t n e r s h i p B e y o n d T h e T o o l s

Launch support

TCO optimization

We’re Excited To Continue

Hulu Innovations In The Cloud!


Broadcast Media Management in the Cloud

Chris Blandy, EVP Technology Solutions, Fox

Broadcast Media Management in the Cloud

Chris BlandyEVP, Technology Solutions

Fox Networks Engineering & Operations

Background on Fox NE&O

Our Mission: Technical operations platforms for Fox Networks Group

- Studio Production Facilities- Editing and Graphics Creation- Master Control / Playout- Linear Channel Distribution- Media Services- VOD

Our Technology Strategy

‘2020 Vision’

• Self-service tools upstream

• Assembly downstream

• Automation

• Cloud First

Why Cloud?

• Lower overhead

• Faster deployment

• Scale

• Security

Current Cloud Applications

Ingest and QC Automation

• By EOY 2017 ALL program and promo deliveries will flow through

AWS

• Delivery to our 2 primary playout centers in the formats we need

• Most assets pass QC checks without human intervention

• Instant replication = operational resiliency

Cloud Rendering

• Deadline Render Manager

• Enables VFX teams to use virtually unlimited render capacity

• Supports on-prem and cloud render farms

• True ‘burst’ capacity management

• Operations teams are in full control of where jobs run, how many instances, etc.

LTO Replacement

-

0.500

1.000

1.500

2.000

6/1/17 7/1/17 7/31/17 8/30/17 9/29/17 10/29/17

Storage Used (Petabytes)• We’ve achieved a truly ’tapeless’ environment

except for LTO tapes!• LTO very cost-effective for archive,

but over time we started using as a Tier 2 storage pool

• Instead of dozens of daily operations, we were seeing hundreds!

• Inevitable failures resulted• Solution: migrate working library to

S3/Glacier

OTT Distribution

• Dramatic change to Pay TV ecosystem• Initial approach included ‘TV Everywhere’

authentication• Digital MVPD launches:

• 2 in 2015• 1 in 2016• 3 in 2017

• Problem: most don’t have an easy way to acquire our network affiliate signals

OTT Distribution

Future Cloud Applications

Automated Versioning & Packaging

• Our assets are increasingly already in the cloud

• Vendor systems for media processing are migrating

• Transcoding options are mature

• Next steps:• Work order management

• Metadata drives versioning

• Object-based container format like IMF

Non-linear Editing

• Why not enable editing anywhere

• Current workflows require pushing media back and forth between edit workstations and production or playout

• Some ‘cloud’ options – but often proprietary

• Testing zeroclient with AWS WorkSpaces for NLE workflows

• Assets and metadata live in AWS, edit anywhere, full control

Machine Learning / AI

• Limited metadata exists if we have it at all!

• Can we apply ML/AI to generate richer metadata?

• AWS Rekognition• Frame-level image analysis

• Auto-tag scenes with talent metadata (RecognizeCelebrities API)

• Other applications:• Find efficiencies by mining workflow analytics

• Automate logging

• Predictive archive retrieval based on news headlines

• Social media sentiment analysis

Cloud Playout

• Final step to full cloud migration

• Recipe: Program assets in S3

Promos in S3

Ad creatives in S3

Log delivery to cloud

Playout automation

Encoding

Secure, error-free delivery to partners

• Gap remains – Live feed integration

Thank you!

[email protected]


Securing Content Storage and Workflows to

Meet Studio Requirements

Eli Mezie, Lead Architect Executive at ISE

Dan Meacham, CISO for Legendary Pictures

S e c u r i n g c o n t e n t s t o r a g e a n d w o r k f l o w s t o m e e t s t u d i o r e q u i r e m e n t s

Dan Meachem

@DanTechieTweets

[email protected]

Eli Mezei

@ISESecurity

[email protected]

Introductions

46

Dan Meacham

CISO– Legendary Pictures

Dan oversees cyber security and content protection for Legendary, a

nearly 100% cloud-based organization.

Eli Mezei

Executive Partner – Independent Security Evaluators

Eli Mezei develops and executes advanced security programs

for clients across several industries, including especially media &

entertainment.

Workflows Dr ive Secur ity

• Security must support the

workflow, not the other way

around

• The workflow must be understood

in depth before security controls

can be defined

• The simplest solution is generally

the most secure

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Trust Model Vs. Threat Model

Know Your Adversary

On-Location Production Facility

On site Networked Digital Storage and Collaborative Processing Environments

Off-Site

On-Location Film Set

Audio Storage

Compact FlashSound

Film Set

Actor

AVID/ISIS

Vendors

VFX, Graphics,

Arts, Set Design,

etc.

Long Term

Offsite Storage

Second Unit

Production

HQ Production

and Reviews

Actor

On Set Review

Video Assist er

Qtake Client

Director

Qtake Client

Asst. Directors

Qtake Client

On-Set Crew On-Set Crew

Color Corrected

Live feed - SDICameras

Raw video via SDI

Raw Camera Capture

SSD Mags to

Production Facility

Network Appliance -

On-Set Disaster Recovery

DIT

Color Correction

on Set

Camera SSD

Misc. On-Set Digital Content

Creation and Reviews

Makeup Set Design Sound

Camera SSD

On Set VFX-Editorial Review

Director

VFX-Editorial Dailies

Asst. Directors Asst. Directors On-Set Crew On-Set CrewDOP

Camera SSD Mags

Editorial

Asst. Directors

Content Review

Content Editing

Editor

Dailies On-Set Review

VFX Editorial

VFX Editing VFX Review

VFX Editors

VFX and Art

Film VFX and Art

Set Design

VFX Art Set Designer

Production Operations

Script Distribution

Production Office Management

Contracts

On site Networked Digital Storage

File Server

Video Lab

Color Correction Verification

Video Lab Engineer

Archival (LTO) Database

In-house Film Lab

In house Post Production - Editorial

Ingest of Data to ISIS bucket from Camera storage

Transcode the Processed Film to formats and upload to ISIS

Post Production

Air-gapped Network

ISIS ISIS

Edit Scenes

Cut in VFX, Art, and Second Unit Footage

Prepare Review Cut

Post Production

Transcription

Sound Mixing

Color and Video Finishing

Translation

Graphics

Commercial Cloud ServicesPIX/DAX Filet rack

Speeder

bikeAspera

5th kind

Commercial

email

Simpl if ied Product ion Workflow1. Content Creation and Capture

2. On-Location Production –

Uses Cloud Based tools3. Post Production –

Relies Heavily on Cloud based Collaboration Tools

Zooming in a sub workflow – VFX

Off-Site

Editorial-VFX

Editing in ART

Editing in VFX

VFX

Post Production Departments

Editorial

Isolated NetworkISIS ISIS

Mount and Move of Original Video to staging area

Prepare Editorial Tools for Editing

Edit Film Scenes

Edit Film Sequences

Post Production

Off-Site Crew

Studio HQVFX Vendor

Art

WANCommercial Cloud

Collaboration Services

PixsFotokem ShotgunSpeederbike

Post Production/On-Location VFX-Editorial

Dailies Review Pre-visualization VFX ReviewEditorial Review

Physical Devices

On-Location Post Production Facility

EditorialArt Editorial-VFXVFX Vendor

Commercial Cloud Collaboration ServicesPixsFotokem Shotgun

Mac Mini Server

On-Premises Render Workf low

The Chal lenge

What are we trying to solve?

SPEEDStorage

S P A C E

Security

Render Work f low in AWS

Private Compute and Storage Subnet 10.0.2.0/24

AWS Batch

Spot Fleet

Spot Instances

instances

VPN Subnet 10.0.1.0/24

router

Bastion ServerAmazon CloudWatch

AWS KMS

Content Ingress/

Egress

Amazon EFS

route table

On-premises network

192.168.0.0/16

customer gateway

Active

Directory

Region

Amazon VPC

VPN gateway

VPN connection

route table

Compute Subnet

Render Farm Compute

Domain

ControllerAD

Connect

Sync

Access Control

Principals of Secure Design in Content

Product ion workf lows

Principal : Least Pr iv i lege

Principal : Pr iv i lege Separat ion

Robust Ident ity Controls

Subnet

AWS KMS

Region

IAM

Amazon VPC

route table

IAM

S3 Bucket

role

permissions

Users

Define role

Userpolicy

permissions

permissions

Encryptionkeys

AWS CloudFormation

Access Data

Monitor

Attach policy

AWS Encrypt ion & Key Management

AWS KMS

PRINCIPAL: DEFENSE IN DEPTH

VPC Segmentat ion

• Controls for an application or

publicly accessible service

• Controls that are currently used

can be applied to most AWS

implementations

VPC Peer ing

• Extends trust and controls

to Counterparties

• Restricts access to only

known infrastructure

• Used for workflows that

involve trusted counterparties

& shared infrastructure

VPC Sub-Account

• Trust is completely centralized

• Tantamount to a physical network

• Most restrictive trust model

Principal : Trust Reluctance

Principal : Trust Reluctance

Virtual private cloud

Amazon VPC

flow logsAmazon CloudWatch

bucket

AWS CloudTrail

Monitor API

VPC Logs

Asset Subnet

AWS KMS

route table

S3 Bucket

Encryptionkeys

Logging data

Key Logging

Logging datainstance with

CloudWatch

Amazon

Elasticsearch

Service

Event drive analytics

analytics'

Amazon RedshiftData warehousequeries

AWS LambdaCentralized Logging Proxy

Proxy ServerAdministrator

Log Management

API Calls

Summing it a l l up

SECURE DESIGN IS INTERCHANGEABLE

What’s next

• Collaborate with vendors and with productions to identify workflows to port to

the cloud

• Use published guidance

• Leverage AWS services and documentation

• Blaze a new trail

[email protected]

THANK YOU!!

@ISESecurity

[email protected]@ DanTechieTweets


Afternoon AgendaTime Track #1: Production & Content Creation Track #2: Broadcast & OTT Track

1:00 PM

Track Session #1

Rendering in Cloud with Deadline 10

Chris Bond, AWS Thinkbox CEO

Jason Fotter, Co-Founder and CTO, FuseFX

Track Session #1

AI Distribution - C-SPAN

Dan Mbanga, AWS BD Artificial Intelligence

1:45 PM

Track Session #2

Panel Discussion: Remote Creatives - Storytelling In a Distributed Age

Igor Boshoer, Founder/CEO for Linumio;

David Benson, CTO for Bebop

David Andrade, Cofounder for Theory Studios

Andy Stack, COO for Arcturus Studio

Track Session #2

Real-time Media Analysis with AI in Media


2:30 PM Break (15mins)

2:45 PM

Track Session #3

Enhanced Content Workflows using Amazon AI


Track Session #3

Delivery Side Supply Chain and OTT

Ken Kanagaraj, Principal Solutions Architect,

AWS Elemental

3:15 PM

Track Session #4

attn: Scaling to 500 million monthly views

Jake McGraw CTO Attn

Peter Agelasto, Founder Digital ReLab;

Luke Adams, CTO Digital ReLab;

Track Session #4

Low Latency Streaming via AWS and Wowza

Mike Talvensaari, Wowza VP of Product;

Chris Gruiz, WW Business Dev & Ops, AWS Marketplace

4:00 PM Closing Comments

4:15 PM Networking Reception


L e a r n m o r e a b o u t t h e i n n o v a t i v e M & E s o l u t i o n s b u i l t o n A W S b y v i s i t i n g

o u r p a r t n e r s p o n s o r s i n t h e r e c e p t i o n a r e a d u r i n g t h e i n t e r m i s s i o n .

L U N C H : R E S T A R T @ 1 P M

Low-Latency Streamingvia AWS and Wowza

Founded in 2005, Wowza offers a complete portfolio to power today’s video streaming ecosystem from encoding to delivery. Wowza provides both software and managed streaming services for producers, developers and engineers to build unique streaming experiences for any device or audience size. Wowza Streaming Engine is Wowza’sflagship award-winning media server software. Wowza Streaming Cloud is an end-to-end live streaming service built on AWS. Wowzaalso offers Wowza GoCoder a mobile SDK and free app for live streaming and Wowza Player, a modern HTML5 video player that is tightly integrated with other Wowza products.

What is Latency?

• Latency is a time interval between the stimulation and response, or, from a more general point of view, a time delay between the cause and the effect of some physical change in the system being observed.

• In terms of streaming, latency is the delay between the initial capture of the video and the viewer.

Other Related Terms

• Time to first frame - Time delay from when a person clicks the play button and when video appears

• Broadcast Delay – The practice of intentionally delaying the broadcast of live material to prevent profanity, bloopers, or violence (wikipedia)

• Drift – when latency increases during a broadcast

• Quality - Higher quality = higher resolution = more data to send

• Scale - How many inputs/participants, how far, how many viewers

• Bandwidth - How much traffic can the infrastructure ideally handle

• Throughput - How much traffic is the infrastructure really delivering

• Bitrate - How many bits of data are being processed over time

Why is there latency?

Q: Why is latency a problem?

A: Latency is not usually a problem – but people think it is.

• For many (or possibly most) live streams, latency does not matter

• HTTP streaming intentionally introduces latency for improved reliability.

• For some live streams, latency is critical.

Latency Demo

What Factors Impact Latency

Main factors affecting latency are:1. Quality2. Scale

Low Latency

Large Scale

High Quality

As either (or both) increase latency increases

What Factors Impact LatencyLow

Latency

Large Scale

High Quality

Quality• Resolution• Two-way• Multi-thread• Frame rate• Smooth playback• Chunk/Block Size• Buffering

Scale• Distance• Participants• Viewers• Streams• Complexity• Variability• Diverse endpoints

How Quality Injects LatencyHigher quality = higher resolution = more data to send• Resolution increases the amount of data in any frame, block, chunk or time

slice. If we hold infrastructure constant as resolution goes up the window and latency to deliver a segment increases.

• Trade-offs: Some mitigation techniques

Increasing ThroughputIncreasing throughput with more bandwidth(network) and bitrate(compute) = More infrastructure, more overhead, more cost

Reducing BufferReducing the buffer requirement will reduce latency but it also makes network fluctuations more visible to the viewer – e.g. a 1ms buffer ideally means content can be viewed as little as 1ms after capture but a 0.1-second network interruption will result in 100 buffer segments being un-recoverable in a live stream.

Reducing FramerateReducing the (frame rate) number of frames per second reduces the smoothness of the viewing experience – e.g. 15 fps is generally a good viewing experience for low movement broadcasts like presentation screen capture/sharing or many church services, it will produce a choppy experience for viewing sports

How Scale Injects LatencyScale increases distances, complexity and variability• As distance, the number of streams, and viewers increase, network

imperfections, variability, and degradation increase and can amplify latency.

• Trade-offs: Some mitigation techniques

Increasing ThroughputIncreasing throughput with more bandwidth(network) and bitrate(compute) = More infrastructure, more complexity, more overhead, more cost

Increasing BufferIncreasing buffer size adds ability for both encoders, transcoders and players to deal with variability smoothly but increasing buffer inherently increases latency. By definition, buffer size is the minimum amount of data for which a process can be initiated. If more data must be accumulated, then processing will take longer.

TCP vs UDPBecause it is a lighter weight protocol without error checking, monitoring, order of messaging and headers roughly 1/3 the size of TCP, UDP is inherently faster but there is no guarantee data is received. SRT can help here

Where streaming latency has been

• Windows Media• MMS, RTSP, HTTP (using UDP or

TCP)

• Encoder, Server, and Player buffer size management

• “Low delay” audio codecs

• Fast Start, Advanced Fast Start, Fast Recovery, Fast Reconnect, FEC

• 2-3 seconds (on a good network)

• Real Time Streaming Protocol (RTSP)

• Developed and by RealNetworks

• Relies on RTP and RTCP

• Frames can be sent one at a time in real time

• Can leverage UDP

• Potential latency could be as low as ~125 ms with minimal buffering (2-3 frames behind)

Where streaming latency has been

• Real Time Messaging Protocol (RTMP)

• Developed and open sourced by Adobe

• 1-3 seconds and sometimes below 1 second

• HTTP• Apple HLS

• 30+ seconds on iOS (using 10 second chunks)

• MPEG-DASH• 10-20 seconds (variable)

Where latency is going

• WebRTC• Designed for real-time audio,

video and data delivery over less-reliable connections

• Leverages TCP or UDP

• Multiple protocols related to RTSP/RTP

• 1 second or less (as low as 200 ms)

• WebSocket• Designed to provide a

standardized, two-way, reliable communications channel between a browser and a server

• Works with TCP

• Can be used with other streaming protocols including RTMP, WebRTC, Haivision SRT, Wowza WOWZ and Aspera FASP

• As low as 200 ms


• HTTP• Reducing to 4 seconds (using 1 second chunks) for DASH and below 8

seconds (using 2 second chunks) for HLS (includes GOP adjustments as well)

• CMAF + HTTP/1.1 (using HTTP Chunked Transfer Coding) enables video transfer, decode, and display before the end of the chunk encoding (“chunks of chunks”)

• Optimizing DASH MPD attributes (availabilityStartTime and minBufferTime)

• Video encoding enhancements that do not impact decoding (H.264 GDR)


• Quick UDP Internet Connections (QUIC)• Uses UDP (with TCP fallback)

• Focuses on security and reliability

• Tries to be like TCP while reducing connect and round trip times, packet loss, congestion control, and more using intelligent retransmissions and storage and delivery of information

• Built with HTTP/2 in mind

• And….

© 2017 Wowza Media Systems, LLC. All rights reserved. Confidential & Proprietary

Network Characteristic that KILL real time video

Packet loss – packets being discarded by routers

Jitter – packets arriving at different times than expected

Delay – the time from sender to receiver

Bandwidth – the fluctuating capacity between points

Overcoming Adverse Network Conditions


SRT Fundamentals

Public InternetSource Destination

SRT Workflow

AWS AWS


SRT Fundamentals

bps

tConstant frame rate

Encoder Variable Bitrate

Unreliable “Dirty" NetworkPacket Loss, Network Jitter,

Variable Delay, Bandwidth Fluctuation

tErratic bitrate and frame rate

Decoder’s Worst Nightmare!

SRT Fundamentals


bps



tReconstructed video frame rate

Much Happier Decoder

SenderBuffer

ReceiverBuffer

SRTEncapsulation & Network Time Stamps

Feedback

SRT Fundamentals

SRT Fundamentals


bps



tReconstructed video frame rate

Much Happier Decoder Video

Audio

Metadata

Packet acknowledgement – error recovery

Timing – low latency streaming

Network connection characteristics

Bi-directional

Firewall Friendly

SRT Fundamentals

SRT Fundamentals

SRT Demo

SRT in Wowza Streaming Engine

Greater reliability of throughput and packet loss

Improved bandwidth utilization with network bandwidth fluctuations & long-haul video transport

RESULTS - Germany to Australia, using a consumer DSL connection: • 7.5x increase of throughput with SRT, compared to RTMP connections over TCP.• RTMP: 800/10,000 kbits throughput• SRT: and 6000/10,000 kbits throughput

Independent security / encryption

Minimized buffer 4x RTT (re-transmission time)

Logic for packet loss recovery and jitter only requires 4 x the network latency to the server

Wide adoption: free open source, known & trusted

SRT in Wowza Streaming Engine Wowza Streaming EngineSRT Fundamentals

Benefits of SRT | Results

AND

WITH

AND

» Needs / Challenges

» Provide ESPN-like sports broadcasting to any screen

» High reliability and scalability to support load fluctuations

» Affordable for Division II & III universities and high schools

» Reliable first-mile network connections from remote venues

» Solution

» Wowza + AWS + SRT

» SRT integration significantly increases first-mile throughput

» Wowza transcoding enables ABR, simplifies on-site capture

» EC2 gives on-demand scalability to match broadcast load

» CloudFront supplies cost-effective global delivery

» Benefits

» Increases reliability of connections, reducing support calls

» Enables new opportunities for end users at remote venues

» Reduces costs from alternative solutions by 50%

Business Case

Enabling Customer Success for BlueFrame Technology

Common BlueFrame Technology Customer Scenario

Wireless connection without SRT HLS

SDI, HDMI, or NDI

EC2

S3

CloudFront

Wireless connection using SRT

Live broadcast from a college athletics venue with unreliable Internet connectivity

SRT Alliance MembersMembers

Try SRT in Wowza Streaming Engine

• Sign up for a free trial

• www.wowza.com/products/streaming-engine

• Images available in AWS Marketplace

http://www.wowza.com/products/streaming-engine

Questions?

Thank you!


AWS Media & Entertainment Cloud SymposiumKen Kanagaraj, Principal Solutions Architect, AWS Elemental

MEDIA SUPPLY CHAIN ON AWS

March 14, 2018


108

AWS acquired Elemental Technologies October, 2015

Enterprise Government

Broadcasters

Content Programmers

Pay TV Operators

MARKETSCONTENT

ON DEMAND

Storage

Amazon S3

LIVE

DISPLAYS

DEVICES

AWS ELEMENTAL – PERFECTING THE MEDIA EXPERIENCE


THE ECOSYSTEM: POWERING MEDIA WORKFLOWS

Content

Production

Content Ingest

& Storage

Processing

&

Management

Content

Distribution &

Consumption

Consumer

Insight &

Analytics

Partner Solutions

Shared Services

Infrastructure Security Network Operations

• Modelling

• Rendering

• Video Editing

• Post Production

• Broadcast Signal

Acquisition

• Digital Dailies /

Approvals

• High Speed Ingest

• Library Storage &

Archive

• Tier Management

• Content / Asset

Management

• Machine Learning /

Artificial Intelligence

• En/Transcode

• Caption, Subtitles

• Packaging

• Encryption;

Watermarking

• Digital Rights

Management

• Automation,

Workflow



• B2C Streaming of

Live and VOD

Content

• B2B Distribution

• Video Advertising

Insertion

• Analytics, Reporting,

Log Analysis

• Real-time Monitoring

• Content Discovery

• Content

Recommendations




DELIVERY MEDIA SUPPLY CHAIN

Ingestion

Manual QC

Media Info

Metadata Extraction

Transcoding

Delivery

Packaging

Cataloging

CMS/DAM

Normalization

Content Acquisition

Post Production

DAM & Archiving


OTT

Analytics

Media Supply Chain

“The digital supply chain is new media term which encompasses the processes involved for

the preparation and delivery of media, be it video or music, by electronic means from point of

origin (content provider) to destination ( consumer) “


111

Content&AssetManagement

ContentPrep&DeliverySources

Entitlements

Multi-DRM

SubscriberManagement

DeviceManagement

MetricsandAnalytics

Search&Discovery

Events/AlertsHeartbeat

ProductCatalogs

Authentication

WebCMS/AppPublishing

Recording/Scheduling

CustomerOSS/BSS Recommendations GeoServices SocialNetworks

Self-Care

AdServerMetadataEnrichment

VideoBackOffice AggregationAPIs

Personalization

OffersandPackages

DVRSchedulingRecordings

Tablets

Mobile

SmartTVs

IPSTB

GamingConsoles

Client/PlayerApplications

PC/Mac

Customerand3rd PartyInterfaces

JITP

DAI/StreamConditioning

IngestWorkflows

AWSConsoleAPIs

Logging OperationalConsoles NetworkMonitoring VideoQuality

VideoApplications,NetworkandBroadcastOperations

CDN

LiveTranscoding

VODEncoding

JSONAPI

ADIXMLTV

APIJSON/XML

AWSCoreServices

LiveFeeds

VODAssets

ADI/EPGMetadata

ContentPolicies

EC2 S3 DynamoDB CloudWatch Lambda

Tagging/Categories/Discovery

AnalystApplications

E2EOTTArchitecture

Billing


10

CASE STUDY: NEWS BROADCAST PROVIDER

Background:

Existing on-premises workflow to enable news producers from

all over the globe to submit videos to be processed and prepped

for OTT delivery across multiple devices.

Challenge:

Outgrown capacity on current infrastructure. Processing 200-

300 clips / day. Improve ‘time-to-web’.

Solution:

Develop content workflow leveraging Elemental Cloud, AWS native

components along with 3rd party offerings to increase capacity, handle

unpredictable workloads and meet SLA requirements – all by paying for

what they use.


MEDIA SUPPLY CHAIN

Ingest QCInput AnalysisOutput

ProcessingDeliver

WORKFLOW STATE MANAGEMENT

Asset Registration

CMS


• Build it yourself vs. PaaS vs. SaaS

• Existing resource skills

• Elemental Cloud vs. partner/competitor processing

• Redundancy models

• No Multicast in the Cloud!

DESIGN CONSIDERATIONS


More than 90 services from

• compute,

• storage,

• networking,

• database,

• analytics,

• application services,

• deployment, management,

• Internet of Things (IoT),

• Artificial Intelligence (AI),

• security, hybrid and enterprise applications

SERVICE BREADTH


• Dedicated, 1 or 10 GE private pipes into AWS

• Create private (VPC) or public virtual interfaces to AWS

• Reduced data-out rates (data-in still free)

• Consistent network performance

• At least 1 location to each AWS region

• Option for redundant connections

• Uses BGP to exchange routing information over a VLAN

INGEST

WHAT IS AWS DIRECT CONNECT?


S3 Standard S3 Standard - Infrequent

AccessAmazon Glacier

Data Transfer (Ingest/Egress)

AWS Direct

Connect

AWS Snowball ISV Connectors Amazon Kinesis

Firehose

S3 VPC

EndPoint

AWS Storage

Gateway

Events

S3 Event

Notifications

S3 Transfer

Acceleration

STORAGE

OBJECT STORAGE OPTIONS


MEDIA WORKLOADS (REDEFINED)

EBS


119

AWS ELEMENTAL ENABLES ELASTIC, FLEXIBLE, MODULAR WORKFLOWS

PROCESS

AWS Elemental LIVEReal-time video processing

for linear broadcast

AWS Elemental DELTAOrigin and video delivery platform

AWS Elemental CLOUDManaged service providing video

processing and delivery in the cloud

AWS Elemental SERVERFile-based video processing for broadcast

and IP content

AWS Elemental CONDUCTORUnified video management

DELIVERYPROCESSING AND MANAGEMENT

INTEROPERABLE COMPONENTS

SUPPORTED INFRASTRUCTURES

Ad Decision CDN CMS/WFM DRM Fingerprinting QC Storage Watermarking

DEVICESDEVICES

AWS Elemental SERVER

AWS Elemental LIVE

AWS Elemental CONDUCTOR LIVE

AWS Elemental CONDUCTOR FILE

AWS Elemental DELTA

AWS Elemental DELTA

MANAGED NETWORK

AmazonCLOUDFRONT

SOURCE

FILE

LIVE

MPEG-2, AVC, HEVC, J2K

Storage

Amazon S3

Appliance On-premises

Virtual Machine AWS Cloud

© 2017, Amazon Web Services, Inc. or its Affiliates. © 2017, Amazon Web Services, Inc. or its Affiliates. As of February 2017

DELIVERY

Ashburn, VA (3)

Atlanta, GA (2)

Chicago, IL

Dallas/Fort Worth, TX (2)

Hayward, CA

Jacksonville, FL

Los Angeles, CA (2)

Miami, FL

Minneapolis, MN

Montreal, QC

Newark, NJ

New York, NY (3)

Palo Alto, CA

San Jose, CA

Seattle, WA

South Bend, IN

St. Louis, MO

Toronto, ON

Rio de Janeiro, Brazil (2)

São Paulo, Brazil

Amsterdam, The Netherlands

(2)

Berlin, Germany

Dublin, Ireland

Frankfurt, Germany (5)

London, England (4)

Madrid, Spain

Marseille, France

Milan, Italy

Munich, Germany

Paris, France (2)

Stockholm, Sweden

Vienna, Austria

Warsaw, Poland

Chennai, India

Hong Kong, China (3)

Manila, the Philippines

Melbourne, Australia

Mumbai, India (2)

New Delhi, India

Osaka, Japan

Seoul, Korea (3)

Singapore (2)

Sydney, Australia

Taipei, Taiwan

Tokyo, Japan (3)

70 CloudFront Edge Locations (PoPs), 45 Cities, 5 Continents

CLOUDFRONT IS GLOBALLY DISTRIBUTED


to manageSimple

Agile

Available

Scalable

Zero effort to administer

on demand

without ”clustering”

coding practices

Instance-lessCOMPUTE


SERVERLESS – DELIVER ON DEMAND, NEVER PAY FOR IDLE


BUILDING BLOCKS FOR SERVERLESS APPLICATIONS

AWS Lambda Amazon DynamoDB

Amazon SNS

Amazon API GatewayAmazon SQS

Amazon Kinesis

Amazon S3

Orchestration and State Management

API Proxy Messaging and Queues Analytics

Logging

Compute Storage Database

AWS Cloud

WatchAWS Step Functions

Processing

AWS Elemental

FfMEG


Amazon

STORAGE

Amazon

DELIVERY

Amazon

COMPUTE

Amazon

PROCESSING

Amazon

INGEST

CloudFront – Global Content Delivery Network with Monitoring, Analytics and customization at the edge

S3, Glacier, EFS & EBS – Durable, performant, scalable and secure solutions for on-line and archival content storage

Elastic Transcoder; AWS Elemental – Scalable, and cost-effective video processing on prem and in the cloud

EC2; Lambda– Resizable general purpose compute capacity featuring instance types optimized for rendering, editing, processing video, analytics

Direct Connect; Snowball; S3 Transfer Accelerator – Upload options for content and files of all sizes

Amazon

ANALYTICS/ AI

Machine Learning, Kinesis, AI – Data ingest/processing/analysis; machine learning, artificial intelligence for text to speech, image processing & analytics, natural language.

AWS SERVICES FOR M&E HIGHLIGHTS


MEDIA ECOSYSTEM OF PARTNERS

INGEST STORE MANAGE SECUREPROCESS

CREATEMONETIZE

INTEGRATEDELIVER

SaaS BYOLPaaS


Source

Amazon S3

Media FileRegister

Asset

Media Info

QC

Encode

Publish

Elemental

Cloud

AWS Elastic

Transcoder

Output

Amazon S3

CloudFront

Amazon S3

• Images• Captions• Feature

Asset• Metadata

• MP4 –Bitrates

• ABR - HLS

Register

Asset

MediaInf

o

AWS

Step Functions

DynamoDB

Encode

HLS MP3

Publish

Q

C

VOD WORKFLOW – REFERENCE SOLUTION


AWS LIVE STREAMING

AWS provides a live streaming video solution that combines AWS Elemental Cloud, a service that enables customers to rapidly deploy multiscreen offerings for live and on-demand content, with Elastic Load Balancing, Auto Scaling, and Amazon CloudFront to build a highly resilient and scalable architecture that delivers your live

content worldwide.


• Modular

• Flexible – Avoid Vendor Lock In

• Elasticity

• Increased Development Velocity

• Leverage existing skills ( Java/Javascript)

NEW SOLUTION REQUIREMENTS


VOD DIGITAL SUPPLY CHAIN

Source

Amazon S3

Amazon

Dynamo DB

Amazon

CloudWatch

Dashboard &

ConsoleAmazon

SNS/SQS

Amazon

Step FunctionsAmazon SQS

Amazon Lambda

MediaInfo QC Integration

AWS Elemental xCode

AWS Elemental JITP ( Delta )

Outputs

Amazon S3

MP4 / HLS Output

Events / Logs Notifications

Multi-CDN

DRM

Service

Fast File

Transfer

DRM

Amazon

Elastic Search

Metadata Index

External Head End

Asset Metadata

Technical Metadata

Encoding Profiles

QC reports


CLOUD ADOPTION APPROACHES

• Deploy application on cloud infrastructure services

Lift-and-Shift

• Refactor application to leverage some Cloud abstractions

Cloud-Optimized

• Re-imagine application architecture with many cloud abstractions

Cloud-Native


INGEST FLOW


PROCESS FLOW


PUBLISH


BENEFITS AND CHALLENGES SOLVED

AWS helps you scale your media

storage and compute needs

Handle unpredictable &

bursty media needsPay for media you store

and process, as you go

Global availability instantly,

with no commit

Focus your resources on

your media needs

Shorten time-to-air,

increased agility, automate

Cost PressuresContent Growth Variable Demand

Operational Pressures Global Market Core Competency


AWS alignment to MPAA Security Best Practices reviewed October 2012, Re-Reviewed August

2015 Based on the MPAA's revised Best practices for

Content security and AWS Shared Responsibility Model

AWS Services In Scope

“All major AWS Services”

Content Types In Scope

RAW master, high/low-resolution, watermarked/spoiled, full/partial feature content, stills, clips, frames,

shots, sequences, scripts, storyboards, as well as production and post-production deliverable formats in

pre-and post-theatrical release windows

MPAA Hub page on AWS Compliance http://aws.amazon.com/compliance/mpaa/

SECURITY BEST PRACTICES – MPAA ETC.

http://aws.amazon.com/compliance/mpaa/


DYNAMIC/JUST-IN-TIME PACKAGING

Encoder Packager Origin Player

1 2

Pre-packaging

Costs +: Each segmented format stored separately. Store whole library 3-4 times.

Costs - : Stored Once, processed on demand.

3


MULTI-CDN – DYNAMIC PACKAGING

240 @ 600kbps

1080 @ 5Mbps

720 @ 3Mbps

576 @ 1.4Mbps

360 @ 1Mbps

AAC Stereo

Dolby Digital

STB & CONNECTED TV

VIDEO DELIVERY PLATFORM

1080 @ 5Mbps

720 @ 3Mbps

576 @ 1.4Mbps

360 @ 1Mbps

AAC Stereo

720 @ 3Mbps

576 @ 1.4Mbps

360 @ 1Mbps

240 @ 600kbps

AAC Stereo

There is no point in streaming HD

1080p content to a smartphone:

even if the device can render it,

users do not see the difference

with 720p on small screens.

Reduction in bandwidth costs

and initial video starting time.

On Delta, the same URL

http://<delta>/out/u/hls.m3u8

will automatically apply the

selection of the audio/video

streams, size of the video

segments and JIT encryption

based on the User Agent of the

client.

1080 @ 5Mbps

720 @ 3Mbps

576 @ 1.4Mbps

360 @ 1Mbps

240 @ 600kbps

AAC Stereo

Dolby Digital

MPEG-TS

5 video

2 audio

2-sec. segments

ENCODERSWINDOWS PC & MAC

ANDROID & IOS

HDS + Access

5 video

1 audio

6-sec. segments

HLS + PlayReady 2.2

4 video

1 audio

10-sec. segments

HLS + PlayReady 1.5

4 video

2 audio

2-sec. segments

1080 @ 5Mbps

Dolby Digital

Dolby Digital


Best practices and well architected - Video Solutions

Serverless content workflow

Get started with a pilot project – AWS Media specialists support

Show you how to build and iterate through your use case

138

GETTING STARTED

https://aws.amazon.com/answers/media-entertainment/


• Part of AWS Elemental, Global Services is focused on delivering market leading

solutions on top of AWS & Elemental product offers.

• We are not a substitute for product, rather we deliver capabilities on top of core

product to extend, integrate, configure, deploy, and provide “System Integrator” like

solutions for our customers.

• We enable our customers to take our building blocks and deliver a solution to

fit their needs.

• Building Blocks are forms of AWS native services, Elemental suite of products and

3rd Parties.

GLOBAL SERVICES – WHO WE ARE


THANK YOU


MEDIA MICRO-SERVICES


Amazon EFS

File

Amazon EBSAmazon EC2

Instance Store

Block

Amazon

S3

Amazon

Glacier

Object

Data Transfer (Ingest/Egress)

AWS Direct Connect AWS Snowball ISV Connectors Amazon Kinesis

Firehose

S3 VPC

EndPoint

AWS Storage

Gateway

S3 – Infrequent

Access

Events

S3 Event

Notifications

S3 Transfer

Acceleration

STORAGE

AWS STORAGE SERVICES MATURITY

ATTN:’s Journey to the

Cloud © 2017 Digital ReLab LLC

&

About Us (Digital ReLab and ATTN)

© 2017 Digital ReLab LLC

ATTN: Scaling (Choosing Starchive &

AWS)


Business Context:

1. Scaling content creation2. Desire to enable greater fidelity3. New types of content creation4. Desire to centralize and optimize

storage5. Approaching half a billion video

views monthly

ATTN:’s Situation


Scaling higher-quality content means:

1. More valuable B-roll to move to the cloud and monetize2. More A-Roll to publish3. Fast, reliable, and flexible transcoding is necessary4. Easy search and discovery is critical – requiring better metadata standards5. Future proofing to leverage “best-in-class” AI is critical6. “Increased fidelity = larger files” – necessitating scalable storage

Power Of Starchive Leveraging AWS


• Sitting on hard drives• Disparate data silos• Lost and duplicate files

• Making and Saving Money via:

• New Publishing channels

• Licensing• New Products• Greater Efficiency

• Transcoding• Metadata Probing• New metadata from AI• Search• Playback• Distribution

Digital Content Digital

AssetsValue Creation

Starchive in the Cloud (AWS)


Enabling New Tools in Media Transformation


The Media Transformation stage in the pipeline allows ATTN: to:

1. Transcode, probe metadata, connect business logic

2. Run AWS Rekognition3. Leverage new AI Tools via Marketplace4. Re-process files with updated AI

Results

With Starchive & AWS, ATTN: can now:

1. Move both A-Roll and B-Roll to the cloud more efficiently2. Automatically transcode files for easier publishing3. Make use of surfaced metadata for search and archiving4. Automate the adding of business logic data to all video as

it is created5. Pipeline new data transformation 6. Stay ahead of the curve by leveraging new AI as tools

emerge


Lessons Learned

Q & A


Thank You


Deep Media Analysis Using Amazon AI

Konstantin Wilms, Principal Solutions Architect, M&E, AWS

November 7, 2017


Media & Entertainment Cloud

Symposium | Los Angeles


L ive2VOD Enr i chment

L ive Sent iment Ana lys i s

Deep V ideo Ana lys i s

Fas te r than Rea l t ime In fe rence

Agenda

ASSET INGEST AMAZON S3 AWS LAMBDA

CLIENT APP AMAZON ELASTICSEARCH

AMAZON REKOGNITION

Mobile app uploads

the image to S3

A Lambda function is triggered

and calls RekognitionRekognition retrieves the image from S3 and

returns labels for the property and amenities

Other users can search properties by

landmarks, category, etc.

MEDIA ASSET

MANAGEMENT

S3TA or 3rd Party Asset

Ingest

Event-Based Processing

Via S3 Notifications

Lambda Extracts Frames

& Conforms Image FramesReturn Object, Scene & Context

Data for Images

Inject Asset Tags via REST API

Into specific fieldsSearchable Tags stored as bag of

words collections

Authenticated Users can

Search Assets via Custom/Web App

Metadata Enrichment using Amazon Rekognition



Since an early flush of optimism in the 1950’s, smaller subsets of artificial intelligence - first machine

learning, then deep learning, a subset of machine learning - have created even larger disruptions.

DEEP

LEARNING

MACHINE

LEARNING

ARTIFICIAL

INTELLIGENCEEarly Artificial Intelligence

stirs excitement

to flourish

Machine Learning begins

drive AI boom

Deep Learning breakthroughs

1950’s 1960’s 1970’s 1980’s 1990’s 2000’s 2010’s

© NVIDIA


The 10,000ft Intro to Deep Learning

Raw Data Low Level Features Mid Level Features High Level Features

Result

Application

Components

Task

Identify a Face

Training

10-100M images

Network

~ 10 layers

1B parameters

Learning

~ 30 Exaflops

~ 30 GPU days

© 2016 NVIDIA

Input


Services & Partners

Amazon

Polly

Life-like

Speech

Amazon

Rekognition

Amazon

Lex

Deep Image

Analysis

Conversational

Engine

Deep Learning

AMI

Multiple

Frameworks

Amazon

ML

Predictive

Analytics

Managed DIY

Partner


The Amazon AI Stack

Services

Platforms

Frameworks

Infrastructure

MXNet TorchCTKKerasTheanoCaffeTensorFlow

AWS Deep Learning AMI

Amazon ML ECSSpark & EMR Kinesis Batch

Vision Speech Language

GPU / FPGA ServerlessCPU IoT Mobile


Choosing the right Instance Type for AI

Instance

Name

GPU

Count

vCPU

CountMemory Network EBS

p3.xlarge 1 8 61 GiB ~10Gbps 1.5 Gbps

p3.8xlarge 84 32 244 GiB 10Gbps 7Gbps


P2 & P3: Distributed Training & InferenceHyper-scale performance on NVIDIA V100s

G3: Multi-User ModelingNVIDIA M60 GPUs, 16,384 cores

F1: High Speed InferenceXilinx Ulstrascale Plus, 6,800 engines

X1: Specialized AI/ML/DL128 vCPUs, 3,904 GiB RAM

P3 Instances Provide up to 1 Petaflop of mixed precision performance, and

125 Teraflops of single precision floating point


Deep Learning (& AI) for Media – Glass to Glass

Playout &

DistributionFiltering & Quality

Control

Visual Effects &

EditingApplication & Filesystem

Texture & Asset Search

AnalyticsSentiment Analysis

Other Amazon AI

Services

(Lex, Polly)

DAM & ArchiveAuto-categorization

Metadata Augmentation

Digital Supply ChainTag on Ingest

Live and VOD Feature

Extraction

Celebrity Detection

PublishingValue Add

API-based services

OTTFiltering &

Quality Control

AcquisitionPre-processing

& optimization


Building A Deep Learning Pipeline in the Cloud

Training

Version Controlled

Datasets

Amazon

S3

Amazon

API Gateway

Amazon

EC2

Amazon

S3

AWS Lambda

Amazon

EFSAWS

BatchAmazon

ECS

Amazon

Glacier

AWS

Snowball

Data

Scientist

Amazon

Mechanical Turk


Training a Deep Learning Pipeline

Youtube-8M CelebAPlaces CIFAR-10/100

hybrid cloud processing is well suited to large datasets

Object, Network & Gateway

Storage Services

Frameworks, 3rd Party Enablement, & Industry Initiatives


Managed Serv i ce , Or D IY?

Deep learning-based image recognition serviceSearch, verify, and organize millions of images

Object and Scene

DetectionFacial

Analysis

Face

Comparison

Facial

Recognition

Celebrity

Recognition

Image

Moderation

Amazon Rekognit ion


• Object & Scene Detection

Metadata Enrichment for

MAM/DAM/CMS/UGC services

• Facial Analysis

User Sentiment Analysis

• Face Comparison

Associate content with ‘non-celbrity celebrities’

• Content Moderation

Automatically scan & approve content


Media Analysis using Rekognition

Maple

Villa

Plant

Garden

Water

Swimming Pool

Tree

Potted Plant

Backyard

Flower

ChairCoffee Table

Living Room

Indoors

Object and scene detection makes it easy for you to add features that search,

filter, and curate large image libraries.

Object & Scene Detection


Demographic Data

Facial Landmarks

Sentiment Expressed

Image Quality

Brightness: 23.6

Sharpness: 99.9

General Attributes

Facial & Sentiment Analysis


Detect explicit and suggestive contentRecognize thousands of famous individuals

Celebrity Tagging Content Filtering

Celebrity Recognition & Image Moderation



3rd Party Software

• AWS AI AMI

• OpenCV

• ImageMagick

• FFMPEG

• Many Others

AWS Services

• Amazon S3

• AWS Lambda

• AWS SQS & SNS

• Amazon DynamoDB

• Amazon EC2 / SPOT

AWS Partners

• Asset Management

• Media Workflow

• Content Processing

• Image Optimization

• Feature Extraction

Extending Rekognition

{

"FaceMatches": [

{"Face": {"BoundingB

"Height":

0.2683333456516266,

"Left":

0.5099999904632568,

"Top":

0.1783333271741867,

"Width":

0.17888888716697693},

"{

"FaceMatches": [


"Height":

0.2683333456516266,

"Left":

0.5099999904632568,

"Top":

0.1783333271741867,

"Width":

0.17888888716697693},

"

{

"FaceMatches": [


"Height":

0.2683333456516266,

"Left":

0.5099999904632568,

"Top":

0.1783333271741867,

"Width":

0.17888888716697693},

"

{

"FaceMatches": [


"Height":

0.2683333456516266,

"Left":

0.5099999904632568,

"Top":

0.1783333271741867,

"Width":

0.17888888716697693},

"

{

"FaceMatches": [


"Height":

0.2683333456516266,

"Left":

0.5099999904632568,

"Top":

0.1783333271741867,

"Width":

0.17888888716697693},

"

Decoupling

Amazon

SQS

Amazon

SNS

Amazon

Kinesis

Amazon Elastic

Transcoder

Media Processing

Amazon API

GatewayAWS Batch

Amazon

EC2 Amazon

ECS

Compute

Glue

AWS

Lambda

Storage

Amazon

EFSAmazon

S3

Persistence

Amazon

DynamoDB

Amazon

ES

AWS Elemental

Serv ice Integrat ion



Determinist ic Response T ime

~500ms Object & Scene Detect ion

~1.5s Search for 1mil Face Col lect ion

moderation level = safe

Building Rich Metadata Indexes using Rekognition


Name: You?


Back to our Agenda …


Live2VOD Enr ichment

Transport

• API calls have a cost – conform & extract frames

• Scene change detection vs. frame extraction

• Streaming & Byterange Processing for S3

• Open Source & Commercial Ingest

Media

• Image enhancement, extraction & stabilization

• Unsharp mask, deconvolution – CPU impact

• Frame Extraction Offsets - PTS extraction

• e.g. FFMPEG w/deshake, vidstab, OpenCV

Live & On-Demand Conformance



Live Sentiment Analysis

Amazon RedshiftAmazon

Quicksight

Live Subject Camera Embedded Application

Amazon S3

Analyze Faces

Shoppers enter and

browse in retail store

In-store cameras capture

live images of shoppers


and calls Rekognition Rekognition analyzes the image and returns

facial attributes detected, which include

emotion and demographic detail

Return data is normalized and

staged in S3 en route to

Redshift

Analysis

Reports Periodic ingest of data into

RedshiftRegular analysis to identify trends in

demographic activity and in-store

sentiment over time

Live Sentiment Analysis



Deep Video Analys is


Distributed Deep Learning Systems Architecture

LO

AD

BA

LA

NC

ER

S

ORKER #1CPU WORKERS

#NDocker(algorithm#1)

Docker(algorithm#2)

..

Docker(algorithm#n)

AP

I

SE

RV

ER

S

AP

I

SE

RV

ER

S

GPU WORKERS

Docker(deep-algo#1)

Docker(deep-algo#2)

..

Docker(deep-algo#n)

m4

m4

m4

x1

• B2B & B2C have differing

workflows - orchestrator, vs. fan

out APIs with host pinning

• SOA, Microservices, Serverless

• Fractional billing is a must for

fleet diversification when

training

• Training - centralized storage

• Inference – ‘shared nothing’

• Hardware-targetable Containers


Faster Than Realt ime Inference


High Speed FPGA Inference

Standard

Training

FPGAZebra

Int16

/int8

Caffe: Inference

GPUcuDNN

FP32

Caffe: Learning

TrainingData

NNDescription

TrainedNN

TrainedNN

IncomingData

Results

Zebra for

Inference

Application

Primitive

CL Execution Region

DDR4 DIMM

16GiB

DDR4 DIMM

16GiB

AWS

Shell

Multi-Port DDR

Controller

Multi-Port DDR

Controller

Input Data Loader Input Data UnLoaderInput Data

Packetizer

Results Data

DePacketizerResults Data LoaderResults Data UnLoader

DMA Controller

AXI Slave

• Deterministic, velocity-based processing

• Real-time sentiment analysis

• AWS F1 FPGA w/Xilinx Ultrascale

• 976GB RAM for large models

• ~1200fps (unoptimized) inference

• Caffe & MxNet DL Framework

• Train on G2/G3, run on F1

• Transparent GPU to FPGA targeting

using HDK/AFI & DL framework


FPGA Advantages for Deep Learning

Un-Optimized Performance Performance with Tuning

Machine Learning on FPGAs

Jason Cong, Chancellor’s Professor, UCLA

Director, Center for Domain-Specific Computing

http://cadlab.cs.ucla.edu/~cong


Where does Med ia Secur i ty f i t in ?


Machine Learning-based Data Security

Machine Learning Service to help

customers prevent data loss in AWS

Security Goals

• Categorize new or unknown threats

based on known and theorized

examples

• High coverage (true positive volume)

• High Accuracy (few false positives)

• Adaptive

Understand Your Data

Natural Language Processing (NLP)

Understand Data Access

Machine Learning


Why is this important for Media?

Data outside of the scope of asset

archives & content can be the largest risk

Features

• Discovery, Classification & Protection

• Basic vs. Predictive Alerting

• Per-data object risk level assessment

• Compliance, Disruption, Ransomware,

Privilege escalation, Permissions,

Suspicious Access, PII data exposure


Key Takeaways

• Managed services can be used to eliminate

‘undifferentiated heavy lifting’, allowing for niche AI focus

• Many traditional ‘in the cloud’ paradigms map to deep

learning

• AI-based technology provides unique opportunities to

enhance existing media delivery services

• Design using hybrid modeling, in-cloud training, & micro-

services

• Utilize compute diversification across GPU, CPU & FPGA,

combined with Object Storage (S3) & Fractional Billing

(SPOT)


Thank You!

Enhanced Content Workflows Using Amazon AI

Konstantin Wilms, Principal Solutions Architect, M&E, AWS

November 7, 2017


Media & Entertainment Cloud

Symposium | Los Angeles


Confidence Labels

99.2%

Animal

Dog

Chihuahua

98.6%

Food

Dessert

Muffin

97.9% Collage

Dog or Muffin?


Algorithm Viability

OCR

Are you

feeling

lucky?

Perceptual

Hash

Not a

chance

Deep Logo

AnalysisBingo

Word or Logo?


Since an early flush of optimism in the 1950’s, smaller subsets of artificial intelligence - first machine

learning, then deep learning, a subset of machine learning - have created even larger disruptions.

DEEP

LEARNING

MACHINE

LEARNING

ARTIFICIAL

INTELLIGENCEEarly Artificial Intelligence

stirs excitement

to flourish

Machine Learning begins

drive AI boom

Deep Learning breakthroughs

1950’s 1960’s 1970’s 1980’s 1990’s 2000’s 2010’s

© NVIDIA


The 10,000ft Intro to Deep Learning

Raw Data Low Level Features Mid Level Features High Level Features

Result

Application

Components

Task

Identify a Face

Training

10-100M images

Network

~ 10 layers

1B parameters

Learning

~ 30 Exaflops

~ 30 GPU days

© 2016 NVIDIA

Input


Services & Partners

Amazon

Polly

Life-like

Speech

Amazon

Rekognition

Amazon

Lex

Deep Image

Analysis

Conversational

Engine

Deep Learning

AMI

Multiple

Frameworks

Amazon

ML

Predictive

Analytics

Managed DIY

Partner


The Amazon AI Stack

Services

Platforms

Frameworks

Infrastructure







Frameworks & Infrastructure

Services

Platforms

Frameworks

Infrastructure







Choosing the right Instance Type for AI

Instance

Name

GPU

Count

vCPU

CountMemory Network EBS

p3.xlarge 1 8 61 GiB ~10Gbps 1.5 Gbps



P2 & P3: Distributed Training & InferenceHyper-scale performance on NVIDIA V100s

G3: Multi-User ModelingNVIDIA M60 GPUs, 16,384 cores

F1: High Speed InferenceXilinx Ulstrascale Plus, 6,800 engines

X1: Specialized AI/ML/DL128 vCPUs, 3,904 GiB RAM

P3 Instances Provide up to 1 Petaflop of mixed precision performance, and

125 Teraflops of single precision floating point


Why is this Important?

Amazon EC2 Compute & EBS block storage supports second-level billing.

Combined with EC2 SPOT Fleet, this provides a up to ~90% cost savings over on-demand.


Deep Learning Compute


MXNet

Torch

CTKKeras

TheanoCaffe

TensorFlow

Amazon EC2

AnacondaIntel MKL

CUDA+cuDNN Python 2+3

Caffe2

• One Click launch

• Machine Image or Stack-based

• Single node or distributed

• GPU, CPU (& FPGA)

• NVIDIA & Intel acceleration

• Anaconda Data Science Platform

• Python w/ AI/ML/DL libraries


Building A Deep Learning Pipeline in the Cloud

Training

Version Controlled

Datasets

Amazon

S3

Amazon

API Gateway

Amazon

EC2

Amazon

S3

AWS Lambda

Amazon

EFSAWS

BatchAmazon

ECS

Amazon

Glacier

AWS

Snowball

Data

Scientist

Amazon

Mechanical Turk


Training a Deep Learning Pipeline

Youtube-8M CelebAPlaces CIFAR-10/100

hybrid cloud processing is well suited to large datasets

Object, Network & Gateway

Storage Services

Frameworks, 3rd Party Enablement, & Industry Initiatives


Large Scale Document Analysis

• NLP Topic Modeling @ Clemson University

• 533,560 Documents, 32,551,540 Words

• 1.1 million vCPUs over ~3hrs

• EC2 Spot, Single AWS Region

• SLURM scheduler - overlay

virtual workflow automation

• Per second billing for

EBS & EC2

17 years of computer science journal abstracts and full text papers

from the NIPS (Neural Information Processing Systems) Conference (2,484 documents and 3,280,697 words)

Provisioning & Workflow Automation Framework


Deep Learning (& AI) for Media – Glass to Glass

Playout &

DistributionFiltering & Quality

Control

Visual Effects &

EditingApplication & Filesystem

Texture & Asset Search

AnalyticsSentiment Analysis

Other Amazon AI

Services

(Lex, Polly)

DAM & ArchiveAuto-categorization

Metadata Augmentation

Digital Supply ChainTag on Ingest

Live and VOD Feature

Extraction

Celebrity Detection

PublishingValue Add

API-based services

OTTFiltering &

Quality Control

AcquisitionPre-processing

& optimization

Deep learning-based image recognition serviceSearch, verify, and organize millions of images

Object and Scene

DetectionFacial

Analysis

Face

Comparison

Facial

Recognition

Celebrity

Recognition

Image

Moderation

Amazon Rekognit ion



Determinist ic Response T ime

~500ms Object & Scene Detect ion

~1.5s Search for 1mil Face Col lect ion

moderation level = safe

Building Rich Metadata Indexes using Rekognition


Name: You?

ASSET INGEST AMAZON S3 AWS LAMBDA

CLIENT APP AMAZON ELASTICSEARCH

AMAZON REKOGNITION

Mobile app uploads

the image to S3


and calls RekognitionRekognition retrieves the image from S3 and

returns labels for the property and amenities

Other users can search properties by

landmarks, category, etc.

MEDIA ASSET

MANAGEMENT

S3TA or 3rd Party Asset

Ingest

Event-Based Processing

Via S3 Notifications

Lambda Extracts Frames

& Conforms Image FramesReturn Object, Scene & Context

Data for Images

Inject Asset Tags via REST API

Into specific fieldsSearchable Tags stored as bag of

words collections

Authenticated Users can

Search Assets via Custom/Web App

Metadata Enrichment using Amazon Rekognition



Service Enhancement using Amazon Lex + Polly

AMAZON LEX API GATEWAY + AWS LAMBDA

AMAZON VPC

API GATEWAY + AWS LAMBDA

User Requests

Transcode Job

Status

Lambda Recognizes

Request

Lex Responds with

Job Completion ETA3rd Party SaaS3rd Party Stovepipe

Connector

Internal Transcoding

Service

Transcode Cluster

Job Inspection


Key Takeaways

• Running Deep Learning Infrastructure is hard

• Managed services can be used to eliminate ‘undifferentiated

heavy lifting’, allowing for niche AI focus

• AI for media is a cross-functional tech undertaking

• Many traditional ‘in the cloud’ paradigms map to deep learning

• AI-based technology provides unique opportunities to enhance

existing media delivery services

• Design using hybrid modeling, in-cloud training, & micro-services

• Utilize compute diversification across GPU, CPU & FPGA,

combined with Object Storage (S3) & Fractional Billing (SPOT)


Marinus Analytics provides law

enforcement with tools, founded in

artificial intelligence, to turn big data

into actionable intelligence. The

Marinus flagship software, Traffic

Jam, is a suite of tools for use by

law enforcement agencies on sex

trafficking investigations.


Thank You!


Dan Romuald Mbanga

Business Development Manager

AI Platforms and Engines

An Overview of AI in M&E

@dmbanga

Thank You

A Flywheel For Data

More Data Better Analytics

A Flywheel For Data


Better Products

A Flywheel For Data


Better ProductsMore Users

A Flywheel For Data



Click stream

User activity

Generated content

Purchases

Clicks

Likes

Sensor data

A Flywheel For Data



Click stream

User activity

Generated content

Purchases

Clicks

Likes

Sensor data

Object Storage

Databases

Data warehouse

Streaming analytics

BI

Hadoop

Spark/Presto

Elasticsearch

A Flywheel For Data



Click stream

User activity

Generated content

Purchases

Clicks

Likes

Sensor data

Object Storage

Databases

Data warehouse

Streaming analytics

BI

Hadoop

Spark/Presto

Elasticsearch

Artificial

Intelligence

A Flywheel For Data



Click stream

User activity

Generated content

Purchases

Clicks

Likes

Sensor data

Object Storage

Databases

Data warehouse

Streaming analytics

BI

Hadoop

Spark/Presto

Elasticsearch

Artificial

IntelligencePinpoint

AI: TL;DR

Fast Compute

Ubiquitous Data

Advanced Learning Algorithms

Artificial Intelligence At AmazonThousands Of Employees Across The Company Focused on AI

Discovery &

Search

Fulfilment &

Logistics

Enhance

Existing ProductsDefine New

Categories OfProducts

Bring Machine

Learning To All

• Applied Research

• Core Research

• Alexa

• Demand Forecasting

• Risk Analytics

• Search

• Recommendations

• AI Services | Rek, Lex, Polly

• Q&A Systems

• Supply Chain Optimization

• Advertising

• Machine Translation

• Video Content Analysis

• Robotics

• Lots of Computer Vision..

• NLP / NLU

Just a few Deep Learning Use Cases at Amazon…

~ 1997

Recommendations & Ranking At Netflix

Personalized ranking,

page generation,

search, similarity, ratings

In 140 new countries,

simultaneously

Neural Style Transfer

http://mashable.com

Search

Watch

Listen

Play

Download

Purchase

Rate It

Review It

Sharing

Tagging

Bookmarking

Transactional Behavioral

Common M&E Analytic Use CasesConte

nt

•Top Content

•Engagement

•Play

Audience

•Acquisitions

•Churn

•Wher

Operations

•How much buffering

•Best CDN paths

Other

•Monetization

•Ad Spend

Latency: Realtime, NRT, Batch

Audience: Execs, Production, Content Programming, Marketing, Ops, App/Dev

Amazon AIIntelligent Services Powered By Deep Learning

More in

2017

Infrastructure CPU

Engines MXNet TensorFlow Caffe Theano Pytorch CNTK

ServicesAmazon Polly

Platforms

IoT

Speech

Mobile

Amazon

ML

Spark &

EMRKinesis Batch ECS

GPU

More in

2017

Chat

Amazon Lex

Amazon AI: Machine Learning In The Hands Of Every Developer

Amazon Rekognition

Vision

Automating Footage Tagging with Amazon

Rekognition

Built in 3 weeks

Index against 99,000 people

Saving ~9,000 hours a year in labor

More in

2017

Infrastructure CPU

Engines MXNet TensorFlow Caffe Theano Pytorch CNTK

ServicesAmazon Polly

Platforms

IoT

Speech

Mobile

Amazon

ML

Spark &

EMRKinesis Batch ECS

GPU

More in

2017

Chat

Amazon Lex

Amazon AI: Machine Learning In The Hands Of Every Developer

Amazon Rekognition

Vision

Solving Some Of The Hardest Problems In Computer Science

Learning Language Perception Problem

Solving

Reasoning

AWS Deep Learning AMI: One-Click GPU Deep Learning

Up To 40,000

CUDA Cores

Python 3 Notebooks

& Examples

(and others)(Volta at launch)

Scale for

Training

TensorFlow,

Apache MXNet

With Thanks To The Apache MXNet Open Source Contributors!

Apple

Bing Xu

MIT

Chiyuan Zhang

Qihoo 360

Yizhi Liu

Microsoft

Tianjun Xiao

Indiana

Tianjun Xiao

“Sounds great, Dan!

How can we get started?”

Direct Connect

Collect & Organize Data

KinesisStreams, Firehose,

Analytics

Upload

AWS Snowball Edge

100T Data Transport Device

With On-board Compute

Snowball Selfies: Customers Love AWS Snowball

Direct Connect

Kinesis

Snowball

Snowmobile

IoT

Database Migration

Streams, Firehose,

Analytics

Upload


Direct Connect

S3

Kinesis

Snowball

Snowmobile

IoT

Database Migration

DynamoDB

Elastic Search

Cloud Search

Glacier

RDS MySQL, PostgreSQL, MariaDB

Oracle, SQL Server

Aurora MySQL

PostgreSQL

Streams, Firehose,

Analytics

Upload


Direct Connect

S3

Kinesis

Snowball

Snowmobile

IoT

Database Migration

EC2

DynamoDB

EMR

Elastic Search

Cloud Search

Glacier


Oracle, SQL Server

Aurora MySQL

PostgreSQL

Streams, Firehose,

Analytics

Redshift & Spectrum

Athena

QuickSight

Upload

Amazon AI

Lambda


Direct Connect

S3

Kinesis

Snowball

Snowmobile

IoT

Database Migration

EC2

DynamoDB

EMR

Elastic Search

Cloud Search

Glacier


Oracle, SQL Server

Aurora MySQL

PostgreSQL

Streams, Firehose,

Analytics

Athena

QuickSight

Upload

Amazon AI

Lambda

Mech Turk

Glue


Redshift & Spectrum

Security & Encryption Integrated Across The Platform

Fine grained

access controls

Broad KMS

integrationServer-side

encryption

with CMK

Audit key

usage by

user & role

Import

keysPolicy

validation

& simulation

Use Case: Recommendations

The Concept

Capture Audience Signal Data

Create history of user and item preferences

Estimate similar users and items

Record these in Search Engine

Query Search Engine with User History

Enjoy recommendations!

Log

Storage

ETL

User

Interface

Serving

Layerusers

Recommend

Engine

users

Media platforms

Mobile

Apache MXNet

SearchPlayBuyRate

Recommendations

Step 1: Logs History Matrix

User1 Thing1

User2 Thing2

User3 Thing3

User2 Thing4

User5 Thing1

User1 Thing2

User1 Thing3

Mike

Jon

Mary

Phil

Kris

Logs History Matrix

Step 2: Estimate Similar Things

History Matrix

2 8

2 4

8

4

Item-Item Matrix

Step 3: Reduce to Interesting Pairs

2 8

2 4

8

4

Item-Item Matrix

LLR

Indicators

(“Items Similar To This….”)

Step 3: Reduce to Interesting Pairs

Indicators

(“Items Similar To This….”)

Items Similar To This

Step 4: Store Indicators in a Search Engine (BATCH)

Superman Highlander, Dune

Star Wars Raiders, Minority

Report

Highlander Superman

Mulan Home Alone, Mermaid

Star Trek …

… …

4587 223, 5234

748 5345, 235

12 8234

245 9543, 7673

3456 4587

… …

Index

Indicators

Step 5: Query Search Engine w/ User History

748 Star Wars

12 Highlander

245 Mulan

4587 Superman

3456 Star Trek

Query

“12”

5345

3456

12

Amazon AI Lab

Partner with ML engineers

from Amazon to:

Prototype ML with your data

Learn through hands-on

workshops & training

Build custom models

in production

[email protected]

Thank you!

[email protected]

THANK YOU!

@dmbanga