moving your media supply chain to the aws cloud
TRANSCRIPT
MediaSupplyChainsinAWSTermsUsmanShakeel |AmazonWebServices
Playout&Distribution
VisualEffects&Editing Analytics
DAM&Archive
DigitalSupplyChain
Publishing
OTT
Acquisition
AWSMediaSegments
Playout&DistributionElemental,EC2,S3/Glacier
EFS,CloudFront
VisualEffects&EditingEC2,S3,EFS,EBS,GPU,DX
AnalyticsKinesis,DynamoDBAML,RedShift,EMR
DAM&ArchiveSnowball,S3,GlacierEC2,DX,LambdaDynamoDB,RDS
DigitalSupplyChainElemental,EC2,S3/GlacierCloudFront,EFS,Snowball
GPU,DX
PublishingEC2,S3,Glacier,ELB,CloudFront
OTTElementalEC2,S3
CloudFront
AcquisitionElemental,EC2,
S3/GlacierCloudFront,Snowball,DX
AWSMediaSegments
MediaSupplyChains
Playout&Distribution
Production/PostProduction B2B/CinematicDelivery
DAM&Archive
DigitalSupplyChain
Publishing
OTT
ContentAcquisition
Ingestion
ManualQC
Auto-QC
MetadataExtraction
Transcoding
DeliveryPackaging
Clips/Edits
Complexworkflow/StatemanagementBursty sub-workflowtasks(transcoding/auto-QC,Packaging)ElasticLicensingContentmovementbetweenworkflowtasks(acommonrepository)DifferentdeliveryformatsforeachaffiliateanddownstreamManualtasks– (contentmovementorinteractivity)
CMS/DAM
Normalization
Normalization
All-InCloud-EnabledDominoEffect
Playout&DistributionLiveTV
Production/PostProduction
EditorialDailiesPost
SocialMediaMarketing/Promo
B2B/CinematicDelivery
DPP,IMF,NABA
DAM&ArchiveActiveArchiveGoldenCopyPreservation
DigitalSupplyChain
Publishing
OTTLive-To-VOD
AVOD,SVOD,TVODCatchupTVCloudPVR
ContentAcquisitionLiveFile
Ingestion
ManualQC
Auto-QC
MetadataExtraction
Transcoding
DeliveryPackaging
Clips/Edits
CMS/DAM
Ingest Store Edit/ManualTasks Process
PUSHORPULLCONTENT&ARTIFACTS
ACCESSSOURCEFILESLOCATEDINS3
CREATE,EDIT,QC&COLLABORATEINTHECLOUD
SCALEOUTONELASTICCAPACITYFORPROCESSING
(Transcoding,QC,Packaginganddistribution)
MediaproductionanddeliverycompaniesareleveragingAWStoacceleratetheirmediasupplychains,ingestion,storage,editing/manualQCofcontentandprocessingwithhighlyscalablecloudcomputingandstorage.
UsingCloudforMediaSupplyChains
Ingest
PUSHORPULLCONTENT&ARTIFACTS
AWS Services
• Amazon S3 Transfer Acceleration: Fast data transfers• AWS Direct Connect: Dedicated network connectivity• AWS Snowball: Petabyte scale data transfer• Amazon Cloudfront: Global content delivery network• Amazon Storage Gateway: On-prem to cloud storage connectivity• Amazon Marketplace: One-click partner software deployments• Elemental: Content contribution from on-premises encoders
Partners
MediaSupplyChains
AWS Services
• Amazon S3/SIA: Highly scalable, durable object storage• AWS Storage Gateway: On-prem to cloud storage connectivity• Amazon Glacier: Secure, durable, petabyte-scale data archival• Amazon EFS: Simple, elastic, scalable file system• Amazon EBS: Persistent block storage
Partners
Store
ACCESSSOURCEFILESLOCATEDINS3
MediaSupplyChains
Edit/ManualTasks
CREATE,EDIT,Clips,ManualQC&COLLABORATEINTHECLOUD
AWS Services
• Amazon EC2: Scalable CPU/GPU computing capacity• Amazon Workspaces: Virtual desktops in the cloud• Amazon Appstream: Remote Application Streaming• Amazon EFS: Simple, elastic, scalable file system• Amazon EBS: Persistent block storage• Amazon S3: Highly scalable, durable object storage
Partners
MediaSupplyChains
Process
SCALEOUTONELASTICCAPACITYFORPROCESSING
AWS Services
• Amazon EC2: Scalable CPU/GPU/Spot computing capacity• Amazon EFS: Simple, elastic, scalable file system• Amazon EBS: Persistent block storage• Amazon S3: Highly scalable, durable object storage• Elemental Cloud: On-demand, scalable video processing
Partners
MediaSupplyChains
AWS Services
• Amazon S3/SIA: Highly scalable, durable object storage• AWS Storage Gateway: On-prem to cloud storage connectivity• Amazon Glacier: Secure, durable, petabyte-scale data archival• Amazon EFS: Simple, elastic, scalable file system• Amazon EBS: Persistent block storage
Partners
AssetManagement
MetadataforAssetsandsupplychainworkflow
MediaSupplyChains
INFRASTRUCTURE
TaskSpecificComputeInstancesComputeIntensiveIntelES-2666v3(Haswell)optimizedspecificallyforEC2
MemoryIntensiveLowestpricepointperGiB ofRAM
GPUs1,536CUDAcores4GBofvideomemory
EnhancedNetworkingHigherPPS,Lowernetworkjitter,lowlatency
IOIntensiveSSDStorage,EBSOptimized
HighStorage24x2000GiB perinstance
AmazonS3
SQSQueues(orchestratedbyAmazonSWF)
M4
M3
C4
C3 I3G2
R4
P2 D2R3X1
Reserved
Make a low, one-time payment and receive a significant discount on the hourly charge
For committed utilization
On-Demand
Pay for compute capacity by the hour
with no long-term commitments
For spiky workloads, or to define needs
Spot
Bid for unused capacity at a Spot Price -
fluctuates based on supply & demand
For time-insensitive or transient workloads
ConsumptionModels
AWS Storage Services Maturity
AmazonEFS
File
AmazonEBS AmazonEC2InstanceStore
Block
AmazonS3 AmazonGlacier
Object
DataTransfer(Ingest/Egress)
AWSDirectConnect AWSSnowball ISVConnectors AmazonKinesisFirehose S3VPCEndPoint
AWSStorageGateway
S3– InfrequentAccess
Events
S3EventNotifications
S3TransferAcceleration
PostProductiontasksNLE/Review&Approve/VFX(Hybrid,onpremisesorincloud)
RawAssetsSecondaryCopy
AmazonGlacierinanotherRegion
RawAssetsPrimaryCopyAmazonGlacier
Finished(GoldenCopy)ArchivePrimaryCopyAmazonGlacier
AssetM
anagem
ent(DA
M/M
AM)
(Hybrid
,onprem
iseso
rinclou
d)
ActiveArchiveAmazonS3(Standard)
ActiveArchiveAmazonS3
(InfrequentAcccess)
Secondary&TertiaryCopiesAmazonGlacierinanotherRegion
Asecondaryoffsitecopyoftheactivearchive– usuallycoldUsedforDRpurposes
Med
iaSup
plyCh
ain
(Hybrid
,onprem
iseso
rinclou
d)
B2BDe
liveryoverDXorInternet
AmazonGlacieringest
Affiliates/ThirdPartyVendors
ingest
AWSDirectConnect
AWSSnowball
S3TransferAcceleration
(PrimaryCopy)AWSRegion1
(Secondary&TertiaryCopies– optional)AWSRegion2oron-premises
Glacier
S3InfrequentAccess
S3StandardLifecycleM
gmt
ProductionArchive
Availability Zone A
Private subnet Private subnet
AWS region
VirtualPrivateGateway
ContentValue-addService
ContentValue-addService
Availability Zone B
Amazon S3
VPC
VPNconnection
VPCEndpointsNoIGWNoNATNopublicIPsFreeRobustaccesscontrol
Customernetwork
Access Control with S3 VPC Endpoint
corporate data center AWS cloud
users
Content
Servers
disk
tape storage
Processing Layer
Amazon S3
Amazon EBS
Amazon Glacier
KMS/HSMClient side
encryption
roleIAM
EncryptedContent
roleDirectConnect
S3VPC
End
point
HybridMediaNetwork
STORAGEPERFORMANCE
EBSVolumesSSDBased HDDBased
gp2GeneralPurposeSSD
io1ProvisionedIOPSSSD
st1ThroughputOptimizedHDD
sc1ColdHDD
Baseline: 3IOPSperGBupto10,000Burst:3000IOPS(forvolumesupto1TB)Throughput:160MB/sLatency:SingledigitmsCapacity:1GBto16TB
UsageBootVolumesBursty workloads(Rendering)Databases(Transactional)Lowlatencyoperations(Streaming)
Baseline: 100IOPSupto20,000Throughput:320MB/sLatency:SingledigitmsCapacity:4GBto16TB
UsageSustainedIOPSCriticalApplicationswithIOPSGuarantee(noburst)
Baseline:40MB/sperTBupto500MB/sBurst:250MB/sperTBupto500MB.sCapacity:500GBto16TB
UsageLargeBlock,highthroughputsequentialTranscoding,QC,PackagingRendering
Baseline:12MB/sperTBupto192MB/sBurst:80MB/sperTBupto250MB.sCapacity:500GBto16TB
UsageSequentialLogging,ActiveArchive/Backup
ObjectstoreasProcessingtier
Partition1
Partition2 Partitionn
AnS3BackendStorage
Auto-scaled/HAFront-end
TCP
DistributedKey-map
Optimizations
ReplicationacrossmultipleAvailabilityZones
NoFS– everythingGET/PUT/DELETE• Writeonce– Readmany• Globallyuniquenamespaces(objectnamesuniquewithinabucket)
RequestsPerSecond(>300)• Partitioning• RandomPrefixPutsvs.Gets• CDN– GETheavyworkloadsNetworkThroughput• TCPWindowScaling• TCPSelectiveAcknowledgements• S3SessionUsage• Multi-partUploads• BufferedWrites• RangeGETsConsistency/CommitLatency• ReadafterWrite• ObjectsreplicatedacrossmultipleAZs• Notsuitabletodayforworkloadswith<MSlatency/IOrequirement
Client(s)
Client-sidebandwidth
Elastic File System (EFS) • DesignedtosupportPeta-Bytescalefilesystems• Throughputscaleslinearlytostorage
– Burst:100MB/secperTBofstoragefor12hourseachday– Baseline:50MB/secperTBofstoragecontinuously
A10TBfilesystemcanburstupto1GB/sfor12hourseachdayanddrive500MB/secaggregateread/writethroughputcontinuously.
• SamelatencyspecacrosseachAZ• ThousandsofconcurrentNFSconnections• WorksgreatforLargeI/Osizes• Payforonlywhatyouusenotwhatyouprovision• Managedwithmulti-copydurability
RenderFarms
Bigdataanalytics
TranscodeFarms
Homedirectories
Contentmanagement
WebservingMetadata-intensive
jobs
HighthroughputandparallelI/O
LowlatencyandserialI/O
WorkloadSpecificStorageObject Store(AmazonS3)
BlockStore(Amazon EBS)
SharedFS(Amazon EFS)
Performance
Per OperationLatency Variable Lowest,Consistent LowConsistent
ThroughputScale Multiple GBspersec Single GBspersec MultipleGBsper sec
DataAvailability/Durability
AcrossMulti-AZ Redundantwithin anAZ AcrossMulti-AZ
Characteristics
Access 1-1000’sofClients AsingleEC2 instance 1-1000’sofClients
UseCases
Active ArchiveandlifecycletoColdArchiveMedia Processing(Transcoding,rendering,QC,Farms)dependingonApplicationconformity
DatabasesBoot VolumesMedia Processing(Transcoding,rendering,QC,Farms)SingletonStreaming
DataAnalyticsMedia Processing(Transcoding,rendering,QC,Farms)DistributedContentManagementWebServing/Publishing
SOLUTIONS
AWSImport/Export
S3multi-partAPIForSource,Renditions,MetadataSidecarFiles
Ingest
AWSElasticBeanstalk
DAMWebService
CloudFrontDistribution
AmazonDynamoDB
AmazonCloudSearch
DAMCatalog
Rendition WorkersEC2ASG
AmazonSNSTopic
AmazonSQSQueueRenditionJobs
AmazonSQSQueueMetadataProcessingJobs
S3Notification
ElasticTranscoderProxy/
Thumbnail/RenditionsGeneration
Processing
Metadata WorkersEC2ASG
SimpleMediaSupplyChain
AWS Import/ Export
S3 multi-part APIS3 as backend storage for Content Files
accessible to other processing tasks
Amazon Elastic Transcoder
S3 Notification
Trigger a Lambda Function to Start a transcoding job
Ingest
S3 Notification
Lambda function to generate a signed
URL to share the file
Update CMS or Metadata
Serverless MediaSupplyChain(concept)
§ Expand computecapacity of Manager node(s)
§ 100ms billing & short run-time
§ Ideal for chunk storage, pre- & post-processing
§ Don’t forget binary encapsulation!
https://github.com/binoculars/aws-lambda-ffmpeg
LambdaPreandPostProcessing
Ahigh-levelMediaSupplyChaininthecloud(concept)
EC2,ETScanrequestthedata-keyonbehalfofcustomerProvideCPKforS3encryptionatrest
AWSImport/Export
S3multi-partAPIAmazonS3ForSource,Renditions,Metadata,SidecarFiles
Ingest
AWSElasticBeanstalk
CloudFrontDistribution
DynamoDB CloudSearch
S3Notification
Content Processing
AmazonSWF,SQS,SNSandAWSLambdacanbeusedtoorchestratetheprocessingworkflowtasks
CloudFrontDistribution
UserLogs
AmazonDynamoDB
AmazonRedshift
Users Connected Devices
Analytics and Recommendations Engine
AmazonEMR
AmazonDataPipeline
ElasticTranscoder
EC2ASGRendition Workers
EC2ASGMetadata Workers
MetadataProc queueRenditionsQueue
Amazon Elastic Transcoder
Rights Management
EC2ASGELB
DRM WrapEncryptPackageWatermarkQC
EC2ASG
AmazonSNSTopic
DAM Web Service
PersonalizationRecommendations
EC2ASG RDS
Front End Web Application
DAM CatalogContentArchive
AmazonGlacier
Key Management Service Content owner provides the master key
BI & ReportingUsers Web Browser
Content Owner
IMFPACKAGINGStack:s3-ta/snowball/?Partners:teradek /nuke(vr/360)/?
Cloud Based Editing
• NVIDIAGPUbasedEC2instances• Teradici PCoIP• AmazonAppStream/NiceDCV• AmazonWorkspaces
Artistapplications
Artist workstation(Dumb Client)
Remote Application running on a G2 instance
G2
ThankYou!