itv& bashton
TRANSCRIPT
Building a Shared Pipeline for Content DistributionAWS Media and Entertainment Cloud Symposium
23rd March 2016
Jim Taylor & Sam Bashton
Who are we? - ITV/Jim
● Jim Taylor - Technical Product Owner, VOD Distribution● [email protected]
ITV is the biggest commercial television network in the UK. We love telly, we’re passionate about content and most importantly we understand our audiences.
As a growing international company, we’re commercially minded and have a global focus. We’re all about pushing the boundaries and being innovative.
Who are we? - Sam Bashton
VOD Distribution - Push model
ITV
Partner 1Push content
Partner 2Push content
Scale issues with:● Bandwidth● Processing● Unique delivery methods● Redelivery requests
...but same content to each partner!
Partner 3
...
Business problem
How can we continuously distribute large volumes of high quality video to several destinations, without spiralling distribution costs?
● Partners need high quality mezzanine so they can transcode to multi-platform● ITV has about 200 hours/week of Catch Up ≈ 6TB/week● 1Gbps link per partner = high cost/long lead time● Push model unsustainable as distribution points increase
Solution - Cloud delivery - pull model
Use AWS to deliver once using a templated delivery and offer partners a fixed menu via a pull model
ITV AWSITV
Partner 1 AWS
Partner 1
Upload once
Downloadcontent
Notifications
Notifications
Partner 2
Partner 2 AWS
Notifications
Downloadcontent
Notifications
Solution - Package
Shared:Series/Episode Images 16:9 & 4:3HD (XDCam50) or SD (IMX30) asset
Private:Metadata file containing● Tech metadata, asset locations● Licence details● Editorial metadata
Technology used
ITV AWS
ITV
Partner AWS
Partner
Signiant Flight
SNS
Amazon S3AWS CloudFormationAmazon LambdaELK stack on EC2
Splunk
Partnership - ITV & Bashton
Partner Setup: AWS
● Each partner has their own AWS account● Files in the bucket use ‘requester pays’ billing - partners pay for data transferred
outside of AWS● Partners notified via SNS when new content is available
Partner setup: SNS
● ‘Simple Notification Service’● Push notifications
○ Webhooks○ SQS (Simple Queue Service)○ Lambda
Lambda
● Code without a server*● Trigger simple tasks● Used as part of shared pipeline
○ Receives notifications when videos are uploaded○ Looks up partners, sends SNS notifications to those that are relevant
Test environments
● Creating resources via web console fine for trying things out● Proof of concept environment created in this way● In production, we need to build multiple copies of the environment● Changing everything manually quickly becomes tiresome
CloudFormation
● CloudFormation to the rescue!● Infrastructure as Code● Specify what an environment looks like in a JSON text file
○ Networks, subnets○ Firewall rules○ S3 Buckets○ Instances, volumes○ Access control
● Commit to source control (GIT)
CloudFormation: Rich documentation
● Every change has a commit message● History of every setting can be viewed back to the beginning of the project
○ With comments about every change to say why they were made
CloudFormation: Testing
● Change a template● Apply to test environment● Run the same change against staging, production● Confidence that every change will work, and that all tests are valid
● Lambda - code without servers*● Code still needs to be tested● For software deployed to instances we use Jenkins for CI
○ Automated Testing○ Deployment and ‘promotion’ between environments (test->stage->prod)
● What to use with Lambda?
Software Testing
● Jenkins● Code committed to git repo● Jenkins checks out code from git● Runs automated tests● Deploys to environment● Deploys to ‘next’ environment (test->stage) at the click of a button
Jenkins + Lambda
Future plans
ITV AWSITV
Partner 1 AWS
Partner 1
Upload once
Downloadcontent
Notifications/metadata
Notifications
Partner 2
Partner 2 AWS
Notifications
Downloadcontent
Notifications/metadata
Metadata API
Added ourselves as a Shared Pipeline partner!
Add access services
Cloud transcoding & stitching
● Deliver a ‘reference’ test system for partners - easier to look at a working example than documentation● Issues using “requester pays” in egress - make this clearer● Avoid logging overload
What would we do differently?
● Delivery time of project - 5 months● Reduced time to deliver a new partner from 3-6 months to 1-2 months
○ ...providing they use our system● Reduced cost to deliver a new partner by up to 75%
○ ...providing they use our system● Can start testing within a week
Success? - Yes!