bbc's graphdb (formerly owlim) aws cloud migration
DESCRIPTION
BBC's presentation for the GraphDB user groupTRANSCRIPT
Moving OWLIMto Amazon Web Services
Moving OWLIMto Amazon Web Services
GraphDB
Introduction to LDP
● Stands for Linked Data Platform● Building APIs and Tools for semantic
publishing at the BBC● Team of 8-12 people (5-8 developers) ● Based in London (moving to Salford)
Introduction to LDP
● Tagging and surfacing content
Why move to AWS?
● Started with taking ownership of Owlim● BBC shift to move to AWS● Loss of key people● Freedom with responsibility
Current Architecture
Cloud Architecture
Keeping in sync
OpsWorks
● Part of AWS● Easily deploy things to EC2
OpsWorks
● Define infrastructure in a stack● The stack is comprised of layers● Each layer has a role (eg. master, slave)● Layers contain a number of instances
OpsWorks – Advantages
● Conceptually, fairly simple.● Entire stack defined as JSON.
○ Can be (and is) version controlled.
● Very easy to clone an entire stack.○ We use this to spin up ‘experimental’ environments.○ Next step: blue/green deployments?
OpsWorks – Disadvantages
● It can be a little slow.● Chef support can be tricky to debug.
● “Infrastructure as code”● Cookbook, made up of recipes.
○ Each recipe installs/configures a specific thing.
● Each layer has a set of recipes that define it.
Chef
include_recipe 'tomcat::default'
cookbook_file 'owlim-production.license' do path File.join(node[:owlim][:basedir], 'owlim.license') mode 0644 owner 'tomcat' group 'tomcat'end
execute 'create_default_cluster_repository' do command SystemRepository.create_repo 'default-cluster' not_if SystemRepository.repo_exists? 'default-cluster' action :nothingend
Lifecycleevents
Recipes for each event type
● A common and well-documented tool.○ And it’s Ruby, a skill we possess.
● Can re-use existing cookbooks.○ eg. Tomcat, Yum.
● Should be platform-agnostic.○ Can theoretically use the same Chef recipes against
any OS, if written well.
Chef – Advantages
● OpsWorks support is a little lacking.○ Using Berkshelf for dependencies is tricky.○ Many of the error messages are a bit cryptic.
Chef – Disadvantages
What next?
What next?
● Autoscaling○ Not supported by OpsWorks.○ For now: predict spikes, use time-based instances.
What next?
● Autoscaling○ Not supported by OpsWorks.○ For now: predict spikes, use time-based instances.
● Backups○ We still do these from our DCs.
What next?
● Autoscaling○ Not supported by OpsWorks.○ For now: predict spikes, use time-based instances.
● Backups○ We still do these from our DCs.
● Chef improvements○ The recipes do too much at the moment.
What next?
● Improve Chef recipes○ The recipes do too much at the moment.○ Example: cluster management.
Questions?
