migrating a running service to aws - drupalcon · “aws cloudformation is a service that helps you...
TRANSCRIPT
![Page 1: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/1.jpg)
![Page 2: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/2.jpg)
Migrating a running service to AWS
Nick Veenhof
DevOps Track
https://events.drupal.org/barcelona2015/sessions/migrating-running-service-mollom-aws-without-service-interruptions-and-reduce
Ricardo Amaro
![Page 3: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/3.jpg)
@Nick_vh
Ghent
Barcelona
Boston
Lisbon
+8 Years in Drupal
Search++
4 years at Acquia
Principal Software Engineer
The Developer
![Page 4: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/4.jpg)
So good to be back...
![Page 5: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/5.jpg)
Mollom● Detecting Spam from Ham
○ Reducing your moderation efforts● Very fast response times (avg under 50 msec)● Fully Managed SAAS service● Free and paid version● Downtime means unprotected sites, which is
bad for reputation and adoption● Built in Java●
![Page 6: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/6.jpg)
@ricardoamaro
Portugal
Lisbon
Drupal Community
Family
+7 years Drupal
90’s Linux Adopter
4 years at Acquia
Senior Tier2 Ops Engineer
The Opsian
![Page 7: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/7.jpg)
Roses, Roses everywhere...
Pre-Migration
![Page 8: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/8.jpg)
How we got the news...
”Operations is now responsible for Mollom servers being up or down, and basic services being available (such as SSH, apache, nginx, etc). If further problems persist above the services layer into the application layer, Ops is to escalate to Mollom Engineering immediately. “
![Page 9: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/9.jpg)
Highly complex piece of engineeringon top of non-cloud hosting.
![Page 10: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/10.jpg)
![Page 11: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/11.jpg)
?
?
?
?
?
?
![Page 12: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/12.jpg)
20 million http requests per day8 million of spam requests / dayworst day: 300+ alerts...
![Page 13: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/13.jpg)
One clear guidance example...Question: “Is disk usage above 95%?”
Answer: “Remove all files that start with the same prefix as the data file...”
rm -rf Mollom-session_history-he-78609-*
“... and restart Cassandra”
/etc/init.d/cassandra restart
![Page 14: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/14.jpg)
Look before you leap
Architecture Exercise
![Page 15: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/15.jpg)
Exercise
● One row = One Component.● I need to be able to “take down”
someone and still be up and running
● Order is important. I will be a site visitor, so I want you to start from the front to the end.
![Page 16: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/16.jpg)
Exercise
● Reverse Proxy (VARNISH)● Web Server (WEB)● DNS● Load Balancer (LB)● Database (DB)● Object Caching (Cache)
![Page 17: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/17.jpg)
Ephemeralism
![Page 18: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/18.jpg)
Eye-opener
Describes the optimal environment and how this relates to reality. Warning, there is no perfect.
A very digestible book for designing distributed systems. This book exposes software patterns that every cloud infrastructure engineer should know.
The Practice of Cloud System Administration
![Page 19: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/19.jpg)
CAP Theorem
It is impossible for a distributed computer system to simultaneously
provide all three of the following guarantees:
● Consistency (all nodes see the same data at the same time)
● Availability (a guarantee that every request receives a response
about whether it succeeded or failed)
● Partition tolerance (the system continues to operate despite
arbitrary partitioning due to network failures)
The Practice of Cloud System Administration
![Page 20: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/20.jpg)
![Page 21: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/21.jpg)
Cloudformation
“AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less time managing those resources and more time focusing on your applications that run in AWS.”
Stackin’ it up
![Page 22: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/22.jpg)
Cloudformation
● AutoScaling Groups (ASG)● Elastic Load Balancer (ELB)● Elastic Compute 2 (EC2)● AMI (VM of Ubuntu 14.04)● Java
Stackin’ it up
![Page 23: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/23.jpg)
Cloudformation
![Page 24: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/24.jpg)
Virtual Private Cloud (VPC)
Amazon VPC lets you provision a logically isolated section of the Amazon Web Services (AWS) Cloud where you can launch AWS resources in a virtual network that you define.
Isolation isn’t bad, mkay?
![Page 25: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/25.jpg)
Virtual Private Cloud (VPC)
● Private Subnets● Internal Load Balancers● Public IP addresses● Security Groups
Isolation isn’t bad, mkay?
![Page 26: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/26.jpg)
Virtual Private Cloud (VPC)Isolation isn’t bad, mkay?
![Page 27: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/27.jpg)
Relational Database ServiceIt’s not a triptych
● Fully Managed● H/A possible● Within your VPC, non public● Option to use MariaDB, Postgres, Aurora, …● Highly configurable
![Page 28: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/28.jpg)
Relational Database ServiceIt’s not a triptych
![Page 29: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/29.jpg)
AWS says: “DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability.”
We read: Cassandra without maintenance (and serious reduction in alerts)!
DynamoDBDatawarehousing for the masses
![Page 30: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/30.jpg)
● Really fast● Fully Managed● No TTL, so we use rotation based tables● Pricy, but maintenance-free.
DynamoDBDocument storage for the masses
![Page 31: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/31.jpg)
● Dynamic DynamoDB○ https://github.com/sebdah/dynamic-dynamodb
● Dynamic DynamoDB Manager○ https://github.com/Mollom/dynamic-dynamodb-manager
DynamoDBDatawarehousing for the masses
![Page 32: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/32.jpg)
Elastic Load Balancing (Amazon ELB) automatically distributes incoming application traffic across multiple Amazon EC2 instances in the cloud.
EC2 = a VM, hosted on AWS’s supervisor system.
EC2 + Load BalancingVMception
![Page 33: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/33.jpg)
EC2 + Load BalancingVMception
Elastic Load Balancing (Amazon ELB) automatically distributes incoming application traffic across multiple Amazon EC2 instances in the cloud.
EC2 = a VM, hosted on AWS’s supervisor system.
![Page 34: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/34.jpg)
● Linux as you know it● AMI-based● Can disappear or crash. Don’t try to do non-stateless
apps.● Triggers to auto-scale (read: add/remove a ec2
machine) on predefined inputs.● Update scheme involves disposable EC2 instances
EC2 + ELBVMception
![Page 35: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/35.jpg)
EC2 + ELBVmception
![Page 36: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/36.jpg)
EC2 + ELBVmception
● Access Logging● Health Check● H/A (multiple zones)● Connection Draining● IPTables-like functionality● Multiple listeners (read: port
forwarding)● SSL Termination (port 443, check
cert and forward to HTTP port 80, eg SSL termination at the load balancer level)
![Page 37: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/37.jpg)
● No puppet● No Chef● No Ansible● Everything is fully rebuilt on launch, every update is a
new machine● We do not update single packages, we remove and add
machines. ● Allows for returning to a point in time as the full “state” is
preserved. Note: Data backups are still necessary if this is required.
EC2 + ELBSo puppet or chef right?
![Page 38: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/38.jpg)
● AWS Cloudwatch● Diamond + Custom Handlers
○ https://github.com/python-diamond/Diamond● StatsD / Graphite● Creating AWS Cloudwatch alarms
per instance for non AWS-specific services
MetricsEver seen a cloud with a watch?
![Page 39: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/39.jpg)
● Nagios + Pagerduty● Integration with Cloudwatch● Ordering of alerts, to help those who are
on-call to prioritize.
AlarmsEvery Pager has its duty
![Page 40: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/40.jpg)
Returning a different IP based on your region
DNS
![Page 41: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/41.jpg)
● Using all these techniques to “hand off” unknown to SAAS services we were able to drastically reduce the alerts in our system.
● We no longer have frustration that only 10% of our time can go into development.
● Chaos Monkey is welcome, fully ephemeral.
ResultHappy Devving, Happy Opsing
![Page 42: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/42.jpg)
Questions?
![Page 43: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/43.jpg)
Sprint: Friday
https://www.flickr.com/photos/amazeelabs/9965814443/in/fav
es-38914559@N03/
Sprint with the Community on Friday.
We have tasks for every skillset.
Mentors are available for new contributors.
An optional Friday morning workshop for first-time sprinters will help you get set up.
Follow @drupalmentoring.
![Page 44: Migrating a running service to AWS - DrupalCon · “AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less](https://reader034.vdocuments.us/reader034/viewer/2022042223/5ec973d79259b21db32deec9/html5/thumbnails/44.jpg)