shipping rails on docker - leanpubsamples.leanpub.com/rails-on-docker-sample.pdfrequirements this...

33

Upload: others

Post on 21-May-2020

1 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon
Page 2: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

Shipping Rails on DockerDeploying Ruby on Rails Applications with Amazon EC2Container Service

Pablo Acuña

Dieses Buch wird verkauft, unter http://leanpub.com/rails-on-docker

Diese Version wurde veröffentlicht am 2016-02-21

This is a Leanpub book. Leanpub empowers authors and publishers with the Lean Publishingprocess. Lean Publishing is the act of publishing an in-progress ebook using lightweight tools andmany iterations to get reader feedback, pivot until you have the right book and build traction onceyou do.

© 2015 - 2016 Pablo Acuña

Page 3: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

Contents

Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii

Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Setting up the project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2Getting the source code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

The Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4Creating the cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4Launching the EC2 instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Our first task definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Our first ELB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Page 4: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

AcknowledgmentsI’d like to thank all of the people that for any reason have taken part in my learning journey.Especially Becky, for always being there for me and supporting all of my ambitions.

i

Page 5: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

PrefaceContainers are the new thing. After several years fighting for their place in the software developmentworld, Docker is now positioned as one of the most promising tendencies. Using containers we canhave a database server, a search engine or a web application, running in minutes. This has allowedto bring the system administration world closer to the developers, giving even more power to thenew DevOps generation.

Nevertheless, there are still some dark parts in the containers world. Especially in the deploymentstep of the software development process. You’ll find several technologies out there battling fortaking this place. Of course Amazon Web Services couldn’t be out of this battle and launched theirservice called Amazon EC2 Container Service. Using this service, it’s possible to create productionenvironments that are robust, safe and scalable without losing the power of the classic resourcesfrom AWS such as Elastic Load Balancing, Auto-Scaling, Security Groups, Virtual Private Cloud,etc.

In this book I’ll show you how to deploy a complex Ruby on Rails application provided by theelasticsearch-rails repository¹ as an example for connecting elasticsearch with Rails. We’ll deploythis application using Amazon EC2 Container Service. This is not going to be a useless and non-realistic application, it’s going to have dependencies widely used in big applications out there. We’llconnect our application with a MySQL database server, a search engine provided by ElasticSearch,Redis and Sidekiq for managing background jobs, and also a cache layer using Memcached.

Since ECS is a new technology and it’s constantly changing and improving, I’ve decided to showall the examples by using the AWS Command Line Interface and also the Web Console. Sometimesyou’ll find one more useful than the other, so it’s important to master both of them.

¹https://github.com/elastic/elasticsearch-rails

ii

Page 6: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

RequirementsThis book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon WebServices. Is not necessary for experts, but there are some basic concepts I’m not going to spend timetalking about. If you don’t know the basics about AWS, I recommend you to go to the getting startedpage², create an account and start playing with the web console.

Some of the things that you’ll need in order to follow the instructions:

• A GitHub³ account and Git properly configured on your machine• An AWS⁴ account• API credentials for your AWS account• The AWS CLI⁵ tool• A DockerHub⁶ account

You’re also going to need Docker and Docker Compose, but those tools will be installed on the virtualmachine that comes with the source code. From there, you’ll be able to run Docker commands andpush the image to your DockerHub account.

²https://aws.amazon.com/getting-started/³http://github.com⁴https://aws.amazon.com⁵https://aws.amazon.com/cli⁶https://hub.docker.com

1

Page 7: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

Setting up the projectThis is not a book about Ruby on Rails development, so we are not going to build somethingfrom scratch. Instead, we’re going to deploy a sample application from the elasticsearch-rails gemrepository⁷. This application is an example of how to integrate Elasticsearch with Ruby on Rails.

Our final application will look like this:

Complete Web Application

We’re basically deploying a faceted browser for articles from the New York Times. These articlesare seeded into the database and indexed into an Elasticsearch cluster. It also comes with a workerthat sends jobs to a Sidekiq queue using redis. Finally, I’ve added a cache storage to the applicationusing memcached just for making it even more realistic.

This application is distributed through several Rails templates available in the official repository ofthe gem. I just made some small modifications in order to dockerize the application and also pushedthe final result to a public repository under my GitHub account. We will use Vagrant⁸ for creatingan environment ready for running the dockerized application. This way you don’t have to installanything except Vagrant and its dependencies on your local machine.

If you don’t have Vagrant installed, go to the download page⁹ and download the proper package foryour system. You’re also going to need a virtualization software such as VirtualBox¹⁰.

⁷https://github.com/elastic/elasticsearch-rails/tree/master/elasticsearch-rails⁸https://www.vagrantup.com/⁹https://www.vagrantup.com/downloads.html¹⁰https://www.vagrantup.com/downloads.html

2

Page 8: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

Setting up the project 3

Getting the source code

First, clone the final application repository:

git clone [email protected]:pacuna/searchapp.git

This repository contains all the necessary files for testing the application locally and for deployingit with Amazon ECS.

We also need a DockerHub account for hosting the Docker image for our project. If you don’t havean account, go to the DockerHub¹¹ sign up page and fill out the form with your information.

Once you have your DockerHub account ready, create a new public repository and name itsearchapp. We are going to push our container images to that repository in order to pull themfrom ECS.

Finally, you’ll need the AWS CLI tool for accessing the Amazon AWS api. For setup instructions,visit this link¹²

Once you installed this tool, you can configure it to use your credentials by running

$ aws configure

¹¹https://hub.docker.com/¹²https://aws.amazon.com/cli/

Page 9: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

The ClusterCreating the cluster

Let’s start by creating the central piece in the ECS architecture, the cluster. The cluster is goingto be a pool of resources we can use to launch tasks or services. This pool will contain regular EC2instances that run the Amazon ECS agent. We can add or remove instances from the cluster anytimewe want.

We can create the cluster using one simple api command. The only parameter that we need is thename:

$ aws ecs create-cluster --cluster-name "searchapp"

Output:

{

"cluster": {

"status": "ACTIVE",

"clusterName": "searchapp",

"registeredContainerInstancesCount": 0,

"pendingTasksCount": 0,

"runningTasksCount": 0,

"activeServicesCount": 0,

"clusterArn": "arn:aws:ecs:us-east-1:387705308362:cluster/searchapp"

}

}

The cluster was created successfully, and as you can see from the output, it has zero EC2 containerinstances registered, zero tasks running and no services.

For creating the cluster using the ECS Web Console, go to the ECS Container Service panel. If this isthe first time you use ECS, you’ll have to skip the wizard for creating the test project. For this, justclick cancel. Now you should see the button for creating a new cluster:

4

Page 10: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

The Cluster 5

New Cluster

If you click the button, you can create the cluster by giving the name.

Creating a new cluster

The idea behind the ECS cluster is we can forget about where and how our tasks and services will bedeployed. We just have to launch them and the ECS scheduler will take care of allocate them inside

Page 11: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

The Cluster 6

the cluster. The only thing that we have to do now is add some EC2 instances to the cluster.

Launching the EC2 instances

Now that we have our cluster, we can launch a couple of instances and register them in the cluster.Some of the things we’re going to need:

• A Key Pair for ssh access• A Security Group• An Instance Role

Let’s start by creating the key pair. We can create the key and send the output to a .pem file by using:

$ aws ec2 create-key-pair --key-name ClusterKeyPair --query 'KeyMaterial' --output text > ClusterKey\

Pair.pem

Set the permissions for the file:

$ chmod 400 ClusterKeyPair.pem

And move the file to your ssh keys directory or some place safe. Remember that this file will benecessary for accessing the instances via ssh.

If you prefer to use the Web Console, go the EC2 panel and click on Key Pairs in the left sidebar

Page 12: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

The Cluster 7

Creating a Key Pair

Then click on Create Key Pair and fill out the form with ClusterKeyPair or the name you’d like touse:

Creating a Key Pair

For the instances security group, we can use the security group that comes with our the default VPC.Now every account should have a default VPC. You can use this command for getting your VPCsinformation:

$ aws ec2 describe-vpcs

Output:

Page 13: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

The Cluster 8

{

"Vpcs": [

{

"VpcId": "vpc-120cd476",

"InstanceTenancy": "default",

"State": "available",

"DhcpOptionsId": "dopt-5bf9023e",

"CidrBlock": "172.31.0.0/16",

"IsDefault": true

}

]

}

Let’s get our default VPC id and its Security Group Id and save these to variables so we can usethem later:

$ read GroupId VpcId <<< $(aws ec2 describe-security-groups --group-names default --query "SecurityG\

roups[*].{GroupId:GroupId,VpcId:VpcId}" --output=text)

Now if you echo the variables you’ll see that we have the information saved:

$ echo "$GroupId | $VpcId"

Output:

sg-c70ff2a1 | vpc-120cd476

This security group by default allows all traffic between resources that belong to the group. That’sgood but we also going to need access to the port 22 for getting SSH access to our instances. We’lldo that later.

Our instances are also going to need an IAM Role in order to allow certain actions on the ECSinfrastructure. Lucky for us, Amazon AWS has already defined the set of policies we need, in anative policy. We only have to create the Role, attach this native policy to that Role, and then addthat Role to an instance profile so we can assign the Role while the instances are being launched.

Inside the deploy folder, you’ll find an standard trust policy for creating the Role. The name of the fileis AmazonEC2ContainerServiceforEC2Role-Trust-Policy.json. Let’s create our ecsInstanceRoleby running this command (use the correct policy path for your case):

$ aws iam create-role --role-name ecsInstanceRole --assume-role-policy-document file://policies/Amaz\

onEC2ContainerServiceforEC2Role-Trust-Policy.json

Output:

Page 14: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

The Cluster 9

{

"Role": {

"AssumeRolePolicyDocument": {

"Version": "2012-10-17",

"Statement": {

"Action": "sts:AssumeRole",

"Effect": "Allow",

"Principal": {

"Service": "ec2.amazonaws.com"

}

}

},

"RoleId": "AROAIIH2CKLXDXCQ3YS4Q",

"CreateDate": "2015-10-08T23:01:52.285Z",

"RoleName": "ecsInstanceRole",

"Path": "/",

"Arn": "arn:aws:iam::387705308362:role/ecsInstanceRole"

}

}

The name of the managed policy we need is AmazonEC2ContainerServiceforEC2Role. Let’s querythe API in order to more information about this policy:

$ aws iam list-policies --query 'Policies[?PolicyName==`AmazonEC2ContainerServiceforEC2Role`]'

Output:

[

{

"PolicyName": "AmazonEC2ContainerServiceforEC2Role",

"CreateDate": "2015-03-19T18:45:18Z",

"AttachmentCount": 1,

"IsAttachable": true,

"PolicyId": "ANPAJLYJCVHC7TQHCSQDS",

"DefaultVersionId": "v2",

"Path": "/service-role/",

"Arn": "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role",

"UpdateDate": "2015-08-17T23:33:23Z"

}

]

We just need the policy ARN identifier for attaching it to our ecsInstanceRole:

$ aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerSer\

viceforEC2Role --role-name ecsInstanceRole

Now let’s create the instance profile:

Page 15: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

The Cluster 10

$ aws iam create-instance-profile --instance-profile-name ecsInstanceRole

Output:

{

"InstanceProfile": {

"InstanceProfileId": "AIPAI5CUZHG54Y7P6V5SG",

"Roles": [],

"CreateDate": "2015-10-08T23:04:44.535Z",

"InstanceProfileName": "ecsInstanceRole",

"Path": "/",

"Arn": "arn:aws:iam::387705308362:instance-profile/ecsInstanceRole"

}

}

And finally let’s attach the Role to that profile:

$ aws iam add-role-to-instance-profile --instance-profile-name ecsInstanceRole --role-name ecsInstan\

ceRole

If you want to create this resources using the Web Console, you can go the IAM panel, click on Roleand create a new Role:

New Role

Page 16: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

The Cluster 11

Then, in the Role Type section, search for Amazon EC2 Role for EC2 Container Service and clickselect:

Role Type

Next, attach the policy:

Attaching a policy

And finally click on Create Role. This process, unlike using the AWS CLI, will create an InstanceProfile automatically.

Page 17: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

The Cluster 12

Role Summary

Now we are ready to launch the instances for the cluster.

The AMI that we’ll use it’s called amzn-ami-2015.03.g-amazon-ecs-optimizedwith and id of ami-4fe4852a. This image comes ready and optimized for ECS.

We are going to launch two instances of type t2.micro and use the keys and instance Role that werecreated previously, one of our subnets and the security group for the default VPC.

For getting the available subnets let’s use the describe-subnets command and filter the result byvpc-id using the variable that contains our default VPC id:

$ aws ec2 describe-subnets --filters "Name=vpc-id,Values=$VpcId" --query "Subnets[*].SubnetId"

Output:

[

"subnet-39706b4e",

"subnet-fa724fa3",

"subnet-4660426d",

"subnet-477ea97a"

]

I have 4 available subnets. Let’s choose the first one for launching the instances and also let’s recordthe id into a variable:

Page 18: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

The Cluster 13

$ SubnetId=`aws ec2 describe-subnets --filters "Name=vpc-id,Values=$VpcId" --query "Subnets[0].Subne\

tId" --output=text`

$ echo $SubnetId

Output:

subnet-39706b4e

Great!. Now, let’s talk about user-data. The user-data parameter allows us to add some initialconfiguration to the instances. In our case, we are passing the configuration for registering theinstances in our searchapp cluster. The user-data.sh file contains these two lines:

#!/bin/bash

echo ECS_CLUSTER=searchapp >> /etc/ecs/ecs.config

You can find this file inside the deploy folder of the source code.

Now we finally can run the command for launching the instances. If you didn’t save your securitygroup and subnet ids into variables, remember to replace them in the command:

$ aws ec2 run-instances --image-id ami-4fe4852a --count 2 --instance-type t2.micro --key-name Cluste\

rKeyPair --security-group-ids $GroupId --subnet-id $SubnetId --iam-instance-profile Name=ecsInstanc\

eRole --associate-public-ip-address --user-data file://user-data/user-data.sh

Output:

{

"OwnerId": "387705308362",

"ReservationId": "r-202269dd",

"Groups": [],

"Instances": [

{

"Monitoring": {

"State": "disabled"

},

"PublicDnsName": "",

"RootDeviceType": "ebs",

"State": {

"Code": 0,

"Name": "pending"

},

"EbsOptimized": false,

"LaunchTime": "2015-10-10T02:13:50.000Z",

"PrivateIpAddress": "172.31.1.132",

"ProductCodes": [],

Page 19: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

The Cluster 14

"VpcId": "vpc-120cd476",

"StateTransitionReason": "",

"InstanceId": "i-8272af56",

"ImageId": "ami-4fe4852a",

"PrivateDnsName": "ip-172-31-1-132.ec2.internal",

"KeyName": "ClusterKeyPair",

"SecurityGroups": [

{

"GroupName": "default",

"GroupId": "sg-c70ff2a1"

}

],

"ClientToken": "",

"SubnetId": "subnet-39706b4e",

"InstanceType": "t2.micro",

...

You can poll the status of the launched instances by using the instance-id that you got from theoutput for any of the instances:

$ aws ec2 describe-instance-status --instance-ids i-8272af56

Output:

{

"InstanceStatuses": [

{

"InstanceId": "i-8272af56",

"InstanceState": {

"Code": 16,

"Name": "running"

},

"AvailabilityZone": "us-east-1d",

"SystemStatus": {

"Status": "initializing",

"Details": [

{

"Status": "initializing",

"Name": "reachability"

}

]

},

"InstanceStatus": {

"Status": "initializing",

"Details": [

{

"Status": "initializing",

"Name": "reachability"

}

Page 20: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

The Cluster 15

]

}

}

]

}

From the output you can see that this instance is being initialized.

After a few minutes, try running this other command for polling the state of the cluster and see ifthe instances were correctly registered:

$ aws ecs describe-clusters --clusters searchapp

Output:

{

"clusters": [

{

"status": "ACTIVE",

"clusterName": "searchapp",

"registeredContainerInstancesCount": 2,

"pendingTasksCount": 0,

"runningTasksCount": 0,

"activeServicesCount": 0,

"clusterArn": "arn:aws:ecs:us-east-1:387705308362:cluster/searchapp"

}

],

"failures": []

}

Great, we have 2 instances registered in our cluster.

Now, I don’t recommend to launch the instances via the EC2 Web Console. As your system scales,you’ll need to add more and more instances to your cluster, and the process using the UI can becomevery tedious as opposed to using just one command that you can document. Anyway, it’s alwaysgood to know all the options, so I’ll also going to show how to run the instances using the WebConsole.

First, go the EC2 panel and click on Launch Instance. Click on Community AMIs on the left sidebarand search for amazon-ecs-optimized:

Page 21: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

The Cluster 16

Choose AMI

Search for amzn-ami-2015.03.g-amazon-ecs-optimized in the list and click Select.

Next, select the t2.micro instance type and click on “Next: Configure Instance Details”.

Choose Instace Type

In the instance configuration, use 2 for the number of instances, choose your default VPC and subnetfor the network and subnet fields, and the ecsInstanceRole as the IAM Role:

Page 22: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

The Cluster 17

Configure Instance

Go down and click on Advance Details. In the User Data section, add the same configuration that’sin the user-data.sh file:

User Data

Then, click on “Next: Add Storage”, skip to “Next: Tag Intance” and “Next: Configure SecurityGroup”. In the security group section, select existing security group and look for the default VPCsecurity group:

Configure Security Group

Page 23: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

The Cluster 18

Click on Review And Launch, and finally click on Launch. Before the instances are launched, you’llbe asked for which key pair to use. Search for your ClusterKeyPair, check the acknowledgementfield and Click on Launch Instances.

Select Key Pair

And that’s it! Your instances will be launched and attached to the cluster.

Page 24: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

Our first task definitionA task definition contains the information about the container that you want to run. You can createthe task definition using a json file, and then register the task in your cluster using the AWS CLI.

All task definitions used in this book can be found in the deploy/task-definitions directory ofthe source code.

{

"containerDefinitions": [

{

"volumesFrom": [],

"memory": 256,

"portMappings": [

{

"hostPort": 3306,

"containerPort": 3306,

"protocol": "tcp"

}

],

"essential": true,

"entryPoint": [],

"mountPoints": [

{

"containerPath": "/var/lib/mysql",

"sourceVolume": "mysql-data",

"readOnly": false

}

],

"name": "searchapp-db",

"environment": [

{

"name": "MYSQL_DATABASE",

"value": "searchapp_production"

},

{

"name": "MYSQL_PASSWORD",

"value": "mysecretpassword"

},

{

"name": "MYSQL_ROOT_PASSWORD",

"value": "mysupersecretpassword"

},

{

"name": "MYSQL_USER",

"value": "searchappusr"

19

Page 25: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

Our first task definition 20

}

],

"links": [],

"image": "mysql:5.7",

"command": [],

"cpu": 128

}

],

"volumes": [

{

"name": "mysql-data"

}

],

"family": "searchapp-db"

}

This template might look complicated, but there’s a lot of optional information there. Actually, ifyou use the ECS web console UI for registering task definitions, you’ll see that most of the containerinformation can be omitted.

Page 26: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

Our first task definition 21

Creating a task definition via ECS web console

If you used docker-compose in the past, you’ll see some similarities between the structures of thedefinitions. Here you also have port mapping, environment variables, links to other containers andvolumes for persisting the container data. Keep inmind that if you link containers in a task definition,those containers will be deployed on the same EC2 instance. Docker still doesn’t support link acrossdifferent hosts, and the way ECS manges this kind of service discovery, is through Elastic LoadBalancers.

This task defines a map between the port 3306 of the container and the port 3306 of the host. Italso declares the environment variables the MySQL image¹³ requires, and creates a volume formounting the /var/lib/mysql directory. This approach will allow us to avoid data lost when theMySQL container is restarted. The problem is we are now coupled to a particular instance, sinceECS doesn’t sync the data between instances in the cluster. This is still a dark area in the containersworld and it’s why some people prefer to run only stateless application inside containers and usemore typical approaches for their database resources.

The command for register a task definition is:

¹³https://hub.docker.com/_/mysql

Page 27: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

Our first task definition 22

$ aws ecs register-task-definition --cli-input-json file://task-definitions/mysql.json

Output:

{

"taskDefinition": {

"status": "ACTIVE",

"family": "searchapp-db",

"volumes": [

{

"host": {},

"name": "mysql-data"

}

],

"taskDefinitionArn": "arn:aws:ecs:us-east-1:387705308362:task-definition/searchapp-db:1",

"containerDefinitions": [

{

"environment": [

{

"name": "MYSQL_DATABASE",

"value": "searchapp_production"

},

{

"name": "MYSQL_PASSWORD",

"value": "mysecretpassword"

},

{

"name": "MYSQL_ROOT_PASSWORD",

"value": "mysupersecretpassword"

},

{

"name": "MYSQL_USER",

"value": "searchappusr"

}

],

"name": "searchapp-db",

"links": [],

"mountPoints": [

{

"sourceVolume": "mysql-data",

"readOnly": false,

"containerPath": "/var/lib/mysql"

}

],

"image": "mysql:5.7",

"essential": true,

"portMappings": [

{

"protocol": "tcp",

"containerPort": 3306,

Page 28: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

Our first task definition 23

"hostPort": 3306

}

],

"entryPoint": [],

"memory": 256,

"command": [],

"cpu": 128,

"volumesFrom": []

}

],

"revision": 1

}

}

Great, our task definition was registered successfully. Now every time we make changes to it, a newrevision will be created as long as we use the same family value when registering the new json file.

Also, now we can create a service scheduler for running a task that uses this task definition. Theservice will be in charge of allocating the task somewhere in the cluster and restarting the task if thecontainer fails for some reason. Since the service associated with this task will deploy the containeranywhere in the cluster, we need a mechanism for getting some fixed value that points to the taskhostname, in this case to the MYSQL server endpoint. The native way to accomplish this in ECS isby creating an Elastic Load Balancer (ELB). Then, in the service definition you can add an optionfor attaching the service to the ELB. This way, even if the service allocates the task in a differentinstance between deploys, it will always be attached to the ELB.

The way you can create and register a task definition using the ECS Web Console is pretty easy. Goto the ECS Panel and click on Task Definitions in the left sidebar. Then click on “Create new TaskDefinition”:

Create Task Definition

Choose a name for the task definition, the click on “Add Volume”. We need to create the volumefirst so we can use it in our container definition. Fill out the volume form with the same name fromthe json definition and then click Add:

Page 29: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

Our first task definition 24

Adding a Volume

We can leave the Source Path field empty so ECS can assign a directory for us inside of the instance.

Now click on “Add Container Definition”. Fill out the standard section with the same informationthat we have in the json task definition:

Container Definition

Click on ‘Advanced container configuration’. In the CPU units field, add 128. Make sure the essentialcheckbox is selected. In the Env Variables fields, add the same environment variables that we usedin the json file:

Page 30: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

Our first task definition 25

Container Advanced Configuration

In the same pop-up window, scroll down to the ‘Storage and Logging’ section and add a mount pointusing the Volume that we created. You also have to add the container path that you want to mount.In our case we have to mount the path that holds the MySQL data:

Adding a Volume

Finally click Add, and then Create.

Page 31: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

Our first ELBThe way we are going to manage Service Discovery inside ECS is by using Elastic Load Balancersfor our resources. As I mentioned before, we can run our tasks by using services that run behindan ELB. This ELB will have a static DNS, so we don’t have to worry if the service runs the task ondifferent instances during deployments.

Let’s create an ELB for the MySQL container. For this, we’ll need our subnet and security group ids.Remember you can store these values into variables by using:

$ read GroupId VpcId <<< $(aws ec2 describe-security-groups --group-names default --query "SecurityG\

roups[*].{GroupId:GroupId,VpcId:VpcId}" --output=text)

$ SubnetId=`aws ec2 describe-subnets --filters "Name=vpc-id,Values=$VpcId" --query "Subnets[0].Subne\

tId" --output=text`

Now you can use $GroupId, $VpcId and $SubnetId inside of the AWS CLI commands. This ELBjust needs to be reached by our cluster instances, so it can be internal. Also, the MySQL containerwill expose the port 3306, so we can add a listener to that port but use the port 80 in our ELB. Thecommand for create this ELB is:

$ aws elb create-load-balancer --load-balancer-name mysql-elb --listeners Protocol=TCP,LoadBalancerP\

ort=80,InstanceProtocol=TCP,InstancePort=3306 --subnets $SubnetId --scheme internal --security-group\

s $GroupId

Output:

{

"DNSName": "internal-mysql-elb-1700927984.us-east-1.elb.amazonaws.com"

}

The output gives us the endpoint for the database server. We will use this address in the Railsdatabase configuration. This ELB will point to an EC2 instance running a MySQL container thatexposes and binds its port 3306 to the host. On the other side, the ELB has a listener on its port 80to the port 3306 of the instance, so if we want to connect to this ELB, remember to always use theport 80.

Finally, hosts outside of the VPC won’t be able to connect to this ELB since it was launched insideof our VPC using its default security group.

Now, let’s use the Web Console for this same process. Go to the EC2 panel, and in the left sidebar,click on Load Balancers. Then click on “Create Load Balancer”:

26

Page 32: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

Our first ELB 27

Create ELB

Give the ELB a name, choose your default VPC and select the internal load balancer option. Also,add a listener from the port 80 of the ELB, to the port 3306 of the instance and click on “Next: AssignSecurity Groups”:

ELB Basic Configuration

In the security group configuration, make sure you select the default VPC security Group and clicknext:

ELB Security Group

Click next until you reach the Review and Create button. Verify that all it’s correct:

Page 33: Shipping Rails on Docker - Leanpubsamples.leanpub.com/rails-on-docker-sample.pdfRequirements This book is for people that have a basic knowledge of Ruby on Rails, Docker and Amazon

Our first ELB 28

ELB Review

And click create.