container-based virtualization: docker · corso di sistemi distribuiti e cloud computing ... server...
TRANSCRIPT
Container-based virtualization: Docker
Università degli Studi di Roma “Tor Vergata” Dipartimento di Ingegneria Civile e Ingegneria Informatica
Corso di Sistemi Distribuiti e Cloud Computing A.A. 2018/19
Valeria Cardellini
Case study: Docker • Lightweight, open and secure container-based
virtualization – Containers include the application and all of its
dependencies, but share the kernel with other containers – Containers run as an isolated process in userspace on the
host operating system – Containers are also not tied to any specific infrastructure
Valeria Cardellini - SDCC 2018/19
1
Docker internals • Docker is written in Go language
• With respect to other OS-level virtualization solutions, Docker is a higher-level platform that exploits Linux kernel mechanisms such as cgroups and namespaces – First versions based on LXC – Then based on its own libcontainer runtime that uses Linux
kernel namespaces and cgroups directly – libcontainer (now included in opencontainers/runc): cross-
system abstraction layer aimed to support a wide range of isolation technologies
• Dockers adds to LXC – Portable deployment across machines – Versioning, i.e., git-like capabilities – Component reuse – Shared libraries, see Docker Hub hub.docker.com
Valeria Cardellini - SDCC 2018/19
2
Docker engine
• Docker Engine: client-server application composed by: – A server, called daemon
process – A REST API which specifies
interfaces that programs can use to control and interact with the daemon
– A command line interface (CLI) client
Valeria Cardellini - SDCC 2018/19
3
See https://docs.docker.com/engine/docker-overview/
Docker architecture • Docker uses a client-server architecture
– The Docker client talks to the Docker daemon, which builds, runs, and distributes Docker containers
– Client and daemon communicate via sockets or REST API
Valeria Cardellini - SDCC 2018/19
4
Docker image • Read-only template containing instructions for creating a
Docker container – Described in text file called Dockerfile, with simple, well-defined
syntax – The Build component of Docker – Enables the distribution of applications with their runtime
environment • Incorporates all the dependencies and configuration necessary for
it to run, eliminating the need to install packages and troubleshoot – Target machine must be Docker-enabled
• The Docker Image – Can be pulled and pushed towards a registry – Image names have the form [registry/][user/]name[:tag] – Default for tag is latest
Valeria Cardellini - SDCC 2018/19
5
Docker image: Dockerfile • Images can be created from a Dockerfile and a context:
– Dockerfile: instructions to assemble the image – Context: set of files (e.g., application, libraries) – Often, an image is based on another image (e.g., ubuntu)
• Example of a Dockerfile
6 Valeria Cardellini - SDCC 2018/19
Docker image: build
• Build an image from a Dockerfile
$dockerbuild -t hello-world .
Valeria Cardellini - SDCC 2018/19 7
$dockerrun[OPTIONS]PATH|URL|-
Docker image: layers • Layered image
– Each image consists of a series of layers – Docker uses union file systems to combine these layers into a
single unified view • Layers are stacked on top of each other to form a base for a
container’s root file system • Based on the copy-on-write (COW) principle
Valeria Cardellini - SDCC 2018/19
8
Docker image: layers • Layering pros
- Enable layer sharing and reuse, installing common layers only once and saving bandwidth and storage space
- Manage dependencies and separate concerns - Facilitate software specializations - See https://docs.docker.com/storage/storagedriver/
Valeria Cardellini - SDCC 2018/19
9
Docker image: storage
• Containers should be stateless. Ideally: – Very little data is written to the container’s writable layer – Data should be written on Docker volumes – Nevertheless: some workloads require to write data to the
container’s writable layer
• The storage driver controls how images and containers are stored and managed on the Docker host
• Multiple choices for the storage driver - Including AuFS and OverlayFS (both operate at file level), Device
Mapper, Btrfs and zfs (that operate at block level) - Storage driver’s choice can affect the performance of the
containerized applications - See https://dockr.ly/2FstUe6
Valeria Cardellini - SDCC 2018/19
10
Docker container and registry • Docker container: runnable instance of a Docker
image – Run, start, stop, move, or delete a container using Docker API
or CLI commands – The Run component of Docker
• Docker registry: stateless server-side application that stores and lets you distribute Docker images - Provides an open library of images - The Distribute component of Docker - Docker-hosted registries: Docker Hub, Docker Store (open
source and enterprise verified images) Valeria Cardellini - SDCC 2018/19
11
- Docker containers are stateless: when a container is deleted, any data written that is not stored in a data volume is deleted along with the container
Docker: run command
• When you run a container whose image is not yet installed but is available on Docker Hub
Valeria Cardellini - SDCC 2018/19 12
Courtesy of “Docker in Action” by J. Nickoloff
State transitions of Docker containers
Valeria Cardellini - SDCC 2018/19 13
Courtesy of “Docker in Action” by J. Nickoloff
Commands: Docker info
• Obtain detailed info on your Docker installation $dockerinfo
E.g., to know the used storage driver (e.g., AuFS)
14
Valeria Cardellini - SDCC 2018/19
Commands: image handling
• List images on host (i.e., local repository) $dockerimages or $dockerimagels- To list every image, including intermediate image layers: $dockerimages–a or $dockerimagels–a– Options to list images by name and tag, to list image digests
(sha256), to filter images, to format the output, e.g., $dockerimages--filterreference=ubuntu
• Inspect an image – Display detailed information, including image layers $docker[image]inspectimageid
• Remove an image $dockerrmiimageid or $dockerimagermimageid
15 Valeria Cardellini - SDCC 2018/19
Can also use imagenameinstead of imageid
Command: run
• Most common options --nameassign a name to the container –ddetached mode (in background) –iinteractive (keep STDIN open even if not attached) -tallocate a pseudo-tty--exposeexpose a range of ports inside the container -ppublish a container's port or a range of ports to the host –vbind and mount a volume –eset environment variables --linkadd link to other containers
• The “Hello World” container $dockerrunalpine/bin/echo'Helloworld'
- Alpine: lightweight Linux distro with reduced image size
16
$dockerrun[OPTIONS]IMAGE[COMMAND][ARGS]
Valeria Cardellini - SDCC 2018/19
Commands: containers management • List containers
– Only running containers: $dockerps• Alternatively,$dockercontainerls
– All containers (even stopped or killed containers): $dockerps-a
• Container lifecycle – Stop running container $dockerstopcontainerid
– Start stopped container $dockerstartcontainerid
– Kill running container $dockerkillcontainerid
– Remove container (need to stop the container before attempting removal) $dockerrmcontainerid
17
Can also use containernameinstead of containerid
Valeria Cardellini - SDCC 2018/19
Commands: containers management • Inspect a container
– Most detailed view of the environment in which a container was launched
$dockerinspectcontainerid
• Copy files from and to docker container $dockercpcontainerid:pathlocalpath$dockercplocalpathcontainerid:path
18
Valeria Cardellini - SDCC 2018/19
Examples of using Docker • Run a nginx Web server inside a container
- Also bind the container to a specific port $dockerrun–d–p80:80--namewebnginx
• Send HTTP request through Web browser - First retrieve the hostname of the host machine
• Send HTTP request through an interactive container using the Docker internal network$dockerrun-i-t--linkweb:web--nameweb_testbusybox
/#wget-O-http://web:80/
/#exit
• To not use --link, let us define a bridge network $dockernetworkcreatemy_net
$dockerrun-d–p80:80--nameweb--net=my_netnginx
$dockerrun–i-t--net=my-net--nameweb_testbusybox
/#... Valeria Cardellini - SDCC 2018/19
19
--link: legacy flag to manually create links between the containers
wget: -OFILESavetoFILE('-'forstdout)
Examples of using Docker • Sending an HTTP request through an Alpine Linux
container with curl installed and set as entrypoint$dockerrun--rmbyrnedo/alpine-curlhttp://…
• Checking the logs of the container $dockerlogscontainerid_or_name
Valeria Cardellini - SDCC 2018/19
20
Examples of using Docker • Running Apache web server with minimal index page
– Define container image with Dockerfile • Define image starting from Ubuntu, install and configure Apache • Incoming port set to 80 using EXPOSE instruction
Vale
ria C
arde
llini
- S
DC
C 2
018/
19
21
FROMubuntu
#Installdependencies
RUNapt-getupdate
RUNapt-get-yinstallapache2
#Installapacheandwritehelloworldmessage
RUNecho'HelloWorld!'>/var/www/html/index.html
#Configureapache
RUNecho'./etc/apache2/envvars'>/root/run_apache.sh
RUNecho'mkdir-p/var/run/apache2'>>/root/run_apache.sh
RUNecho'mkdir-p/var/lock/apache2'>>/root/run_apache.sh
RUNecho'/usr/sbin/apache2-DFOREGROUND'>>/root/run_apache.sh
RUNchmod755/root/run_apache.sh
EXPOSE80
CMD/root/run_apache.sh
Examples of using Docker
– Build container image from Dockerfile $dockerbuild-thello-apache.
– Run container and bind $dockerrun-d-p80:80hello-apache
• Option –p: publish container port (80) to host port (80)
Valeria Cardellini - SDCC 2018/19 22
Examples of using Docker
23
Valeria Cardellini - SDCC 2018/19
$dockerps
CONTAINERIDIMAGECOMMANDCREATEDSTATUSPORTSNAMES
2b2b39d05fe0hello_world"/bin/sh-c/root/ru…"3secondsagoUp2seconds0.0.0.0:80->80/tcppedantic_austin
• Stopping and removing a container $dockerps$dockerstopcontainerid_or_name$dockerps-a$dockerrmcontainerid_or_name
• Stopping and removing all container $dockerstop$(dockerps-a–q)$dockerrm$(dockerps-a–q)
Examples of using Docker
24
Valeria Cardellini - SDCC 2018/19
• Running a Python web app in Docker – See https://docs.docker.com/get-started/part2/ – Define container image with Dockerfile
• Define image with Python runtime and Python app • Incoming port set to 80 using EXPOSE instruction
– Write app code in Python using Flask and Redis packages and create file requirements.txt to specify those packages needed by app
• Redis: in-memory data store • Redis is used to keep the counter of the number of visits to web app
– Build container image $dockerbuild-tfriendlyhello.
– Run container and bind $dockerrun-d-p4000:80friendlyhello
• Option –p: publish container port (80) to host port (4000)
Multi-container Docker applications
• How to run multi-container Docker apps? • Docker Compose
– Deployment on single host
• Docker Swarm – The native orchestration tool for Docker – Deployment on multiple hosts
• Kubernetes
Valeria Cardellini - SDCC 2018/19 25
Docker Compose • To coordinate the execution of multiple containers,
we can use Docker Compose – See https://docs.docker.com/compose/
• Docker Compose – Not bundled within Docker installation (on Linux) – Can be installed following the official Docker documentation
https://docs.docker.com/compose/install/
• Allows to easily express the containers to be instantiated at once, and the relations among them
• Runs the composition on a single machine (i.e., single Docker engine)
• Use Docker Swarm to deploy containers on multiple nodes
26
Valeria Cardellini - SDCC 2018/19
Docker Compose • We specify how to compose containers in a easy-to-
read file, by default named docker-compose.yml
• To start the Docker composition (background -d):
• To stop the Docker composition:
• By default, Docker Compose looks for the docker-compose.ymlfile in the current working directory
– Change file using -fflag
27
$docker-composeup-d
$docker-composedown
Valeria Cardellini - SDCC 2018/19
Docker Compose • Different versions of the Docker compose file format
– Latest: version 3 is supported from Docker Compose 1.13
28
Docker compose file format: https://docs.docker.com/compose/compose-file/ Valeria Cardellini - SDCC 2018/19
Docker Compose: example • Simple Python web app running on Docker Compose
– Two containers: Python web app and Redis – Use Flask framework and maintain a hit counter in Redis – See https://docs.docker.com/compose/gettingstarted/
• Steps: – Write Python app – Define Python container image with Dockerfile – Define services in docker-compose.yml file
• Two services: web (image defined by Dockerfile) and redis (image pulled from Docker Hub)
– Build and run your app with Compose $docker-composeup–d
– Send HTTP requests using curl (now counter is increased) – Stop Compose
$docker-composedown Valeria Cardellini - SDCC 2018/19
29
Docker: Swarm mode • Docker includes the swarm mode for natively
managing a cluster of Docker Engines, which is called swarm – See https://docs.docker.com/engine/swarm/
• Task: running container part of a swarm service • Basic features of swarm mode:
– Scaling: allows to declare the number of tasks for each service
– State reconciliation: Swarm monitors the cluster state and reconciles any differences w.r.t. the expressed desired state
– Multi-host networking: allows to specify an overlay network among services
– Load balancing: allows to expose the ports for services to an external load balancer. Internally, the swarm lets you specify how to distribute service containers among nodes
30
Valeria Cardellini - SDCC 2018/19
Docker: Swarm mode • A swarm consists of multiple Docker hosts which run in swarm
mode • Node: an instance of the Docker engine
– Manager node dispatches tasks to worker nodes – Worker nodes receive and execute tasks
• Load balancing – The swarm manager can automatically assign the service a
(configurable) PublishedPort – External components can access the service on the PublishedPort.
All nodes in the swarm route ingress connections to a running task
31 Valeria Cardellini - SDCC 2018/19
Commands: Swarm cluster • Create a swarm: manager node
• Create a swarm: worker node
• Inspect status
32 Valeria Cardellini - SDCC 2018/19
$dockerswarminit--advertise-addr<MANAGER-IP>
Swarminitialized:currentnode(<nodeid>)isnowamanager.
Toaddaworkertothisswarm,runthefollowingcommand:
dockerswarmjoin--token<token><manager-ip>:port
$dockerswarmjoin--token<token><manager-ip>:port
$dockerinfo
$dockernodels
ID HOSTNAME STATUS AVAILABILITY MANAGERSTATUS<nodeid>* controller Ready Active Leader<nodeid> storage Ready Active
Commands: Swarm cluster • Leave the swarm
If the node is a manager node, you will receive a warning about maintaining the quorum. To override the warning, pass the --forceflag
• After a node leaves the swarm, you can run dockernoderm command on a manager node to remove the node from the node list
33 Valeria Cardellini - SDCC 2018/19
$dockerswarmleave
$dockernodermnode-id
Commands: manage services • Deploy a service to the swarm (from the manager node)
• List running services
34 Valeria Cardellini - SDCC 2018/19
$dockerservicecreate-d--replicas1\
--namehelloworldalpinepingdocker.com
$dockerservicels
ID NAME MODE REPLICAS IMAGE PORTS<serviceid> helloworld replicated 1/1 alpine:latest
Commands: manage services • Inspect the service
• Inspect the container
35 Valeria Cardellini - SDCC 2018/19
$dockerps<cont.id1>
#Managernode
CONTAINERID IMAGE COMMAND CREATED STATUS ...NAMES<cont.id1> alpine:latest "pingdocker.com"2minago Up2minhelloworld.1.iuk1sj…
#Workernode
CONTAINERID IMAGE COMMAND CREATED STATUS ...NAMES<cont.id2> alpine:latest "pingdocker.com"2minago Up2minhelloworld.2.skfos4…
$dockerserviceinspect--pretty<SERVICE-ID>
$dockerserviceps<SERVICE-ID>
ID NAME IMAGE NODE DESIREDSTCURRENTST ERROR PORTS<cont.id1>helloworld.1 alpine:latest controller Running Running…<cont.id2>helloworld.2 alpine:latest storage Running Running…
Commands: manage services • Scale the service
The swarm manager will automatically enact the updates
• Apply rolling updates to a service
• Roll back an update
• Remove a service
36 Valeria Cardellini - SDCC 2018/19
$dockerserviceupdate--imageredis:3.0.7redis
$dockerserviceupdate--replicas2helloworld
$dockerservicerm<SERVICE-ID>
$dockerservicerollback[OPTIONS]<SERVICE-ID>
$dockerservicescale<SERVICE-ID>=<NUMBER-OF-TASKS>
Docker Swarm: example • Let’s use Docker Swarm to scale Python web app on the same
host and load balance the traffic among the container replicas – See https://docs.docker.com/get-started/part3/
• Steps: – Define services in docker-compose.yml file
• Only one service called web • Set replication degree to 5 • Set resource limits to be used by each single replica • Set restart policy to on-failure so that a failed container is
automatically restarted • Docker supports 4 restart policies
» no» on-failure» unless-stopped» always
• Instruct web’s containers to share port 80 via a load-balanced network called webnet
• Define webnet network (load-balanced overlay network) Vale
ria C
arde
llini
- S
DC
C 2
018/
19
37
Docker Swarm: example – Start the swarm $dockerswarminit
– Deploy the stack $dockerstackdeploy-cdocker-compose.ymlgetstartedlab
– Send multiple requests to Python web app and see how they are served by different replicas chosen in a round-robin fashion
– Take down the app and the swarm $dockerstackrmgetstartedlab
$dockerswarmleave--force
38 Valeria Cardellini - SDCC 2018/19