supporting docker in emulab-based network …...supporting docker in emulab-based network testbeds...

Supporting Docker inEmulab-Based Network Testbeds

DavidJohnson,ElijahGrubb,EricEideUniversityofUtah

• overthecourseofastudy…

• prototypeonlaptop

• networktestbed

• commercialcloud

• needtomoveexperimental artifactsaround

�3

This talk

• extendedEmulabsouserscancreateexperimentsinwhichsomeorallnodesareDockercontainers

• challenges• preservingusers’“testbedexperience”• meshingwithEmulab’sinfrastructure

• results• justworks:52/60topDockerHubimagesautomaticallyadapted• supportslarge(5K-node)experiments

�4

Docker

�5

Docker

• basedoncontainers

• filesystemspopulatedviaimages

�5

PhysicalHostHostOSDocker

Container

Filesystem

App

Docker

• basedoncontainers

• filesystemspopulatedviaimages

• imagescreatedviaDockerfiles

�5

PhysicalHostHostOSDocker

Container

Filesystem

App

ImageDockerfile

Emulab

• testbedmanagementsoftware

• allocatesphysicaland virtualresourcestousers

• configuresresources

• isolatesusersfromeach other

�6

Emulab

�7

Emulab

• organizedaroundprofiles

• profilesareinstantiated tomakeexperiments

�7

Profile

Emulab



• nodes’diskspopulatedviadiskimages

�7

Profile

Emulab



• nodes’diskspopulatedviadiskimages

• in-experimentservices

�7

Profile Diskimages

Goal: Emulab + Docker should “just work”

�8


• containersinEmulabarejustanotherkind ofvirtualnode

�8

Dockerimages


• containersinEmulabarejustanotherkind ofvirtualnode• EmulabusercanchooseanyDockerimage

• preserveEmulab’sexperimenterservices• e.g.,SSH,local/remotestorageaccess,…

�8

Dockerimages




• preserveEmulab’snetworkservices• e.g.,controlnetwork,trafficshaping,…

�8

Dockerimages




• preserveEmulab’snetworkservices• e.g.,controlnetwork,trafficshaping,…

• preserveDockeruserexperience• e.g.,“dockercommit”

�8

Dockerimages

�9

httpd:latest

Preserving Emulab’s experimenter services

• shellaccesstonodes

• remoteandlocalstorage

• networkconfiguration• addressing,routing,shaping

• startupprograms

�10

Preserving Emulab’s experimenter services

• shellaccesstonodes

• remoteandlocalstorage

• networkconfiguration• addressing,routing,shaping

• startupprograms

• typicalDockerimagesareminimalappliances

• runtheapplicationonly• notpreparedtohostotherservices

�10

httpd:latest

augmentation

generateanewDockerfile,startingfromtheuser’schosenimage,

andaddingtestbedsoftware

Augment the startup

�12

httpd:latest

Augment the startup

• maketemporarycontainer

�12

httpd:latest

Container

Augment the startup

• maketemporarycontainer• addbuildtoolchain

�12

httpd:latest

Containerbuildtools

Augment the startup

• maketemporarycontainer• addbuildtoolchain• compileandpackagerunit

�12

httpd:latest

Containerbuildtools

runit

Augment the startup


�12

httpd:latest

Containerbuildtools

runit

runit

Augment the startup


�12

httpd:latest

runit

Augment the startup


�12

httpd:latest

runit

FROMhttpd:latest

COPY…runit…RUN…runit-setup…

Augment the startup

• maketemporarycontainer• addbuildtoolchain• compileandpackagerunit• addDockerfileinstructionstoinstallrunit

�12

httpd:latest

runit

FROMhttpd:latest


Augment the startup

• maketemporarycontainer• addbuildtoolchain• compileandpackagerunit• addDockerfileinstructionstoinstallrunit• configurerunittoruntheoriginalENTRYPOINT

�12

httpd:latest

runit

FROMhttpd:latest


Augment the startup

• maketemporarycontainer• addbuildtoolchain• compileandpackagerunit• addDockerfileinstructionstoinstallrunit• configurerunittoruntheoriginalENTRYPOINT• whenaugmentedimageisused,setENTRYPOINTtorunit

�12

httpd:latest

runit

FROMhttpd:latest


Add the Emulab “client-side” software

• maketemporarycontainer• compileandpackageEmulabclient-sidesoftware• addDockerfileinstructionstoinstallthesoftware

• user-selectablelevelsofaugmentation

�13

FROMhttpd:latest

COPY…RUN…runit-setup…&&…emulab-setup…

Local registry

�14

Dockerregistry

Local registry

• cacheaugmentedimagesinatestbed-localDockerregistry

• speedssubsequentexperimentcreation

• integratedwithEmulab’suserauthentication&authorizationmodel

�14

Dockerregistry

httpd:latest perl:5.28

redis:4.0 node:8.11.3

mysql:5.7 erlang:21.0

Preserving Emulab’s network services

• separatecontrolnetwork

• experimenttrafficshaping

• control-networkfirewalls

• DNS

�15

Preserving Emulab’s network services

• separatecontrolnetwork

• experimenttrafficshaping

• control-networkfirewalls

• DNS

• Docker’sContainerNetworkModel(CNM)ismismatchedtodemandsofanetworktestbed

• tooabstract

• triestocontroltoomuch

• missingfeatures

�15

leverage the physical host

managenetworkservicesonthephysical-hostsideof

containers’virtualnetworkinterfaces

Control network

�17

PhysicalHost

physicalcontrolnetwork

Control network

• atphysical-hostboot• createdockercnetvirtualnetwork

• bridgetothephysicalcontrolnetwork

�17

PhysicalHost

dockercnet


Control network

• atphysical-hostboot• createdockercnetvirtualnetwork

• bridgetothephysicalcontrolnetwork

• atcontainerstartup• connecttodockercnet• setupNATtoexposeSSHoverthephysicalhost’spublicIPaddress

�17

PhysicalHost

Container Container Container

dockercnet


Traffic shaping and firewalls

�18

PhysicalHost

dockercnet

controlnetwork

experimentnetworks


• Emulabsubscribestolife-cycleeventsofeachcontainer

• atcontainerstartup• installtcrulesforexpt.-networktrafficshaping

• installiptablesrulesforcontrol-networkfirewalling

�18

PhysicalHost

dockercnet

controlnetwork

experimentnetworks


• Emulabsubscribestolife-cycleeventsofeachcontainer

• atcontainerstartup• installtcrulesforexpt.-networktrafficshaping

• installiptablesrulesforcontrol-networkfirewalling

• atcontainershutdown• removetherules

�18

PhysicalHost


dockercnet

controlnetwork

experimentnetworks

firewall

shaping

Dedicated and shared modes

�19

PhysicalHost


experimentnetworks


• dedicated—containersrunonphysicalmachinereservedtooneexperiment

�19

PhysicalHost

Experiment


experimentnetworks


• dedicated—containersrunonphysicalmachinereservedtooneexperiment• shared—physicalmachinemayhostcontainersfromseveralexperiments

�19

PhysicalHost

Expt.1 Expt.2


experimentnetworks



�19

PhysicalHost

Expt.1 Expt.2


experimentnetworks

192.168

.1.1

192.168

.1.1



• wemodifiedDockertosupportmultiple,isolatedlayer2netsonasinglephysicalhost

�19

PhysicalHost

Expt.1 Expt.2


experimentnetworks

192.168

.1.1

192.168

.1.1

Implemented & deployed

• supportedOSes• AlpineLinux3.6,3.7,3.8• CentOS7• Debian8,9,sid• Ubuntu14.04,16.04,18.04

• registriesat

�20

…

evaluation•60mostpopularimagesfromDockerHub

•fourresearchDockerimages•timetoaugmentDockerimages•timetocreatelargeexperiments

�22

Category DockerImages

Linuxdistro alpine,centos,debian,ubuntu,amazonlinux,busybox,fedora

Debian buildpack-deps,cassandra,chronograf,drupal,elasticsearch,ghost,golang,gradle,groovy,haproxy,httpd,influxdb,java,jenkins,jruby,kibana,logstash,mariadb,maven,memcached,mongo,mysql,nextcloud,nginx,node,openjdk,owncloud,percona,perl,php,postgres,python,rabbitmq,redis,rethinkdb,rocket.chat,ruby,sentry,solr,sonarqube,tomcat,wordpress,telegraf

Alpine consul,docker,kong,neo4j,vault,registry

Scratch hello-world,nats,swarm,traefik

�22






fullysupportedpartiallysupportednotsupported

�23







�23







Emulabautomaticallyadapted52/60imagesintothetestbedenvironmentandinstantiatedcontainersfromthem.

Scalability

• createlargeexperimentswithDockercontainers

• ineachtrial• 200containersperphysicalhost• eachcontainerrunsaugmentedubuntu:14.04imagefromtestbed’slocalregistry

• allcontainersattachedtoaLAN

• physicalhosts:CloudLabxl170nodesrunningUbuntu16.04

�24

Scalability





• 1–25physicalhosts• yielding200–5,000containers

�24

Scalability






• measure• elapsedtimetofirstcontainer• avg.creationtimeforeachcontainerafterthefirst

• elapsedtimetocreateallcontainersoneachphysicalhost

• elapsedtimetocreatefullexpt.

�24

Scalability






• measure• elapsedtimetofirstcontainer• avg.creationtimeforeachcontainerafterthefirst

• elapsedtimetocreateallcontainersoneachphysicalhost

• elapsedtimetocreatefullexpt.

• repeateachtrial3×,reportavgs.

�24

�25

200containers 14minutes

�25


5,000containers 1.87hours

�25


5,000containers 1.87hours

Conclusion:acceptableperformance,butmoreserver-sideoptimizationwillbeneededforlargeexperiments.

�25

parallelized

�25

one-timesetup

Conclusion

• Emulab+Docker“justworks”• experimenterservices—automaticaugmentation• networkservices—physicalhostcontrol&minorDockermods

• supportsexistingDockerimages• promotesartifactportability• promotesresearchrepeatability

• availableinEmulab-basedtestbedsnow!

�26

EricEidewww.cs.utah.edu/~eeide/email:[email protected]:@eeide

supporting docker in emulab-based network …...supporting docker in emulab-based network testbeds...

Documents