MonitoringDocker

TableofContents

MonitoringDocker

Credits

AbouttheAuthor

AbouttheReviewer

www.PacktPub.com

Supportfiles,eBooks,discountoffers,andmore

Whysubscribe?

FreeaccessforPacktaccountholders

Preface

Whatthisbookcovers

Whatyouneedforthisbook

Whothisbookisfor

Conventions

Readerfeedback

Customersupport

Downloadingtheexamplecode

Downloadingthecolorimagesofthisbook

Errata

Piracy

Questions

1.IntroductiontoDockerMonitoring

Pets,Cattle,Chickens,andSnowflakes

Pets

Cattle

Chickens

Snowflakes

Sowhatdoesthisallmean?

Docker

Launchingalocalenvironment

Cloningtheenvironment

Runningavirtualserver

Haltingthevirtualserver

Summary

2.UsingtheBuilt-inTools

Dockerstats

RunningDockerstats

Whatjusthappened?

Whataboutprocesses?

Dockertop

Dockerexec

Summary

3.AdvancedContainerResourceAnalysis

WhatiscAdvisor?

RunningcAdvisorusingacontainer

CompilingcAdvisorfromsource

Collectingmetrics

TheWebinterface

Overview

Processes

CPU

Memory

Network

Filesystem

Viewingcontainerstats

Subcontainers

Driverstatus

Images

Thisisallgreat,what’sthecatch?

Prometheus

LaunchingPrometheus

QueryingPrometheus

Dashboard

Thenextsteps

Alternatives?

Summary

4.ATraditionalApproachtoMonitoringContainers

Zabbix

InstallingZabbix

Usingcontainers

Usingvagrant

Preparingourhostmachine

TheZabbixwebinterface

Dockermetrics

Createcustomgraphs

Comparecontainerstoyourhostmachine

Triggers

Summary

5.QueryingwithSysdig

WhatisSysdig?

InstallingSysdig

UsingSysdig

Thebasics

Capturingdata

Containers

Furtherreading

UsingCsysdig

Summary

6.ExploringThirdPartyOptions

Awordaboutexternallyhostedservices

DeployingDockerinthecloud

WhyuseaSaaSservice?

SysdigCloud

Installingtheagent

Exploringyourcontainers

Summaryandfurtherreading

Datadog

Installingtheagent

Exploringthewebinterface

Summaryandfurtherreading

NewRelic

Installingtheagent

Exploringthewebinterface

Summaryandfurtherreading

Summary

7.CollectingApplicationLogsfromwithintheContainer

Viewingcontainerlogs

ELKStack

Startingthestack

Logspout

Reviewingthelogs

Whataboutproduction?

Lookingatthirdpartyoptions

Summary

8.WhatAretheNextSteps?

Somescenarios

Pets,Cattle,Chickens,andSnowflakes

Pets

Cattle

Chickens

Snowflakes

Scenarioone

Scenariotwo

Scenariothree

Alittlemoreaboutalerting

Chickens

CattleandPets

Sendingalerts

Keepingup

Summary

Index

MonitoringDocker

MonitoringDockerCopyright©2015PacktPublishing

Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthepublisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.

Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyoftheinformationpresented.However,theinformationcontainedinthisbookissoldwithoutwarranty,eitherexpressorimplied.Neithertheauthor,norPacktPublishing,anditsdealersanddistributorswillbeheldliableforanydamagescausedorallegedtobecauseddirectlyorindirectlybythisbook.

PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesandproductsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.

Firstpublished:December2015

Productionreference:1041215

PublishedbyPacktPublishingLtd.

LiveryPlace

35LiveryStreet

BirminghamB32PB,UK.

ISBN978-1-78588-275-3

www.packtpub.com

CreditsAuthor

RussMcKendrick

Reviewer

MarceloCorreiaPinheiro

CommissioningEditor

VeenaPagare

AcquisitionEditor

RahulNair

ContentDevelopmentEditor

AnishSukumaran

TechnicalEditor

SaurabhMalhotra

CopyEditor

TrishyaHajare

ProjectCoordinator

IzzatContractor

Proofreader

SafisEditing

Indexers

MariammalChettiyar

PriyaSane

ProductionCoordinator

ShantanuN.Zagade

CoverWork

ShantanuN.Zagade

AbouttheAuthorRussMcKendrickisanexperiencedsolutionsarchitectwhohasbeenworkinginITandIT-relatedindustriesforthebetterpartof23years.Duringhiscareer,hehashadvariedresponsibilitiesinanumberofindustries,rangingfromlookingafterentireITinfrastructurestoprovidingfirstline,secondline,andseniorsupportinclientfacing,andinternalteamsforcorporateorganizations.

HeworksalmostexclusivelywithLinux,usingopensourcesystemsandtoolsonvariousplatformsrangingfromdedicatedhardwareandvirtualmachinestopublicclouds.

AbouttheReviewerMarceloCorreiaPinheiroisaBraziliansoftwareengineerfromPortoAlegre.Hestartedtoworkasawebdesignerandprogrammerin2000withASPandPHP,naturallygettingintouchwiththeMicrosoft.NETframeworkandJavarunningrespectivedatabasesofchoiceforwebapplications.Since2003,hehasusedLinuxandUNIX-relatedoperationalsystems,fromSlackwaretoGoboLinux,Archlinux,CentOS,Debian,andtodayOSX,havingsomecontactwithBSDdistributionstoo.HehaslostsomenightscompilingandapplyingpatchestotheLinuxkerneltomakeitsdesktopwork.Sincethebeginning,hehasbeenactingasaproblemsolver,nomatterwhattheprogramminglanguage,database,orplatformis—opensourceenthusiast.

Afterafewyears,hedecidedtoliveinSãoPaulotoworkwithnewertechnologiessuchasNoSQL,cloudcomputing,andRuby,wherehestartedtoconducttechtalkswiththislanguageinLocaweb.HecreatedsometoolstostandardizedevelopmentusingtoolssuchasvagrantandRubygems—someoftheseintheirGitHub—inLocawebtoensurefastapplicationpackagingandreduceddeploymentrollbacks.In2013,hechangedhiscareertobeafull-stackdeveloperfollowingtheDevOpsmovement.Since2012,hehasattended,asaspeaker,someofthebiggestsoftwareconferencesinBrazil—RSonRails,QConSP,TheDeveloper’sConference,andRubyConfBrazil—talkingnotonlyaboutRuby,butalsoaboutsomeofthewell-knownDevOpstoolssuchasTerraform,Packer,Ansible,andDocker.Today,heworksasaDevOpsconsultantintheircompany.

Inhisfreetime,helovesplayingtheguitar,havingsomefunwithcats,traveling,anddrinkingbeer.Hecanbefoundonhisblog(http://salizzar.net),Twitter(https://twitter.com/salizzar),GitHub(https://github.com/salizzar)andLinkedin(https://www.linkedin.com/in/salizzar).

HehasworkedasareviewerforVagrantVirtualDevelopmentEnvironmentCookbook,aPacktPublishingbookwithusefulrecipesusingvagrantwithconfigurationmanagementtoolssuchasPuppet,Chef,Ansible,andSaltStack.

Iwanttothankallmyfriends,whobelievedinmypotentialsincethebeginningandwhostillfollowmedespitethedistance.Iwouldalsoliketothankmymentors,GleiconMoraes,RobertoGaiser,andRodrigoCampos,whogavemetheincentiveandtipstobeabettersoftwareengineerandperson.

www.PacktPub.com

Supportfiles,eBooks,discountoffers,andmoreForsupportfilesanddownloadsrelatedtoyourbook,pleasevisitwww.PacktPub.com.

DidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFandePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwithusat<service@packtpub.com>formoredetails.

Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signupforarangeoffreenewslettersandreceiveexclusivediscountsandoffersonPacktbooksandeBooks.

https://www2.packtpub.com/books/subscription/packtlib

DoyouneedinstantsolutionstoyourITquestions?PacktLibisPackt’sonlinedigitalbooklibrary.Here,youcansearch,access,andreadPackt’sentirelibraryofbooks.

Whysubscribe?FullysearchableacrosseverybookpublishedbyPacktCopyandpaste,print,andbookmarkcontentOndemandandaccessibleviaawebbrowser

FreeaccessforPacktaccountholdersIfyouhaveanaccountwithPacktatwww.PacktPub.com,youcanusethistoaccessPacktLibtodayandview9entirelyfreebooks.Simplyuseyourlogincredentialsforimmediateaccess.

PrefaceWiththeincreaseintheadoptionofDockercontainers,theneedtomonitorwhichcontainersarerunning,whatresourcestheyareconsuming,andhowitaffectstheoverallperformanceofthesystem,hasbecomeatime-relatedneed.MonitoringDockerwillteachyouhowmonitoringcontainersandkeepingakeeneyeontheworkingofapplicationshelptoimprovetheoverallperformanceoftheapplicationsthatrunonDocker.

ThisbookwillcovermonitoringcontainersusingDocker’snativemonitoringfunctions,variousplugins,andalsothird-partytoolsthathelpinmonitoring.Thebookwillfirstcoverhowtoobtaindetailedstatsfortheactivecontainers,resourcesconsumed,andcontainerbehavior.Thisbookwillalsoshowthereadershowtousethesestatstoimprovetheoverallperformanceofthesystem.

WhatthisbookcoversChapter1,IntroductiontoDockerMonitoring,discusseshowdifferentitistomonitorcontainerscomparedtomoretraditionalserverssuchasvirtualmachines,baremetalmachines,andcloudinstances(PetsversusCattleandChickensversusSnowflakes).Thischapteralsodetailstheoperatingsystemscoveredintheexampleslaterinthisbookandalsogivesalittleinformationonhowtogetalocaltestenvironmentupandrunningusingvagrant,sothatinstallationinstructionsandpracticalexamplescanbeeasilyfollowed.

Chapter2,UsingtheBuilt-inTools,helpsyoulearnaboutthebasicmetricsyoucangetoutofthevanillaDockerinstallationandhowyoucanusethem.Also,wewillunderstandhowtogetreal-timestatisticsonourrunningcontainers,howtousecommandsthatarefamiliartous,andhowtogetinformationontheprocessesthatarelaunchedaspartofeachcontainer.

Chapter3,AdvancedContainerResourceAnalysis,introducescAdvisorfromGoogle,whichaddsalotmoreprecisiontothebasictoolsprovidedbyDocker.YouwillalsolearnhowtoinstallcAdvisorandstartcollectingmetrics.

Chapter4,ATraditionalApproachtoMonitoringContainers,looksatatraditionaltoolformonitoringservices.Bytheendofthischapter,youshouldknowyourwayaroundZabbixandthevariouswaysyoucanmonitoryourcontainers.

Chapter5,QueryingwithSysdig,describesSysdigas“anopensource,system-levelexplorationtooltocapturesystemstateandactivityfromarunningLinuxinstance,thensave,filter,andanalyzeit.”Inthischapter,youwilllearnhowtouseSysdigtobothviewyourcontainers’performancemetricsinrealtimeandalsorecordsessionstoquerylater.

Chapter6,ExploringThirdPartyOptions,walksyouthroughafewoftheSoftwareasaService(SaaS)optionsthatareavailable,whyyouwouldusethem,andhowtoinstalltheirclientsonthehostserver.

Chapter7,CollectingApplicationLogsfromwithintheContainer,looksathowwecangetthecontentofthelogfilesfortheapplicationsrunningwithinourcontainerstoacentrallocationsothattheyareavailableevenifyouhavetodestroyandreplaceacontainer.

Chapter8,WhatAretheNextSteps?,looksatthenextstepsyoucantakeinmonitoringyourcontainersbytalkingaboutthebenefitsofaddingalertingtoyourmonitoring.Also,wewillcoversomedifferentscenariosandlookatwhichtypeofmonitoringisappropriateforeachofthem.

WhatyouneedforthisbookToensuretheexperienceisasconsistentaspossible,wewillbeinstallingvagrantandVirtualBoxtorunthevirtualmachinethatwillactasahosttorunourcontainers.VagrantisavailableforLinux,OSX,andWindows;fordetailsonhowtoinstallthis,seethevagrantwebsiteathttps://www.vagrantup.com/.ThedetailsofhowtodownloadandinstallVirtualBoxcanbefoundathttps://www.virtualbox.org/;again,VirtualBoxcanbeinstalledonLinux,OSX,andWindows.

WhothisbookisforThisbookisforDevOpsengineersandsystemadministratorswhowanttomanageDockercontainers,bettermanagethesecontainersusingexperttechniquesandmethods,andbettermaintainapplicationsbuiltonDocker.

ConventionsInthisbook,youwillfindanumberoftextstylesthatdistinguishbetweendifferentkindsofinformation.Herearesomeexamplesofthesestylesandanexplanationoftheirmeaning.

Codewordsintext,databasetablenames,foldernames,filenames,fileextensions,pathnames,dummyURLs,userinput,andTwitterhandlesareshownasfollows:“Wecanincludeothercontextsthroughtheuseoftheincludedirective.”

Ablockofcodeissetasfollows:

{

"fields":{

"@timestamp":[

1444567706641

]

},

"sort":[

1444567706641

]

}

Whenwewishtodrawyourattentiontoaparticularpartofacodeblock,therelevantlinesoritemsaresetinbold:

{

"fields":{

"@timestamp":[

1444567706641

]

},

"sort":[

1444567706641

]

}

Anycommand-lineinputoroutputiswrittenasfollows:

cd~/Documents/Projects/monitoring-docker/vagrant-ubuntu

vagrantup

Newtermsandimportantwordsareshowninbold.Wordsthatyouseeonthescreen,forexample,inmenusordialogboxes,appearinthetextlikethis:“ClickingtheNextbuttonmovesyoutothenextscreen.”

NoteWarningsorimportantnotesappearinaboxlikethis.

TipTipsandtricksappearlikethis.

ReaderfeedbackFeedbackfromourreadersisalwayswelcome.Letusknowwhatyouthinkaboutthisbook—whatyoulikedordisliked.Readerfeedbackisimportantforusasithelpsusdeveloptitlesthatyouwillreallygetthemostoutof.

Tosendusgeneralfeedback,simplye-mail<feedback@packtpub.com>,andmentionthebook’stitleinthesubjectofyourmessage.

Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineitherwritingorcontributingtoabook,seeourauthorguideatwww.packtpub.com/authors.

CustomersupportNowthatyouaretheproudownerofaPacktbook,wehaveanumberofthingstohelpyoutogetthemostfromyourpurchase.

DownloadingtheexamplecodeYoucandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.comforallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.

DownloadingthecolorimagesofthisbookWealsoprovideyouwithaPDFfilethathascolorimagesofthescreenshots/diagramsusedinthisbook.Thecolorimageswillhelpyoubetterunderstandthechangesintheoutput.Youcandownloadthisfilefrom:http://www.packtpub.com/sites/default/files/downloads/Monitoring_Docker_ColorImages.pdf

ErrataAlthoughwehavetakeneverycaretoensuretheaccuracyofourcontent,mistakesdohappen.Ifyoufindamistakeinoneofourbooks—maybeamistakeinthetextorthecode—wewouldbegratefulifyoucouldreportthistous.Bydoingso,youcansaveotherreadersfromfrustrationandhelpusimprovesubsequentversionsofthisbook.Ifyoufindanyerrata,pleasereportthembyvisitinghttp://www.packtpub.com/submit-errata,selectingyourbook,clickingontheErrataSubmissionFormlink,andenteringthedetailsofyourerrata.Onceyourerrataareverified,yoursubmissionwillbeacceptedandtheerratawillbeuploadedtoourwebsiteoraddedtoanylistofexistingerrataundertheErratasectionofthattitle.

Toviewthepreviouslysubmittederrata,gotohttps://www.packtpub.com/books/content/supportandenterthenameofthebookinthesearchfield.TherequiredinformationwillappearundertheErratasection.

PiracyPiracyofcopyrightedmaterialontheInternetisanongoingproblemacrossallmedia.AtPackt,wetaketheprotectionofourcopyrightandlicensesveryseriously.IfyoucomeacrossanyillegalcopiesofourworksinanyformontheInternet,pleaseprovideuswiththelocationaddressorwebsitenameimmediatelysothatwecanpursuearemedy.

Pleasecontactusat<copyright@packtpub.com>withalinktothesuspectedpiratedmaterial.

Weappreciateyourhelpinprotectingourauthorsandourabilitytobringyouvaluablecontent.

QuestionsIfyouhaveaproblemwithanyaspectofthisbook,youcancontactusat<questions@packtpub.com>,andwewilldoourbesttoaddresstheproblem.

Chapter1.IntroductiontoDockerMonitoringDockerhasbeenarecentbutveryimportantadditiontoaSysAdminstoolbox.

Dockerdescribesitselfasanopenplatformforbuilding,shipping,andrunningdistributedapplications.Thismeansthatdeveloperscanbundletheircodeandpassittotheiroperationsteam.Fromhere,theycandeploysafeintheknowledgethatitwillbedonesoinawaythatintroducesconsistencywiththeenvironmentinwhichthecodeisrunning.

Whenthisprocessisfollowed,itshouldmaketheage-olddevelopersversusoperationsargumentof“itworkedonmylocaldevelopmentserver”—athingofthepast.Sincebeforeits“productionready”1.0releasebackinJune2014,therehadbeenover10,000Dockerizedapplicationsavailable.Bytheendof2014,thatnumberhadrisentoover71,000.YoucanseehowDockergrewin2014bylookingattheinfographicthatwaspublishedbyDockerinearly2015,whichcanbefoundathttps://blog.docker.com/2015/01/docker-project-2014-a-whirlwind-year-in-review/.

Whilethedebateisstillragingabouthowproductionreadythetechnologyis,Dockerhasgainedanimpressivelistoftechnologypartners,includingRedHat,Canonical,HP,andevenMicrosoft.

CompaniessuchasGoogle,Spotify,Soundcloud,andCenturyLink,haveallopensourcedtoolsthatsupportDockerinsomeway,shape,orformandtherehasalsobeennumerousindependentdeveloperswhohavereleasedappsthatprovideadditionalfunctionalitytothecoreDockerproductset.Also,allthecompanieshavesprunguparoundtheDockerecosystem.

Thisbookassumesthatyouhavehadsomelevelofexperiencebuilding,running,andmanagingDockercontainers,andthatyouwouldnowliketostarttometricsfromyourrunningapplicationstofurthertunethem,orthatyouwouldliketoknowwhenaproblemoccurswithacontainersothatyoucandebuganyongoingissues.

IfyouhaveneverusedDockerbefore,youmaywanttotryoneoftheexcellentbooksthatserveandintroduceyoutoallthethingsthatDockerprovides,bookssuchasLearningDocker,PacktPublishing,orDocker’sownintroductiontocontainers,whichcanbefoundattheirdocumentationpages,asfollows:

LearningDocker:https://www.packtpub.com/virtualization-and-cloud/learning-dockerOfficialDockerdocs:https://docs.docker.com/

Now,wehaveabroughtourselvesuptospeedwithwhatDockeris;therestofthischapterwillcoverthefollowingtopics:

Howdifferentisittomonitorcontainersversusmoretraditionalserverssuchasvirtualmachines,baremetalmachine,andcloudinstances(Pets,Cattle,Chickens,andSnowflakes).

WhataretheminimumversionsofDockeryoushouldberunning?HowtofollowinstructionsonbringingupanenvironmentlocallyusingVagrantinordertofollowthepracticalexercisesinthisbook

Pets,Cattle,Chickens,andSnowflakesBeforewestartdiscussingthevariouswaysinwhichyoucanmonitoryourcontainers,weshouldgetanunderstandingofwhataSysAdminsworldlookslikethesedaysandalsowherecontainersfitintoit.

AtypicalSysAdminwillprobablybelookingafteranestateofserversthatarehostedineitheranon-siteorthird-partydatacenter,somemayevenmanageinstanceshostedinapubliccloudsuchasAmazonWebServicesorMicrosoftAzure,andsomeSysAdminsmayjugglealltheirserverestatesacrossmultiplehostingenvironments.

Eachofthesedifferentenvironmentshasitsownwayofdoingthings,aswellasperformingbestpractices.BackinFebruary2012,RandyBiasgaveatalkatCloudscalingthatdiscussedarchitecturesforopenandscalableclouds.Towardstheendoftheslidedeck,RandyintroducedtheconceptofPetsversusCattle(whichheattributestoBillBaker,whowasthenanengineeratMicrosoft).

Youcanviewtheoriginalslidedeckathttp://www.slideshare.net/randybias/architectures-for-open-and-scalable-clouds.

PetsversusCattleisnowwidelyacceptedasagoodanalogytodescribemodernhostingpractices.

PetsPetsareakintotraditionalphysicalserversorvirtualmachines,asfollows:

Eachpethasaname;forexample,myserver.domain.com.Whenthey’renotwell,youtakethemtothevettohelpthemgetbetter.YouemploySysAdminstolookafterthem.Youpaycloseattentiontothem,sometimesforyears.Youtakebackups,patchthem,andensurethattheyarefullydocumented.

CattleCattle,ontheotherhand,representmoremoderncloudcomputinginstances,asfollows:

You’vegottoomanytoname,soyougivethemnumbers;forexample,theURLcouldlooksomethinglikeip123123123123.eu.public-cloud.com.Whentheygetsick,youshootthemandifyourherdrequiresit,youreplaceanythingyou’vekilled:Aservercrashesorshowssignsthatitishavingproblems,youterminateitandyourconfigurationautomaticallyreplacesitwithanexactreplica.Youputtheminafieldandwatchthemfromfarandyoudon’texpectthemtolivelong.Ratherthanmonitoringtheindividualinstances,youmonitorthecluster.Whenmoreresourcesareneeded,youaddmoreinstancesandoncetheresourceisnolongerrequired,youterminatetheinstancestogetyoubacktoyourbaseconfiguration.

ChickensNextupisatermthatisagoodwayofdescribinghowcontainersfitintothePetsversusCattleworld;inablogposttitle“CloudComputing:Pets,Cattleand…Chickens?”onActiveState,BernardGoldendescribescontainersasChickens:

They’remoreefficientthancattlewhenitcomestoresourceuse.Acontainercanbootinsecondswhereainstanceorservercantakeminutes;italsouseslessCPUpowerthanatypicalvirtualmachineorcloudinstance.Therearemanymorechickensthancattle.Youcanquitedenselypackcontainersontoyourinstancesorservers.Chickenstendtohaveashorterlifespanthancattleandpets.Containerslendthemselvestorunningmicros-services;thesecontainersmayonlybeactiveforafewminutes.

Theoriginalblogpostcanbefoundathttp://www.activestate.com/blog/2015/02/cloud-computing-pets-cattle-and-chickens.

SnowflakesThefinaltermisnotanimal-relatedanditdescribesatypeofserverthatyoudefiantlydon’twanttohaveinyourserverestate,aSnowflake.ThistermwaspennedbyMartinFowlerinablogposttitled“SnowflakeServer”.Snowflakesisatermappliedto“legacy”or“inherited”servers:

Snowflakesaredelicateandaretreatedwithkidgloves.Typically,theserverhasbeeninthedatacentersinceyoustarted.Nooneknowswhooriginallyconfigureditandthereisnodocumentationofit;allyouknowisthatitisimportant.Eachoneisuniqueandisimpossibletoexactlyreproduce.EventhemosthardenedSysAdminfearstorebootthemachineincaseitdoesn’tbootafterwards,asitisrunningend-of-lifesoftwarethatcannoteasilybereinstalled.

Martin’spostcanbefoundathttp://martinfowler.com/bliki/SnowflakeServer.html.

Sowhatdoesthisallmean?Dependingonyourrequirementsandtheapplicationyouwanttodeploy,yourcontainerscanbelaunchedontoeitherpetorcattlestyleservers.Youcanalsocreateaclutchofchickensandhaveyourcontainersrunmicro-services.

Also,intheory,youcanreplaceyourfearedsnowflakeserverswithacontainer-basedapplicationthatmeetsalltheend-of-lifesoftwarerequirementswhileremainingdeployableonamodernsupportableplatform.

Eachofthedifferentstylesofserverhasdifferentmonitoringrequirements,inthefinalchapterwewilllookatPets,Cattle,Chickens,andSnowflakesagainanddiscussthetoolswehavecoveredinthecomingchapters.Wewillalsocoverbestpracticesyoushouldtakeintoconsiderationwhenplanningyourmonitoring.

DockerWhileDockerhititsversion1.0milestoneoverayearago,itisstillinit’sinfancy;witheachnewreleasecomesnewfeatures,bugfixes,andevensupportforsomeearlyfunctionalitythatisbeingdepreciated.

Dockeritselfisnowacollectionofseveralsmallerprojects;theseincludethefollowing:

DockerEngineDockerMachineDockerComposeDockerSwarmDockerHubDockerRegistryKitmatic

Inthisbook,wewillbeusingDockerEngine,DockerCompose,andtheDockerHub.

DockerEngineisthecorecomponentoftheDockerprojectanditprovidesthemainbulkoftheDockerfunctionality.WheneverDockerorthedockercommandismentionedinthisbook,IwillbereferringtoDockerEngine.

ThebookassumesyouhaveDockerEngineversion1.71orlaterinstalled;olderversionsofDockerEnginemaynotcontainthenecessaryfunctionalityrequiredtorunthecommandsandsoftwarecoveredintheupcomingchapters.

DockerComposestarteditslifeasathird-partyorchestrationtoolcalledFigbeforebeingpurchasedbyDockerin2014.Itisdescribedasawayofdefiningamulti-containerapplicationusingYAML(http://yaml.org).Simplyput,thismeansthatyouquicklydeploycomplexapplicationsusingasinglecommandthatcallsahumanreadableconfigurationfile.

WeassumethatyouhaveDockerCompose1.3.3orlaterinstalled;thedocker-compose.ymlfilesmentionedinthisbookhavebeenwrittenwiththisversioninmind.

Finally,themajorityoftheimageswewillbedeployingduringthisbookwillbesourcedfromtheDockerHub(https://hub.docker.com/),whichnotonlyhousesapublicregistrycontainingover40,000publicimagesbutalso100officialimages.ThefollowingscreenshotshowstheofficialrepositorieslistingontheDockerHubwebsite:

YoucanalsosignupandusetheDockerHubtohostyourownpublicandprivateimages.

LaunchingalocalenvironmentWhereverpossible,Iwilltrytoensurethatthepracticalexercisesinthisbookwillbeabletoberunonalocalmachinesuchasyourdesktoporlaptop.Forthepurposesofthisbook,IwillassumethatyourlocalmachineisrunningeitherarecentversionOSXoranup-to-dateLinuxdistributionandhasahighenoughspecificationtorunthesoftwarementionedinthischapter.

ThetwotoolswewillbeusingtolaunchourDockerinstanceswillalsorunonWindows;therefore,itshouldbepossibletofollowtheinstructionswithinthis,althoughyoumayhavetorefertheusageguidesforanychangestothesyntax.

DuetothewayinwhichDockerisarchitected,alotofthecontentofthisbookwillhaveyourunningcommandsandinteractingwiththecommandlineonthevirtualserverthatisactingasthehostmachine,ratherthanthecontainersthemselves.Becauseofthis,wewillnotbeusingeitherDockerMachineorKitematic.

BothofthesearetoolsprovidedbyDockertoquicklybootstrapaDocker-enabledvirtualserveronyourlocalmachine,asunfortunatelythehostmachinesdeployedbythesetoolscontainastrippeddownoperatingsystemthatisoptimizedforrunningDockerwiththesmallestfootprintaspossible.

Aswewillbeinstallingadditionalpackagesonthehostmachines,astrippeddown“Dockeronly”operatingsystemmaynothavethecomponentsavailabletomeettheprerequisitesofthesoftwarethatwewillberunninginthelaterchapters;therefore,toensurethattherearenoproblemsfurtheron,weberunningafulloperatingsystem.

Personally,IpreferaRPM-basedoperatingsystemsuchasRedHatEnterpriseLinux,Fedora,orCentOS,asIhavebeenusingthemprettymuchsincethedayIfirstloggedintoaLinuxserver.

However,asalotofreaderswillbefamiliarwiththeDebian-basedUbuntu,Iwillbeprovidingpracticalexamplesforbothoperatingsystems.

Toensuretheexperienceisasconsistentaspossible,wewillbeinstallingVagrantandVirtualBoxtorunthevirtualmachinethatwillactasahosttorunourcontainers.

Vagrant,writtenbyMitchellHashimoto,isacommandlinetoolforcreatingandconfiguringreproducibleandportablevirtualmachineenvironments.TherehavebeennumerousblogpostsandarticlesthatactuallypitchDockeragainstVagrant;however,inourcase,thetwotechnologiesworkquitewelltogetherinprovidingarepeatableandconsistentenvironment.

VagrantisavailableforLinux,OSX,andWindows.Fordetailsonhowtoinstall,gototheVagrantwebsiteathttps://www.vagrantup.com/.

VirtualBoxisagreatallroundopensourcevirtualizationplatformoriginallydevelopedbySunandnowmaintainedbyOracle.Itallowsyoutorunboth32-bitand64-bitguestoperatingsystemsonyourlocalmachine.DetailsonhowtodownloadandinstallVirtualBoxcanbefoundathttps://www.virtualbox.org/;again,VirtualBoxcanbeinstalled

onLinux,OSX,andWindows.

CloningtheenvironmentThesourcefortheenvironmentalongwiththepracticalexamplescanbefoundonGitHubintheMonitoringDockerrepositoryathttps://github.com/russmckendrick/monitoring-docker.

Toclonetherepositoryonaterminalonyourlocalmachine,runthefollowingcommands(replacingthefilepathasneeded):

mkdir~/Documents/Projects

cd~/Documents/Projects/

gitclonehttps://github.com/russmckendrick/monitoring-docker.git

Oncecloned,youshouldseeadirectorycalledmonitoring-dockerandthenenterthatdirectory,asfollows:

cd~/Documents/Projects/monitoring-docker

RunningavirtualserverIntherepository,youwillfindtwofolderscontainingthenecessaryVagrantfiletolauncheitheraCentOS7oraUbuntu14.04virtualserver.

IfyouwouldliketousetheCentOS7vagrantbox,changethedirectorytovagrant-centos:

cdvagrant-centos

Onceyouareinthevagrant-centosdirectory,youwillseethatthereisaVagrantfile;thisfileisallyouneedtolaunchaCentOS7virtualserver.Afterthevirtualserverhasbeenbooted,thelatestversionofdockeranddocker-composewillbeinstalledandthemonitoring-dockerdirectorywillalsobemountedinsidethevirtualmachineusingthemountpoint/monitoring-docker.

Tolaunchthevirtualserver,simplytypethefollowingcommand:

vagrantup

Thiswilldownloadthelatestversionofthevagrantboxfromhttps://atlas.hashicorp.com/russmckendrick/boxes/centos71andthenbootthevirtualserver;it’sa450MBdownloadsoitmaytakeseveralminutestodownload;itonlyhastodothisonce.

Ifallgoeswell,youshouldseesomethingsimilartothefollowingoutput:

Nowthatyouhavebootedthevirtualserver,youcanconnecttoitusingthefollowingcommand:

vagrantssh

Onceloggedin,youshouldverifythatdockeranddocker-composearebothavailable:

Finally,youcantryrunningthehello-worldcontainerusingthefollowingcommand:

dockerrunhello-world

Ifeverythinggoesasexpected,youshouldseethefollowingoutput:

Totrysomethingmoreambitious,youcanrunanUbuntucontainerwiththefollowingcommand:

dockerrun-itubuntubash

BeforewelaunchandentertheUbuntucontainer,letsconfirmthatwearerunningtheCentOShostmachinebycheckingthereleasefilethatcanbefoundin/etc:

Now,wecanlaunchtheUbuntucontainer.Usingthesamecommand,wecanconfirmthatweareinsidetheUbuntucontainerbyviewingitsreleasefile:

Toexitthecontainerjusttypeinexit.Thiswillstopthecontainerfromrunning,asithasterminatedtheonlyrunningprocesswithinthecontainer,whichwasbash,andreturnedyoutothehostCentOSmachine.

AsyoucanseeherefromourCentOS7host,wehavelaunchedandremovedanUbuntucontainer.

BoththeCentOS7andUbuntuVagrantfileswillconfigureastaticIPaddressonyourvirtualmachine.Itis192.168.33.10;also,thereisaDNSrecordforthisIPaddress

availableatdocker.media-glass.es.Thesewillallowyoutoaccessanycontainersthatexposethemselvestoabrowserateitherhttp://192.168.33.10/orhttp://docker.media-glass.es/.

TipTheURLhttp://docker.media-glass.es/willonlyworkwhilethevagrantboxisup,andyouhaveacontainerrunningwhichservesWebpages.

Youcanseethisinactionbyrunningthefollowingcommand:

dockerrun-d-p80:80russmckendrick/nginx-php

TipDownloadingtheexamplecode

Youcandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.comforallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.

ThiswilldownloadandlaunchacontainerrunningNGINX.Youcanthengotohttp://192.168.33.10/orhttp://docker.media-glass.es/inyourbrowser;youshouldseeaforbiddenpage.ThisisbecausewehavenotyetgivenNGINXanycontenttoserve(moreonthiswillbecoveredlaterinthebook):

Formoreexamplesandideas,gotothewebsiteathttp://docs.docker.com/userguide/.

HaltingthevirtualserverTologoutofthevirtualserverandreturntoyourlocalmachine,youtypeexit.

Youshouldnowseeyourlocalmachine’sterminalprompt;however,thevirtualserveryoubootedwillstillberunninginthebackgroundhappily,usingresources,untilyoueitherpoweritdownusingthefollowingcommand:

vagranthalt

Terminatethevirtualserveraltogetherusingvagrantdestroy:

vagrantdestroy

Tocheckthecurrentstatusofthevirtualserver,youcanrunthefollowingcommand:

vagrantstatus

Theresultoftheprecedingcommandisgiveninthefollowingoutput:

Eitherpoweringthevirtualserverbackonorcreatingitfromscratchagain,canbeachievedbyissuingthevagrantupcommandagain.

TheprecedingdetailsshowhowtousetheCentOS7vagrantbox.IfyouwouldprefertolaunchanUbuntu14.04virtualserver,youcandownloadandinstallthevagrantboxbygoingintothevagrant-ubuntudirectoryusingthefollowingcommand:

cd~/Documents/Projects/monitoring-docker/vagrant-ubuntu

vagrantup

Fromhere,youwillbeablerunvagrantupandfollowthesameinstructionsusedtobootandinteractwiththeCentOS7virtualserver.

SummaryInthischapter,wetalkedaboutdifferenttypesofserverandalsodiscussedhowyourcontainerizedapplicationscanfitintoeachofthecategories.WehavealsoinstalledVirtualBoxandusedVagranttolauncheitheraCentOS7orUbuntu14.04virtualserver,withdockeranddocker-composeinstalled.

Ournewvirtualserverenvironmentwillbeusedthroughouttheupcomingchapterstotestthevariousdifferenttypesofmonitoring.Inthenextchapter,wewillstartourjourneybyusingDocker’sin-builtfunctionalitytoexploremetricsaboutourrunningcontainers.

Chapter2.UsingtheBuilt-inToolsInthelaterchaptersofthisbook,wewillexplorethemonitoringpartsofthelargeeco-systemthathasstartedtoflourisharoundDockeroverthelast24months.However,beforewepressaheadwiththat,weshouldtakealookatwhatispossiblewithavanillainstallationofDocker.Inthischapter,wewillcoverthefollowingtopics:

UsingthetoolsbuiltintoDockertogetreal-timemetricsoncontainerperformanceUsingstandardoperatingsystemcommandstogetmetricsonwhatDockerisdoingGeneratingatestloadsoyoucanviewthemetricschanging

DockerstatsSinceversion1.5,therehasbeenabasicstatisticcommandbuiltintoDocker:

dockerstats--help

Usage:dockerstats[OPTIONS]CONTAINER[CONTAINER…]

Displayalivestreamofoneormorecontainers'resourceusagestatistics

--help=falsePrintusage

--no-stream=falseDisablestreamingstatsandonlypullthefirst

result

Thiscommandwillstreamdetailsoftheresourceutilizationofyourcontainersinrealtime.Thebestwaytofindoutaboutthecommandistoseeitinaction.

RunningDockerstatsLet’slaunchacontainerusingthevagrantenvironment,whichwecoveredinthelastchapter:

[russ@mac~]$cd~/Documents/Projects/monitoring-docker/vagrant-centos/

[russ@mac~]$vagrantup

Bringingmachine'default'upwith'virtualbox'provider…

==>default:Importingbasebox'russmckendrick/centos71'...

==>default:MatchingMACaddressforNATnetworking…

==>default:Checkingifbox'russmckendrick/centos71'isuptodate…

.....

==>default:=>Installingdocker-engine…

==>default:=>Configuringvagrantuser…

==>default:=>Startingdocker-engine…

==>default:=>Installingdocker-compose…

==>default:=>FinishedinstallationofDocker

[russ@mac~]$vagrantssh

Nowthatyouareconnectedtothevagrantserver,launchthecontainerusingtheDockercomposefilein/monitoring_docker/Chapter01/01-basic/:

[vagrant@centos7~]$cd/monitoring_docker/Chapter01/01-basic/

[vagrant@centos701-basic]$docker-composeup-d

Creating01basic_web_1…

Youhavenowpulleddownandlaunchedacontainerinthebackground.Thecontaineriscalled01basic_web_1anditrunsNGINXandPHPservingasinglePHPinformationpage(http://php.net/manual/en/function.phpinfo.php).

Tocheckwhethereverythinghasbeenlaunchedasexpected,rundocker-composeps.YoushouldseeyoursinglecontainerwithStateofUp:

[vagrant@centos701-basic]$docker-composeps

NameCommandStatePorts

---------------------------------------------------------------

01basic_web_1/usr/local/bin/runUp0.0.0.0:80->80/tcp

Finally,youshouldbeabletoseethepagecontainingtheoutputofthePHPinformationathttp://192.168.33.10/(thisIPaddressishardcodedintothevagrantconfiguration),ifyouputitinyourlocalbrowser:

Now,youhaveacontainerupandrunning;let’slookatsomeofthebasicstats.Weknowfromtheoutputofdocker-composethatourcontaineriscalled01basic_web_1,soenterthefollowingcommandtostartstreamingstatisticsinyourterminal:

dockerstats01basic_web_1

Itwilltakeasecondtoinitiate;afterthisisdone,youshouldseeyourcontainerlistedalongwiththestatisticsforthefollowing:

CPU%:ThisshowsyouhowmuchoftheavailableCPUresourcethecontaineriscurrentlyusing.MEMUSEAGE/LIMIT:ThistellsyouhowmuchRAMthecontainerisutilizing;italsodisplayshowmuchallowancethecontainerhas.Ifyouhaven’texplicitlysetalimit,itwillshowthetotalamountofRAMonthehostmachine.MEM%:ThisshowsyouwhatpercentageoftheRAMallowancethecontainerisusing.

NETI/O:Thisgivesarunningtotalofhowmuchbandwidthhasbeentransferredinandoutofthecontainer.

Ifyougobacktoyourbrowserwindowandstarttorefreshhttp://192.168.33.10/,youwillseethatthevaluesineachofthecolumnsstarttochange.Tostopstreamingthestatistics,pressCtrl+c.

Ratherthankeepingonhittingrefreshoverandoveragain,let’sgeneratealotoftrafficto01basic_web_1,whichshouldputthecontainerunderaheavyload.

Here,wewilllaunchacontainerthatwillsend10,000requeststo01basic_web_1usingApacheBench(https://httpd.apache.org/docs/2.2/programs/ab.html).Althoughitwilltakeaminuteortwotoexecute,weshouldrundockerstatsassoonaspossible:

dockerrun-d--name=01basic_load--link=01basic_web_1russmckendrick/abab

-k-n10000-c5http://01basic_web_1/&&dockerstats01basic_web_1

01basic_load

AftertheApacheBenchimagehasbeendownloadedandthecontainerthatwillbecalled01basic_loadstarts,youshouldseethestatisticsforboth01basic_web_1and01basic_loadbegintostreaminyourterminal:

CONTAINERCPU%MEMUSAGE/LIMITMEM%NETI/O

01basic_load18.11%12.71MB/1.905GB0.67%335.2MB/5.27MB

01basic_web_1139.62%96.49MB/1.905GB5.07%5.27MB/335.2MB

Afterawhile,youwillnoticethatmostofthestatisticsfor01basic_loadwilldropofftozero;thismeansthatthetesthascompletedandthatthecontainerrunningthetesthasexited.Thedockerstatscommandcanonlystreamstatisticsfortherunningcontainers;onesthathaveexitedarenolongerrunningand,therefore,donotproduceoutputwhenrunningdockerstats.

ExitfromdockerstatsusingCtrl+c;toseetheresultsoftheApacheBenchcommand,youcantypedockerlogs01basic_load;youshouldseesomethinglikethefollowingscreenshot:

Youshouldn’tworryifyouseeanyfailureslikeintheprecedingoutput.ThisexercisewaspurelytodemonstratehowtoviewthestatisticsoftherunningcontainersandnottotuneawebservertohandletheamountoftrafficwesenttoitusingApacheBench.

Toremovethecontainersthatwelaunched,runthefollowingcommands:

[vagrant@centos701-basic]$docker-composestop

Stopping01basic_web_1…

[vagrant@centos701-basic]$docker-composerm

Goingtoremove01basic_web_1

Areyousure?[yN]y

Removing01basic_web_1…

[vagrant@centos701-basic]$dockerrm01basic_load

01basic_load

Tocheckwhethereverythinghasbeenremovedsuccessfully,rundockerps-aandyoushouldnotbeabletoseeanyrunningorexitedcontainersthathave01basic_intheirnames.

Whatjusthappened?WhilerunningtheApacheBenchtest,youmayhavenoticedthattheCPUutilizationonthecontainerrunningNGINXandPHPwashigh;intheexampleintheprevioussection,itwasusing139.62percentoftheavailableCPUresource.

Aswedidnotattachanyresourcelimitstothecontainerswelaunched,itwaseasyforourtesttousealloftheavailableresourcesonthehostVirtualMachine(VM).IfthisVMwasbeingusedbyseveralusers,allrunningtheirowncontainers,theymayhavestartedtonoticethattheirapplicationshadstartedtoslowdownor,evenworse,theapplicationshadstartedshowingerrors.

Ifyoueverfindyourselfinthissituation,youcanusedockerstatstohelptrackdowntheculprit.

Runningdockerstats$(dockerps-q)willstreamthestatisticsforallthecurrentlyrunningcontainers:

CONTAINERCPU%MEMUSAGE/LIMITMEM%NETI/O

361040b7b33e0.07%86.98MB/1.905GB4.57%2.514kB/738B

56b459ae9092120.06%87.05MB/1.905GB4.57%2.772kB/738B

a3de616f84ba0.04%87.03MB/1.905GB4.57%2.244kB/828B

abdbee7b52070.08%86.61MB/1.905GB4.55%3.69kB/738B

b85c49cf740c0.07%86.15MB/1.905GB4.52%2.952kB/738B

Asyoumayhavenoticed,thisdisplaysthecontainerIDratherthanthename;thisinformationshould,however,beenoughtospottheresourcehogsothatyoucanquicklystopit:

[vagrant@centos701-basic]$dockerstop56b459ae9092

56b459ae9092

Oncestopped,youcanthengetthenameoftheroguecontainerbyrunningthefollowingcommand:

[vagrant@centos701-basic]$dockerps-a|grep56b459ae9092

56b459ae9092russmckendrick/nginx-php"/usr/local/bin/run"9

minutesagoExited(0)26secondsagomy_bad_container

Alternatively,formoredetailedinformation,youcanrundockerinspect56b459ae9092,whichwillgiveyoualltheinformationyouneedonthecontainer.

Whataboutprocesses?OneofthegreatthingsaboutDockeristhatitisn’treallyvirtualization;asmentionedinthepreviouschapter,itisagreatwayofisolatingprocessesratherthanrunninganentireoperatingsystem.

Thiscangetconfusingwhenrunningtoolssuchastoporps.Togetanideajusthowconfusingthiscanget,letslaunchseveralcontainersusingdocker-composeandseeforourselves:

[vagrant@centos7~]$cd/monitoring_docker/Chapter01/02-multiple

[vagrant@centos702-multiple]$docker-composeup-d

Creating02multiple_web_1…

[vagrant@centos702-multiple]$docker-composescaleweb=5

Creating02multiple_web_2…

Creating02multiple_web_3…

Creating02multiple_web_4…

Creating02multiple_web_5…

Starting02multiple_web_2…

Starting02multiple_web_3…

Starting02multiple_web_4…

Starting02multiple_web_5…

Now,wehavefivewebserversthathaveallbeenlaunchedfromthesameimageusingthesameconfiguration.OneofthefirstthingsIdowhenloggingintoaservertotroubleshootaproblemisrunps-aux;thiswillshowalltherunningprocesses.Asyoucansee,whenrunningthecommand,therearealotprocesseslisted.

EvenjusttryingtolookattheprocessesforNGINXisconfusing,asthereisnothingtodifferentiatetheprocessesfromonecontainertoanother,asshowninthefollowingoutput:

So,howcanyouknowwhichcontainerownswhichprocesses?

DockertopThiscommandlistsalltheprocessesthatarerunningwithinacontainer;thinkofitasawayoffilteringtheoutputofthepsauxcommandweranonthehostmachine:

Asdockertopisanimplementationofthestandardpscommand,anyflagsyouwouldnormallypasstopsshouldworkasfollows:

[vagrant@centos702-multiple]$dockertop02multiple_web_3–aux

[vagrant@centos702-multiple]$dockertop02multiple_web_3-faux

DockerexecAnotherwaytoviewwhatisgoingonwithinacontaineristoenterit.Toenableyoutodothis,Dockerintroducedthedockerexeccommand.Thisallowsyoutospawnanadditionalprocesswithinanalreadyrunningcontainerandthenattachtotheprocess;so,ifwewantedtolookatwhatiscurrentlyrunningon02multiple_web_3,weshouldusethefollowingcommandspawnabashshellwithinanalreadyrunningcontainer:

dockerexec-t-i02multiple_web_3bash

Onceyouhaveanactiveshellonthecontainer,youwillnoticethatyourprompthaschangedtothecontainer’sID.Yoursessionisnowisolatedtothecontainer’senvironment,meaningthatyouwillonlybeabletointeractwiththeprocessesbelongingtothecontaineryouentered.

Fromhere,youcanrunthepsauxortopcommandasyouwoulddoonthehostmachine,andonlyseetheprocessesassociatedwiththecontaineryouareinterestedin:

Toleavethecontainer,typeinexit,youshouldseeyourpromptchangebackinyourhostmachine.

Finally,youcanstopandremovethecontainersbyrunningdocker-composestopanddocker-composekill.

SummaryInthischapter,wesawhowwecangetreal-timestatisticsonourrunningcontainersandhowwecanusecommandsthatarefamiliartous,togetinformationontheprocessesthatarelaunchedaspartofeachcontainer.

Onthefaceofit,dockerstatsseemslikeareallybasicpieceoffunctionalitythatisn’treallyanythingmorethanatooltohelpyouidentifywhichcontainerisusingalltheresourceswhileaproblemisoccurring.However,theDockercommandisactuallypullingtheinformationfromaquitepowerfulAPI.

ThisAPIformsthebasisforalotofthemonitoringtoolswewillbelookingatinthenextfewchapters.

Chapter3.AdvancedContainerResourceAnalysisInthelastchapter,welookedathowyoucanusetheAPIbuiltintoDockertogainaninsighttowhatresourcesyourcontainersarerunning.Now,wearetoseehowwecantakeittothenextlevelbyusingcAdvisorfromGoogle.Inthischapter,youwillcoverthefollowingtopics:

HowtoinstallcAdvisorandstartcollectingmetricsLearnallaboutthewebinterfaceandreal-timemonitoringWhatyouroptionsareforshippingmetricstoaremotePrometheusdatabaseforlong-termstorageandtrendanalysis

WhatiscAdvisor?GoogledescribescAdvisorasfollows:

“cAdvisor(ContainerAdvisor)providescontainerusersanunderstandingoftheresourceusageandperformancecharacteristicsoftheirrunningcontainers.Itisarunningdaemonthatcollects,aggregates,processes,andexportsinformationaboutrunningcontainers.Specifically,foreachcontainer,itkeepsresourceisolationparameters,historicalresourceusage,histogramsofcompletehistoricalresourceusage,andnetworkstatistics.Thisdataisexportedbyacontainerandismachine-wide.”

TheprojectstartedofflifeasaninternaltoolatGoogleforgaininganinsightintocontainersthathadbeenlaunchedusingtheirowncontainerstack.

NoteGoogle’sowncontainerstackwascalled“LetMeContainThatForYou”orlmctfyforshort.TheworkonlmctfyhasbeeninstalledasaGoogleportfunctionalityovertolibcontainerthatispartoftheOpenContainerInitiative.Furtherdetailsonlmctfycanbefoundathttps://github.com/google/lmctfy/.

cAdvisoriswritteninGo(https://golang.org);youcaneithercompileyourownbinaryoryoucanusethepre-compiledbinarythataresuppliedviaacontainer,whichisavailablefromGoogle’sownDockerHubaccount.Youcanfindthisathttp://hub.docker.com/u/google/.

Onceinstalled,cAdvisorwillsitinthebackgroundandcapturemetricsthataresimilartothatofthedockerstatscommand.Wewillgothroughthesestatsandunderstandwhattheymeanlaterinthischapter.

cAdvisortakesthesemetricsalongwiththoseforthehostmachineandexposesthemviaasimpleandeasy-to-usebuilt-inwebinterface.

RunningcAdvisorusingacontainerThereareanumberofwaystoinstallcAdvisor;theeasiestwaytogetstartedistodownloadandrunthecontainerimagethatcontainsacopyofaprecompiledcAdvisorbinary.

BeforerunningcAdvisor,let’slaunchafreshvagranthost:

[russ@mac~]$cd~/Documents/Projects/monitoring-docker/vagrant-centos/

[russ@mac~]$vagrantup

Bringingmachine'default'upwith'virtualbox'provider…

==>default:Importingbasebox'russmckendrick/centos71'...

==>default:MatchingMACaddressforNATnetworking…

==>default:Checkingifbox'russmckendrick/centos71'isuptodate…

.....

==>default:=>Installingdocker-engine…

==>default:=>Configuringvagrantuser…

==>default:=>Startingdocker-engine…

==>default:=>Installingdocker-compose…

==>default:=>FinishedinstallationofDocker

[russ@mac~]$vagrantssh

TipUsingabackslash

Aswehavealotoptionstopasstothedockerruncommand,weareusing\tosplitthecommandovermultiplelinessoit’seasiertofollowwhatisgoingon.

Onceyouhaveaccesstothehostmachine,runthefollowingcommand:

dockerrun\

--detach=true\

--volume=/:/rootfs:ro\

--volume=/var/run:/var/run:rw\

--volume=/sys:/sys:ro\

--volume=/var/lib/docker/:/var/lib/docker:ro\

--publish=8080:8080\

--privileged=true\

--name=cadvisor\

google/cadvisor:latest

YoushouldnowhaveacAdvisorcontainerupandrunningonyourhostmachine.Beforewestart,let’slookatcAdvisorinmoredetailbydiscussingwhywehavepassedalltheoptionstothecontainer.

ThecAdvisorbinaryisdesignedtorunonthehostmachinealongsidetheDockerbinary,sobylaunchingcAdvisorinacontainer,weareactuallyisolatingthebinaryinitsdownenvironment.TogivecAdvisoraccesstotheresourcesitrequiresonthehostmachine,wehavetomountseveralpartitionsandalsogivethecontainerprivilegedaccesstoletthecAdvisorbinarythinkitisbeingexecutedonthehostmachine.

NoteWhenacontainerislaunchedwith--privileged,Dockerwillenablefullaccesstodevicesonthehostmachine;also,DockerwillconfigurebothAppArmororSELinuxtoallowyourcontainerthesameaccesstothehostmachineasaprocessrunningoutsidethecontainerwillhave.Forinformationonthe--privilegedflag,seethispostontheDockerblogathttp://blog.docker.com/2013/09/docker-can-now-run-within-docker/.

CompilingcAdvisorfromsourceAsmentionedintheprevioussection,cAdvisorreallyoughttobeexecutedonthehostmachine;thismeans,youmayhavetouseacasetocompileyourowncAdvisorbinaryandrunitdirectlyonthehost.

TocompilecAdvisor,youwillneedtoperformthefollowingsteps:

1. InstallGoandMercurialonthehostmachine—version1.3orhigherofGoisneededtocompilecAdvisor.

2. SetthepathforGotoworkfrom.3. GrabthesourcecodeforcAdvisorandgodep.4. SetthepathforyourGobinaries.5. BuildthecAdvisorbinaryusinggodeptosourcethedependenciesforus.6. Copythebinaryto/usr/local/bin/.7. DownloadeitheranUpstartorSystemdscriptandlaunchtheprocess.

Ifyoufollowedtheinstructionsintheprevioussection,youwillalreadyhaveacAdvisorprocessrunning.Beforecompilingfromsource,youshouldstartwithacleanhost;let’slogoutofthehostandlaunchafreshcopy:

[vagrant@centos7~]$exit

logout

Connectionto127.0.0.1closed.

[russ@mac~]$vagrantdestroy

default:Areyousureyouwanttodestroythe'default'VM?[y/N]y

==>default:ForcingshutdownofVM…

==>default:DestroyingVMandassociateddrives…

==>default:Runningcleanuptasksfor'shell'provisioner…

[russ@mac~]$vagrantup

Bringingmachine'default'upwith'virtualbox'provider…

==>default:Importingbasebox'russmckendrick/centos71'...

==>default:MatchingMACaddressforNATnetworking…

==>default:Checkingifbox'russmckendrick/centos71'isuptodate…

.....

==>default:=>Installingdocker-engine…

==>default:=>Configuringvagrantuser…

==>default:=>Startingdocker-engine…

==>default:=>Installingdocker-compose…

==>default:=>FinishedinstallationofDocker

[russ@mac~]$vagrantssh

TobuildcAdvisorontheCentOS7host,runthefollowingcommand:

sudoyuminstall-ygolanggitmercurial

exportGOPATH=$HOME/go

goget-dgithub.com/google/cadvisor

gogetgithub.com/tools/godep

exportPATH=$PATH:$GOPATH/bin

cd$GOPATH/src/github.com/google/cadvisor

godepgobuild.

sudocpcadvisor/usr/local/bin/

sudowgethttps://gist.githubusercontent.com/russmckendrick/f647b2faad5d92c96

771/raw/86b01a044006f85eebbe395d3857de1185ce4701/cadvisor.service-O

/lib/systemd/system/cadvisor.service

sudosystemctlenablecadvisor.service

sudosystemctlstartcadvisor

OntheUbuntu14.04LTShost,runthefollowingcommand:

sudoapt-get-yinstallsoftware-properties-common

sudoadd-apt-repositoryppa:evarlast/golang1.4

sudoapt-getupdate

sudoapt-get-yinstallgolangmercurial

exportGOPATH=$HOME/go

goget-dgithub.com/google/cadvisor

gogetgithub.com/tools/godep

exportPATH=$PATH:$GOPATH/bin

cd$GOPATH/src/github.com/google/cadvisor

godepgobuild.

sudocpcadvisor/usr/local/bin/

sudowgethttps://gist.githubusercontent.com/russmckendrick/f647b2faad5d92c96

771/raw/e12c100d220d30c1637bedd0ce1c18fb84beff77/cadvisor.conf-O

/etc/init/cadvisor.conf

sudostartcadvisor

YoushouldnowhavearunningcAdvisorprocess.Youcancheckthisbyrunningpsaux|grepcadvisorandyoushouldseeaprocesswithapathof/usr/local/bin/cadvisorrunning.

CollectingmetricsNow,youhavecAdvisorrunning;whatdoyouneedtodotoconfiguretheserviceinordertostartcollectingmetrics?Theshortansweris,nothingatall.WhenyoustartedthecAdvisorprocess,itinstantlystartedpollingyourhostmachinetofindoutwhatcontainersarerunningandgatheredinformationonboththerunningcontainersandyourhostmachine.

TheWebinterfacecAdvisorshouldberunningonthe8080port;ifyouopenhttp://192.168.33.10:8080/,youshouldbegreetedwiththecAdvisorlogoandanoverviewofyourhostmachine:

Thisinitialpagestreamslivestatsaboutthehostmachine,thougheachsectionisrepeatedwhenyoustarttodrilldownandviewthecontainers.Tostartwith,let’slookateachsectionusingthehostinformation.

OverviewThisoverviewsectiongivesyouabird’s-eyeviewofyoursystem;itusesgaugessoyoucanquicklygetanideaofwhichresourcesarereachingtheirlimits.Inthefollowingscreenshot,thereisverylittleinthewayofCPUutilizationandthefilesystemusageisrelativelylow;however,weareusing64%oftheavailableRAM:

ProcessesThefollowingscreenshotdisplaysacombinedviewoftheoutputofthepsaux,dockerpsandtopcommandsweusedinthepreviouschapter:

Hereiswhateachcolumnheadingmeans:

User:ThisshowswhichuserisrunningtheprocessPID:ThisistheuniqueprocessIDPPID:ThisisthePIDoftheparentprocessStartTime:ThisshowswhattimetheprocessstartedCPU%:ThisisthepercentageoftheCPUtheprocessiscurrentlyconsumingMEM%:ThisisthepercentageoftheRAMtheprocessiscurrentlyconsumingRSS:ThisshowshowmuchofthemainmemorytheprocessisusingVirtualSize:ThisshowshowmuchofthevirtualmemorytheprocessisusingStatus:Thisshowsthecurrentstatusoftheprocess;thisarethestandardLinuxprocessstatecodesRunningTime:ThisshowshowlongtheprocesshasbeenrunningCommand:ThisshowswhichcommandtheprocessisrunningContainer:Thisshowswhichcontainertheprocessisattachedto;thecontainerlistedas/isthehostmachine

Astherecouldbeseveralhundredprocessesactive,thissectionissplitintopages;youcannavigatetothesewiththebuttonsonthebottom-left.Also,youcansorttheprocessesbyclickingonanyoftheheadings.

CPUThefollowinggraphshowstheCPUutilizationoverthelastminute:

Hereiswhateachtermmeans:

TotalUsage:ThisshowsanaggregateusageacrossallcoresUsageperCore:ThisgraphbreaksdowntheusagepercoreUsageBreakdown(notshownintheprecedingscreenshot):Thisshowsaggregateusageacrossallcores,butbreaksitdowntowhatisbeingusedbythekernelandwhatisbeingusedbytheuser-ownedprocesses

MemoryTheMemorysectionissplitintotwoparts.Thegraphtellsyouthetotalamountofmemoryusedbyalltheprocessesforthehostorcontainer;thisisthetotalofthehotandcoldmemory.TheHotmemoryisthecurrentworkingset:pagesthathavebeentouchedbythekernelrecently.TheColdmemoryisthepagethathasn’tbeentouchedforawhileandcouldbereclaimedifneeded.

TheUsageBreakdowngivesavisualrepresentationofthetotalmemoryinthehostmachine,orallowanceinthecontainer,alongsidethetotalandhotusage:

NetworkThissectionshowstheincomingandoutgoingtrafficoverthelastminute.Youcanchangetheinterfaceusingthedrop-downboxonthetop-left.Thereisalsoagraphthatshowsanynetworkingerrors.Typically,thisgraphshouldbeflat.Ifitisn’t,thenyouwillbeseeingperformanceissueswithyourhostmachineorcontainer:

FilesystemThefinalsectiongivesabreakdownofthefilesystemusage.Inthefollowingscreenshot,/dev/sda1isthebootpartition,/dev/sda3isthemainfilesystem,and/dev/mapper/docker-8…isanaggregateofthewritefilesystemsofyourrunningcontainers:

ViewingcontainerstatsAtthetopofthepage,thereisalinkofyourrunningcontainers;youcaneitherclickonthelinkorgodirectlytohttp://192.168.33.10:8080/docker/.Oncethepageloads,youshouldseealistofallyourrunningcontainers,andalsoadetailedoverviewofyourDockerprocess,andfinallyalistoftheimagesyouhavedownloaded.

SubcontainersSubcontainersshowsalistofyourcontainers;eachentryisaclickablelinkthatwilltakeyoutoapagethatwillgiveyouthefollowingdetails:

Isolation:

CPU:ThisshowsyoutheCPUallowancesofthecontainer;ifyouhavenotsetanyresourcelimits,youwillseethehost’sCPUinformationMemory:Thisshowsyouthememoryallowancesofthecontainer;ifyouhavenotsetanyresourcelimits,yourcontainerwillshowanunlimitedallowance

Usage:

Overview:ThisshowsgaugessoyoucanquicklyseehowclosetoanyresourcelimitsyouareProcesses:ThisshowstheprocessesforjustyourselectedcontainerCPU:ThisshowstheCPUutilizationgraphsisolatedtojustyourcontainerMemory:Thisshowsthememoryutilizationofyourcontainer

DriverstatusThedrivergivesthebasicstatsonyourmainDockerprocess,alongwiththeinformationonthehostmachine’skernel,hostname,andalsotheunderlyingoperatingsystem.

Italsogivesinformationonthetotalnumberofcontainersandimages.Youmaynoticethatthetotalnumberofimagesisamuchlargerfigurethanyouexpectedtosee;thisisbecauseitiscountingeachfilesystemasanindividualimage.

NoteFormoredetailsonDockerimages,seetheDockeruserguideathttps://docs.docker.com/userguide/dockerimages/.

Italsogivesyouadetailedbreakdownofyourstorageconfiguration.

ImagesFinally,yougetalistoftheDockerimageswhichareavailableonthehostmachine.ItliststheRepository,Tag,Size,andwhentheimagewascreated,alongwiththeimages’uniqueID.Thisletsyouknowwheretheimageoriginatedfrom(Repository),whichversionoftheimageyouhavedownloaded(Tag)andhowbigtheimageis(Size).

Thisisallgreat,what’sthecatch?Soyouaremaybethinkingtoyourselfthatallofthisinformationavailableinyourbrowserisreallyuseful;beingabletoseereal-timeperformancemetricsinaneasilyreadableformatisareallyplus.

ThebiggestdrawbackofusingthewebinterfaceforcAdvisor,asyoumayhavenoticed,isthatitonlyshowsyouoneminute’sworthofmetrics;youcanquiteliterallyseetheinformationdisappearinginrealtime.

Asapaneofglassgivesareal-timeviewintoyourcontainers,cAdvisorisabrillianttool;ifyouwanttoreviewanymetricsthatareolderthanoneminute,youareoutofluck.

Thatis,unlessyouconfiguresomewheretostoreallofyourdata;thisiswherePrometheuscomesin.

PrometheusSowhat’sPrometheus?Itsdevelopersdescribeitasfollows:

Prometheusisanopen-sourcesystem’smonitoringandalertingtoolkitbuiltatSoundCloud.Sinceitsinceptionin2012,ithasbecomethestandardforinstrumentingnewservicesatSoundCloudandisseeinggrowingexternalusageandcontributions.

OK,butwhatdoesthathavetodowithcAdvisor?Well,Prometheushasquiteapowerfuldatabasebackendthatstoresthedataitimportsasatimeseriesofevents.

Wikipediadescribesatimeseriesasfollows:

“Atimeseriesisasequenceofdatapoints,typicallyconsistingofsuccessivemeasurementsmadeoveratimeinterval.Examplesoftimeseriesareoceantides,countsofsunspots,andthedailyclosingvalueoftheDowJonesIndustrialAverage.Timeseriesareveryfrequentlyplottedvialinecharts.”

https://en.wikipedia.org/wiki/Time_series

OneofthethingscAdvisordoes,bydefault,isexposeallthemetricsitiscapturingonasinglepageat/metrics;youcanseethisathttp://192.168.33.10:8080/metricsonourcAdvisorinstallation.Themetricsareupdatedeachtimethepageisloaded:

Asyoucanseeintheprecedingscreenshot,thisisjustasinglelongpageofrawtext.ThewayPrometheusworksisthatyouconfigureittoscrapethe/metricsURLatauser-definedinterval,let’ssayeveryfiveseconds;thetextisinaformatthatPrometheusunderstandsanditisingestedintothePrometheus’stimeseriesdatabase.

Whatthismeansisthat,usingPrometheus’spowerfulbuilt-inquerylanguage,youcanstarttodrilldownintoyourdata.Let’slookatgettingPrometheusupandrunning.

LaunchingPrometheusLikecAdvisorthereareseveralwaysyoucanlaunchPrometheus.Tostartwith,wewilllaunchacontainerandinjectourownconfigurationfilesothatPrometheusknowswhereourcAdvisorendpointis:

dockerrun\

--detach=true\

--

volume=/monitoring_docker/Chapter03/prometheus.yml:/etc/prometheus/promethe

us.yml\

--publish=9090:9090\

--name=prometheus\

prom/prometheus:latest

Onceyouhavelaunchedthecontainer,PrometheuswillbeaccessibleonthefollowingURL:http://192.168.33.10:9090.WhenyoufirstloadtheURL,youwillbetakentoastatuspage;thisgivessomebasicinformationonthePrometheusinstallation.Theimportantpartofthispageisthelistoftargets.ThisliststheURLthatPrometheuswillbescrappingtocapturemetrics;youshouldseeyourcAdvisorURLlistedwithastateofHEALTHY,asshowninthefollowingscreenshot:

Anotherinformationpagecontainsthefollowing:

Runtimeinformation:ThisdisplayshowlongPrometheushasbeenupandpollingdata,ifyouhaveconfiguredanendpointBuildinformation:ThiscontainsthedetailsoftheversionofPrometheusthatyouhavebeenrunningConfiguration:ThisisacopyoftheconfigurationfileweinjectedintothecontainerwhenitwaslaunchedRules:Thisisacopyofanyrulesweinjected;thesewillbeusedforalertingStartupflags:Thisshowsalltheruntimevariablesandtheirvalues

QueryingPrometheusAsweonlyhaveafewcontainersupandrunningatthemoment,let’slaunchonethatrunsRedissowecanstarttolookatthequerylanguagebuiltintoPrometheus.

WewillusetheofficialRedisimageforthisandasweareonlygoingtousethisasanexamplewewon’tneedtopassitanyuservariables:

dockerrun--namemy-redis-server-dredis

Wenowhaveacontainercalledmy-redis-serverrunning.cAdvisorshouldalreadybeexposingmetricsaboutthecontainertoPrometheus;let’sgoaheadandsee.InthePrometheuswebinterface,gototheGraphlinkinthemenuatthetopofthepage.Here,youwillbepresentedwithatextboxintowhichyoucanenteryourquery.Tostartwith,let’slookattheCPUusageoftheRediscontainer.

Inthebox,enterthefollowing:

container_cpu_usage_seconds_total{job="cadvisor",name="my-redis-server"}

Then,afterclickingonExecute,youshouldhavetworesultsreturned,listedintheConsoletabofthepage.Ifyouremember,cAdvisorrecordstheCPUusageofeachoftheCPUcoresthatthecontainerhasaccessto,whichiswhywehavetwovaluesreturned,onefor“cpu00”andonefor“cpu01”.ClickingontheGraphlinkwillshowyouresultsoveraperiodoftime:

Asyoucanseeintheprecedingscreenshot,wenowhaveaccesstotheusagegraphsforthelast25minutes,whichisabouthowlongagoIlaunchedtheRedisinstancebefore

generatingthegraph.

DashboardAlso,whencreatingoneofthegraphsusingthequerytoolinthemainapplication,youcaninstallaseparateDashboardapplication.ThisrunsinasecondcontainerthatconnectstoyourmainPrometheuscontainerusingtheAPIasadatasource.

BeforewestarttheDashboardcontainer,weshouldinitializeaSQLite3databasetostoreourconfiguration.Toensurethatthedatabaseispersistent,wewillstorethisonthehostmachinein/tmp/prom/file.sqlite3:

dockerrun\

--volume=/tmp/prom:/tmp/prom\

-eDATABASE_URL=sqlite3:/tmp/prom/file.sqlite3\

prom/promdash./bin/rakedb:migrate

Oncewehaveinitializedthedatabase,wecanlaunchtheDashboardapplicationproperly:

dockerrun\

--detach=true\

--volume=/tmp/prom:/tmp/prom\

-eDATABASE_URL=sqlite3:/tmp/prom/file.sqlite3\

--publish=3000:3000\

--name=promdash\

prom/promdash

Theapplicationshouldnowbeaccessibleathttp://192.168.33.10:3000/.Thefirstthingweneedtodoissetupthedatasource.Todothis,clickontheServerslinkatthetopofthescreenandthenclickonNewServer.Here,youwillbeaskedtoprovidethedetailsofyourPrometheusserver.NametheserverandenterthefollowingURL:

Name:cAdvisorURL:http://192.168.33.10:9090ServerType:Prometheus

OnceyouclickonCreateServer,youshouldreceiveamessagesayingServerwassuccessfullycreated.Nextup,youneedtocreateadirectory;thisiswhereyourdashboardswillbestored.

ClickontheDashboardslinkinthetopmenuandthenclickonNewdirectoryandcreateonecalledTestdirectory.Now,youarereadytostartcreatingDashboards.ClickonNewDashboard,callitMyDashboard,placeitinTestdirectory.OnceyouclickonCreateDashboard,youwillbetakentothepreviewscreen.

Fromhere,youcanbuildupdashboardsusingthecontrolinthetopright-handsideofeachsection.Toadddata,yousimplyenterthequeryyouwouldliketoseeinthedashboardsection:

NoteFordetailedinformationonhowtocreateDashboards,seethePROMDASHsectionofthePrometheusdocumentationathttp://prometheus.io/docs/visualization/promdash/.

ThenextstepsAtthemoment,wearerunningPrometheusinasinglecontaineranditsdataisbeingstoredwithinthatsamecontainer.Thismeans,ifforanyreasonthecontaineristerminated,ourdataislost;italsomeansthatwecan’tupgradewithoutloosingoutdata.Togetaroundthisproblem,wecancreateadatavolumecontainer.

NoteAdatavolumecontainerisaspecialtypeofcontainerthatonlyexistsasstorageforothercontainers.Formoredetails,seetheDockeruserguideathttps://docs.docker.com/userguide/dockervolumes/#creating-and-mounting-a-data-volume-container.

Firstofall,let’smakesurewehaveremovedalltherunningPrometheuscontainers:

dockerstopprometheus&&dockerrmPrometheus

Nextup,let’screateadatacontainercalledpromdata:

dockercreate\

--volume=/promdata\

--name=promdata\

prom/prometheus/bin/true

Finally,launchPrometheusagain,thistime,usingthedatacontainer:

dockerrun\

--detach=true\

--volumes-frompromdata\

--

volume=/monitoring_docker/Chapter03/prometheus.yml:/etc/prometheus/promethe

us.yml\

--publish=9090:9090\

--name=prometheus\

prom/prometheus

Thiswillensurethat,ifyouhavetoupgradeorrelaunchyourcontainer,themetricsyouhavebeencapturingaresafeandsound.

WehaveonlytouchedonthebasicsofusingPrometheusinthissectionofthebook;forfurtherinformationontheapplication,Irecommendthefollowinglinksasagoodstartingpoint:

Documentation:http://prometheus.io/docs/introduction/overview/Twitter:https://twitter.com/PrometheusIOProjectpage:https://github.com/prometheus/prometheusGooglegroups:https://groups.google.com/forum/#!forum/prometheus-developers

Alternatives?TherearesomealternativestoPrometheus.OnesuchalternativeisInfluxDBthatdescribesitselfasfollows:

Anopen-sourcedistributedtimeseriesdatabasewithnoexternaldependencies.

However,atthetimeofwriting,cAdvisorisnotcurrentlycompatiblewiththelatestversionofInfluxDB.TherearepatchesinthecodebaseforcAdvisor;however,theseareyettomakeitthroughtotheGoogle-maintainedDockerImage.

FormoredetailsonInfluxDBandit’snewvisualizationcomplainapplicationChronograf,seetheprojectwebsiteathttps://influxdb.com/andformoredetailsonhowtoexportcAdvisorstatisticstoInfluxDB,seethesupportingdocumentationforcAdvisorathttps://github.com/google/cadvisor/tree/master/docs.

SummaryInthischapter,welearnedhowtotaketheviewingreal-timestatisticsofourcontainersoffthecommandlineandintothewebbrowser.WeexploredsomedifferentmethodstoinstallGoogle’scAdvisorapplicationandalsohowtouseitswebinterfacetokeepaneyeonourrunningcontainers.WealsolearnedhowtocapturemetricsfromcAdvisorandstorethemusingPrometheus,amoderntimeseriesdatabase.

Thetwomaintechnologieswehavecoveredinthischapterhaveonlybeenpublicallyavailableforlessthantwelvemonths.Inthenextchapter,wewilllookatusingamonitoringtoolthathasbeeninaSysAdminstoolboxforover10years—Zabbix.

Chapter4.ATraditionalApproachtoMonitoringContainersSofar,wehavelookedatonlyafewtechnologiestomonitorourcontainers,sointhischapter,wewillbelookingmoreatatraditionaltoolformonitoringservices.Bytheendofthischapter,youshouldknowyourwayaroundZabbixandthevariouswaysyoucanmonitoryourcontainers.Wewillcoverthefollowingtopicsinthischapter:

HowtorunaZabbixServerusingcontainersHowtolaunchaZabbixServeronavagrantmachineHowtoprepareourhostsystemformonitoringcontainersusingtheZabbixagentHowtofindyourwayaroundtheZabbixwebinterface

ZabbixFirstthingsfirst,whatisZabbixandwhyuseit?

Ihavepersonallybeenusingitsinceversion1.2;theZabbixsitedescribesitasfollows:

“WithZabbix,itispossibletogathervirtuallylimitlesstypesofdatafromthenetwork.High-performancereal-timemonitoringmeansthattensofthousandsofservers,virtualmachines,andnetworkdevicescanbemonitoredsimultaneously.Alongwithstoringthedata,visualizationfeaturesareavailable(overviews,maps,graphs,screens,andsoon),aswellasveryflexiblewaysofanalyzingthedataforthepurposeofalerting.

Zabbixoffersgreatperformancefordatagatheringandcanbescaledtoverylargeenvironments.DistributedmonitoringoptionsareavailablewiththeuseofZabbixproxies.Zabbixcomeswithaweb-basedinterface,secureuserauthentication,andaflexibleuserpermissionschema.Pollingandtrappingissupported,withnativehigh-performanceagentsgatheringdatafromvirtuallyanypopularoperatingsystem;agent-lessmonitoringmethodsareavailableaswell.”

AtthetimeIstartedusingZabbix,theonlyrealviableoptionswereasfollows:

Nagios:https://www.nagios.org/Zabbix:http://www.zabbix.com/Zenoss:http://www.zenoss.org/

Outofthethesethreeoptions,Zabbixseemedtobethemoststraightforwardoneatthetime.ItwasdoingenoughworktomanagetheseveralhundredserversIwasgoingtomonitorwithouthavingtohavetheextraworkoflearningthecomplexitiesofsettingupNagiosorZenoss;afterall,giventhetaskthesoftwarehad,IneededtobeabletotrustthatIhadsetitupcorrectly.

Inthischapter,whileIamgoingtogointosomedetailaboutthesetupandthebasicsofusingZabbix,wewillonlybetouchingonsomeofthefunctionalities,whichcandoalotmorethanjustmonitoryourcontainers.Formoreinformation,Iwouldrecommendthefollowingasagoodstartingpoint:

Zabbixblog:http://blog.zabbix.comZabbix2.4manual:https://www.zabbix.com/documentation/2.4/manualFurtherreading:https://www.packtpub.com/all/?search=zabbix

InstallingZabbixAsyoumayhavenoticedfromthelinksintheprevioussection,therearealotofmovingpartsinZabbix.Itleveragesseveralopensourcetechnologies,andaproduction-readyinstallationneedsalittlemoreplanningthanwecangointointhischapter.BecauseofthiswearegoingtolookattwowaysofinstallingZabbixquicklyrathergointotoomuchdetail.

UsingcontainersAtthetimeofwriting,thereareoverahundredDockerimagesavailableontheDockerHub(https://hub.docker.com)thatmentionsZabbix.Theserangefromfullserverinstallationstojustthevariousparts,suchastheZabbixagentorproxyservices.

Outoftheoneslisted,thereisonethatisrecommendbyZabbixitself.So,wewilllookatthisone;itcanbefoundatthefollowingURLs:

DockerHub:https://hub.docker.com/u/zabbix/Projectpage:https://github.com/zabbix/zabbix-community-docker

TogettheZabbixServercontainerupandrunning,wemustfirstlaunchadatabasecontainer.Let’sstartafreshwithourvagrantinstancebyrunningthefollowingcommand:

[russ@mac~]$cd~/Documents/Projects/monitoring-docker/vagrant-centos/

[russ@mac~]$vagrantdestroy

default:Areyousureyouwanttodestroythe'default'VM?[y/N]y

==>default:ForcingshutdownofVM…

==>default:DestroyingVMandassociateddrives…

==>default:Runningcleanuptasksfor'shell'provisioner…

[russ@mac~]$vagrantup

Bringingmachine'default'upwith'virtualbox'provider…

==>default:Importingbasebox'russmckendrick/centos71'...

==>default:MatchingMACaddressforNATnetworking…

==>default:Checkingifbox'russmckendrick/centos71'isuptodate…

.....

==>default:=>Installingdocker-engine…

==>default:=>Configuringvagrantuser…

==>default:=>Startingdocker-engine…

==>default:=>Installingdocker-compose…

==>default:=>FinishedinstallationofDocker

[russ@mac~]$vagrantssh

Now,wehaveacleanenvironmentandit’stimetolaunchourdatabasecontainer,asfollows:

dockerrun\

--detach=true\

--publish=3306\

--env="MARIADB_USER=zabbix"\

--env="MARIADB_PASS=zabbix_password"\

--name=zabbix-db\

million12/mariadb

Thiswilldownloadthemillion12/mariadbimagefromhttps://hub.docker.com/r/million12/mariadb/andlaunchacontainercalledzabbix-db,runningMariaDB10(https://mariadb.org)withausercalledzabbixwhohasapasswordzabbix_password.WehavealsoopenedtheMariaDBport3306uponthecontainer,butaswewillbeconnectingtoitfromalinkedcontainer,thereisnoneedtoexposethatportonthehostmachine.

Now,wehavethedatabasecontainerupandrunning,wenowneedtolaunchourZabbixServercontainer:

dockerrun\

--detach=true\

--publish=80:80\

--publish=10051:10051\

--link=zabbix-db:db\

--env="DB_ADDRESS=db"\

--env="DB_USER=zabbix"\

--env="DB_PASS=zabbix_password"\

--name=zabbix\

zabbix/zabbix-server-2.4

Thisdownloadstheimage,whichatthetimeofwritingisover1GBsothisprocesscouldtakeseveralminutesdependingonyourconnection,andlaunchesacontainercalledzabbix.Itmapsthewebserver(port80)andtheZabbixServerprocess(port10051)onthehosttothecontainer,createsalinktoourdatabasecontainer,setsupthealiasdb,andinjectsthedatabasecredentialsasenvironmentvariablessothatthescriptsthatlaunchwhenthecontainerbootscanpopulatethedatabase.

Youcanverifythateverythingworkedasexpectedbycheckingthelogsonthecontainer.Todothis,enterdockerlogszabbix.Thiswillprintdetailsofwhathappenedwhenthecontainerlaunchedonscreen:

Now,oncewehavethecontainerupandrunning,itistimetomovetothebrowserforourfirsttasteofthewebinterface.Gotohttp://192.168.33.10/inyourbrowserandyouwillbegreetedbyawelcomepage;beforewecanstartusingZabbix,weneedtocompletetheinstallation.

Onthewelcomepage,clickonNexttobetakentothefirststep.ThiswillverifythateverythingweneedtorunaZabbixServerisinstalled.Aswehavelauncheditinacontainer,youshouldseeOKnexttoalloftheprerequisites.ClickonNexttomoveontothenextstep.

Now,weneedtoconfigurethedatabaseconnectionforthewebinterface.Here,youshouldhavethesamedetailsasyoudidwhenyoulaunchedthecontainer,asillustratedinthefollowingscreenshot:

Onceyouhaveenteredthedetails,clickonTestconnectionandyoushouldreceiveanOKmessage;youwillnotbeabletoproceeduntilthistestcompletessuccessfully.OnceyouhaveenteredthedetailsandhaveanOKmessage,clickonNext.

Nextup,arethedetailsontheZabbixServerthatthewebinterfaceneedstoconnectto;clickonNexthere.Nextup,youwillreceiveasummaryoftheinstallation.Toproceed,clickonNextandyouwillbegetconfirmationthatthe/usr/local/src/zabbix/frontends/php/conf/zabbix.conf.phpfilehasbeencreated.ClickonFinishtobetakentotheloginpage.

UsingvagrantWhilewritingthischapter,IthoughtalotaboutprovidinganothersetofinstallationinstructionsfortheZabbixServerservice.WhilethebookisallaboutMonitoringDockercontainers,havingaserviceasresourceintensiveasZabbixrunninginsideacontainerfeelsalittlecounterintuitive.Becauseofthis,thereisavagrantmachinethatusesPuppettobootstrapaworkinginstallationofZabbixServer:

[russ@mac~]$cd~/Documents/Projects/monitoring-docker/vagrant-zabbix/

[russ@mac~]$vagrantup

Bringingmachine'default'upwith'virtualbox'provider…

==>default:Importingbasebox'russmckendrick/centos71'...

==>default:MatchingMACaddressforNATnetworking…

==>default:Checkingifbox'russmckendrick/centos71'isuptodate…

.....

==>default:Debug:Receivedreporttoprocessfromzabbix.media-glass.es

==>default:Debug:Evictingcacheentryforenvironment'production'

==>default:Debug:Cachingenvironment'production'(ttl=0sec)

==>default:Debug:Processingreportfromzabbix.media-glass.eswith

processorPuppet::Reports::Store

Asyoumayhavenoticed,thereisalotofoutputstreamedtotheterminal,sowhatjusthappened?Firstofall,aCentOS7vagrantinstancewaslaunchedandthenaPuppetagentwasinstalled.Onceinstalled,theinstallationwashandedofftoPuppet.UsingtheZabbixPuppetmodulebyWernerDijkerman,ZabbixServerwasinstalled;formoredetailsonthemodule,seeitsPuppetForgepageathttps://forge.puppetlabs.com/wdijkerman/zabbix.

UnlikethecontainerizedversionofZabbixServer,thereisnoadditionalconfigurationrequired,soyoushouldbeabletoaccesstheZabbixloginpageathttp://zabbix.media-glass.es/(anIPaddressof192.168.33.11ishardcodedintotheconfiguration).

PreparingourhostmachineFortheremainderofthischapter,IwillassumethatyouareusingtheZabbixServerthatisrunningonitsownvagrantinstance.ThishelpstoensurethatyourenvironmentisconsistentwiththeconfigurationoftheZabbixagentwewillbelookingat.

TopassthestatisticsfromourcontainerstotheZabbixagent,whichwilltheninturnexposethemtotheZabbixServer,wewillbeinstallingusingtheZabbix-Docker-MonitoringZabbixagentmodulethathasbeendevelopedbyJanGaraj.Formoreinformationontheproject,seethefollowingURLs:

TheProjectpage:https://github.com/monitoringartist/Zabbix-Docker-Monitoring/TheZabbixsharepage:https://share.zabbix.com/virtualization/docker-containers-monitoring

Togettheagentandmoduleinstalled,configured,andrunning,weneedtoexecutethefollowingsteps:

1. InstalltheZabbixpackagerepository.2. InstalltheZabbixagent.3. Installtheprerequisitesforthemodule.4. AddtheZabbixagentusertotheDockergroup.5. Downloadtheauto-discoverybashscript.6. Downloadtheprecompiledzabbix_module_dockerbinary.7. ConfiguretheZabbixagentwiththedetailsofourZabbixServerandalsotheDocker

module.8. Setthecorrectpermissionsonallthefileswehavedownloadedandcreated.9. StarttheZabbixagent.

WhilethestepsremainthesameforbothCentOSandUbuntu,theactionstakentodotheinitialpackageinstallationdifferslightly.Ratherthangoingthroughtheprocessofshowingthecommandstoinstallandconfiguretheagent,thereisascriptforeachofthehostoperatingsystemsinthe/monitoring_docker/chapter04/folder.Toviewthescripts,runthefollowingcommandfromyourterminal:

cat/monitoring_docker/chapter04/install-agent-centos.sh

cat/monitoring_docker/chapter04/install-agent-ubuntu.sh

Now,youhavetakenalookatthescriptsitstimetorunthem,todothistypeoneofthefollowingcommands.IfyouarerunningCentOS,runthiscommand:

bash/monitoring_docker/chapter04/install-agent-centos.sh

ForUbuntu,runthefollowingcommand:

bash/monitoring_docker/chapter04/install-agent-ubuntu.sh

Toverifythateverythingranasexpected,checktheZabbixagentlogfilebyrunningthefollowingcommand:

cat/var/log/zabbix/zabbix_agentd.log

Youshouldseethattheendofthefileconfirmsthattheagenthasstartedandthatthezabbix_module_docker.somodulehasbeenloaded:

BeforewemoveontotheZabbixwebinterface,let’slaunchafewcontainersusingthedocker-composefilefromChapter2,UsingtheBuilt-inTools:

[vagrant@docker~]$cd/monitoring_docker/chapter02/02-multiple/

[vagrant@docker02-multiple]$docker-composeup-d

[vagrant@docker02-multiple]$docker-composescaleweb=3

[vagrant@docker02-multiple]$docker-composeps

WeshouldnowhavethreewebservercontainersrunningandarunningZabbixagentonthehost.

TheZabbixwebinterfaceOnceyouhaveZabbixinstalledyoucanopentheZabbixwebinterfacebygoingtohttp://zabbix.media-glass.es/inyourbrowser,thislinkwillonlyworkwhenyouhavetheZabbixvagrantboxupandrunning,ifyoudon’thaveitrunningthepagewilltimeout.Youshouldbepresentedwithaloginscreen.Enterthedefaultusernameandpasswordhere,whichisAdminandzabbix(notethattheusernamehasacapitalA),tologin.

Onceloggedin,youwillneedtoaddthehosttemplates.ThesearepreconfiguredenvironmentsettingsandwilladdsomecontextaroundthestatisticsthattheZabbixagentissendingtotheserver,alongwiththeauto-discoveryofcontainers.

Toaddthetemplates,gototheConfigurationtabinthetopmenuandselectTemplate;thiswillbringupalistofallthetemplatesthatarecurrentlyinstalled.ClickontheImportbuttonintheheaderanduploadacopyofthetwotemplatefilesyoucanfindinthe~/Documents/Projects/monitoring-docker/chapter04/templatefolderonyourmainmachine;thereisnoneedtochangetheruleswhenuploadingthetemplates.

Oncebothtemplateshavebeensuccessfullyimported,itistimetoaddourDockerhost.Again,gototheConfigurationtab,butthistimeselectHosts.Here,youneedtoclickonCreatehost.Then,enterthefollowinginformationintheHosttab:

Herearethedetailsoftheprecedinginformation:

Hostname:ThisisthehostnameofourDockerhostVisiblename:Here,thenameserverwillappearasinZabbixGroups:WhichgroupwithinZabbixtheserveryouwouldliketheDockerhosttobepartofAgentInterfaces:ThisistheIPaddressortheDNSnameofourDockerhostEnabled:Thisshouldbeticked

BeforeclickingonAdd,youshouldclickontheTemplatestabandlinkthefollowingtwotemplatestothehost:

TemplateAppDockerTemplateOSLinux

Hereisthescreenshotofthehost:

Onceyouhaveaddedthetwotemplates,clickonAddtoconfigureandenablethehost.Toverifythatthehosthasbeenaddedcorrectly,youshouldgototheMonitoringtabandthenLatestdata.Fromhere,clickonShowfilterandenterthehostmachineintheHostsbox.Youshouldthenstarttoseeitemsappearing:

Don’tworryifyoudon’tseetheDockersectionimmediately,bydefault,Zabbixwillattempttoauto-discovernewcontainerseveryfiveminutes.

DockermetricsForeachcontainer,Zabbixdiscoversthefollowingmetricsthatwillberecorded:

Container(yourContainersname)isrunningCPUsystemtimeCPUusertimeUsedcachememoryUsedRSSmemoryUsedswap

Apartfrom“Usedswap”,thesearethesamemetricsrecordedbycAdvisor.

CreatecustomgraphsYoucanaccessatime-basedgraphforanyofthemetricscollectedbyZabbix;youcanalsocreateyourowncustomgraphs.Inthefollowinggraph,IhavecreatedagraphthatplotsalltheCPUSystemstatsfromthethreewebcontainerswelaunchedearlierinthechapter:

Asyoucansee,IperformedafewtestsusingApacheBenchtomakethegraphalittlemoreinteresting.

Formoreinformationonhowtocreatecustomgraphs,seethegraphssectionofthedocumentationsiteathttps://www.zabbix.com/documentation/2.4/manual/config/visualisation/graphs.

ComparecontainerstoyourhostmachineAsweaddedtheLinuxOStemplateandtheDockertemplatetothehostandwearealsorecordingquitealotofinformationaboutthesystem,herewecantelltheeffectthetestingwithApacheBenchhadontheoverallprocessorload:

Wecandrilldownfurthertogetinformationontheoverallutilization:

TriggersAnotherfeatureofZabbixistriggers:youcandefineactionstohappenwhenametricmeetsacertainsetofcriteria.Inthefollowingexample,ZabbixhasbeenconfiguredwithatriggercalledContainerDown;thischangesthestatusofthemonitoreditemtoProblemwithaseverityofDisaster:

Thischangeinstatusthentriggersane-mailtoinformthat,forsomereasonthecontainerisnolongerupandrunning:

Thiscouldhavealsotriggeredothertasks,suchasrunningacustomscript,sendinganinstantmessageviaJabber,oreventriggeringathird-partyservicesuchasPagerDuty(https://www.pagerduty.com)orSlack(https://slack.com).

FormoreinformationonTriggers,Events,andNotifications,seethefollowingsectionsofthedocumentation:

https://www.zabbix.com/documentation/2.4/manual/config/triggershttps://www.zabbix.com/documentation/2.4/manual/config/eventshttps://www.zabbix.com/documentation/2.4/manual/config/notifications

SummarySo,howdoesthistraditionalapproachtomonitoringfitintoacontainer’slifecycle?

GoingbacktothePetsversusCattleanalogy,atfirstglance,ZabbixseemstobegearedmoretowardsPets:itsfeaturesetisbestsuitedtomonitoringservicesthatarestaticoveralongperiodoftime.Thismeansthatthesameapproachtomonitoringapetcanalsobeappliedtolong-runningprocessesrunningwithinyourcontainers.

Zabbixisalsotheperfectoptionformonitoringmixedenvironments.Maybeyouhaveseveraldatabaseserversthatarenotrunningascontainers,butyouhaveseveralhostsrunningDocker,andhaveequipmentsuchasswitchesandSANsthatyouneedtomonitor.Zabbixcanprovideyouwithasinglepaneofglassshowingyoumetricsforallyourenvironments,alongwithbeingabletoalertyoutoproblems.

Sofar,wehavelookedatusingAPIsandmetricsprovidedbyDockerandLXC,butwhataboutothermetricscanweuse?Inthenextchapter,wewilllookatatoolthathooksstraightintothehostmachine’skerneltogatherinformationonyourcontainers.

Chapter5.QueryingwithSysdigTheprevioustoolswehavelookedathaveallreliedonmakingAPIcallstoDockerorreadingmetricsfromLXC.Sysdigworksdifferentlybyhookingitselfintothehostsmachine’skernelwhilethisapproachdoesgoagainstDocker’sphilosophyofeachservicebeingruninitsownisolatedcontainer,theinformationyoucangetbyrunningSysdigonlyforafewminutesfaroutweighsanyargumentsaboutnotusingit.

Inthischapter,wewilllookatthefollowingtopics:

HowtoinstallSysdigandCsysdigonthehostmachineBasicusageandhowtoqueryyourcontainersinrealtimeHowtocapturelogssotheycanbequeriedlater

WhatisSysdig?BeforewestarttogetintoSysdig,let’sfirstunderstandwhatitis.WhenIfirstheardaboutthetool,Ithoughttomyselfthatitsoundedtoogoodtobetrue;thewebsitedescribesthetoolasfollows:

“Sysdigisopensource,system-levelexploration:capturesystemstateandactivityfromarunningLinuxinstance,thensave,filterandanalyze.SysdigisscriptableinLuaandincludesacommandlineinterfaceandapowerfulinteractiveUI,csysdig,thatrunsinyourterminal.Thinkofsysdigasstrace+tcpdump+htop+iftop+lsof+awesomesauce.Withstateoftheartcontainervisibilityontop.”

Thisisquiteaclaimasallthetoolsthatitisclaimingtobeaspowerfulwereallinasetofgotocommandstorunwhenlookingintoproblems,soIwasalittleskepticalatfirst.

Asanyonewhohashadtotryandtrackdownahaywireprocessoftryandtrackdownanissuethatisn’tbeingveryverboseinitserrorlogsonaLinuxserverwillknowthatusingtoolssuchasstrace,lsof,andtcpdumpcangetcomplicatedveryquicklyanditnormallyinvolvescapturingawholelotofdataandthenusingacombinationofseveraltoolstoslowly,andmanually,tracetheproblembyreducingtheamountofdatayoucaptured.

ImaginemydelightwhenSysdig’sclaimsturnedouttobetrue.ItmademewishIhadthetoolbackwhenIwasafrontlineengineer;itwouldhavemademylifealoteasier.

Sysdigcomesintwodifferentflavors,firstistheOpenSourceversionavailableathttp://www.sysdig.org/;thiscomeswithanncursesinterfacesothatyoucaneasilyaccessandquerydatafromaterminal-basedGUI.

NoteWikipediadescribesncurses(newcurses)asaprogramminglibrarythatprovidesanAPIthatallowstheprogrammertowritetext-baseduserinterfacesinaterminal-independentmanner.Itisatoolkitfordeveloping“GUI-like”applicationsoftwarethatrunsunderaterminalemulator.Italsooptimizesscreenchangesinordertoreducethelatencyexperiencedwhenusingremoteshells.

ThereisalsoacommercialservicethatallowsyoutostreamyourSysdigtotheirexternallyhostedservice;thisversionhasaweb-basedinterfaceforviewingandqueryingyourdata.

Inthischapter,wewillbeconcentratingontheopensourceversion.

InstallingSysdigConsideringhowpowerfulSysdigis,ithasoneofthemoststraightforwardinstallationandconfigurationprocessesIhavecomeacross.ToinstallSysdigoneitheraCentOSorUbuntuserver,typethefollowingcommand:

curl-shttps://s3.amazonaws.com/download.draios.com/stable/install-sysdig

|sudobash

Afterrunningtheprecedingcommand,youwillgetthefollowingoutput:

That’sit,youarereadytogo.Thereisnothingmoretoconfigureordo.Thereisamanualinstallationprocessandalsoawayofinstallingthetoolusingcontainerstobuildthenecessarykernelmodules;formoredetails,seetheinstallationguideasfollows:

http://www.sysdig.org/wiki/how-to-install-sysdig-for-linux/

UsingSysdigBeforewelookathowtouseSysdig,let’slaunchafewcontainersusingdocker-composebyrunningthefollowingcommand:

cd/monitoring_docker/chapter05/wordpress/

docker-composeup–d

ThiswilllaunchaWordPressinstallationrunningadatabaseandtwowebservercontainersthatareloadbalancedusinganHAProxycontainer.YouwillbeabletoviewtheWordPressinstallationathttp://docker.media-glass.es/oncethecontainershavelaunched.Youwillneedtoentersomedetailstocreatetheadminuserbeforethesiteisvisible;followtheon-screenpromptstocompletethesesteps.

ThebasicsAtitscore,Sysdigisatoolforproducingastreamofdata;youcanviewthestreambytypingsudosysdig(toquit,pressCtrl+c).

Thereisalotinformationtheresolet’sstarttofilterthestreamdownandrunthefollowingcommand:

sudosysdigevt.type=chdir

Thiswilldisplayonlyeventsinwhichauserchangesdirectory;toseeitinaction,openasecondterminalandyouwillseethatwhenyoulogin,youseesomeactivityinthefirstterminal.Asyoucansee,itlooksalotlikeatraditionallogfile;wecanformatoutputtogiveinformationsuchastheusername,byrunningthefollowingcommand:

sudosysdig-p"user:%user.namedir:%evt.arg.path"evt.type=chdir

Then,inyoursecondterminal,changethedirectoryafewtimes:

Asyoucansee,thisisaloteasiertoreadthantheoriginalunformattedoutput.PressCtrl+ctostopfiltering.

CapturingdataIntheprevioussection,welookedatfilteringdatainrealtime;itisalsopossibletostreamSysdigdatatoafilesothatyoucanquerythedataatalatertime.Exitfromyoursecondterminalandrunthefollowingcommandonyourfirstone:

sudosysdig-w~/monitoring-docker.scap

Whilethecommandisrunningonthefirstterminal,logintothehostonthesecondoneandchangethedirectoryafewtimes.Also,whilewearerecording,clickaroundtheWordPresssitewestartedatthebeginningofthissection,theURLishttp://docker.media-glass.es/.Onceyouhavedonethat,stoptherecordingbypressingCrtl+c;youshouldhavenowdroppedbacktoaprompt.YoucancheckthesizeofthefilecreatedbySysdigbyrunningthefollowing:

ls-lha~/monitoring-docker.scap

Now,wecanusethedatathatwehavecapturedtoapplythesamefilteraswedidwhenlookingatthereal-timestream:

sudosysdig-r~/monitoring-docker.scap-p"user:%user.name

dir:%evt.arg.path"evt.type=chdir

Byrunningtheprecedingcommand,youwillgetthefollowingoutput:

Noticehowwegetsimilarresultstowhenwewereviewingthedatainrealtime.

ContainersOneofthethingsthatwasrecordedin~/monitoring-docker.scapwasdetailsonthesystemstate;thisincludesinformationonthecontainerswelaunchedatthestartofthechapter.Let’susethisfiletogetsomestatsonthecontainers.Tolistthecontainersthatwereactiveduringthetime,wecapturedthedatafilerun:

sudosysdig-r~/monitoring-docker.scap-clscontainers

ToseewhichofthecontainersutilizedtheCPUmostofthetime,wewereclickingaroundtheWordPresssiterun:

sudosysdig-r~/monitoring-docker.scap-ctopcontainers_cpu

Tohavealookatthetopprocessesineachofthecontainersthathave“wordpress”intheirnames(whichisalloftheminourcase),runthefollowingcommand:

sudosysdig-r~/monitoring-docker.scap-ctopprocs_cpucontainer.name

containswordpress

Finally,whichofourcontainerstransferredthemostamountofdata?:

sudosysdig-r~/monitoring-docker.scap-ctopcontainers_net

Byrunningtheprecedingcommand,youwillgetthefollowingoutput:

Asyoucansee,wehaveextractedquiteabitofinformationonourcontainersfromthedatawecaptured.Also,usingthefile,youcanremovethe-r~/monitoring-docker.scappartofthecommandtoviewthecontainermetricsinrealtime.

It’salsoworthpointingoutthattherearebinariesforSysdigthatworkonbothOSXandWindows;whilethesedonotcaptureanydata,theycanbeusedtoreaddatathatyouhaverecordedonyourLinuxhost.

FurtherreadingFromthefewbasicexercisescoveredinthissection,youshouldstarttogetanideaofjusthowpowerfulSysdigcanbe.TherearemoreexamplesontheSysdigwebsiteathttp://www.sysdig.org/wiki/sysdig-examples/.Also,Irecommendyoutoreadtheblogpostathttps://sysdig.com/fishing-for-hackers/;itwasmyfirstexposuretoSysdiganditreallydemonstratesitsusefulness.

UsingCsysdigAseasyasitistoviewdatacapturedbySysdigusingthecommandlineandmanuallyfilteringtheresults,itcangetmorecomplicatedasyoustarttostringmoreandmorecommandstogether.TohelpmakethedatacapturedbySysdigasaccessibleaspossible,SysdigshipswithaGUIcalledCsysdig.

LaunchingtheCsysdigisdonewithasinglecommand:

sudocsysdig

Oncetheprocesshaslaunched,itshouldinstantlylookfamiliartoanyonewhohasusedtoporcAdvisor(minusthegraphs);itsdefaultviewwillshowyoureal-timeinformationontheprocessesthatarerunning:

Tochangethisview,knownastheProcessesview,pressF2toopentheViewsmenu;fromhere,youcanusetheupanddownarrowsonyourkeyboardtoselectaview.Asyoumayhavealreadyguessed,wewouldliketoseetheContainersview:

However,beforewedrilldownintoourcontainers,let’squitCsysdigbypressingqandloadupthefilewecreatedintheprevioussection.Todothis,typethefollowingcommand:

sudocsysdig-r~/monitoring-docker.scap

OnceCsysdigloads,youwillnoticethatSourcehaschangedfromLiveSystemtothefilepathofourdatafile.Fromhere,pressF2andusetheuparrowtoselectcontainersandthenhitEnter.Fromhere,youcanusetheupanddownarrowstoselectoneofthetwowebservers,thesewouldbeeitherwordpress_wordpress1_1orwordpress_wordpress2_1asshowninthefollowingscreen:

NoteTheremainingpartofthischapterassumesthatyouhaveCsysdigopenin-frontofyou,itwilltalkyouthroughhowtonavigatearoundthetool.Pleasefeelfreetoexploreyourselfaswell.

Onceyouhaveselectedaserver,hitEnterandyouwillbepresentedwithalistofprocessesthatthecontainerwasrunning.Again,youcanusethearrowkeystoselectaprocesstodrilldownfurtherinto.

IsuggestedlookingatoneoftheApacheprocessesthathasavaluelistedintheFilecolumn.Thistime,ratherthanpressingEntertoselecttheprocess,let’s“Echo”whattheprocesswasuptoatthetimewecapturedthedata;withtheprocessselected,pressF5.

Youcanusetheupanddownarrowstoscrollthroughtheoutput:

Tobetterformatthedata,pressF2andselectPrintableASCII.Asyoucanseefromtheprecedingscreenshot,thisApacheprocessperformedthefollowingtasks:

AcceptedanincomingconnectionAccessedthe.htaccessfileReadthemod_rewriterulesGotinformationfromthehostsfileMadeaconnectiontotheMySQLcontainerSenttheMySQLpassword

Byscrollingthroughtheremainderofthedatainthe“Echo”resultsfortheprocess,youshouldbeabletoeasilyfollowtheinteractionswiththedatabaseallthewaythroughtothepagebeingsenttothebrowser.

Toleavethe“Echo”screen,pressBackspace;thiswillalwaystakeyoualevelback.

Ifyouwantamoredetailedbreakdownonwhattheprocesswasdoing,thenpressF6toentertheDigview;thiswilllistthefilesthattheprocesswasaccessingatthetime,alongwiththenetworkinteractionandhowitisaccessingtheRAM.

Toviewafulllistofcommandsandformorehelp,youcanpressF1atanytime.Also,togetabreakdownonanycolumnsthatareonscreen,pressF7.

SummaryAsImentionedatthestartofthischapter,SysdigisprobablyoneofthemostpowerfultoolsIhavecomeacrossinrecentyears.

Partofitspoweristhewaythatitexposesalotofinformationandmetricsinawaythatneverfeelsoverwhelming.It’sclearthatthedevelopershavespentalotoftimeensuringthatboththeUIandthewaythatcommandsarestructuredfeelnaturalandinstantlyunderstandable,evenbythenewestmemberofanoperationsteam.

Theonlydownsideisthat,unlessyouwanttoviewtheinformationinrealtimeorlookintoaproblemindevelopmentstoringtheamountofdatathatisbeinggeneratedbySysdig,itcanbequitecostlyintermsofdiscspacebeingused.

ThisissomethingthatSysdighasrecognized,andtohelpwiththis,thecompanyoffersacloud-basedcommercialservicecalledSysdigCloudforyoutostreamyourSysdigdatainto.Inthenextchapter,wewilllookatthisserviceandalsosomeofitscompetitors.

Chapter6.ExploringThirdPartyOptionsSofar,wehavebeenlookingatthetoolsandservicesyouhostyourself.Alongwiththeseself-hostedtools,alargeamountofcloud-basedsoftwarehasdevelopedaroundDockerasaserviceecosystem.Inthischapter,wewilllookatthefollowingtopics:

WhyuseaSaaSserviceoverself-hostedorreal-timemetrics?Whatservicesareavailableandwhatdotheyoffer?InstallationofagentsforSysdigCloud,Datadog,andNewReliconthehostmachinesConfigurationoftheagentstoshipmetrics

AwordaboutexternallyhostedservicesSofar,toworkthroughtheexamplesinthisbook,wehaveusedlocallyhostedvirtualserversthatarelaunchedusingvagrant.Duringthischapter,wearegoingtouseservicesthatneedtobeabletocommunicatewithyourhostmachine,soratherthantryingtodothisusingyourlocalmachine,itsabouttimeyoutookyourhostmachineintothecloud.

Aswearegoingtostartandstoptheremotehostswhilewelookattheservices,itpaystouseapubliccloud,asweonlygetchargedforwhatweuse.

Thereareseveralpubliccloudservicesthatyoucanusetoevaluatethetoolscoveredinthischapter,whichoneyouchoosetouseisuptoyou,youcoulduse:

DigitalOcean:https://www.digitalocean.com/AmazonWebServices:https://aws.amazon.com/MicrosoftAzure:https://azure.microsoft.com/VMwarevCloudAir:http://vcloud.vmware.com/

Oruseyourownpreferredprovider,theonlypre-requisiteisthatyourserverispublicallyaccessible.

ThischapterassumesthatyouarecapableoflaunchingeitheraCentOS7orUbuntu14.04cloudinstanceandyouunderstandthatyouwilllikelyincurchargeswhilethecloudinstanceisupandrunning.

DeployingDockerinthecloudOnceyouhavelaunchedyourcloudinstance,youcanbootstrapDockerinthesamewaythatyouinstalledusingvagrant.Inthechapter6folderoftheGitrepository,therearetwoseparatescriptstodownloadandinstalltheDockerengineandcomposeitonyourcloudinstance.

ToinstallDocker,ensurethatyourcloudinstanceisupdatedbyrunning:

sudoyumupdate

FortheCentOSinstanceofyourUbuntu,runthefollowingcommand:

sudoapt-getupdate

Onceupdated,runthefollowingcommandtoinstallthesoftware.Duetothedifferencesinthewaydifferentcloudenvironmentsareconfigured,itisbesttoswitchovertotherootusertoruntheremainderofthecommands,todothis,run:

sudosu-

Nowyouwillbeabletoruntheinstallscriptusingthefollowingcommand:

curl-fsShttps://raw.githubusercontent.com/russmckendrick/monitoring-

docker/master/chapter06/install_docker/install_docker.sh|bash

Tocheckthateverythingworksasexpected,runthefollowingcommand:

dockerrunhello-world

Youshouldseesomethingsimilartotheterminaloutput,asshowninthefollowingscreenshot:

WecanstarttolookattheSasSservicesonceyouhaveDockerupandrunning.

WhyuseaSaaSservice?Youmayhavenoticedwhileworkingwiththeexamplesinthepreviouschaptersthatthetoolswehaveusedcanpotentiallyusemanyresourcesifweneededtostartcollectingmoremetrics,especiallyiftheapplicationswewanttomonitorareinproduction.

TohelpshiftthisloadfrombothstorageandCPU,anumberofcloud-basedSaaSoptionshavestartedofferingsupporttorecordmetricsforyourcontainers.Manyoftheseserviceswerealreadyofferingservicestomonitorservers,soaddingsupportforcontainersseemedanaturalprogressionforthem.

Thesetypicallyrequireyoutoinstallanagentonyourhostmachine,onceinstalled,theagentwillsitinthebackgroundandreporttotheservices,normallycloud-basedandAPIservices.

AfewoftheservicesallowyoutodeploytheagentsasDockercontainers.Theyoffercontainerizedagentssothattheservicecanrunonstrippeddownoperatingsystems,suchas:

CoreOS:https://coreos.com/RancherOS:http://rancher.com/rancher-os/Atomic:http://www.projectatomic.io/UbuntuSnappyCore:https://developer.ubuntu.com/en/snappy/

Theseoperatingsystemsdifferfromtraditionalones,asyoucannotinstallservicesonthemdirectly;theironlypurposeistorunaservice,suchasDocker,sothatyoucanlaunchtheservicesorapplicationsyouneedtoberunascontainers.

Aswearerunningfulloperatingsystemsasourhostsystems,wedonotneedthisoptionandwillbedeployingtheagentsdirectlytothehosts.

TheSaaSoptionsthatwearegoingtolookatinthischapterareasfollows:

SysdigCloud:https://sysdig.com/product/Datadog:https://www.datadoghq.com/NewRelic:http://newrelic.com

Theyallofferfreetrialsandtwoofthemofferfreecut-downversionsofthemainservice.Onthefaceofit,theymightallappeartooffersimilarservices;however,whenyoustarttousethem,youwillimmediatelynoticethattheyareinfactallverydifferentfromeachother.

SysdigCloudInthepreviouschapter,wehadalookattheopensourceversionofSysdig.WesawthatthereisagreatncursesinterfacecalledcSysdiganditallowsustonavigatethroughallthedatathatSysdigiscollectingaboutourhost.

ThesheeramountofmetricsanddatacollectedbySysdigmeansthatyouhavetotrytostayontopofiteitherbyshippingyourfilesofftheserver,maybetoAmazonSimpleStorageService(S3),ortosomelocalsharedstorage.Inaddition,youcanquerythedatainthecommandlineonthehostitselforonyourlocalmachineusinganinstallationofthecommand-linetools.

ThisiswhereSysdigCloudcomesintoplay;itoffersaweb-basedinterfacetothemetricsthatSysdigcapturesalongwiththeoptionstoshiptheSysdigcapturesoffyourhostmachineeithertoSysdig’sownstorageortoyourS3bucket.

Sysdigcloudoffersthefollowingfunctionality:

ContainerVision™Real-TimeDashboardHistoricalReplayDynamicTopologyAlerting

Aswellas,theoptiontotriggeracaptureonanyofyourhostsandatanytime.

SysdigdescribesContainerVisionas:

“SysdigCloud’spatent-pendingcoretechnology,ContainerVision,istheonlymonitoringtechnologyonthemarketdesignedspecificallytorespecttheuniquecharacteristicsofcontainers.ContainerVisionoffersyoudeepandcomprehensivevisibilityintoallaspectsofyourcontainerizedenvironment-applications,infrastructures,servers,andnetworks-allwithouttheneedtopolluteyourcontainerswithanyextrainstrumentation.Inotherwords,ContainerVisiongivesyou100%visibilityintotheactivityinsideyourcontainers,fromtheoutside.”

BeforewedelveintoSysdigCloudanyfurther,Ishouldpointoutthatthisisacommercialserverandatthetimeofwriting,itcosts$25perhostpermonth.Thereisalsoa14-dayfullyfeaturedtrialavailable.Ifyouwishtoworkthroughtheagentinstallationandfollowtheexampleinthischapter,youwillneedanactiveaccountthatrunseitheronthe14-daytrialorapaidsubscription.

Signupfora14-dayfreetrial:https://sysdig.com/Detailsonpricing:https://sysdig.com/pricing/Introductiontothecompany:https://sysdig.com/company/

InstallingtheagentTheagentinstallationissimilartoinstallingtheopensourceversion;youneedtoensurethatyourcloudhostisrunninganup-to-datekernelandthatyouarealsobootedintothekernel.

Somecloudproviderskeepatightcontrolonthekernelsyoucanbootinto(forexample,DigitalOcean),andtheydonotallowyoutomanageyourkernelonthehostitself.Instead,youneedtochoosethecorrectversionthroughtheircontrolpanel.

Onceyouhavethecorrectkernelinstalled,youshouldbeabletorunthefollowingcommandtoinstalltheagent.Ensurethatyoureplacetheaccesskeyattheendofthecommandwithyourownaccesskey,whichcanbefoundonyourUserProfilepageorontheagentinstallationpages;youcanfindtheseat:

UserProfile:https://app.sysdigcloud.com/#/settings/userAgentInstallation:https://app.sysdigcloud.com/#/settings/agentInstallation

Thecommandtorunis:

curl-shttps://s3.amazonaws.com/download.draios.com/stable/install-agent|

sudobash-s—--access_keywn5AYlhjRhgn3shcjW14y3yOT09WsF7d

Theshelloutputshouldlooklikethefollowingscreen:

Oncetheagenthasbeeninstalled,itwillimmediatelystarttoreportthedatabacktoSysdigCloud.IfyouclickonExplore,youwillseeyourhostmachineandtherunning

containers:

Asyoucanseehere,IhavemyhostmachineandfourcontainersrunningaWordPressinstallationsimilartotheoneweusedinthepreviouschapter.Fromhere,wecanstarttodrilldownintoourmetrics.

TolaunchtheWordPressinstallationonyourcloud-basedmachine,runthefollowingcommandsastherootuser:

sudosu-

mkdir~/wordpress

curl-Lhttps://raw.githubusercontent.com/russmckendrick/monitoring-

docker/master/chapter05/wordpress/docker-compose.yml>~/wordpress/docker-

compose.yml

cd~/wordpress

docker-composeup-d

ExploringyourcontainersTheSysdigCloudwebinterfacewillfeelinstantlyfamiliar,asitsharesasimilardesignandoverallfeelingwithcSysdig:

Onceyoustarttodrilldown,youcanseethatabottompaneopensupandthisiswhereyoucanviewthestatistics.OneofthethingsIlikedaboutSysdigCloudisthatitopensupawealthofmetricsandthereshouldbeverylittlethatyouneedtoconfigurefromhere.

Forexample,ifyouwanttoknowwhatprocesseshavebeenconsumingthemostCPUtimeinthelast2hours,clickon2HinthesecondarymenuandthenfromtheViewstabinthebottom-leftclickonSystem:TopProcesses;thiswillgiveyouatableoftheprocesses,orderedbytheonesthathaveusedthemosttime.

Toapplythisviewtoacontainer,clickonacontainerinthetop-sectionandthebottom-sectionwillbeinstantlyupdatedtoreflectthetopCPUutilizationforjustthatcontainer;asmostcontainerswillonlyrunoneortwoprocesses,thismaynotbethatinteresting.So,let’shaveadeeplookattheprocessesthemselves.Let’ssay,weclickedonourdatabase

containerandwewantedinformationonwhatishappeningwithinMySQL.

SysdigCloudcomeswithapplicationoverlays,thesewhenselectedgiveyoumoregranularinformationontheprocesseswithinthecontainer.SelectingtheApp:MySQL/PostgreSQLviewgivesyouaninsightintowhatyourMySQLprocessesarecurrentlydoing:

Here,youcanseethatviewinthebottomsectionhasinstantlyupdatedtogiveawealthofinformationonwhathasbeenhappeninginthelast5minuteswithinMySQL.

SysdigCloudsupportsanumberofapplicationviews,including:

ApacheHAProxy

NGINXRabbitMQRedisTomcat

Eachonegivesyouimmediateaccesstometrics,whicheventhemostexperiencedSysAdminswillfindvaluable.

Youmayhavenoticedthatatthetopofthesecondpaneltherearealsoafewicons,theseallowyouto:

AddAlert:Createsanalertbasedontheviewyouhaveopen;itletsyoutweakthethresholdandalsochoosehowyouarenotified.SysdigCapture:Pressingthisbringsupadialog,whichletsyourecordaSysdigsession.Oncerecorded,thesessionistransferredtoSysdigCloudoryourownS3bucket.Oncethesessionisavailable,youdownloaditorexploreitwithinthewebinterface.SSHConnect:GetsaremoteshellontheserverfromtheSysdigCloudwebinterface;itisusefulifyoudonothaveimmediateaccesstoyourlaptopordesktopmachineandyouwanttodosometroubleshooting.Pintodashboard:Addsthecurrentviewtoacustomdashboard.

Outtheseoptionsicons,the“AddAlert”and“SysdigCapture”optionsareprobablytheonesthatyouwillendupusingthemost.OnefinalviewthatIfoundinteresting,isthetopologyone.Itgivesyouabird’seyeviewofyourhostandcontainers,thisisusefultooseetheinteractionbetweencontainersandhosts:

Here,youcanseemerequestapagefromtheWordPresssite(it’sintheboxontheleft),thisrequesthitsmyhostmachine(theboxontheright).Onceit’sonthehostmachine,itisroutedtotheHAProxycontainer,whichthenpassesthepagerequesttotheWordpress2container.Fromhere,theWordpress2containerinteractswiththedatabasethatisrunningontheMySQLcontainer.

SummaryandfurtherreadingAlthoughSysdigCloudisquiteanewservice,itfeelsinstantlyfamiliarandfullyfeaturedasitisbuiltontopofanalreadyestablishedandrespectedopensourcetechnology.Ifyoulike,thelevelofdetailyougetfromtheopensourceversionofSysdig,thenSysdigCloudisanaturalprogressionforyoutostartstoringyourmetricsoffsiteandalsotoconfigurealerts.SomegoodstartingpointsforlearningmoreaboutSysdigCloudare:

VideoIntroduction:https://www.youtube.com/watch?v=p8UVbpw8n24SysdigCloudBestPractices:http://support.sysdigcloud.com/hc/en-us/articles/204872795-Best-PracticesDashboards:http://support.sysdigcloud.com/hc/en-us/articles/204863385-DashboardsSysdigblog:https://sysdig.com/blog/

TipIfyouhavelaunchedacloudinstanceandarenolongerusingit,nowwouldbeagoodtimetopowertheinstancedownorterminateitaltogether.Thiswillensurethatyoudonotgetbilledforservicesthatyouarenotusing.

DatadogDatadogisafullmonitoringplatform;itsupportsvariousservers,platforms,andapplications.Wikipediadescribestheserviceas:

“DatadogisaSaaS-basedmonitoringandanalyticsplatformforITinfrastructure,operationsanddevelopmentteams.Itbringstogetherdatafromservers,databases,applications,toolsandservicestopresentaunifiedviewoftheapplicationsthatrunatscaleinthecloud.”

Itusesanagentthatisinstalledonyourhostmachine;thisagentsendsmetricsbacktotheDatadogserviceperiodically.Italsosupportmultiplecloudplatforms,suchasAmazonWebServices,MicrosoftAzure,andOpenStacktonameafew.

Theaimistobringallofyourservers,applications,andhostprovidermetricsintoasinglepaneofglass;fromhere,youcancreatecustomdashboardsandalertssothatyoucanbenotifiedofanyproblematanylevelwithinyourinfrastructure.

Youcansignupforafreetrialofthefullserviceathttps://app.datadoghq.com/signup.Youwillneedatleastatrialaccounttoconfigurethealtering,andifyourtrialhasalreadyexpiredtheliteaccountwilldo.FormoredetailonDatadog’spricingstructure,pleaseseehttps://www.datadoghq.com/pricing/.

InstallingtheagentTheagentcanbeinstalledeitherdirectlyonthehostmachineorasacontainer.Toinstalldirectlyonthehostmachine,runthefollowingcommandandmakesurethatyouuseyourownuniqueDD_API_KEY:

DD_API_KEY=wn5AYlhjRhgn3shcjW14y3yOT09WsF7dbash-c"$(curl-L

https://raw.githubusercontent.com/DataDog/dd-

agent/master/packaging/datadog-agent/source/install_agent.sh)"

Toruntheagentasacontainer,usethefollowingcommandandagainmakesurethatyouuseyourownDD_API_KEY:

sudodockerrun-d--namedd-agent-h`hostname`-v

/var/run/docker.sock:/var/run/docker.sock-v

/proc/mounts:/host/proc/mounts:ro-v/sys/fs/cgroup/:/host/sys/fs/cgroup:ro

-eAPI_KEY=wn5AYlhjRhgn3shcjW14y3yOT09WsF7ddatadog/docker-dd-agent

Oncetheagenthasbeeninstalled,itwillcallbacktoDatadogandthehostwillappearinyouraccount.

IftheagenthasbeeninstalleddirectlyonthehostmachinethenwewillneedtoenabletheDockerintegration,ifyouinstalledtheagentusingthecontainerthenthiswillhavebeendoneforyouautomatically.

Todothis,youfirstneedtoallowtheDatadogagentaccesstoyourDockerinstallationbyaddingthedd-agentusertotheDockergroupbyrunningthefollowingcommand:

usermod-a-Gdockerdd-agent

Thenextstepistocreatethedocker.yamlconfigurationfile,luckilytheDatadogagentshipswithanexampleconfigurationfilethatwecanuse;copythisinplaceandthenrestarttheagent:

cp-pr/etc/dd-agent/conf.d/docker.yaml.example/etc/dd-

agent/conf.d/docker.yaml

sudo/etc/init.d/datadog-agentrestart

Nowtheagentonourhostmachinehasbeenconfiguredandthefinalstepistoenabletheintegrationthroughthewebsite.Todothis,gotohttps://app.datadoghq.com/andclickonIntegrations,scrolldownandthenclickoninstallonDocker:

Onceyouclickinstall,youwillbepresentedwithanoverviewoftheintegration,clickontheConfigurationtab,thisgivesinstructionsonhowtoconfiguretheagent;aswehavealreadydonethisstep,youcanclickonInstallIntegration.

YoucanfindmoreinformationoninstallingtheagentandtheintegrationsatthefollowingURLs:

https://app.datadoghq.com/account/settings#agenthttps://app.datadoghq.com/account/settings#integrations

ExploringthewebinterfaceNow,youhaveinstalledtheagentandenabledtheDockerintegration,youcanstarttohavealookaroundthewebinterface.Tofindyourhost,clickon“Infrastructure”intheleft-handsidemenu.

Youshouldbetakentoascreenthatcontainsamapofyourinfrastructure.Likeme,youprobablyonlyhaveasinglehostmachinelisted,clickonitandsomebasicstatsshouldappearatthebottomofthescreen:

Ifyoudon’talreadyhavethecontainerslaunched,nowwouldbeagoodtimetodoso,letslaunchtheWordPressinstallationagainusing:

sudosu-

mkdir~/wordpress

curl-Lhttps://raw.githubusercontent.com/russmckendrick/monitoring-

docker/master/chapter05/wordpress/docker-compose.yml>~/wordpress/docker-

compose.yml

cd~/wordpress

docker-composeup-d

Now,gobacktothewebinterface,fromthereyoucanclickonanyoftheserviceslistedonthehexagon.Thiswillbringupsomebasicmetricsfortheserviceyouhaveselected.Ifyouclickondocker,youwillseealinkforaDockerDashboardamongthevariousgraphsandsoon;clickingthiswilltakeyoutoamoredetailedviewofyourcontainers:

Asyoucansee,thisgivesusournowfamiliarbreakdownoftheCPUandmemorymetrics,alongwithinthetoprightofthedashboardabreakdownofthecontaineractivityonthehostmachine;thislogsevents,suchasstoppingandstartingcontainers.

Datadogcurrentlyrecordsthefollowingmetrics:

docker.containers.running

docker.containers.stopped

docker.cpu.system

docker.cpu.user

docker.images.available

docker.images.intermediate

docker.mem.cache

docker.mem.rss

docker.mem.swap

FromtheMetricsexploreroptionintheleft-handsidemenu,youcanstarttographthesemetricsandonceyouhavethegraphs,youcanthenstarttoaddthemtoyourowncustomdashboardsorevenannotatethem.Whenyouannotateagraph,asnapshotiscreatedandthegraphshowsupintheeventsqueuealongwiththeotherevents,thathavebeenrecorded,suchascontainerstoppingandstarting:

Also,withinthewebinterfaceyoucanconfiguremonitors;theseallowyoutodefinetriggers,whichalertyouifyourconditionsarenotmet.Alertscanbesentase-mailsorviathirdpartyservices,suchasSlack,Campfire,orPagerDuty.

SummaryandfurtherreadingWhileDatadog’sDockerintegrationonlygivesyouthebasicmetricsonyourcontainers,itdoeshaveawealthoffeaturesandintegrationwithotherapplicationsandthirdparties.IfyouneedtomonitoranumberofdifferentservicesalongsideyourDockercontainers,thenthisservicecouldbeforyou:

Homepage:https://www.datadoghq.comOverview:https://www.datadoghq.com/product/MonitoringDockerwithDatadog:https://www.datadoghq.com/blog/monitor-docker-datadog/Twitter:https://twitter.com/datadoghq

TipPleaseRemember

Ifyouhavelaunchedacloudinstanceandarenolongerusingitthennowwouldbeagoodtimetopowertheinstancedownorterminateitaltogether.Thiswillensurethatyoudonotgetbilledforanyservicesyouarenotusing.

NewRelicNewReliccouldbeconsideredthegranddaddyofSaaSmonitoringtools,chancesarethatifyouareadeveloperyouwillhaveheardofNewRelic.IthasbeenaroundforawhileanditisthestandardtowhichotherSaaStoolscomparethemselves.

NewRelichasgrownintoseveralproductsovertheyear,currently,theyoffer:

NewRelicAPM:Themainapplicationperformance-monitoringtool.ThisiswhatmostpeoplewillknowNewRelicfor;thistollgivesyouthecodelevelvisibilityofyourapplication.NewRelicMobile:Asetoflibrariestoembedintoyournativemobileapps,givingAPMlevelsofdetailforyouriOSandandroidapplication.NewRelicInsights:Ahigh-levelviewofallofthemetricscollectedbyotherNewRelicservices.NewRelicServers:Monitorsyourhostservers,recordingmetricsaroundCPU,RAM,andstorageutilization.NewRelicBrowser:Givesyouaninsightintowhathappenswithyourweb-basedapplicationsoncetheyleaveyourserversandenteryourenduser’sbrowserNewRelicSynthetics:Monitorsyourapplicationsresponsivenessfromvariouslocationsaroundtheworld.

RatherthanlookingatalloftheseofferingsthatgiveusaninsightintowhatishappeningwithourDocker-basedcode,asthat’sprobablyawholebookonitsown,wearegoingtotakealookattheserverproduct.

TheservermonitoringserviceofferedbyNewRelicisavailablefreeofcharge,youjustneedanactiveNewRelicaccount,youcansignupforanaccountathttps://newrelic.com/signup/detailsonNewRelicspricingcanbefoundattheirhomepageathttp://newrelic.com/.

InstallingtheagentLiketheotherSaaSofferingswehavelookedatinthischapter,NewRelicServershasahost-basedclient,whichneedstobeabletoaccesstheDockerbinary.ToinstallthisonaCentOSmachine,runthefollowing:

yuminstallhttp://download.newrelic.com/pub/newrelic/el5/i386/newrelic-

repo-5-3.noarch.rpm

yuminstallnewrelic-sysmond

ForUbuntu,runthefollowingcommand:

echo'debhttp://apt.newrelic.com/debian/newrelicnon-free'|sudotee

/etc/apt/sources.list.d/newrelic.list

wget-O-https://download.newrelic.com/548C16BF.gpg|sudoapt-keyadd-

apt-getupdate

apt-getinstallnewrelic-sysmond

Nowthatyouhavetheagentinstalled,youneedtoconfiguretheagentwithyourlicensekey.Youcandothiswiththefollowingcommandandmakesurethatyouaddyourlicense,whichcanbefoundinyoursettingspage:

nrsysmond-config--setlicense_key=wn5AYlhjRhgn3shcjW14y3yOT09WsF7d

Nowthattheagentisconfigured,weneedtoaddthenewrelicusertothedockergroupsothattheagenthasaccesstoourcontainerinformation:

usermod-a-Gdockernewrelic

Finally,weneedtostarttheNewRelicServeragentandrestartDocker:

/etc/init.d/newrelic-sysmondrestart

/etc/init.d/dockerrestart

TipRestartingDockerwillstoptherunningcontainersthatyouhave;makesurethatyoumakeanoteoftheseusingdockerpsandthenstartthemmanuallyandbackuponcetheDockerservicerestarts.

YoushouldseeyourserverappearonyourNewReliccontrolpanelafterafewminutes.

ExploringthewebinterfaceOnceyouhavetheNewRelicserveragentinstalled,configured,andrunningonyourhostmachine,youwillseesomethingsimilartothefollowingscreenshotwhenclickingonServersinthetopmenu:

Selectingtheserverwillallowyoutostartexploringthevariousmetricsthattheagentisrecording:

Fromhere,youhavetheoptiontodrilldownfurther:

Overview:GivesaquickoverviewofyourhostmachineProcesses:ListsalloftheprocessesthatarerunningbothonthehostmachineandwithinyourcontainersNetwork:LetsyouseethenetworkactivityforyourhostmachineDisks:GivesyoudetailsonhowmuchspaceyouareusingDocker:ShowsyoutheCPUandmemoryutilizationforyourcontainers

Asyoumayhaveguessed,wearegoingtobelookingattheDockeritemnext,clickonitandyouwillseealistofyouractiveimages:

YoumayhavenoticedadifferencebetweenNewRelicandtheotherservices,asyoucanseeNewRelicdoesnotshowyoutherunningcontainers,insteaditshowsyoutheutilizationbyDockerimage.

Intheprecedingscreenshot,IhavefourcontainersactiveandrunningtheWordPressinstallationwehaveusedelsewhereinthebook.IfIwantedabreakdownpercontainer,thenIwouldbeoutofluck,asdemonstratedbythefollowingscreen:

It’saprettydullscreen,butitgivesyouanideaaboutwhatyouwillseeifyouarerunningmultiplecontainersthathavebeenlaunchedusingthesameimage.Sohowisthisuseful?Well,coupledwiththeotherservicesofferedbyNewRelic,itcangiveyouanindicationofwhatyourcontainerswereuptowhenaproblemoccurredwithinyourapplication.IfyourememberthePetsversusCattleversusChickensanalogyfromChapter1,IntroductiontoDockerMonitoring,wedon’tnecessarilycarewhichcontainerdidwhat;wejustwanttoseetheimpactithadduringtheissuewearelookinginto.

SummaryandfurtherreadingDuetotheamountofproductsitoffers,NewReliccanbealittledauntingatfirst,butifyouworkwithadevelopmentteamthatactivelyusesNewRelicwithintheirday-to-dayworkflow,thenhavingalloftheinformationaboutyourinfrastructurealongsidethisdatacanbebothvaluableandnecessary,especiallyduringanissue:

NewRelicServermonitoring:http://newrelic.com/server-monitoringNewRelicandDocker:http://newrelic.com/docker/Twitter:https://twitter.com/NewRelic

TipIfyouhavelaunchedacloudinstanceandarenolongerusingitthen,nowisagoodtimetopowertheinstancedownorterminateitaltogether,thiswillensureyoudonotgetbilledforanyservicesyouarenotusing.

SummaryWhichSaaSserviceyouchoosedependsonyourcircumstances,thereareanumberofquestionsyoushouldaskyourselfbeforeyoustartevaluatingtheSaaSofferings:

Howmanycontainerswouldyouliketomonitor?Howmanyhostmachinesdoyouhave?Isthereanon-containerizedinfrastructureyouneedtomonitor?Whatmetricsdoyouneedfromthemonitoringservice?Howlongshouldthedataberetainedfor?Couldotherdepartments,suchasdevelopmentandutilizetheservice?

WecoveredjustthreeoftheavailableSaaSoptionsinthischapter,thereareotheroptionsavailable,suchas:

Ruxit:https://ruxit.com/docker-monitoring/Scout:https://scoutapp.com/plugin_urls/19761-docker-monitorLogentries:https://logentries.com/insights/server-monitoring/Sematext:http://sematext.com/spm/integrations/docker-monitoring.html

Monitoringserversandservicesareonlyasgoodasthemetricsyoucollect,ifpossibleandifyourbudgetallows,youshouldtakefulladvantageoftheservicesofferedbyyourchosenproviders,asmoredatabeingrecordedbyasingleproviderwillonlybenefityouwhenitcomestoanalyzingproblemswithnotonlyyourcontainerizedapplications,butalsowithyourinfrastructure,codeandevenyourcloudprovider.

Forexample,ifyouaremonitoringyourhostmachineusingthesameserviceasyouusetomonitoryourcontainers,thenbyusingthecustomgraphingfunctions,youshouldbeabletocreateoverlaygraphsofCPUloadspikesofbothyourhostmachineandyourcontainer.Thisisalotmoreusefulthantryingtocomparetwodifferentgraphsfromdifferentsystemssidebyside.

Inthenextchapter,wewilllookatanoften-overlookedpartofmonitoring:shippingyourlogfilesawayfromyourcontainers/hoststoasinglelocationsothattheycanbemonitoredandreviewed.

Chapter7.CollectingApplicationLogsfromwithintheContainerOneofthemostoverlookedpartsofmonitoringarelogfilesgeneratedbytheapplicationorservicessuchasNGINX,MySQL,Apache,andsoon.SofarwehavelookedatvariouswaysofrecordingtheCPUandRAMutilizationoftheprocesseswithinyourcontainersareatapointintime,nowitstimetodothesameforthelogfiles.

IfyouarerunningyourcontainersasCattleorChickens,thenthewayyoudealwiththeissuestodestroyandrelaunchyourcontainereithermanuallyorautomaticallyisimportant.Whilethisshouldfixtheimmediateproblem,itdoesnothelpwithtrackingdowntherootcauseoftheissueandifyoudon’tknowthatthenhowcanyouattempttoresolveitsothatitdoesnotreoccur.

Inthischapter,wewilllookathowwecangetthecontentofthelogfilesfortheapplicationsrunningwithinourcontainerstothecentrallocationsothattheyareavailable,evenifyouhavetodestroyandreplaceacontainer.Wearegoingtocoverthefollowingtopicsinthischapter:

Howtoviewcontainerlogs?Deployingan“ELK”stackusingaDockercontainersstacktoshipthelogstoReviewingyourlogsWhatthirdpartyoptionsareavailable?

ViewingcontainerlogsLikethedockertopcommand,thereisaverybasicwayofviewinglogs.Whenyouusethedockerlogscommand,youareactuallyviewingtheSTDOUTandSTDERRoftheprocessesthatarerunningwithinthecontainer.

NoteFormoreinformationonStandardStreams,pleaseseehttps://en.wikipedia.org/wiki/Standard_streams.

Asyoucanseefromthefollowingscreenshot,thesimplestthingyouhavetodoisrundockerlogsfollowedbyyourcontainername:

Toseethisonyourownhost,let’slaunchtheWordPressinstallationfromchapter05usingthefollowingcommands:

cd/monitoring_docker/chapter05/wordpress/

docker-composeup–d

dockerlogswordpress_wordpress1_1

Youcanextendthedockerlogscommandbyaddingthefollowingflagsbeforeyourcontainername:

-for--followwillstreamthelogsinrealtime-tor--timestampswillshowatimestampatthestartofeachline--tail="5"willshowthelastxnumberoflines--since="5m00s"willshowonlytheentriesforthelast5minutes

UsingtheWordPressinstallationthatwehavejustlaunched,tryrunningthefollowingcommands:

dockerlogs--tail="2"wordpress_wordpress1_1

Thiswillshowthelasttwolinesofthelogs,youcanaddtimestampsusing:

dockerlogs--tail="2"–timestampswordpress_wordpress1_1

Asyoucanseeinthefollowingterminaloutput,youcanalsostringcommandstogethertoformaverybasicquerylanguage:

Thedownsideofusingdockerlogsisexactlythesameasusingdockertop,inthatitisonlyavailablelocallyandthelogsareonlypresentforthetimethecontainerisaround,youcanviewthelogsofastoppedcontainer,butoncethecontainerisremoved,soarethelogs.

ELKStackSimilartosomeofthetechnologiesthatwehavecoveredinthisbook,anELKstackreallydeservesabookbyitself;infact,therearebooksforeachoftheelementsthatmakeanELKstack,theseelementsare:

Elasticsearchisapowerfulsearchserver,whichhasbeendevelopedwithmodernworkloadsinmindLogstashsitsbetweenyourdatasourceandElasticsearchservices;ittransformsyourdatainrealtimetoaformat,whichElasticsearchcanunderstand.KibanaisinfrontofyourElasticsearchservicesandallowsyoutoqueryyourdatainafeature-richweb-baseddashboard.

TherearealotofmovingpartswithanELKstack,sotosimplifythings,wewilluseaprebuiltstackforthepurposeoftesting;however,youprobablydon’twanttousethisstackinproduction.

StartingthestackLet’slaunchafreshvagranthostonwhichtoruntheELKstack:

[russ@mac~]$cd~/Documents/Projects/monitoring-docker/vagrant-centos/

[russ@mac~]$vagrantup

Bringingmachine'default'upwith'virtualbox'provider…

==>default:Importingbasebox'russmckendrick/centos71'...

==>default:MatchingMACaddressforNATnetworking…

==>default:Checkingifbox'russmckendrick/centos71'isuptodate…

.....

==>default:=>Installingdocker-engine…

==>default:=>Configuringvagrantuser…

==>default:=>Startingdocker-engine…

==>default:=>Installingdocker-compose…

==>default:=>FinishedinstallationofDocker

[russ@mac~]$vagrantssh

Now,wehaveacleanhostthatisupandrunning,wecanstartthestackbyrunningthefollowingcommands:

[vagrant@docker~]$cd/monitoring_docker/chapter07/elk/

[vagrant@dockerelk]$docker-composeup-d

Asyoumayhavenoticed,itdidmorethatjustpulldownsomeimages;whathappenedwas:

AnElasticsearchcontainerwaslaunchedusingtheofficialimagefromhttps://hub.docker.com/_/elasticsearch/.ALogstashcontainerwaslaunchedusingtheofficialimagefromhttps://hub.docker.com/_/logstash/,itwasalsolaunchedwithourownconfiguration,whichmeansthatourinstallationlistensforlogssentfromLogspout(moreaboutthatinaminute).AcustomKibanaimagewasbuiltusingtheofficialimagefromhttps://hub.docker.com/_/kibana/.AllitdidwasaddasmallscripttoensurethatKibanadoesn’tstartuntilourElasticsearchcontainerisfullyupandrunning.Itwasthenlaunchedwithacustomconfigurationfile.AcustomLogspoutcontainerwasbuiltusingtheofficialimagefromhttps://hub.docker.com/r/gliderlabs/logspout/andthenweaddedacustommodulesothatLogspoutcouldtalktoLogstash.

Oncedocker-composehasfinishedbuildingandlaunchingthestackyoushouldbeabletoseethefollowingwhenrunningdocker-composeps:

WenowhaveourELKstackupandrunning,asyoumayhavenoticed,thereisanadditionalcontainerrunningandgivingusanELK-Lstack,sowhatisLogspout?

LogspoutIfweweretolaunchElasticsearch,Logstash,andKibanacontainers,weshouldhaveafunctioningELKstackbutwewillhavealotofconfigurationtodotogetourcontainerlogsintoElasticsearch.

SinceDocker1.6,youhavebeenabletoconfigureloggingdrivers,thismeantthatitispossibletolaunchacontainerandhaveitsenditsSTDOUTandSTDERRtoaSyslogServer,whichwillbeLogstashinourcase;however,thismeansthatyouwillhavetoaddsomethingsimilartothefollowingoptionseachtimewelaunchacontainer:

--log-driver=syslog--log-optsyslog-address=tcp://elk_logstash_1:5000

ThisiswhereLogspoutcomesin,ithasbeendesignedtocollectalloftheSTDOUTandSTDERRmessagesonahostmachinebyinterceptingthemessagesthatarebeingcollectedbytheDockerprocessandthenitroutesthemtoourLogstashinstanceinaformatthatisunderstoodbyElasticsearch.

Justasthelog-driver,itsupportsSyslogoutofthebox;however,thereisathirdpartymodulethattransformstheoutputtoJSON,whichLogstashunderstands.Asapartofourbuildwedownloaded,compiledandconfiguredthemodule.

YoucanfindoutmoreaboutLogspoutandloggingdriversatthefollowing:

OfficialLogspoutimage:https://hub.docker.com/r/gliderlabs/logspout/LogspoutProjectpage:https://github.com/gliderlabs/logspoutLogspoutLogstashmodule:https://github.com/looplab/logspout-logstashDocker1.6releasenotes:https://blog.docker.com/2015/04/docker-release-1-6/DockerLoggingDrivers:https://docs.docker.com/reference/logging/overview/

ReviewingthelogsSonow,wehaveourELKrunningandamechanisminplacetostreamalloftheSTDOUTandSTDERRmessagesgeneratedbyourcontainersintoLogstash,whichinturnroutesthedataintoElasticsearch.NowitstimetoviewthelogsinKibana.ToaccessKibanagotohttp://192.168.33.10:8080/inyourbrowser;whenyouaccessthepage,youwillbeaskedtoConfigureanindexpattern,thedefaultindexpatternwillbefineforourneedssojustclicktheCreatebutton.

Onceyoudo,youwillseealistoftheindexpatterns,thesearetakendirectlyfromtheLogspoutoutput,andyoushouldnoticethefollowingitemsintheindex:

docker.name:Thenameofcontainerdocker.id:ThefullcontainerIDdocker.image:Thenameoftheimageusedtolaunchtheimage

Fromhere,ifyouweretoclickonDiscoverinthetopmenuyouwouldseesomethingsimilartothefollowingpage:

Inthescreenshot,youwillseethatIhaverecentlylaunchedtheWordPressstackandwehavebeenusingitthroughoutthebook,usingthefollowingcommands:

[vagrant@dockerelk]$cd/monitoring_docker/chapter05/wordpress/

[vagrant@dockerwordpress]$docker-composeup–d

Togiveyouanideaofwhatisbeinglogged,hereistherawJSONtakenfromElasticseachforrunningtheWordPressinstallationscript:

{

"_index":"logstash-2015.10.11",

"_type":"logs",

"_id":"AVBW8ewRnBVdqUV1XVOj",

"_score":null,

"_source":{

"message":"172.17.0.11--[11/Oct/2015:12:48:26+0000]\"POST/wp-

admin/install.php?step=1HTTP/1.1\"2002472\"http://192.168.33.10/wp-

admin/install.php\"\"Mozilla/5.0(Macintosh;IntelMacOSX10_11)

AppleWebKit/601.1.56(KHTML,likeGecko)Version/9.0Safari/601.1.56\"",

"docker.name":"/wordpress_wordpress1_1",

"docker.id":

"0ba42876867f738b9da0b9e3adbb1f0f8044b7385ce9b3a8a3b9ec60d9f5436c",

"docker.image":"wordpress",

"docker.hostname":"0ba42876867f",

"@version":"1",

"@timestamp":"2015-10-11T12:48:26.641Z",

"host":"172.17.0.4"

},

"fields":{

"@timestamp":[

1444567706641

]

},

"sort":[

1444567706641

]

}

Fromhere,youcanstarttousethefreetextsearchboxandbuildupsomequitecomplexqueriestodrilldownintoyourcontainer’sSTDOUTandSTDERRlogs.

Whataboutproduction?Asmentionedatthetopofthissection,youprobablydon’twanttorunyourproductionELKstackusingthedocker-composefile,whichaccompaniesthischapter.Firstofall,youwillwantyourElasticsearchdatatobestoredonapersistentvolumeandyoumorethanlikelywantyourLogstashservicetobehighlyavailable.

TherearenumerousguidesonhowtoconfigureahighlyavailableELKstack,aswellas,thehostedservicesfromElastic,whichisthecreatorofElasticsearch,andalsoAmazonWebServices,whichoffersanElasticsearchservice:

ELKtutorial:https://www.youtube.com/watch?v=ge8uHdmtb1MFoundfromElastic:https://www.elastic.co/foundAmazonElasticsearchService:https://aws.amazon.com/elasticsearch-service/

LookingatthirdpartyoptionsThereareafewoptionswhenitcomestohostingcentralloggingforyourcontainersexternaltoyourownserverinstances.Someoftheseare:

LogEntries:https://logentries.com/Loggly:https://www.loggly.com/

Bothoftheseservicesofferafreetier.LogEntriesalsooffersa“LogentriesDockerFree”accountthatyoucanfindoutmoreaboutathttps://logentries.com/docker/

NoteAsrecommendedintheExploringThirdPartyOptionschapter,itisbesttouseacloudservicewhenevaluatingthirdpartyservices.Theremainderofthischapterassumesthatyouarerunningacloudhost.

Let’slookatconfiguringtheLogEntriesonanexternalserver,firstofallyouneedtohavesignedupforanaccountathttps://logentries.com/.Onceyouhavesignedup,youshouldbetakentoapageinwhichyourlogswilleventuallybedisplayed.

Tostart,clickontheAddnewlogbuttoninthetop-rightcornerofthepageandthenclicktheDockerlogointhePlatformssection.

YouhavetonameyoursetoflogsintheSelectsetsection,sogiveanametoyourlogset.YounowhavethechoiceofbuildingyourowncontainerlocallyusingtheDockerfilefromhttps://github.com/logentries/docker-logentries:

gitclonehttps://github.com/logentries/docker-logentries.git

cddocker-logentries

dockerbuild-tdocker-logentries.

Afterrunningtheprecedingcommand,youwillgetthefollowingoutput:

Beforeyoustartyourcontainer,youwillneedtogenerateanaccesstokenforyourlogsetbyclickingonGenerateLogToken.Onceyouhavethis,youcanlaunchyourlocallybuiltcontainersusingthefollowingcommand(replacethetokenwiththeoneyouhavejustgenerated):

dockerrun-d-v/var/run/docker.sock:/var/run/docker.sockdocker-

logentries-twn5AYlh-jRhgn3shc-jW14y3yO-T09WsF7d-j

YoucandownloadtheimagestraightfromtheDockerhubbyrunning:

dockerrun-d-v/var/run/docker.sock:/var/run/docker.sock

logentries/docker-logentries-twn5AYlh-jRhgn3shc-jW14y3yO-T09WsF7d–j

It’sworthpointingoutthattheautomaticallygeneratedinstructionsgivenbyLogEntrieslaunchesthecontainerintheforeground,ratherthandetachingfromthecontaineronceithasbeenlaunchedliketheprecedinginstructions.

Onceyouhavethedocker-logentriescontainerupandrunning,youshouldstarttoseelogsfromyourcontainerstreamedinreal-timetoyourdashboard:

Fromhere,youwillbeabletoqueryyourlogs,createdashboards,andcreatealertsdependingontheaccountoptionyougofor.

SummaryInthischapter,wehavecoveredhowtoquerytheSTDOUTandSTDERRoutputfromyourcontainersusingthetoolbuiltintoDocker,howtoshipthemessagestoanexternalsource,ourELKstack,andhowtostorethemessagesevenafterthecontainerhasbeenterminated.Finally,wehavelookedatafewofthethird-partyserviceswhoofferservicestowhichyoucanstreamyourlogs.

Sowhygotoallofthiseffort?Monitoringisn’tjustaboutkeepingandqueryingCPU,RAM,HDD,andNetworkutilizationmetrics;thereisnopointinknowingiftherewasaCPUspikeanhouragoifyoudon’thaveaccesstothelogfilestoseeifanyerrorswerebeinggeneratedatthattime.

Theserviceswehavecoveredinthischapterofferthequickestandmostefficientinsightsintowhatcanquicklybecomeacomplexdataset.

Inthenextchapter,wewilllookatalloftheservicesandconceptswehavecoveredinthebookandapplythemtosomerealworldscenarios.

Chapter8.WhatAretheNextSteps?Inthisfinalchapter,wewilllookatthenextstepsyoucantaketomonitoryourcontainers,bytalkingaboutthebenefitsofaddingalertstoyourmonitoring.Also,wewillcoversomedifferentscenariosandalsowhichtypeofmonitoringisappropriateforeachofthem:

Commonproblems(performance,availability,andsoon)andwhichtypeofmonitoringisbestforyoursituation.Whatarethebenefitsofalertingonthemetricsyouarecollectingandwhataretheoptions?

SomescenariosTolookatwhichtypeofmonitoringyoumightwanttoimplementforyourcontainer-basedapplications,weshouldworkthroughafewdifferentexampleconfigurationsthatyourcontainer-basedapplicationscouldbedeployinginto.First,let’sremindourselvesaboutPets,Cattle,Chickens,andSnowflakes.

Pets,Cattle,Chickens,andSnowflakesBackintheChapter1,IntroductiontoDockerMonitoring,wespokeaboutPets,Cattle,Chickens,andSnowflakes;inthatchapter,wedescribedwhateachtermmeantwhenitwasappliedtomodernclouddeployments.Here,wewillgointoalittlemoredetailabouthowthetermscanbeappliedtoyourcontainers.

PetsForyourcontainerstobeconsideredaPet,youwillbemorethanlikelytoberunningeitherasingleorasmallnumberoffixedcontainersonadesignatedhost.

Eachoneofthesecontainerscouldbeconsideredasinglepointoffailure;ifanyoneofthemgoesdown,itwillmorethanlikelyresultinerrorsforyourapplication.Worststill,ifthehostmachinegoesdownforanyreason,yourentireapplicationwillbeoffline.

ThisisatypicaldeploymentmethodformostofourfirststepswithDocker,andinnowayshoulditbeconsideredbad,frownedupon,ornotrecommend;aslongasyouareawareofthelimitations,youwillbefine.

Thispatterncanalsobeusedtodescribemostdevelopmentenvironments,asyouareconstantlyreviewingitshealthandtuningasneeded.

YouwillmorethanlikelybehostingthemachineonyourlocalcomputeroronahostingservicesuchasDigitalOcean(https://www.digitalocean.com/).

CattleForthebulkofproductionorbusinesscriticaldeployments,youshouldaimtolaunchyourcontainersinaconfigurationthatallowsthemtoautomaticallyrecoverthemselvesafterafailure,or,whenmorecapacityisneeded,additionalcontainersarelaunchedandthenterminatedwhenthescalingeventisover.

Youwillmorethanlikelybeusingapubliccloud-basedserviceasfollows:

AmazonEC2ContainerService:https://aws.amazon.com/ecs/GoogleContainerEngine:https://cloud.google.com/container-engine/JoyentTriton:https://www.joyent.com/blog/understanding-triton-containers/

Alternatively,youwillbehostingonyourownserversusingaDocker-friendlyandcluster-awareoperatingsystemasfollows:

CoreOS:https://coreos.com/RancherOS:http://rancher.com/rancher-os/

Youwon’tcaresomuchastowhereacontainerislaunchedwithinyourclusterofhosts,aslongasyoucanroutetraffictoit.Toaddmorecapacitytothecluster,youwillbebringingupadditionalhostswhenneededandremovingthemfromtheclusterwhennotneededinordertosaveoncosts.

Chickens

Itsmorethanlikelyyouwillbeusingcontainerstolaunch,processdata,andthenterminate.Thiscanhappenanytimefromonceadaytoseveraltimesaminute.Youwillbeusingadistributedschedulerasfollows:

KubernetesbyGoogle:http://kubernetes.io/ApacheMesos:http://mesos.apache.org/

Becauseofthis,youwillhavealargenumberofcontainerslaunchingandterminatingwithinyourcluster;youdefinitelywon’tcareaboutwhereacontainerislaunchedorevenhowtrafficisroutedtoit,aslongasyourdataisprocessedcorrectlyandpassedbacktoyourapplication.

LiketheclusterdescribedintheCattlesection’sdescription,hostswillbeaddedandremovedautomatically,probablyinresponsetoscheduledpeakssuchasendofmonthreportingorseasonalsalesandsoon.

SnowflakesIhopeoneofthethingsyoutookawayfromthefirstchapteristhatifyouhaveanyserversorservicesthatyouconsiderbeingSnowflakes,thenyoushoulddosomethingtoretirethemassoonaspossible.

Luckily,duetothewaythecontainerizingofyourapplicationsworks,youshouldneverbeabletocreateasnowflakeusingDocker,asyourcontainerizedenvironmentshouldalwaysbereproducible,eitherbecauseyouhavetheDockerfile(everyonemakesbackupsright?)oryouhaveaworkingcopyofthecontainerimagebecauseyouhaveexportedthecontainerasawholeusingthebuilt-intools.

NoteSometimesitmaynotbepossibletocreateacontainerusingaDockerfile.Instead,youcanbackupormigrateyourcontainersbyusingtheexportcommand.Formoreinformationonexportingyourcontainers,seethefollowingURL:

https://docs.docker.com/reference/commandline/export/

Ifyoufindyourselfinthisposition,letmebethefirsttocongratulateyouonmitigatingafuturedisasterbypromotingyourSnowflakeintoaPetorevenCattleaheadofanyproblems.

TipStillrunningaSnowflake?

IfyoufindyourselfstillrunningaSnowflakeserverorservice,Icannotstressenoughthatyoulookatdocumenting,migrating,orupdatingtheSnowflakeassoonaspossible.Thereisnopointinmonitoringaservicethatmaybeimpossibleforyoutorecover.Rememberthattherearecontainersforoldtechnologies,suchasPHP4,ifyoureallyneedtorunthem.

ScenariooneYouarerunningapersonalWordPresswebsiteusingtheofficialcontainersfromtheDockerHub;thecontainershavebeenlaunchedusingaDockerComposefileliketheonewehaveusedseveraltimesthroughoutthisbook.

YouhavetheDockerComposefilestoredinaGitHubrepositoryandyoucantakesnapshotsofthehostmachineasabackup.Asit’syourownblog,youarefinerunningitonasinglecloud-basedhost.

Asuitablemonitoringwillbeasfollows:

DockerstatsDockertopDockerlogscAdvisorSysdig

Asyouarerunningasinglehostmachinethatyouaretreatingasabackup,thereisnorealneedforyoutoshipyourlogfilestoacentrallocationasoddsareyourhostmachines;likethecontainers,itshostingwillbeonlineformonthsorpossiblyevenyears.

Itisunlikelythatyouwillneedtodigtoodeeplyintoyourcontainers’historicalperformancestats,asmostofthetuningandtroubleshootingwillbedoneinrealtimeasproblemsoccur.

Withthemonitoringtoolssuggested,youwillbeabletogetagoodinsightintowhatishappeningwithinyourcontainersinrealtime,andtogetmorethanenoughinformationonprocessesthatareconsumingtoomuchRAMandCPU,alongwithanyerrormessagesfromwithinthecontainers.

YoumaywanttoenableaservicesuchasPingdom(https://www.pingdom.com/)orUptimeRobot(http://uptimerobot.com/).TheseservicespollyourwebsiteeveryfewminutestoensurethattheURLyouconfigurethemto,checkwhetheritsloadingwithinacertaintimeoratall.Iftheydetectanyslowdownorfailureswiththepageloading,theycanbeconfiguredtosendaninitialalerttonotifyyouthatthereisapotentialissue,suchasboththeservicesmentionedhaveafreetier.

ScenariotwoYouarerunningacustome-commerceapplicationthatneedstobehighlyavailableandalsoscaleduringyourpeaktimes.Youareusingapubliccloudserviceandthetoolsetthatcomeswithittolaunchcontainersandroutetraffictothem.

Asuitablemonitoringwillbeasfollows:

cAdvisor+PrometheusZabbixSysdigCloudNewRelicServerMonitoringDatadogELK+LogspoutLogEntriesLoggly

Withthisscenario,thereisabusinessneedtonotonlybenotifiedaboutcontainerandhostfailures,butalsotoholdyourmonitoringdataandlogsawayfromyourhostserverssothatyoucanproperlyreviewhistoricalinformation.YoumayalsoneedtokeeplogsforPCIcomplianceorinternalauditingforafixedperiodoftime.

Dependingonyourbudget,youcanachievethisbyhostingyourownmonitoring(ZabbixandPrometheus)andcentrallogging(ELK)stackssomewherewithinyourinfrastructure.

Youcanalsochoosetorunafewdifferentthird-partytoolssuchascombiningtoolsthatmonitorperformance,forexample,SysdigCloudorDatadog,withacentralloggingservice,suchasLogEntriesorLoggly.

Ifappropriate,youcanalsorunacombinationofself-hostedandthird-partytools.

Whiletheself-hostedoptionmayappeartobethemostbudget-friendlyoption,therearesomeconsiderationstotakeintoaccount,asfollows:

Yourmonitoringneedstobehostedawayfromyourapplication.Thereisnopointinhavingyourmonitoringinstalledonthesamehostasyourapplication;whatwillalertyouifthehostfails?Yourmonitoringneedstobehighlyavailable;doyouhavetheinfrastructuretodothis?Ifyourapplicationneedstobehighlyavailable,thensodoesyourmonitoring.Youneedtohaveenoughcapacity.Doyouhavethecapacitytobeabletostorelogfilesandmetricsgoingbackamonth,6months,orayear?

Ifyouaregoingtohavetoinvestinanyoftheprecedingoptions,thenitwillbeworthweighingupthecostsofinvestinginboththeinfrastructureandthemanagementofyourownmonitoringsolutionagainstusingathird-partythatwilloffertheprecedingoptionsasaservice.

Ifyouareusingacontainer-onlyoperatingsystemsuchasCoreOSorRancherOS,thenyouwillneedtochooseaservicewhoseagentorcollectorcanbeexecutedfromwithinacontainer,asyouwillnotbeabletoinstalltheagentbinariesdirectlyontheOS.

Youwillalsoneedtoensurethatyourhostmachineisconfiguredtostarttheagents/collectorsonboot.Thiswillensurethatassoonasthehostmachinejoinsacluster(whichistypicallywhencontainerswillstarttopopuponthehost),itisalreadysendingmetricstoyourchosenmonitoringservices.

ScenariothreeYourapplicationlaunchesacontainereachtimeyourAPIiscalledfromyourfrontendapplication;thecontainertakestheuserinputfromadatabase,processesit,andthenpassestheresultsbacktoyourfrontendapplication.Oncethedatahasbeensuccessfullyprocessed,thecontaineristerminated.Youareusingadistributedschedulingsystemtolaunchthecontainers.

Asuitablemonitoringwillbeasfollows:

ZabbixSysdigCloudDatadogELK+LogspoutLogEntriesLoggly

Inthisscenario,youmorethanlikelydonotwanttomonitorthingssuchasCPUandRAMutilization.Thesecontainersafterallshouldonlybearoundforafewminutes,andalsoyourschedulerwilllaunchthecontaineronthehostmachinewherethereisenoughcapacityforthetasktoexecute.

Instead,youwillprobablywanttokeeparecordtoverifythatthecontainerlaunchedandterminatedasexpected.YouwillalsowanttomakesurethatyoulogtheSTDOUTandSTDERRfromthecontainerwhileitisactive,asoncethecontainerhasbeenterminated,itwillbeimpossibleforyoutogetthesemessagesback.

Withthetoolslistedintheprecedingpoints,youshouldbeabletobuildsomequiteusefulqueriestogetadetailedinsightintohowyourshortrunprocessesareperforming.

Forexample,youwillbeabletogettheaveragelifetimeofacontainer,asyouknowthetimethecontainerwaslaunchedandwhenitwasterminated;knowingthiswillthenallowyoutosetatriggertoalertyouifanycontainersarearoundforanylongerthanyouwouldexpectthemtobe.

AlittlemoreaboutalertingAlotofthetoolswehavelookedatinthisbookofferatleastsomesortofbasicalertingfunctionality;themillion-dollarquestionisshouldyouenableit?

Alotofthisisdependentonthetypeofapplicationyouarerunningandhowthecontainershavebeendeployed.Aswehavealreadymentionedafewtimesinthischapter,youshouldneverreallyhaveaSnowflakecontainer;thisleavesuswithPets,Cattle,andChickens.

ChickensAsalreadydiscussedintheprevioussection,youprobablydon’tneedtoworryaboutgettingalertsforRAM,CPU,andharddriveperformanceonaclusterthatisconfiguredtorunChickens.

Yourcontainersshouldnotbeuplongenoughtoexperienceanyrealproblems;however,shouldtherebeanyunexpectedspikes,yourschedulerwillprobablyhaveenoughintelligencetodistributeyourcontainerstohoststhathavethemostavailableresourcesatthattime.

Youwillneedtoknowifanyofyourcontainershavebeenrunninglongerthanyouexpectthemtobeup;forexample,aprocessinacontainerthatnormallytakesnomorethan60secondsisstillrunningafter5minutes.

Thisnotonlymeansthatthereisapotentialproblem,italsomeansthatyoufindyourselfrunninghoststhatonlycontainstalecontainers.

CattleandPetsWhenitcomestosettingupalertsonCattleorPets,youhaveafewoptions.

YouwillmorethanlikelywanttoreceivealertsbasedonCPUandRAMutilizationforboththehostmachineandthecontainers,asthiscouldindicateapotentialproblemthatcouldcauseslowdownwithintheapplicationandalsolossofbusiness.

Asmentionedpreviously,youwillprobablyalsowanttobealertedifyourapplicationstartstoservethecontentthatisunexpected.Forexample,ahostandacontainerwillquitehappilysitthereservinganapplicationerror.

YoucanuseaservicesuchasPingdom,Zabbix,orNewRelictoloadapageandcheckforthecontentinthefooter;ifthiscontentismissing,thenanalertcanbesent.

Dependingonhowfluidyourinfrastructureis,inaCattleconfiguration,youwillprobablywanttobealertedwhencontainersspinupanddown,asthiswillindicateperiodsofhightraffic/transactions.

SendingalertsSendingalertsdiffersforeachtool,forexample,analertcouldbeassimpleassendinganemailtoinformyouthatthereisanissuetothesoundingofanaudiblealertinaNetworkOperationsCenter(NOC)whentheCPUloadofacontainergoesabovefive,ortheloadonthehostgoesabove10.

Forthoseofyouwhorequireanon-callteamtobealerted,mostofthesoftwarewehavecoveredhassomelevelofintegrationalertaggregationservicessuchasPagerDuty(https://www.pagerduty.com).

TheseaggregationserviceseitherinterceptyouralertemailsorallowservicestomakeAPIcallstothem.Whentriggered,theycanbeconfiguredtoplacephonecalls,sendSMSmessages,andevenescalatetosecondaryon-calltechnicianifanalerthasnotbeenflaggeddownwithinadefinabletime.

Ican’tthinkofanycaseswhereyoushouldn’tlookatenablingalerting,afterall,it’salwaysbesttoknowaboutanythingthatcouldeffectyourapplicationbeforeyourendusersdo.

Howmuchalertingyouenableisreallydowntowhatyouareusingyourcontainersfor;however,Iwouldrecommendthatyoureviewallyouralertsregularlyandalsoactivelytuneyourconfiguration.

Thelastthingyouwantisaconfigurationthatproducestoomanyfalsepositivesoronethatistootwitchy,asyoudonotwanttheteamwhoreceivesyouralertstobecomedesensitizedtothealertsthatyouaregenerating.

Forexample,ifacriticalCPUalertistriggeredevery30minutesbecauseofascheduledjob,thenyouwillprobablyneedtoreviewthesensitivityofthealert,otherwiseitiseasyfortheengineertosimplydismissacriticalalertwithoutthinkingaboutit,as“thisalertcomeseveryhalfanhourandwillbeokinafewminutes”,whenyourentireapplicationcouldbeunresponsive.

KeepingupWhileDockerhasbeenbuiltontopofwell-establishedtechnologiessuchasLinuxContainers(LXC),thesehavetraditionallybeendifficulttoconfigureandmanage,especiallyfornon-systemadministrators.

Dockerremovesalmostallthebarrierstoentry,allowingeveryonewithasmallamountofcommand-lineexperiencetolaunchandmanagetheirowncontainer-basedapplications.

Thishasforcedalotofthesupportingtoolstoalsolowertheirbarriertoentry.Softwarethatoncerequiredcarefulplanningtodeploy,suchassomeofthemonitoringtoolswecoveredinthisbook,cannowbedeployedandconfiguredinminutesratherthanhours.

Dockerisalsoaveryfast-movingtechnology;whileithasbeenconsideredproduction-readyforawhile,newfeaturesarebeingaddedandexistingfeaturesareimprovedwithregularupdates.

Sofar,in2015,therehavebeen11releasesofDockerEngine;ofthese,onlysixhavebeenminorupdatesthatfixbugs,andtheresthaveallbeenmajorupdates.Detailsofeachreleasecanbefoundintheproject’sChangelog,whichcanbefoundathttps://github.com/docker/docker/blob/master/CHANGELOG.md.

BecauseofthepaceofdevelopmentofDocker,itisimportthatyoualsoupdateanymonitoringtoolsyoudeploy.Thisisnotonlytokeepupwithnewfeatures,butalsotoensurethatyoudon’tlooseanyfunctionalityduetochangesinthewayinwhichDockerworks.

Thisattitudeofupdatingmonitoringclients/toolscanbeabitofachangeforsomeadministratorswhomaybeinthepastwouldhaveconfiguredamonitoringagentonaserverandthennotthoughtaboutitagain.

SummaryAsdiscussedinthischapter,Dockerisafastmovingtechnology.Whilethisbookhasbeeninproduction,therehavebeenthreemajorversionsreleasedfrom1.7to1.9;witheachreleaseDockerhasbecomemorestableandmorepowerful.

Inthischapter,wehavelookedatdifferentwaystoimplementthetechnologiesthathavebeendiscussedinthepreviouschaptersofthisbook.Bynow,youshouldhaveanideaofwhichapproachisappropriatetomonitoryourcontainersandhostmachines,forbothyourapplicationandforthewaytheapplicationhasbeendeployedusingDocker.

Nomatterwhichapproachyouchosetotake,itisimportantthatyoustayup-to-datewithDocker’sdevelopmentandalsothenewmonitoringtechnologiesastheyemerge,thefollowinglinksaregoodstartingpointstokeepyourselfinformed:

DockerEngineeringBlog:http://blog.docker.com/category/engineering/DockeronTwitter:https://twitter.com/dockerDockeronReddit:https://www.reddit.com/r/dockerDockeronStackOverflow:http://stackoverflow.com/questions/tagged/docker

OneofthereasonswhytheDockerprojecthasbeenembracedbydevelopers,systemadministratorsandevenenterprisecompaniesisbecauseitisabletomoveataquickpace,whileaddingmorefeaturesandveryimpressivelymaintainingitseaseofuseandflexibility.

Overthenext12months,thetechnologyissettobeevenmorewidespread;theimportanceofensuringthatyouarecapturingusefulperformancemetricsandlogsfromyourcontainerswillbecomemorecriticalandIhopethatthisbookhashelpedyoustartyourjourneyintomonitoringDocker.

IndexA

alertsenabling/Alittlemoreaboutalertingsettingup,onChickens/Chickenssettingup,onCattle/CattleandPetssettingup,onPets/CattleandPetssending/Sendingalerts

AmazonEC2ContainerServiceURL/Cattle

AmazonWebServicesURL/Awordaboutexternallyhostedservices

ApacheBenchURL/RunningDockerstats

ApacheMesosURL/Chickens

AtomicURL/WhyuseaSaaSservice?

Bbackslash

using/RunningcAdvisorusingacontainer

CcAdvisor

about/WhatiscAdvisor?executing,containerused/RunningcAdvisorusingacontainercompiling,fromsource/CompilingcAdvisorfromsourcereferencelink/Alternatives?

Cattleabout/Cattlecontainers,deployingonto/Cattlealerts,settingup/CattleandPets

Changelog,DockerURL/Keepingup

Chickensabout/Chickenscontainers,deployingonto/Chickensalerts,settingup/Chickens

cloudDocker,deployingin/DeployingDockerinthecloud

CloudComputingreferencelink/Chickens

Cloudscalingreferencelink/Pets,Cattle,Chickens,andSnowflakes

containerused,forexecutingcAdvisor/RunningcAdvisorusingacontainer

ContainerDownabout/Triggers

containerlogsviewing/Viewingcontainerlogs

containersresourceutilization,tracking/Whatjusthappened?comparing,tohostmachine/Comparecontainerstoyourhostmachinedeploying,ontoPets/Petsdeploying,ontoCattle/Cattledeploying,ontoChickens/Chickensdeploying,ontoSnowflakes/Snowflakes

containers,monitoringscenarios/Scenarioone,Scenariotwo,Scenariothree

containerstatsviewing/Viewingcontainerstatssubcontainers/Subcontainersdriverstatus/Driverstatusimages/Images

CoreOS

URL/WhyuseaSaaSservice?,CattleCsysdig

about/UsingCsysdigusing/UsingCsysdig

customgraphscreating/Createcustomgraphsreferencelink/Createcustomgraphs

DDashboard

launching/DashboardDatadog

URL/WhyuseaSaaSservice?,Datadogabout/Datadogagent,installing/Installingtheagentagentinstallation,URL/Installingtheagentwebinterface,exploring/Exploringthewebinterfacereferences/Summaryandfurtherreading

datavolumecontainercreating/Thenextsteps

DigitalOceanURL/Pets

DigitalOceanURL/Awordaboutexternallyhostedservices

Dockerabout/DockerURL,foruserguide/Runningavirtualserver,Thenextstepsdeploying,incloud/DeployingDockerinthecloudadvancements/Keepingup

docker-composefileused,forexecutingELKstack/Whataboutproduction?

DockerComposeabout/Docker

DockerEngineabout/Docker

Dockerexecabout/Dockerexec

dockerexportcommandURL/Snowflakes

DockerHubabout/DockerURL/Docker,WhatiscAdvisor?,Usingcontainers

DockerimagesURL/Driverstatus

Dockerstatsabout/Dockerstatsexecuting/RunningDockerstats

Dockertopabout/Dockertop

driverstatusobtaining/Driverstatus

EElasticsearch

URL,forlaunching/StartingthestackELKstack

about/ELKStackstarting/StartingthestackLogspout/Logspoutlogs,reviewing/Reviewingthelogsexecuting,docker-composefileused/Whataboutproduction?references/Whataboutproduction?

environmentcloning/Cloningtheenvironment

externallyhostedservicesabout/AwordaboutexternallyhostedservicesDocker,deployingincloud/DeployingDockerinthecloud

FFig

about/Docker

GGo

URL/WhatiscAdvisor?GoogleContainerEngine

URL/Cattle

Hhostmachine

containers,comparingto/Comparecontainerstoyourhostmachine

Iimages

listing/ImagesInfluxDB

about/Alternatives?URL/Alternatives?

JJoyentTriton

URL/Cattle

KKibana

URL,forlaunching/StartingthestackKubernetes

URL/Chickens

LLiquidWaxEster(LXE)

about/Keepinguplmctfy

URL/WhatiscAdvisor?localenvironment

launching/LaunchingalocalenvironmentLogEntries

URL/Lookingatthirdpartyoptionsconfiguring/Lookingatthirdpartyoptions

LogglyURL/Lookingatthirdpartyoptions

logsreviewing/Reviewingthelogs

LogspoutURL,forlaunching/Startingthestackabout/Logspoutreferences/Logspout

LogstashURL,forlaunching/Startingthestack

MMariaDB10

URL/Usingcontainersmetrics

collecting/Collectingmetricsrecording/Dockermetricscustomgraphs,creating/Createcustomgraphscontainers,comparingtohostmachine/Comparecontainerstoyourhostmachinetriggers,defining/Triggers

MicrosoftAzureURL/Awordaboutexternallyhostedservices

MonitoringDockerrepositoryURL/Cloningtheenvironment

NNagios

URL/Zabbixncurses

about/WhatisSysdig?NetworkOperationsCenter(NOC)

about/SendingalertsNewRelic

URL/WhyuseaSaaSservice?,NewRelicabout/NewRelicagent,installing/Installingtheagentwebinterface,exploring/Exploringthewebinterfacereferences/Summaryandfurtherreading

P—privilegedflag

about/RunningcAdvisorusingacontainerURL/RunningcAdvisorusingacontainer

PagerDutyURL/Triggers,Sendingalerts

Petsabout/Petscontainers,deployingonto/Petsalerts,settingup/CattleandPets

phpinfoURL/RunningDockerstats

PingdomURL/Scenarioone

processesisolating/Whataboutprocesses?Dockertop/DockertopDockerexec/Dockerexec

Prometheusabout/Prometheuslaunching/LaunchingPrometheusquerying/QueryingPrometheusDashboard,launching/DashboardURL,fordocumentation/Dashboarddatavolumecontainer,creating/Thenextstepsreferences/Thenextsteps

PuppetForgeURL/Usingvagrant

RRancherOS

URL/WhyuseaSaaSservice?,Cattle

SSaaSservice

using/WhyuseaSaaSservice?serverstyles

selecting/Sowhatdoesthisallmean?Slack

URL/TriggersSnowflakes

about/Snowflakesreferencelink/Snowflakescontainers,deployingonto/Snowflakes

StandardStreamsreferencelink/Viewingcontainerlogs

subcontainersabout/Subcontainers

SysAdminabout/Pets,Cattle,Chickens,andSnowflakes

Sysdigabout/WhatisSysdig?installing/InstallingSysdigURL/InstallingSysdigusing/UsingSysdigbasics/Thebasicsdata,capturing/Capturingdatacontainers,listing/Containersreferences/Furtherreading

SysDigCloudURL/WhyuseaSaaSservice?

SysdigCloudabout/SysdigCloudfunctionality/SysdigCloudreferences/SysdigCloud,Summaryandfurtherreadingagent,installing/Installingtheagentagentinstallation,URL/Installingtheagentcontainers,exploring/Exploringyourcontainers

Ttimeseries

about/Prometheusreferencelink/Prometheus

triggersdefining/Triggers

UUbuntuSnappyCore

URL/WhyuseaSaaSservice?UptimeRobot

URL/Scenarioone

VVagrant

URL/Launchingalocalenvironmentvagrantbox

URL/RunningavirtualserverVirtualBox

URL/Launchingalocalenvironmentvirtualserver

executing/Runningavirtualserverhalting/Haltingthevirtualserver

VMwarevCloudAirURL/Awordaboutexternallyhostedservices

WWebinterface

about/TheWebinterfaceoverview/OverviewProcesses/ProcessesCPU/CPUMemory/MemoryNetwork/NetworkFilesystem/Filesystemdrawbacks/Thisisallgreat,what’sthecatch?

WordPressURL,forinstallation/UsingSysdig

YYAML

URL/Docker

ZZabbix

about/ZabbixURL/Zabbix,Usingvagrant,TheZabbixwebinterfacereferences/Zabbixinstalling/InstallingZabbixcontainers,using/Usingcontainersvagrant,using/Usingvagranthostmachine,preparing/Preparingourhostmachinehosttemplates,adding/TheZabbixwebinterface

ZenossURL/Zabbix