7/13/17

1

InfrastructureDesignPatternswithPython,Buildbot,andLinux

ContainersDavidLiuPythonTechnicalConsultantEngineerIntelCorporation

Overview

• Introduction• BreakingoutofCI:InfrastructureDesignpatternswithBuildbotframeworkpieces• Hookingthingsupinweirdways:Ports,multi-masters,andpseudo-RPC• Whenthingsdon’tfit:LinuxContainers,andkeepingthingsmovable• PullingitalltogetherwithPython• Real-worldarchitecturesthathaveworked• Summary

7/13/17

2

Introduction:OnInfrastructure

•Whatdoesitmeantohaveinfrastructure?• Isitautomation?Isitorchestration?Isittaskrunners?• Manyoptionsexistdependingonwhat“needs”youhave• FullonorchestrationwithChef,Puppet• Dask,IPyParallel,Joblib (Thesearemostlynumerical)• Celery,Kafka

•Manyoftheseexamplesareheavy-handedorsquarepeg/roundholeproblems

OnInfrastructure(con’t)

• Examples• TryingtogetadistributedtasksystemsuchasDask torunaCRONisnotexactlythebestusecase• TryingtogetCelerytodoamap-reduceoperation• Tryingtogetpuppettomakeataskgraph

• Inessence,everyoneoftheseframeworksaremeantforvastlydifferentthings!

7/13/17

3

BreakingoutofCI:InfrastructureDesignpatternswithBuildbot• Buildbot isnormallymeantforContinuousIntegration(CI),butyoucanconstructthingsoutoftheelementsinweirdways.• JustlikeLegoblocksforinfrastructure;thisdiffersheavilyfromthingssuchasJenkinsorTeamCity• CITasksnormallyencompassinterestingpieces:Ascheduler,dependencies,aresult• However,thesemaintaskcomponentsareactuallycomposedofmanyotherprimitivesthathavebeenassembledtogether

BreakingoutofCI:InfrastructureDesignpatternswithBuildbot (Con’t)

• Resourcepools• Rolesandtriggers• Taskrunners• DistributedSystem+communications• StateLogic

• Changetriggers• Schedulers• Buildsteps(scriptingsteps)•Master/workersystem• Barriersandsemaphores• Dependencytree

Examples:

7/13/17

4

BreakingoutofCI:InfrastructureDesignpatternswithBuildbot (Con’t)• BecauseBuildbot splitsalltheseitemsup,onemaybeabletowirethecomponentsupinunusualwaystomeetcommonlyoccurringinfrastructurepatterns

• Warning:Beforegoinganyfarther,IwanttoreiteratethatwhatIamabouttoshowisconceptual andusedforproof-of-concepts,andisnoreplacementforsoundorchestrationandpropersecurity• Thisisconsideredavery“offuse”ofBuildbot (andwasnotintendedbythedevelopers),sojustbemindfulofthis

BreakingoutofCI:InfrastructureDesignpatternswithBuildbot (Con’t)• Infrastructuredesignpatternsarethecommontasks/roles,andinterconnectsthatoccurinsoftwaredeployments• UsingBuildbot isjustone wayofsolvingsuchexamples• Onecanutilizethistoprototypesomethingandthenconvertittoenterprise-leveldeployments

• Examples:• CI->Package->Deployment(commonuse)• Enterpriseapplicationdeployment• LicenseServer• LinuxSessionlaunching/landingonServers• HomeWeatherServerw/MachineLearningtasks

7/13/17

5

Hookingthingsupinweirdways:Ports,multi-masters,andpseudo-RPC• Normally,mostCIsystemsdonotexposesuchcontrols,butbecauseoftheflexibilityinBuildbot,onemayuseitquitefreely• Thechange-port allowsforusageofascriptorsymlinked calltotriggeratask-whichgivesuser-leveltriggers• Bypassinginargumentsofthescriptin,onecanessentially“RPC”toaworkerwithaknownresource• i.e.runsometaskwheretherightversionofPython/NumPy is

• Buildbot iscontrolledviathelogicofthemaster.cfg,whichisinterpretedasmajorityPythoncode

Hookingthingsupinweirdways:Ports,multi-masters,andpseudo-RPC(Con’t)• Inthebuildmaster’s configuration,normalchangesourceslooklikethefollowing:• c['change_source']=[]c['change_source'].append(changes.GitPoller('git://github.com/buildbot/pyflakes.git',workdir='gitpoller-workdir',branch='master',pollinterval=300))

• However,youcanaddasecondarytriggersource:• c['change_source'].append(changes.PBChangeSource(port=9999,user=’myApp',passwd=’AppPassword'))

7/13/17

6

Hookingthingsupinweirdways:Ports,multi-masters,andpseudo-RPC(Con’t)

• Matchedwiththe“fakechange.py”scriptinbuildbot-contrib,onecaninitiateandpassarguments(suchasX11info,userinfo)toabuildmaster• Utilizesthetwisted.internet andtwisted.spread capabilities• Sendschangetothescheduler intheBuildbotmaster.cfg

Exampleofusingthechange-porttolaunchapps

7/13/17

7

Hookingthingsupinweirdways:Ports,multi-masters,andpseudo-RPC(Con’t)

• Multi-mastergivestheabilitytochaintasksandresourcepoolstogethertograntcapabilitiessuchasloadbalancerstocertaintasks• Don’thesitatetohaveonetaskkickoffanothersubsetofBuildbotinstances

ImagefromBuildbot Docs

Hookingthingsupinweirdways:Ports,multi-masters,andpseudo-RPC(Con’t)• Useutil.BuildFactory() tosendcommandstoworkersviaShellCommand• Notethattheworkermustbeprivilegedtoruncommandandmusthaveresources,sodefineworkerswell

7/13/17

8

Whenthingsdon’tfit:LinuxContainers,andkeepingthingsmovable• Whathappenswhenthingsdon’twanttofittogether?oryouhavesecurityconceptstoworryabout?• UseLinuxContainerstoprovideadditionaldesignflexibilitythroughcompositiontechniques(docker-compose,asanexample)• UseContainerstoalsocordonofftheriskierbits(preventvolumemaps,etc.)• Provideprivilege/non-privilegedbarrierstoseparateusersfromfull-privilegedresources

Whenthingsdon’tfit:LinuxContainers,andkeepingthingsmovable(con’t)• Atsomepoint,youmayneedorchestrationtopullofftasks,sojustknowwhatresponsibilitiesyouwantinwhattechnologies• Dependingonhowyouapproachtheproblem,youmightbeabletogetawaywithlittleornoorchestration• Ifallelsefails,youcansomewhatcheatbyhavingtheentireBuildbot+logicinsideofacontainer,andusethoseasbuildingblocks

7/13/17

9

PullingitalltogetherwithPython

• SowithPythonattheforefront,youcanutilizethePythonscriptsinjectedintobuildbot itself,orhavethemaster.cfg unpack codethatitreceives• ThescriptingcapabilitiesmeanthatyoucanusecallsinbuildstepstoachievethingsinanRPCformatontheworkers• Pythoncancallthebuildmasterseasily,soscriptingittodoyourbiddingisfree-form• Mixingthiswithfileopening,webcallsandrequests,arejustsomeoftheadvantagesofusingPython“glue”

Real-worldarchitecturesthathaveworked

• Company-wideserverapplicationdeployment• UsedApplicationssetincontainers,calledbysymlinked pythonscriptscallingportstostartprogram

• CompanyusedOrchestrationtoscaleupanddowntheavailableworkersasa“resourcepool”dependingonserverloads

• Licenseserverfora“floatinglicense”• Companyonlyhadonelicense,andsoftwarehadnoabilitytogatephonehomedataorqueue

• Implementedwithbuildbot worker,andamasterthatqueued/scheduled/gatedtheusers.

7/13/17

10

BuildbotWorker

BuildbotWorker

Company-wideserverapplicationdeployment

• Buildmasterhandlesqueue,sessiondetails• Spinupnewworkersforlargerpool• UpdateapplicationsviaDockerrepositoryonWorker

/usr/binmappedchange

portscript

BuildbotMaster

BuildbotWorkerUser

Changeport

RequestApplicationSession

UsersessionorLoginNode Serverwithresource&fullprivilege

LaunchDockerAppMountVolumeForwardXSession(Screen)

ApplicationScreen

Company-wideserverapplicationdeployment

7/13/17

11

Company-wideserverapplicationdeployment(con’t)

Company-wideserverapplicationdeployment(con’t)

7/13/17

12

BuildbotWorker

BuildbotWorker

LicenseServerfora“floatinglicense”

• License(s)canbeheldbymasterortheattachedstockDB• Eitheruseavailablepooltoblocklicenses,orviaDBorlogic• Canalsojustholdalockonthelicensefileinsteadofapplicationscreen

/usr/binmappedchange

portscript

BuildbotMaster

BuildbotWorkerUserA

Changeport

RequestApplicationLicense

UsersessionorLoginNode Serverwithresource&fullprivilege

LaunchDockerAppMountVolumeForwardXSession(Screen)

ApplicationScreen

DBorLicenseLogic

UserB

LicenseServerfora“floatinglicense”

7/13/17

13

Real-worldarchitecturesthathaveworked

• ComputeserverLinuxsessionhandler• CompanyusedBuildbot master/workerwithworkersandX11forwardingtohandsessionstousers;queuesystemviathemaster’sbonestockscheduler

• HomeMachineLearningserver• Usedsuccessfullytocreatea“dashboard”and“computecenter”formyhomesystem,whichpullsinaggregatedataanddoesMLonlargedatasetswithclassifiers

BuildbotWorker

BuildbotWorker

ComputeserverLinuxSessionHandler(Tiersetupw/Multimaster)

/usr/binmapped

changeportscript

BuildbotMaster

BuildbotWorker

UserAChangeport Applications,

othersetupdetailsandadvancedtasks

Usersession Serverwithresource&fullprivilege

LaunchXterm SessionMountVolumeForwardXSession(Screen)

LandedSession(s)

DBSessionQueue

UserB

LoginNode

TaskBuildbotMaster

7/13/17

14

BuildbotWorker

BuildbotWorker

MachineLearningServer

BuildbotMaster

BuildbotWorker

DashboardLocalhost:8080

Localmachine Serverwithresource&fullprivilege

Cleanupdata,closetaskPostnewresultofMLondashboard

DBSessionQueue

Buildmaster layer

GetDatafromURLsandsources

Retrieveddata

Compute

WebUI

Weblayer

Summary

• Throughalittlebitofingenuityandcreativeuseofcomponents,onemayfashionmanyoftheinfrastructuredesignpatternsthatappearinsoftwareandIT• Beingabletorapidlydesignproof-of-conceptsispossiblewiththismethod,andcanrevealdesignconsiderationsbeforemakingapropersolution• Rememberthattheexamplesshownarenotshownwithanysecurityororchestration• Keepinganopenmindandaneyeonupcomingtechnologycanwidentheavailableinfrastructurepatternsonecandesign• Experimentwithnewtoolsoftentoseewhatpatternscanbemadenext

7/13/17

15

References

• http://buildbot.net• https://github.com/buildbot• Repoforexamples(Tobepostedsoon):• https://github.com/triskadecaepyon/infrastructure_patterns

Q&A?https://github.com/triskadecaepyonhttps://triskadecaepyon.github.io/