agenda - cloudbees...source control: git code review: gerrit ci: jenkins slaves: vms + bare metal...

AGENDAIntroductionWhatisoVirtBasicviewofoVirtInfrastructureReleaseflowGerrithooksJenkinsjobstypesJenkinsjobsbuilderPublishingbuildstoarepositoryserverFutureplansQuestions

INTRODUCTIONWhoarewe?

KirilNesenko-InfrastructureMaintainerandreleaseengineer.ResponsibleforreleaseengineeringactivitiesforoVirtproject.

EyalEdri-InfrastructureMaintaineratoVirtandCIteamleadatRedHat.

WHATISOVIRToVirtisavirtualdatacentermanagerDeliverspowerfulmanagementofmultiplevirtualmachinesonmultiplehostsUsesKVM,libvirtandvdsm.CanbeinstalledonFedora,CentOSorRHEL

OVIRTARCHITECTURE

OVIRTUI

OVIRTINFRASTRUCTUREVIEW

SERVICESProvisioning:ForemanConfiguration:PuppetArtifacts(mirror):ArtifactorySourceControl:gitCodeReview:gerritCI:JenkinsSlaves:VMs+BareMetalhostsVirtualization:oVirt,OpenStack

RELEASEPROCESS

COMMITLIFECYCLE

OVIRTCONTINUOUSDELIVERYOnepublisherperversionUseagenericscripttopublishthebuilds

PUBLISHERJOB

PUBLISHERJOB

GERRITWhatisGerrit?

GERRIT:GITWEBCodehosting(gitfrontend)

GERRITHOOKWhatisahook?

Foreachworkflowstep(event)ascriptgetsexecuted

GERRITHOOKSDefaultgerrithooksflaws:

1event<->1hookSamehookforallprojectsFireandforget,nointeraction

Solution:

Buildadispatcher!

GERRITHOOKS:TRIGGER

PatchCreatedCommentAddedChangeMergedChangeAbandonedReferenceupdated(moreonnewerversions)

GERRITHOOKS:EVENTTYPE

ByparameterspassedWewillbeabletodetectcustomevents:

MaintainerapprovedTestspassedMaintainerblocked...

GERRITHOOKS:LABELS

Comment

Bug-Url:http://mytracker/bugid

Rerun-Hooks:all

What?

Label:labelcontent

Where?

Commitmessage

Example:

GERRITHOOKS:CHAINS

Hooknametemplate:

${event}.${chain}.${hook_name}${event}.${hook_name}${event}

UsingeventnameChainassecondtermifwantedHierarchicalconfigfiles

Mainoneat$gerrit_dirPerprojectPerevent

GERRITHOOKS:EXECUTION

Everythingtologfile(configurable):

Eachhookcan:

INFO::STARTING::change-mergedDEBUG::get_hooks::change-mergedonvdsm.git/hooksINFO::::AVAILABLEHOOKS::['change-merged.set_MODIFIED','change-merged.update_tracker']

V:VERIFIEDSCORE\nR:REVIEWSCORE\nCOMMENT

Breakit'shookchainBreakwholeexecution

Commonstdoutformat(eachsectionisoptional):

GERRITHOOKS:REVIEW

Sumofallthereviews:

negativeonesprevailoverpositivesnovotemeansnochange0meansresetnegativevalue

Executionsummaryofallthehooks:

GERRITHOOKS:SOURCECODE

Patchesarealwayswelcome!

Git:gitclonegit://gerrit.ovirt.org/gerrit-admin

Web:http://gerrit.ovirt.org/gitweb?p=gerrit-admin.git

GERRITHOOKS:LIBSSomesimplegenericlibsavailableforthehooks:

Bugzilla

Gerrit

Configuration

Tools

Bashlibusageexample:sourcebz.shsourcegerrit.shgerrit.parse_params"$@"#Setstheparametersintoglobalvars#getthebugidsinthecommitmessageofthecommitpassedas#parameter(--commitSHA)bug_ids=($(bz.get_bug_id"$commit"))

Pythonlibusageexample:fromgerritimportGerritfromconfigimportload_configconfig=load_config()##Loadstheconfighierarchicalyintoadictgerrit=Gerrit(config.get('GERRIT_SRV'))##Connecttothegerritinstance

Modifybugstatuswhenallpatchesmerged

GERRITHOOKS:BUGZILLAUpdateexternaltrackerinbugzilla

Modifybugstatuswhenfirstpatchsent

JENKINSJOBBUILDERProblemswithmanualconfiguration

HardtomaintainHardtotrackHardtoreviewHardtochangecommonsettings

Sometaskssoonbecomequitepainful:

...AddanextrascmtojobsforprojectsA,Band

C

NewbranchforprojectM,duplicateallit'sjobs

Disable all the jobs for aprojectTest a job configuration before

deploying

JENKINSJOBBUILDER

JenkinsjobBuildertotherescue!

JENKINSJOBBUILDERSomepluginsthathelpedalongtheway:

Managejobsstatuses:

Configurationdeduplication:

Matrixconfigurations

Multijobsconfigchanges:

Configurationslice

Configuationchangemanagement:

JobConfighistory

Scriptler

JENKINSJOBBUILDER

DevelopedbyOpenStackfolks(Kudos!!)Configurationascode(yamlformat)Easytoreviewchanges(onGerrit!)Code/configde-duplicationIncludeexternalshell/groovy/...scriptsTestbeforedeployingEasiertoorganize(perdirectory,perfile)Servesasbackup(easilyreplicateonanotherjenkins)

JENKINSJOBBUILDER

Thiswillcreatethejobs:projectA-whatever-fc19projectA-whatever-fc20

-job-group:name:projectAdist:-fc19-fc20jobs:-'{name}-wahtever-{dist}'

-job-template:name:'{name}-whatever-{dist}'node:'slave-{dist}'triggers:-timed:'@midnight'builders:-shell:makewhateverpublishers:-archive:artifacts:'*.log'

RunningatmidnighOnslave-fc19andslave-fc20Executing'makewhatever'Archivingthelogs

JENKINSJOBBUILDER

Oh!ButIwanttoaddanewdistro,el6

-job-group:name:projectAdist:-fc19-fc20-el6jobs:-'{name}-wahtever-{dist}'

Andacleanupafter

-job-template:name:'{name}-whatever-{dist}'node:'slave-{dist}'triggers:-timed:'@midnight'builders:-shell:makewhatever-shell:makecleanpublishers:-archive:artifacts:'*.log'

JENKINSJOBBUILDEROfficialOpenstacklinks:

Checkjob

Code:

Docs: Latestbuilddocs

Httpsgitrepo

oVirtlinks

Gityamlcodetree

Deployjob

OVIRTJENKINSJOBTYPESTriggeredonlyonpatchsetsTriggeredonlyonmergesMatrixjobsCOPRjobs

POLLSCMVSGERRITTRIGGERWeusegerrittriggerplugininsteadofpollSCMAllowstotestpatchesbeforemergeReducestheloadonthegerritserver

MATRIXJOBSAllowstoconfigurethejobstorunondifferentconfigurations

MATRIXRELOADEDAllowstorerunthejobonaspecificconfiguration

COPRCreatedbyFedorafolksIsaneasytouseautomaticbuildsystemprovidingapackagerepositoryasitsoutputReducestheloadonjenkinsslavesCLI+Jenkinsplugin

yum-yinstallcopr-clicopr-clibuildvdsmhttp://jenkins.ovirt.org/view/vdsm-

COPRJENKINSPLUGIN

RELEASEPROCESS

CURRENTPROBLEMS/FUTURESOLUTIONSScarceresources

MovingtoanewinfrastructureOS1(openstack)slaves

JobsmanagementandcontrolMovealljobstoyamlGetdeveloperstohelpwrite/maintainjobs

LeveragemanualproceduresAutomatetheofficialrelease+signingpackagesAutomategerritconfig/projectcreation

CURRENTPROBLEMS/FUTURESOLUTIONSIsolatetestenvironments

Docker/mockonallsimplebuildsOneslaveperjoboncomplexbuilds

Provisioningofslavesondemand

Leveragepackagebuilds

BuildonCopr(hasjenkinsplugin!)

QUESTIONS?

USEFULLINKSoVirt

oVirtgerrit

oVirtJenkins

Copr

JenkinsJobBuilderCode

JenkinsJobBuilderDocs

YAMLcodetree

CONTACTUS