deployment and continous integration of a zope/plone application

;

Deployment & Continuous integrationDeployment & Continuous integration

of a Plone/Zope applicationof a Plone/Zope application

Julien PivottoJulien Pivotto

PyconFR’13PyconFR’13October 26, 2013October 26, 2013

;

whoamiwhoami• sysadmin @ inuitssysadmin @ inuits• open-source defender for 7+ yearsopen-source defender for 7+ years• devops believerdevops believer• @roidelapluie on twitter/github@roidelapluie on twitter/github

Julien Pivotto Deployment and CI of a Plone/Zope application

;

• Plone is an Open-Source CMSPlone is an Open-Source CMS• Written in pythonWritten in python• Vibrant communityVibrant community


;

• Open-Source Web application serverOpen-Source Web application server• Written in PythonWritten in Python• Used by PloneUsed by Plone


;

Plone/Zope applicationPlone/Zope application

• Threads = instancesThreads = instances• Built using BuildoutBuilt using Buildout• Modules (eggs)Modules (eggs)


;

The missionThe mission

• Several Zope/Plone applicationsSeveral Zope/Plone applications• ScalabilityScalability• AutomationAutomation• ReliabilityReliability


;

The old daysThe old days

• Manual workManual work• Failover HAFailover HA• SupervisordSupervisord• No overviewNo overview


;

BUTBUT• Run buildout on each server/environmentRun buildout on each server/environment• Compilation error in the middleCompilation error in the middle• Pypi is downPypi is down• Which commit is faulty?Which commit is faulty?


;

The goalThe goal

• Less failure during buildsLess failure during builds• More testingMore testing• Puppetize all the thingsPuppetize all the things• Reproduce all the buildsReproduce all the builds• More env: dev, test, demo, prodMore env: dev, test, demo, prod


;

Infrastructure changesInfrastructure changes• DRBD for PostgreSQL ⇒ PostgreSQL 9 native replicationDRBD for PostgreSQL ⇒ PostgreSQL 9 native replication• DRBD for files ⇒ GlusterFS for data, RPM for codeDRBD for files ⇒ GlusterFS for data, RPM for code• Supervisord to launch instances ⇒ Puppet to start servicesSupervisord to launch instances ⇒ Puppet to start services• Pound as reverse proxy ⇒ HaproxyPound as reverse proxy ⇒ Haproxy• Internal mirror of pypiInternal mirror of pypi


;

PuppetizationPuppetization

• From nothing to everythingFrom nothing to everything• Plone, but also apache, haproxyPlone, but also apache, haproxy• . . . and anything else. . . and anything else• Mcollective to orchestrateMcollective to orchestrate


;

Development decisionsDevelopment decisions

• Move away from SVNMove away from SVN• Write testsWrite tests• Automate them in JenkinsAutomate them in Jenkins


;

It takes timeIt takes time

• Cultural shiftCultural shift• Legacy stuffLegacy stuff• A lot of surprisesA lot of surprises• Compromises between Dev and OpsCompromises between Dev and Ops


;

JenkinsJenkins

• An open-source CI serverAn open-source CI server• More than 600 pluginsMore than 600 plugins• FlexibleFlexible• Jobs & pipelinesJobs & pipelines


;Julien Pivotto Deployment and CI of a Plone/Zope application

;

JobsJobs

• Reproducible buildsReproducible builds• Single place to find build logsSingle place to find build logs• History of buildsHistory of builds• Creating (RPM) artifactsCreating (RPM) artifacts


;

PipelinePipeline

• Build is only part of the processBuild is only part of the process• Need to package, deploy, . . .Need to package, deploy, . . .• Running testsRunning tests


;

TestsTests

• Running tests on each moduleRunning tests on each module• Feedback: graphes, mailsFeedback: graphes, mails• Code coverageCode coverage• ViolationsViolations• Unit testsUnit tests• zc.recipe.testrunnerzc.recipe.testrunner


;

Translation in JobsTranslation in Jobsvirtualenv --no-site-packages .bin/python bootstrap.pybin/buildout -c jenkins.cfg buildout:eggs-directory=

/var/lib/jenkins/buildout-shared-eggsbin/test-all


;

BuildoutBuildout

• Running buildout in a Spec fileRunning buildout in a Spec file• Creating a RPMCreating a RPM• Using virtualenvUsing virtualenv• Starting from scratch each timeStarting from scratch each time


;

Ops like RPMOps like RPM

• Easy to deploy anywhereEasy to deploy anywhere• Consistency and dependenciesConsistency and dependencies• Find where a file comes fromFind where a file comes from• Orchestrate with puppet/mcollectiveOrchestrate with puppet/mcollective• Version numberVersion number• Conf does not belong to packageConf does not belong to package


;

TimeTime


;

Deploying ZopeDeploying Zope

• Building RPMBuilding RPM• Adding them to a yum repo (with pulp)Adding them to a yum repo (with pulp)• Cleaning yum cache (with mcollective)Cleaning yum cache (with mcollective)• Running puppet (with mcollective)Running puppet (with mcollective)• dev -> staging -> demo and proddev -> staging -> demo and prod


;

ReportsReports

• Jenkins build logJenkins build log• Puppet logsPuppet logs• Zope instances logsZope instances logs• IRC, mails, . . .IRC, mails, . . .


;

Role of PuppetRole of Puppet

• Puppet creates the configurationPuppet creates the configuration• Update the packageUpdate the package• Restarts the instancesRestarts the instances• Puppet (hiera) creates password filesPuppet (hiera) creates password files


;

Database upgradesDatabase upgrades

• WIP, no solution chosen yetWIP, no solution chosen yet• Postscript in RPM?Postscript in RPM?• Commands trigerred by puppet?Commands trigerred by puppet?


;

LogsLogs

• Zope/Plone: logs files/instancesZope/Plone: logs files/instances• 10 instances * 2 servers10 instances * 2 servers⇒ 20 log files20 log files• Plus apache logsPlus apache logs• Plus system logsPlus system logs


;

The deprecated wayThe deprecated way

• tail -ftail -f• multitailmultitail• clustersshclusterssh


;

LogstashLogstash

• An open source toolAn open source tool• Manage your logsManage your logs• Input from anywhereInput from anywhere• FilterFilter• Output to anywhereOutput to anywhere


;

LogstashLogstash


;

Plone and LogstashPlone and Logstash

• Plone output to syslogPlone output to syslog• Mangle to add good fieldsMangle to add good fields• Logstash output to ElasticSearchLogstash output to ElasticSearch• Kibana3 creates dashboardsKibana3 creates dashboards


;

zope.confzope.conf<syslog>address 172.16.28.55:5544facility local3format %(asctime)s ZopApp-Server zope[%(process)s]:

instance1 [%(levelname)s] %(name)s | %(message)sdateformat %b %d %H:%M:%Slevel info</syslog>


;

filter {grok {type => "syslog"pattern => ["(?m)<%= "<%" %>{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp}%{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?:%{GREEDYDATA:syslog_message}"]

add_field => [ "received_at", "%{@timestamp}" ]add_field => [ "received_from", "%{@source_host}" ]add_tag => "syslog-%{syslog_program}"

}}


;

\frametitle{zope.conf}filter {grok {tags => ["syslog-zope"]pattern => ["(?m)(?<instance>[a-z0-9]+)\[(?<loglevel>[A-Z]+)\](?<loggername>[A-Za-z0-9.]+) \|%{GREEDYDATA:logmessage}"]

add_tag => "grokked-zope"}mutate {tags => ["grokked-zope"]rename => [ "@source_host", "application" ]replace => [ "@message", "%{logmessage}" ]add_tag => "loggername-%{loggername}"

}}


;

Kibana3Kibana3

• Plays with ElasticSearch and LogstashPlays with ElasticSearch and Logstash• Easily build dashboardsEasily build dashboards• Share the metric and logs with anyoneShare the metric and logs with anyone• Filter by time, instance, http code. . .Filter by time, instance, http code. . .• View all the logs at the same placeView all the logs at the same place


;

ConclusionsConclusions

• This is a sysadmin point of viewThis is a sysadmin point of view• Still a lot of work to be doneStill a lot of work to be done• From a "black box"From a "black box"

. . . To a fully automated environmentTo a fully automated environment• Not a unique solutionNot a unique solution• You need to get it working for youYou need to get it working for you

. . . in your wayin your way


;

This work is being realized withThis work is being realized withAffinitic SPRL, Plone developers in BelgiumAffinitic SPRL, Plone developers in Belgium

http://affinitic.behttp://affinitic.be


;

Thank youThank youAny question?Any question?


;

ContactContactJulien PivottoJulien [email protected]@inuits.eu@roidelapluie@roidelapluie

INUITS bvbaINUITS bvbaBelgiumBelgium+32 473 441 636+32 473 441 636https://inuits.euhttps://inuits.eu


deployment and continous integration of a zope/plone application

Technology