continuous integration testing for plone using hudson

Continuous Integration Testing with Hudson

Wednesday, October 27, 2010

Continuous Integration

• Extreme Programming principle

• Martin Fowlerhttp://www.martinfowler.com/articles/continuousIntegration.html

Continuous Integration• Maintain a Source

Repository

• Automate the Build

• Make the Build Self-Testing

• Everyone Commits Every Day

• Every Commit Should Be Built

• Keep the Build Fast

• Test in a Clone of the Production Environment

• Make it Easy to Get the Latest Deliverables

• Everyone Can See the Results of the Latest Build

• Automate Deployment

Essentially

• Commit your code

• Test the crap out of it

• Regressions are bad

• Assign blame

• Shame is a great motivator

Hudson vs Buildbot?

*shrug*

Hudson (Pros)

Easy to install/run

%> java -jar hudson.war

Easy to Configure

• Create

• Modify

• Manage

• All TTW

Frequent Releases~ 1 / Week

Inline Documentation

Plugins

Hudson (Cons)

It’s Java.

It’s Java.1. It eats resources

2. I don’t know how to write plugins

Hudson

Project Layout

• Project/Job

Project Layout

• Project/Job

• Build

Project Layout

• Project

• Build

• Trigger

Triggers

• Other projects

• Remote trigger

• Periodically

• SCM polling

Project Layout

• Project/Job

• Build

• Trigger

• Build Step

Build Steps

• Shell script

• Python script

• Ant

• Maven

Project Layout

• Project/Job

• Build

• Trigger

• Build Step

• Post-Build Action

Post-Build Actions

• Trigger builds

• Archive results

• Generate reports

• Send email

Build Status

• Successful

• Unstable

• Failed

Build Stability

Filesystem Layout

Filesystem Layouthudson /

config.xmlhudson.plugins.*.xmljobs /

myjob / builds /

2010-10-21_11-56-402010-10-21_12-20-26

config.xmlworkspace /

myjob / builds /

2010-10-21_11-56-402010-10-21_12-20-26

myjob / builds /

2010-10-21_11-56-402010-10-21_12-20-26

myjob / builds /

2010-10-21_11-56-402010-10-21_12-20-26

Plugins

• Green Balls

• Python

• Subversion

• Trac

• Warnings

• Violations

• Cobertura

• Disk Usage

• Chuck Norris

Plugins

• Green Balls

• Python

• Subversion

• Trac

• Warnings

• Violations

• Cobertura

• Disk Usage

• Chuck Norris

Plone Integration

Buildout & mr.developer

buildout.cfg[buildout]...extensions = mr.developer

buildout.cfg[buildout]...extensions = mr.developersources = sources

[sources]fsd.core = svn https:.../weblion/fsd.core/trunkfsd.membrane = svn https:.../weblion/fsd.membrane/trunk

mr.developer

• bin/develop checkout

• bin/develop status

• bin/develop update

• bin/develop activate (deactivate)

• bin/develop reset

• bin/develop purge

buildout.cfg[buildout]...extensions = mr.developersources = sourcesauto-checkout = fsd.core fsd.membrane

[sources]fsd.core = svn https:.../weblion/fsd.core/trunkfsd.membrane = svn https:.../weblion/fsd.membrane/trunk

buildoutbin/

buildoutdevelop...

bootstrap.pybuildout.cfg...src/

fsd.corefsd.membrane

• bin/test returns text

• Hudson wants JUnit (XML) format

• collective.xmltestreport

hudson.cfg[buildout]extends = buildout.cfg

parts += xmltestrunner

[xmltestrunner]recipe = collective.xmltestreporteggs = ${test:eggs}defaults = ${test:defaults} + ['--xml']

Test Coverage

[coverage]recipe = zc.recipe.eggeggs = coverageinitialization = sys.argv = sys.argv[:] + ['run', 'bin/xmltestrunner', '-k', '-q', '--xml']

Test Coverage[report]recipe = zc.recipe.eggeggs = coveragescripts = coverage=reportinitialization = eggs = '${buildout:eggs-directory}' bin = '${buildout:directory}/bin' exclude = '--omit=' + ','.join([eggs, sys.prefix, bin]) sys.argv = sys.argv[:] + ['xml', '-i', exclude, include]

Test Coverage

Code Analysis

• zptlint

• pyflakes

• pylint

• jslint

zptlint

• http://pypi.python.org/pypi/zptlint

• Scan page templates for parser and output errors

zptlint

[zptlint]recipe = zc.recipe.eggeggs = zptlintentry-points = zptlint=zptlint:run

zptlint

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" xmlns:tal="http://xml.zope.org/namespaces/tal" xmlns:metal="http://xml.zope.org/namespaces/metal" lang="en" metal:use-macro="context/main_template/macros/master"><body> <metal:fill fill-slot="content-core"> <div tal:content="structure view/context/text/output"> <h2>People</h2> <div tal:content="structure context/@@people" /> </metal:fill></body></html>

zptlint

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" xmlns:tal="http://xml.zope.org/namespaces/tal" xmlns:metal="http://xml.zope.org/namespaces/metal" lang="en" metal:use-macro="context/main_template/macros/master"><body> <metal:fill fill-slot="content-core"> <div tal:content="structure view/context/text/output"> <h2>People</h2> <div tal:content="structure context/@@people" /> </metal:fill></body></html>

zptlint

%> bin/zptlint src/.../view.pt

zptlint

%> bin/zptlint src/.../view.pt

*** Error in: src/.../view.pt <class 'zope.tal.taldefs.TALError'>: TAL attributes on <div> require explicit </div> , at line 12, column 9

Parsing Log Messages

• Warnings plug-in

• Regular expression

• Groovy script (groovy.codehaus.org)

Priorities

• HIGH

• NORMAL

• LOW

zptlint[zptlint-test]recipe = collective.recipe.templateinput = inline: #!/bin/sh if [ -e zptlint.log ]; then echo "Old zptlint.log file removed" rm zptlint.log fi echo "Running zptlint-test" PACKAGES="${buildout:package-directories}" for pkg in $PACKAGES do find $pkg -regex ".*\.[c|z]?pt" | xargs -r bin/zptlint | perl -p -e 's/\s+$/ /g;s/\s+/ /g;s/\*{3}s?/\n/g' >> zptlint.log doneoutput = ${buildout:directory}/bin/zptlint-testmode = 755

zptlint[buildout]...# A list of package locations to be examined # by Hudsonpackage-directories = src/fsd.core/fsd/core src/fsd.membrane/fsd/membrane

pyflakes

• Python syntax checking

• Find common errors quickly

• Doesn’t attempt to run Python code

pyflakes

[pyflakes]recipe = zc.recipe.eggeggs = pyflakesentry-points = pyflakes=pyflakes.scripts.pyflakes:main

pyflakes%> bin/pyflakes src/.../person.py

src/.../person.py:25: 'ATTRIBUTE_NAME' imported but unusedsrc/.../person.py:25: 'IAttributeUUID' imported but unused

pyflakes[pyflakes-test]recipe = collective.recipe.templateinput = inline: #!/bin/sh if [ -s pyflakes.log ]; then rm pyflakes.log echo "Old pyflakes.log file removed" fi echo "Running pyflakes" PACKAGES="${buildout:package-directories}" for pkg in $PACKAGES do find -L $pkg -regex ".*\.py"|xargs -r bin/pyflakes >> pyflakes.log doneoutput = ${buildout:directory}/bin/pyflakes-testmode = 755

[pyflakes-test]recipe = collective.recipe.templateinput = inline: #!/bin/sh if [ -s pyflakes.log ]; then rm pyflakes.log echo "Old pyflakes.log file removed" fi echo "Running pyflakes" PACKAGES="${buildout:package-directories}" for pkg in $PACKAGES do find -L $pkg -regex ".*\.py"|xargs -r bin/pyflakes >> pyflakes.log doneoutput = ${buildout:directory}/bin/pyflakes-testmode = 755

pyflakes

pylint

• Syntax checking

• PEP8 compliance

• Imports code

• You’ve done everything wrong

pylint

[pylint]recipe = zc.recipe.eggeggs = logilab.pylintinstallerextra-paths = ${instance:location}/lib/pythonentry-points = pylint=pylint.lint:Runarguments = [ '--output-format=parseable', '--zope=y', '--reports=y', '--disable-msg=

E0611,F0401,W0232,E1101,C0103,C0111,R0201,W0201,R0911,R0904,F0220, E1103,R0901,E0211,E0213,E1002,W0622',

'--generated-members=objects', ] + sys.argv[1:]

pylint

• disable-msg

• R0911: Too many return statements (%s/%s)

• R0201: Method could be a function

• W0201: Attribute %r defined outside of __init__

• http://pylint-messages.wikidot.com/all-messages

pylint[pylint-test]recipe = collective.recipe.templateinput = inline: #!/bin/sh if [ -s pylint.log ]; then rm pylint.log echo "Old pylint.log file removed" fi echo "Running pylint" PACKAGES="${buildout:package-directories}" for pkg in $PACKAGES do find -L $pkg -regex ".*\.py" | xargs bin/pylint >> pylint.log done echo "Finished"output = ${buildout:directory}/bin/pylint-testmode = 755

pylint

Selenium

Selenium• Automated web application testing

Selenium1. Open this page

2. Click that thing

3. Fill out that form

4. Click submit

Did it work?

Selenium Testcase

def test_login_overlay(self): self.open("/") self.wait() self.selenium.click('link=Log in') self.waitForElement('form#login_form') self.selenium.type("name=__ac_name", TEST_USER_NAME) self.selenium.type("name=__ac_password", TEST_USER_PASSWORD) self.selenium.click("submit") self.wait() self.failUnless(self.selenium.is_text_present("Log out"))

Selenium Testcaseclass SeleniumLayer(PloneSandboxLayer): defaultBases = (HOST_ADJUSTABLE_ZSERVER_FIXTURE, PLONE_FIXTURE)

# Connection parameters seleniumHost = os.environ.get('SELENIUM_HOST', 'localhost') seleniumPort = os.environ.get('SELENIUM_PORT', '4444') seleniumBrowser = os.environ.get('SELENIUM_BROWSER', '*firefox')

def setUpZope(self, app, configurationContext): ...

def setUpPloneSite(self, portal): ... url = "http://%s:%s/%s" % (self['host'], self['port'], PLONE_SITE_ID) self['selenium'] = selenium.selenium(self.seleniumHost,

self.seleniumPort, self.seleniumBrowser, url) self['selenium'].start()

def tearDownPloneSite(self, portal): self['selenium'].stop() del self['selenium']

Selenium RC

• Firefox

• Internet Explorer

• Safari

• Chrome

• Opera

• more...

Selenium Grid

“Firefox?”

Thanks!

Selenium Test Runnerfsd/

core/...selenium/

base.pytestSelenium.py

tests/...

Selenium Test Runner

[selenium]recipe = collective.xmltestreporteggs = fsd.core [test] fsd.membrane [test]defaults = ['--tests-pattern', '^f?selenium$', '--xml']

[selenium-firefox]recipe = ${selenium:recipe}eggs = ${selenium:eggs}defaults = ${selenium:defaults} [selenium-safari]recipe = ${selenium:recipe}eggs = ${selenium:eggs}defaults = ${selenium:defaults}

%> bin/selenium-firefox%> bin/selenium-safari

workspace/bin/...parts/

selenium-firefox/testreports/

*.xmlselenium-safari/

testreports/*.xml

Sprint!

• Saturday

• QA Sprint

• Selenium testing and more

• Free food!

Multi-Configuration Projects

• Run a build for each set of n x n ( x n x n) variables.

Wish List

• Load testing

• Sphinx builds

• Chameleon compatibility checking

• Windmill

Best Practices

Check in Your Configs

• Configuration is stored in XML files

• Easy to back up to SVN

Check in Your Configshudson /

myjob / builds /

2010-10-21_11-56-402010-10-21_12-20-26

hudson /config.xmlhudson.plugins.*.xmljobs /

myjob / builds /

2010-10-21_11-56-402010-10-21_12-20-26

Watch Your Disk Space

• “Hudson disk-usage” plugin

Use an Egg Cache

• Set up a default.cfg for your hudson account.

[buildout]eggs-directory =

/home/hudson/.buildout/eggsdownload-cache =

/home/hudson/.buildout/downloads

Pin unittest2

• (If you’re using it)

• plone.app.testing

• 3x faster buildout

Split Big Builds

• How long does the build take?

• How often does each part need to be run?

• Tests?

• Coverage?

Split Big Builds

• Build

• Test

• Syntax Checking

• Coverage

• Selenium

Split Big Builds1. “Build other projects” post-build action

2. Custom workspace

3. Build Fingerprinting

Fingerprinting

• “These builds are related”

• Archive an md5 hash of a file to act as an identifier

FingerprintingIn parent build:

In parent and child builds:

Reporting

Hudson Report Views

iPhone

Extreme Feedback Devices

hudson.plone.org

• Currently builds Plone 4.0.x

• Core developers have access

• Add your core & collective projects

Thanks

• Martin Aspeli

• Timo Stollenwerk

• Hanno Schlichting

QA Sprint!

• Saturday

• QA Sprint

• Selenium testing and more

• Free food!

continuous integration testing for plone using hudson

Technology

continuous integration in the cloud with hudson ·...

plone handouts

plone redesign

plone mockup modernising plone front-end development

plone biblio

eclipse democamp bucharest 2014 - continuous integration...

plone pwns

angularjs & plone

eu-osha: nuestra experiencia con plone - plone day 2012

about plone - enfold systems, the plone experts

jon stahl best plone ever presenting plone 3

troubleshooting plone

plone futures, plone conference 2016 keynote by eric steele

continuous integration testing based on selenium and hudson

simplifying plone

churn, continuous eligibility, and medicaid: the safety net...

practicing continuous delivery using...

functional continuous integration with selenium and hudson

conclusion - plone

world plone day 2013 tokyo, new version of plone