plone testing tools and techniques
DESCRIPTION
Plone Conferrence 2009 talk, an overview of Testing Tools and Techniques with PloneTRANSCRIPT
Plone Testing Tools and TechniquesAn Overview
Presented By: Jordan Baker, ScryentPlone Conference 2009, Budapest
Image CC Encyclopedia Britannica
Wednesday, October 28, 2009
Goals of this Talk
• practical and relevant usage of testing in Plone, not exhaustive
• give you the confidence to start testing, or incorporate it more fully into your development process
• give you some hands-on examples of using testing
• provide a good review of the topic for more experienced developers
• provide some quality resources as reference
Wednesday, October 28, 2009
Wednesday, October 28, 2009
Wednesday, October 28, 2009
About Me (Jordan Baker)
• Using Plone since version 1.0
• Senior Consultant/Founder of Scryent, a Plone service provider based in Toronto, Canada
• Over the years I have gone from writing no tests, to writing lots of tests, to writing tests first, to getting 100% code coverage and beyond
• Quality code and enhanced developer productivity are some of my passions
Wednesday, October 28, 2009
Slides and examples
• These slides are available on Slideshare so you can follow along:
http://tinyurl.com/pttt-slides
• Also the Plone example used in this talk is available from bitbucket. To checkout a copy and build:
$ hg clone https://[email protected]/hexsprite/pttt/$ cd pttt$ python2.4 bootstrap.py && bin/buildout
Wednesday, October 28, 2009
When to test?
• When writing production code. Testing reduces defects as measured by reported bugs.
• Prototyping doesn't always require tests, as there is indeed some overhead to writing and maintaining tests.
• Safe as long as you remember to throw away your prototypes.
Wednesday, October 28, 2009
Testing Results
• improves code quality, reduced defects
• provides more maintainable developer documentation
• creates more loosely coupled code which can offer maintenance and re-use improvements
• “Integrating Unit Testing into a Software Development Team's Process”: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.19.5798&rep=rep1&type=pdf
Wednesday, October 28, 2009
Why Test
• produce more robust code faster
• tests passing is proof that you delivered what you were contracted to produce
• acceptance tests can map directly to your software requirements
• business measurement help prevent scope creep
• finding bugs early makes them cheaper to fix
• refactor with confidence
Wednesday, October 28, 2009
Test First!
• writing tests first forces you to get clear about requirements up front
• facilitates a conversation with your product owners
• generally results in more modular and reusable code
• manual testing is slow and error-prone! The longer you put off writing tests the less efficient you are.
Wednesday, October 28, 2009
Test Levels in the Testing Pyramid
functional, system
integration
unit
Wednesday, October 28, 2009
Test-Code Cycle
Wednesday, October 28, 2009
More tests is not always better
• Tests are software too and must be maintained.
• Too many tests makes running your entire test suite slow.
• Remember the pyramid. Just like the food pyramid you want to focus on having lots of tests from the lower levels which run quickly, and fewer from the top of the pyramid.
• Some people are opportunistic testers. They will write a minimal test for the main path of their code and then only write additional tests if regressions are found.
Wednesday, October 28, 2009
Unit Tests
• Answer the question: Is the code doing the right thing? Written from developer’s perspective
• Tests implementation of a sliver of behaviour
• Fast
• Doesn't rely on any external resources (so use mocks or fakes for external resources)
• Have lots of these, for every important variation of behaviour
• Python’s unittest module - http://docs.python.org/library/unittest.html
Wednesday, October 28, 2009
When to Unit Test
• as often as possible when able to test components in isolation.
• if you depend on a few outside components, think about using a Mock object to represent them
• if you rely on having a full Plone site configured, then you are a better off with an integration test
Wednesday, October 28, 2009
Unit Testing Example
• setUp called before each test method
• tearDown called after each test method
• methods beginning with test* are automatically added
class TestFizzBuzz(unittest.TestCase): """Unit test for the FizzBuzzWriter """
def setUp(self): self.fizzbuzz = FizzBuzzWriter()
def test_numbers_divisable_by_three(self): self.failUnlessEqual(self.fizzbuzz.write_number(3), 'FIZZ')
Wednesday, October 28, 2009
Unit Testing Boilerplate
• Tests are defined in a package named tests. Typically this should be created a subdirectory tests in your package. Don’t forget an empty __init__.py!
• Create a seperate test file for each component you are testing, eg. test_fizzbuzz.py
• Only files starting with test_ are picked up by Zope’s test runner
• Required to find all tests and make them available to the test runner
def test_suite(): return unittest.defaultTestLoader.loadTestsFromName(__name__)
Wednesday, October 28, 2009
Test methods
• Base class unittest.TestCase defines several assertion methods to use:
• failIf(expr[, msg])
• failUnless(expr[, msg])
• failUnlessEqual(first, second[, msg])
• failUnlessRaises(exception, callable, ...)
• there’s also equivalent methods that start with assert...()
• and more, see docs
Wednesday, October 28, 2009
FizzBuzz implementation
class FizzBuzzWriter(object): def write_number(self, number): if number % 3 == 0 and number % 5 == 0: return 'FIZZ BUZZ' if number % 3 == 0: return 'FIZZ' if number % 5 == 0: return 'BUZZ' return str(number)
def write_numbers(self, count): return ( self.write_number(number) for number in range(1, count + 1) )
Wednesday, October 28, 2009
Using Zope's testrunner
• TIP: run only the test you need to run:
$ bin/instance test -s your.module -t test_something
• By using -s you can specify the module to look for tests.
• If you want to run only one particular test method then use -t to specify which one.
• Also to run only unit tests (as opposed to integration and functional tests) use the -u option:
$ bin/instance test -s your.module -u
Wednesday, October 28, 2009
Running the Tests
$ bin/instance test -s pttt.fizzbuzzRunning tests at level 1Running unit tests: Running:.....
Wednesday, October 28, 2009
Sample Applications of Unit Testing in Plone
• View Classes: instantiate your view class, using mock objects as necessary for request and context
• Test validators for all corner cases, eg. phone number validator
• Simple Event handlers
Wednesday, October 28, 2009
Additional Unit Testing Resources
• official unittest docs: http://docs.python.org/library/unittest.html
• http://plone.org/documentation/tutorial/best-practices/unit-testing
• http://www.slideshare.net/Quintagroup/intro-to-testing-in-zope-plone-presentation
• http://trizpug.org/Members/tbryan/unittest_talk
Wednesday, October 28, 2009
Integration Tests
• Plone is a complex system with many moving parts
• integration tests will be necessary to confirm that components are properly registered and work together
• When to use
• when testing integration of various components and the flow in and out of your code. eg. cataloging, relationships, object creation, security, workflow
• Verifying that view was registered correctly and works in context of the Plone site
Wednesday, October 28, 2009
Integration Testing with PloneTestCase
• Builds on top of ZopeTestCase and PortalTestCase.
• All tests happen in a sandbox. Zope cleans up after you and it doesn’t affect any of your on-disk ZODBs.
• They are by their very nature quite slow. So write them as unit tests if possible. Essentially a full Plone site is created to run your tests in.
Wednesday, October 28, 2009
Plone Integration Test Example
• Because users love to run FizzBuzz so often they want to generate a Page of Fizz Buzz results.
• Let’s introduce a new view (using five.grok) which when given an appropriate number and a document id will generate a page with the results.
Wednesday, October 28, 2009
Writing an Integration Test
class FizzBuzzIntegration(PloneTestCase): layer = ptc_layer def test_fizzbuzz_contents(self): view = self.folder.restrictedTraverse('@@fizzbuzz') view.generate_document(5) # document with id fizzbuzz exists self.failUnless('fizzbuzz' in self.folder)
text = self.folder.fizzbuzz.getText()
# verify contents are correct (Document converted to HTML for us) self.failUnlessEqual(text, '<p>1<br />2<br />FIZZ<br />4<br />BUZZ</p>') def test_fizzbuzz_anonymous_cant_generate(self): self.logout() view = self.folder.restrictedTraverse('@@fizzbuzz') self.assertRaises(Unauthorized, view.generate_document, 5)
Wednesday, October 28, 2009
Quick review of PloneTestCase’s API
• self.loginAsPortalOwner()
• .login(user)
• .logout()
• .setRoles([roles])
• .portal
• .folder
• PloneTestCase.default_user
Wednesday, October 28, 2009
collective.testcaselayer and PloneTestCase
• collective.testcaselayer, developed by Ross Patterson, provides layer base classes for Zope, CMF and Plone development.
• Layers allow expensive setup of fixtures, such as a Plone site, to be shared by many tests.
• Simplifies repetitive Plone testing boilerplate code while providing a clean way to extend.
• Also of interest is the ability to create ZODB sandboxes for each layer so that sibling layers are isolated from each other.
Wednesday, October 28, 2009
Installing collective.testcaselayer using testing extras
• When to use: when adding dependencies that are only required for testing
• examples: collective.testcaselayer, mocker library, zc.testbrowser
• dependencies are registered as setuptools extras so they can be optionally included
• you should have a separate development buildout configuration that specifies the extras
Wednesday, October 28, 2009
Adding collective.testcaselayer
• Use the extra_requires functionality of setuptools so that production usage won’t require collective.testcaselayer
•from setuptools import setup, find_packages...tests_require = ['collective.testcaselayer']...setup(name='pttt.fizzbuzz', ... install_requires=[ 'setuptools', # -*- Extra requirements: -*- ], tests_require=tests_require, extras_require={'tests': tests_require}, ... entry_points="""
Wednesday, October 28, 2009
Enabling tests extras in your buildout
• Find the eggs section in your buildout part and add:
...eggs += pttt.fizzbuzz [tests]...
Wednesday, October 28, 2009
Autoinclude your ZCML
• Plone 3.3 includes support via z3c.autoinclude to automatically load ZCML in dependent packages using a setuptools entry point.
• For Plone 3.2 you have to add z3c.autoinclude to your buildout and something like this somewhere in your buildout ZCML:
<includePlugins package="plone" file="configure.zcml" />
• Here’s how we’d register ours in our packages’ setup.py:
entry_points="""[z3c.autoinclude.plugin]target = plone“"",
Wednesday, October 28, 2009
Writing an Integration Test
class FizzBuzzIntegration(PloneTestCase): layer = ptc_layer def test_fizzbuzz_contents(self): view = self.folder.restrictedTraverse('@@fizzbuzz') view.generate_document(5) # document with id fizzbuzz exists self.failUnless('fizzbuzz' in self.folder)
text = self.folder.fizzbuzz.getText()
# verify contents are correct (Document converted to HTML for us) self.failUnlessEqual(text, '<p>1<br />2<br />FIZZ<br />4<br />BUZZ</p>') def test_fizzbuzz_anonymous_cant_generate(self): self.logout() view = self.folder.restrictedTraverse('@@fizzbuzz') self.assertRaises(Unauthorized, view.generate_document, 5)
Wednesday, October 28, 2009
Generating the Fizzbuzz document
class FizzBuzz(grok.View): grok.context(ATFolder) grok.require('zope2.View')
def generate_document(self, count): writer = FizzBuzzWriter() text = '\n'.join(writer.write_numbers(count)) self.context.invokeFactory('Document', id='fizzbuzz', title="Fizzbuzz results", text=text)
Wednesday, October 28, 2009
Running the test
$ bin/instance test -s pttt.fizzbuzz -t test_fizzbuzz_contents -t test_fizzbuzz_anonymous_cant_generate
Wednesday, October 28, 2009
Make it faster with roadrunner
• Roadrunner is a tool that preloads Plone to make running your integration tests much much faster. http://pypi.python.org/pypi/roadrunner
• Due to interactions with z3c.autoinclude it will currently only reload your tests without repeating the Plone site setup.
• Its easy to use too. Just add a roadrunner part to your buildout.cfg like so:
• Roadrunner accepts the same command line arguments as Zope testrunner. So we can run our tests like this:
$ bin/roadrunner -s pttt.fizzbuzz
[roadrunner]recipe = roadrunner:plonepackages-under-test = pttt.*
Wednesday, October 28, 2009
Plone Integration Testing Resources
• Programmatically Manipulate Plone: http://plonemanual.twinapex.fi/content/index.html
• PloneTestCase Tutorial - PloneConf 2004: http://www.zope.org/Members/shh/Tutorial
• http://plone.org/documentation/tutorial/testing/integration-doctests-using-plonetestcase
• PloneTestCase: http://plone.org/products/plonetestcase
Wednesday, October 28, 2009
Doctests
• tests that look like interactive Python sessions
• When to use
• API Documentation, verify examples work
• its a matter of taste, some people prefer unittests especially when there are a lot of variations
• they are especially good for functional tests where the prose explains the acceptance test and the code verifies that it is working
Wednesday, October 28, 2009
Doctest boilerplate
import unittestimport doctest
from Testing import ZopeTestCase as ztc
from example.tests.tests import base
def test_suite(): """This sets up a test suite that actually runs the tests in the class above """ return unittest.TestSuite([
ztc.ZopeDocFileSuite( 'tests/functional.txt', package='example.tests', test_class=base.ExampleFunctionalTestCase, optionflags=doctest.REPORT_ONLY_FIRST_FAILURE | doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS), ])
Wednesday, October 28, 2009
A Simple Doctest
First we'll import and instantiate our FizzBuzzWriter class
>>> from pttt.fizzbuzz.fizzbuzz import FizzBuzzWriter>>> writer = FizzBuzzWriter()
Let's write out single numbers
>>> writer.write_number(1)'1'>>> writer.write_number(3)'FIZZ'>>> writer.write_number(15)'FIZZ BUZZ'
Finally let's write a sequence of numbers
>>> list(writer.write_numbers(15))['1', '2', 'FIZZ', '4', 'BUZZ', 'FIZZ', '7', '8', 'FIZZ', 'BUZZ', '11', 'FIZZ', '13', '14', 'FIZZ BUZZ']
Wednesday, October 28, 2009
Functional and System Tests
• End to end tests
• Typically black-box tests that act as an end-user would and are written from end user’s point of view. eg. click the button labeled “Login” vs. call the login method
• inherently brittle as the UI often changes
• Use zope.testbrowser, twill, Selenium
• typically written as doctests but don’t need to be
Wednesday, October 28, 2009
Functional Tests
• When to use:
• blackbox testing
• automated acceptance testing
• Smoke or face-saving tests - exercise most important paths of business value
• When not to use:
• testing every corner case. use unit tests for that
Wednesday, October 28, 2009
About zope.testbrowser
• testbrowser implements a Python scriptable browser that simulates accessing your Plone site.
• Full docs for zope.testbrowser: http://pypi.python.org/pypi/zope.testbrowser
• Note that using the testbrowser in Plone we should use the Five version which wraps it for use in a Zope 2 environment:
from Products.Five.testbrowser import Browser
• Not Javascript aware. If you need to test DHTML, Javascript, Ajax et al then you will need to use something like Selenium.
Wednesday, October 28, 2009
Using zope.testbrowser
• Don’t hammer Plone.org ;)
from zope.testbrowser.browser import Browser# this version of testbrowser works with external resources# for testing your own apps using internal Browser class:# from Products.Five.testbrowser import Browser
browser = Browser('http://plone.org')browser.getControl(name='SearchableText').value = 'Testing'browser.getForm(id='search').submit()assert browser.isHtmlassert 'Untested Code is Broken Code' in browser.contents
Wednesday, October 28, 2009
Plone Functional testbrowser API basics
• Open your test Plone site: browser.open('http://nohost/plone/')
• Click a link: browser.getLink(link_text).click()
• Click the first link when multiple have the same name: browser.getLink(link_text, index=0).click()
• Get URL: browser.url
• Reload: browser.reload()
• Change form value: browser.getControl(input_name).value = ‘Whatever’
• Submit form: browser.getForm(id)
Wednesday, October 28, 2009
Two ways to login
• One is to use Plone’s login mechanism, arguably more black-box since it should simulate what your users will actually do.
• Alternatively you could add an auth header:
>>> browser.addHeader('Authorization', 'Basic root:secret')
>>> browser.open(portal_url)>>> browser.getControl(name='__ac_name').value = portal_owner>>> browser.getControl(name='__ac_password').value = default_password>>> browser.getControl(name='submit').click()
Wednesday, October 28, 2009
Functional Testing Example
• Let’s take a look at an existing functional testing example from Plone’s source code from plone/app/workflow/tests/sharingpage.txt:
• A couple things about this
• browser.handleErrors = False means you’ll get proper tracebacks instead of cryptic 500 Errors
• demonstrates pure black box testing, no assumptions of internal state
Wednesday, October 28, 2009
Functional Testing Resources
• http://plone.org/documentation/tutorial/testing/functional-tests
• Face Saving Tests: http://gojko.net/2007/09/25/effective-user-interface-testing
Wednesday, October 28, 2009
System Tests using twill
• You can write system tests using testbrowser as well
• Twill provides a simple domain specific language (DSL) for performing web tests. It also provides a Python API.
• When to use
• when doing whole system testing that includes caching proxy, multiple web servers, backend and database
• testing live systems
• deployment testing
Wednesday, October 28, 2009
Installing twill in your buildout
• twill is on pypi, simply use zc.recipe.egg to install its scripts:
• Also make sure to add the part to your [buildout] parts section.
[twill]recipe = zc.recipe.eggeggs = twill
Wednesday, October 28, 2009
Simple Twill Script
go http://plone.orgformvalue searchform SearchableText testingsubmit
# assert that we found the following textfind 'Untested Code is Broken Code'
# assert that we didn't find this text:notfind 'Testing sucks'
Wednesday, October 28, 2009
Running twill interactively
$ bin/twill-sh
-= Welcome to twill! =-
current page: *empty page* >> go http://plone.org==> at http://plone.orgcurrent page: http://plone.org>> formvalue searchform SearchableText testingcurrent page: http://plone.org>> submitcurrent page: http://plone.org/search?SearchableText=testing>> current page: http://plone.org/search?SearchableText=testing>> # assert that we found the following textcurrent page: http://plone.org/search?SearchableText=testing>> find 'Untested Code is Broken Code'current page: http://plone.org/search?SearchableText=testing>> current page: http://plone.org/search?SearchableText=testing>> # assert that we didn't find this text:current page: http://plone.org/search?SearchableText=testing>> notfind 'Testing sucks'current page: http://plone.org/search?SearchableText=testing
Wednesday, October 28, 2009
Writing a System Test
• Verify your SSL configuration:
# TEST: join form redirects to httpsgo http://mysite.comfollow "Join"
# test that the resulting URL is https://url "^https:"url "/join_form$"find "Registration Form"
# fill in registration form and registerfv 2 fullname "Test User"fv 2 email "[email protected]"submitfind "You have been registered."fv 2 form.submitted 1submitfind "You are now logged in" # home page is httpurl "^http://"
Wednesday, October 28, 2009
Mock and Fake objects
• When to use:
• unittest code that has external dependencies such as databases or network connections
• depends on external components
Wednesday, October 28, 2009
Basic Mocks
• Let’s say you have a test method that expects an object to be provided which provides an absolute_url method in order to return a calculated URL:
• Without using an Integration test, how could you do this? Simply create a simple Python class that implements only the methods you need.
• This can get tiresome if you have to create many of these mocks, or if you want to make some assertions about what is called on those mocks with what arguments.
Wednesday, October 28, 2009
Basic Mocking Example
def calc_url(context): return context.absolute_url() + "/calculated_url" class MockDocument: def absolute_url(self): return "http://plone/some_document"
def test_calc_url(self): mock = MockDocument() self.failUnlessEqual(calc_url(mock), "http://plone/some_document/calculated_url")
Wednesday, October 28, 2009
Mocker
• Python library for creating Mock objects and managing expectations we have for those Mock objects.
• Other more advanced features as well such as proxies, patching and proxy replacement.
• Unittest integration
• Martin Aspeli took this further and created plone.mocktestcase which provides some helper methods that are specific to Zope and Plone development.
http://pypi.python.org/pypi/plone.mocktestcase
Wednesday, October 28, 2009
A Mocking Example
• Writing an interface to an external network service
• Using the Component Architecture to create an abstract interface for a remote network service and using MockTestCase’s mock_utility to make this available to application code under test:
In the network_service = self.mocker.mock()self.mock_utility(self.video_provider, INetworkService)
Wednesday, October 28, 2009
Mocking Resources
• http://www.martinaspeli.net/articles/mock-testing-with-mocker-and-plone.mocktestcase
Wednesday, October 28, 2009
Coverage
• answers the question: are my tests covering all the code in my application?
• for production code I aim for 100%. Why? anything less than 100% can hide defects.
• 100% code coverage is considered minimum standard *
• does NOT guarantee the code has no bugs
• * - http://www.plope.com/Members/chrism/a_hundred_percent
Wednesday, October 28, 2009
Coverage won’t catch this• This pseudo-code example would give 100% code coverage. What if I
passed 0 to myfunc?
def test_myfunc(): myfunc(100) def myfunc(n): return 100 / n
Wednesday, October 28, 2009
Generating coverage report
• Zope’s testrunner comes with a builtin coverage report tool, invoke like this:
$ bin/instance test -s pttt.fizzbuzz --coverage=./coverage
• When calculating coverage you should know that code runs much slower. So grab a coffee, tea or water and catch up on your xkcd.
• The results are in:
Ran 5 tests with 0 failures and 0 errors in 0.005 seconds.Tearing down left over layers: Tear down zope.testing.testrunner.layer.UnitTests in 0.000 seconds.lines cov% module (path) 16 100% pttt.fizzbuzz.fizzbuzz (/Users/jbb/co/pttt/src/pttt.fizzbuzz/ pttt/fizzbuzz/fizzbuzz.py)
Wednesday, October 28, 2009
HTML Coverage report using z3c.coverage
• Produces nice looking reports that you can discuss with your team:
Wednesday, October 28, 2009
HTML Coverage report using z3c.coverage
• Add this part to buildout:
• Don’t forget to add z3c-coverage to [buildout] parts =
• Generate reports:
$ bin/coverage ./coverage ./coverage/reports
[z3c-coverage]recipe = zc.recipe.eggeggs = z3c.coverageextra-paths = ${zope2:location}/lib/python
Wednesday, October 28, 2009
Coverage Resources
• http://plone.org/documentation/tutorial/testing/code-coverage
Wednesday, October 28, 2009
Pdb-fu (debugging)
• Getting test failures from Zope Testrunner? Use -D to drop you into Pdb using post-mortem debugging
• Use pdb’s jump command to re-run certain lines of codes. Watch out for side-effects.
• Special variable __return__ is set. Use “continue” to go to the end of the method and then you can easily examine the return result.
• Breakpoint variables. Looping through 1000 items, set a breakpoint with an expression.
• Missing source code? This could help, add to your buildout.cfg: [buildout] unzip = true
Wednesday, October 28, 2009
General Testing Tips
• Write at the lowest level possible. This will ensure that your test will run as quickly as possible which becomes important when you have hundreds or thousands of tests.
• Take your time, write out what you’re trying to do in your commends first before writing actual code:
def test_whatever(self): # verify document was created # ensure that notification was sent
Wednesday, October 28, 2009
General Testing Tips
• Broken tests that lie around for a long time because they are minor issues that no one can be bothered to fix can mask larger bugs due to too much noise and not enough signal. 0 failing tests (and no cheating!)
• Hard to test something? Step back, try to break it up into easier to test pieces.
Wednesday, October 28, 2009
Some Further Resources
• Plone Testing tutorial: http://plone.org/documentation/tutorial/testing/
• plone.reload: http://pypi.python.org/pypi/plone.reload
• http://diveintopython.org/unit_testing/index.html
• example tests of all sorts, mostly still valid: http://svn.plone.org/svn/collective/examples/example.tests/trunk/
Wednesday, October 28, 2009
Open Spaces and Sprinting
• For all of those who want to have open discussions about Testing in Plone we’ll have an Open Space on Friday.
• Also I’ll be sprinting on bringing roadrunner up to spec with Plone 4, Dexterity and Zope 2.12. Interested? Drop me a line.
Wednesday, October 28, 2009
Thanks
• Your Feedback is appreciated: [email protected]
• Find me on twitter: @hexsprite
• Good luck and happy testing!
Wednesday, October 28, 2009
Thanks
• Photos used licensed under Creative Commons 2.0:
• http://www.flickr.com/photos/rupertuk/2102548824/
• http://www.flickr.com/photos/kacey/2626513873
Wednesday, October 28, 2009