software testing with python
TRANSCRIPT
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 1/35
Software Testing with Python
Magnus LyckåThinkware AB
www.thinkware.se
EuroPython Conference 2004Chalmers, Göteborg, Sweden
© 2004, Magnus Lyckå
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 2/35
In the next 30 minutes you should...
● Learn about different aspects of software testingsuch as unit tests and acceptance tests.
●
See how the standard modules for unit tests areused, learn about some alternatives and have agrasp of their respective pros and cons.
● Know about some options for acceptance tests.
● Know about some standard Python modules andthird party tools that are helpful in softwaretesting and related activities.
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 3/35
What is software testing?
● “Testing means verifying that your code isrunning correctly by exercising the code under known conditions and checking that the results
are as expected.” – Alex Martelli, Python in a Nutshell
● If we are serious with our requirements on asoftware system, we should make sure that we
verify them, and testing is the most common kindof verification for software.
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 4/35
Software testing in a context
Software Quality Assurance
Software Verification
Software Testing
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 5/35
Automating tests
● Writing automated tests is usually more work than testing things manually once, but they makeit much easier to...
– Work in a repeatable and predicable way
– Run tests more often
– Run tests unattended
– Find regression bugs
– Test more operating systems
– Manage the test load as the system grows
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 6/35
Example of requirements and
tests on different levels
System Specification Acceptance Test
Detailed Design
API Specification
Unit Test
Integration Test
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 7/35
Requirements and tests on different
levels in Extreme Programming (XP)
System specified through customer tests
Design by writing
API validated through
programmer tests
continuous integration
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 8/35
Consequences of the XP approach
● Requirements are written in a verifiable way.
● No discrepancies between requirements and tests.
●
We get unambiguous and repeatable ways of verifying requirements.
● Continuous picture of project progress
But...● Customers must be able to understand the tests.
● We need test automation frameworks.
●
Automated tests → more software to maintain.
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 9/35
Software Testing with Python
Unit testing
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 10/35
Unit tests in Python
● Unit tests, or programmer tests, are tests the programmer writes to make sure that the codedoes what the programmer intended it to do.
● Unit tests can also help document how a piece of code (e.g. a class) is supposed to be used.
● Python has two standard modules for unit testing,
unittest and doctest.
● There are several third party modules that aim tocomplement or replace unittest.
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 11/35
The unittest module
● The Python unittest module is modeled after
the unit test modules developed within the XPcommunity by Kent Beck and Erich Gamma.
● It's intended for a test-first approach, where testsare written before the actual code that it tests.
● Always writing tests first is a bit like always
eating your meat before you eat your dessert...
● When your tests pass, you're done. (XPerscombine this with refactoring.)
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 12/35
Concepts in unittest
● A test fixture consists of the actions needed tosetup for a test, and clean up afterwards.
●
Atest case
is the smallest unit of testing. It mightfor instance call a function and check the results.
● A test suite is a collection of test cases and/or testsuites that should be executed together.
● A test runner runs test suites (or individual testcases), collects the results and presents it as textor graphically to the user.
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 13/35
The TestCase class
● Test cases sharing a common fixture will beimplemented as methods (whose names start withtest) in a sub class of unittest.TestCase.
● The test runner will create one instance object per test case, and run SetUp, followed by the test casemethod and finally tearDown.
● Checks are made with the methods fail, failIf,assert_/failUnless, assertEqual/failUnlessEqual,assertNotEqual/failIfEqual andassertRaises/failUnlessRaises.
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 14/35
The test runner
● In many cases, running tests is as simple asexecuting unitest.main() in the file
containing the test cases.
● Testing progress will be reported, and the testrunner makes a distinction between FAILUREand ERROR.
– FAILURE means that the check didn't produce theexpected result.
– ERROR means that something else in the test wentwrong. (I.e. You got an exception.)
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 15/35
Unittest template
import unittest import MyModule
class MyModuleTest(unittest.TestCase): def testOneCase(self): pass
if __name__ == '__main__':unittest.main()
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 16/35
Unittest examples
● divtest.py
– Trivial example to demonstrate features of unittest.
●
container_ut.py – Real world example from SystemSpecifyer (see
http://sourceforge.net/projects/systemspecifyer)
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 17/35
The doctest module
● The Python doctest module was written by
Tim Peters to check that coding examples in doc-strings were correct, by testing them.
● It has also found use as a more general unittesting module, and from Python 2.3, this has been made more convenient.
● Using doctest helps you combine tests anddocumentation in a more readable way thanunittest does.
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 18/35
Doctest example
def divide(a,b): """
Return a divided with b.
Divide will return the result of an integer division>>> divide(6,3)
2
Division by zero is handled as expected...>>> divide(1,0)Traceback (most recent call last):
...ZeroDivisionError: integer division or modulo by zero
"""
result = a / b return result
if __name__ == '__main__': import doctest, sys
doctest.testmod(sys.modules[__name__])
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 19/35
Unittest vs doctest
● Unittest mantra: Test a little, code a little
– The logical choice if you use extreme programmingor test-driven development of some other kind.
– Maybe better for complex tests?
● Doctest mantra: Code a little, test a little
– The natural companion if you prefer to experiment
interactively with your classes and functions in thePython interactive environment.
– Probably better for adding tests to already writtencode.
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 20/35
Unittest vs doctest on the web
● Charming Python: Testing frameworks in Python
– http://www-106.ibm.com/developerworks/linux/library/l-cptest.html
● Literate Testing: Automated Testing with doctest
– http://www.python.org/pycon/dc2004/papers/4/
● DocTest at WikiWiki
– http://c2.com/cgi/wiki?DocTest
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 21/35
Other contenders...
● Sancho – a unit testing framework from MEMSExchange. Adds coverage analysis etc.
– http://www.mems-exchange.org/software/sancho/
● Peckcheck by Darius Bacon.
– http://www.accesscom.com/~darius/software/clickcheck.html
● The “second standard library” std.
– Here, 11:00 today...
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 22/35
Unittest complements
● Pester – the Python version of Jester.
– It finds code that is not covered by tests, makes somechange to your code, runs your tests, and if the tests
pass it displays a message saying what it changed. – http://jester.sourceforge.net/
● Test coverage support
–
E.g. http://www.garethrees.org/2001/12/04/python-coverage/● Mock objects: Here, 9:30
● A better test runner: Here, 10:00
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 23/35
Software Testing with Python
Integration testing
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 24/35
Python Integration Test
● No static linking. We must run code to testinterfaces! (On the other hand, compilers andlinkers fail to see lots of integration problems...)
● Since building/linking isn't an issue in a purePython project, we can use either unit test tools or acceptance test tools for integration tests.
● Python is good at gluing things together, and thushelpful in all sorts of integration work. See e.g.http://www.thinkware.se/cgi-bin/thinki.cgi/UsingPythonWithOtherLanguages
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 25/35
Software Testing with Python
Acceptance testing
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 26/35
Test Frameworks using Python
● QMTest – Uses Python for test expressions.
– http://www.codesourcery.com/qmtest/
●
PyFIT – Python clone of Ward Cunningham'sFramework For Integrated Testing (FIT).
– http://www.xprogramming.com/software.htm
● Software Testing Automation Framework (STAF)Big framework from IBM with Python API.
– http://staf.sourceforge.net/
● TextTest – See more here, today 11:30!
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 27/35
Software Testing with Python
Not quite software testing...
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 28/35
The usual suspects...
● Python is excellent for analysis and manipulationof data. A great tool for test related work.
● The re library is useful but use special tools
when available, such as for parsing XML files.
● For dealing with files you might use the modulesgzip, zipfile, codecs, filecmp,
struct, sgmllib, xml.* etc● Python interfaces well with internet services,
database, and all sorts of external programs.
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 29/35
difflib
● More or less like the unix diff utility, but as a
Python module. (New in version 2.1.)
● Useful when we want a more detailed responsethan FAILED from a test – particularly if we aretrying to spot small changes in big amounts of data.
● Great for spotting differences in configurations,for instance database schemas.
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 30/35
Difflib example
>>> import difflib; d=difflib.Differ()>>> diff =d.compare(['Hello World', "This is the same.",
"Time flies like an arrow. Isn't that great?"],['Hello World!', "This is the same.","Fruit flies like a banana. Isn't that great?"])
>>> print "\n".join(diff)- Hello World + Hello World!? +
This is the same.- Time flies like an arrow. Isn't that great?
? ^ ^^ - ^^^^
+ Fruit flies like a banana. Isn't that great?? ^^^ ^^ +++ ^^
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 31/35
AT&T Graphviz
● Tools to generate graphs from C-like text files.
– Dot – for directed graphs.
– Neato – for undirected graphs
– Fairly clever algorithms for decent looking layout
– Generates graphs in many file formats
– Suitable for automatic generation of graphs
– Some control over placement is possible
● Great for dependency analysis!
● Not Python, but good anyway... :)
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 32/35
Remember this?
System Specification Acceptance Test
Detailed Design
API Specification
Unit Test
Integration Test
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 33/35
AT&T Graphviz example 1
digraph G {label = "V-model";
sysspec[label="System Specification"];
systest [label="Acceptance Test"];apispec [label="API Specification"];apitest [label="Integration Test"];unitspec [label="Detailed Design"];unittest [label="Unit Test"];
sysspec -> apispec -> unitspec;unittest -> apitest -> systest;sysspec -> systest;apispec -> apitest;unitspec -> unittest;
}
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 34/35
AT&T Graphviz example 2
digraph G {label = "V-model";node [shape=box, style=filled,
color="#CCCCFF"];sysspec [label=
"System Specification"];apispec [label="API Specification"];unitspec [label="Detailed Design"];node [shape=box, style=filled,
color="#FFFF99"];systest [label="Acceptance Test"];
apitest [label="Integration Test"];unittest [label="Unit Test"];
{rank = same; sysspec; systest};{rank = same; apispec; apitest};{rank = same; unitspec; unittest};
...
7/27/2019 Software Testing With Python
http://slidepdf.com/reader/full/software-testing-with-python 35/35
Useful Books
● Python in a Nutshell, by Alex Martelli
● Text Processing in Python, by David Mertz
●
Software Test Automation, by Fewster & Graham● Just Enough Software Test Automation, by
Daniel J. Mosley & Bruce A. Posey
●
Testing Extreme Programming, by Lisa Crispinand Tip House
● Test-Driven Development By Example, by KentBeck