Download - #LFMF: Tales of Test Automation Gone Wrong
#LFMF
Learn from My Fail:
Tales of Test Automa6on Gone Wrong
(Plus a Li*le Bit about How to Do It Right)
(and others who shall remain anonymous even if they don’t deserve it)!
Elisabeth Hendrickson Quality Tree So7ware, Inc.
@testobsessed
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Fail Pa;ern #0: “No 6me to automate” No automaFon means rapid growth in the
manual tesFng burden. Or untested so7ware.
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Velocity!
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Fail Pa;ern #1: Automa6on for the sake
of automa6on
If you really want a mess, use a count of automated tests as a measure of
progress.
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Image of prinFng office from hJp://www.shorpy.com/node/12652
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Fail Pa;ern #2: All automa6on is end-‐to-‐end through the GUI
End-‐to-‐end tests are slow and fragile. Some are good.
More are not beJer.
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Flooded basement image courtesy of carlpenergy Showerhead image courtesy of deusx Images from Flickr licensed through CreaFve Commons AJribuFon
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Fail Pa;ern #3: Dispropor6onate
Amount of Time Spent Fixing Broken Tests Just when you get one working again, another
breaks.
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Image courtesy of cdrussorusso From hJp://www.flickr.com/photos/23516192@N08/2631474033/ Licensed Under CreaFve Commons AJribuFon
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Fail Pa;ern #4: Fragile Tests
They pass, they fail, but there seems to be no
correlaFon between the test results and the actual so7ware behavior.
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc. Hey, the
build is red.!
Yeah, it’s the 2nd Tuesday with a full moon this
year.!
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Fail Pa;ern #5: Inter-‐dependent Tests
When one fails, they all fail together.
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Run #1!
Run #2!
Run #3!
Run #4!
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Fail Pa;ern #6: Useless Tests
The tests can never fail, so they yield no informaFon.
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc. …
assert True
… But rarely. More common: tests that
verify the software
does what the software does.!
Yes, I have actually seen this in tests.!
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Fail Pa;ern #7: Bad Tests
Obfuscated intent, massive duplicaFon, and Fght coupling between the interface and the
tests makes for an unmaintainable mess.
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Test Login
Input Text first_name fred
Input Text last_name flintstone
Input Text user_name fflint
Input Text password p@ssw0rd
Click Submit
Click Logout
Click Login
Input Text username fflint
Input Text password badpassword
Click Submit
Input Text ...
WAT?!?!
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Fail Pa;ern #8: Bad Code
Smelly test automaFon code that violates SOLID
principles is as much of a problem as smelly producFon code.
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Fail Pa;ern #9: Only specialists
automate
They might be good at automaFon, but they’re fighFng an uphill baJle.
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Fail Pa;ern #10: Forcing devout non-‐
coders to write automa6on
If they lack either interest or apFtude to automate, the
results will be bad.
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Fail Pa;ern #11: Tests That Don’t Get Run
Someone writes them, no one runs them. So when they suffer bitrot, no one can hear them scream.
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Photo by crabchick hJp://www.flickr.com/photos/crabchick/2627370039/ Licensed under CreaFve Commons AJribuFon
Unrun !tests!become !creaky,!dusty,!sad.!
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Fail Pa;ern #12: Failing Tests Ignored
They run.
They fail.
No one cares.
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Image of the 1906 San Francisco Earthquake a7ermath from hJp://www.shorpy.com/node/9866
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc. #LFMF
Automate important expectations!Treat test code with the same level of care as production code!
Run all the automated tests in the CI build (no manual kickoff)!Fix failing tests immediately!Foster a “whole team tests” culture!
Copyright ©
2012 Elisabeth Hen
drickson
, Quality Tree So7
ware, Inc.
Many thanks to …!@hackyourtv!Juha Rantanen!Jyrki Pulliainen!Jarkko Kailanto!Teema Frisk!
