what you are doing wrong with automated testing
TRANSCRIPT
What You are Doing Wrong With Automated Testing
Anti-Patterns in Automated Testingby Martin Folkoff and Shawn Faunce
US
Martin Folkoff US CIS Shawn Faunce
Anti-Patterns• “An anti-pattern is just like a pattern, except
that instead of a solution, it gives something that looks superficially like a solution but isn't one.” - Andrew Koenig
Ice CreamCone Recorders
Low Priority
BugsOwnershipMonolith
Coverage=
Quality
Ice CreamCone
99%UI Tests
70%Unit & Integration Tests
20%API Tests
10%UI Tests
ExploratoryTesting
Looks like a good solution• Easy to create• Mimics user interactions• You see something happening
10%Unit & Integration Tests
20%API Tests
70%UI Tests
ExploratoryTesting
Anti-Pattern: When you automated testing strategy relies mostly or exclusively tests at the user interface (UI) layer
But it introduces problems• Maintenance nightmare• No fast feedback• Complex tests, complex debugging
Looks like a good solution• Tools make it quick and easy• No programming required• Leverage manual testers
Anti-Pattern: When you use a screen recorder to build your automated tests instead of one of the automated testing frameworks that relies on programming.
But it introduces problems• Extremely fragile• Dependent on having a UI• No mock objects
Recorders
Looks like a good solution• There are bigger fish to fry• Only happens occasionally• Not worth our attention
Anti-Pattern: When you ignore intermittent low priority bugs that occasionally affects the automated test, but a user might easily overcome or not notice.
But it introduces problems• Lose confidence in your tests• Flaky tests• Build technical debt
Low Priority
Bugs
Looks like a good solution• Avoid managing dependencies• Always run your full test suite• Just keep appending new tests
Anti-Pattern: When you organize your test suite so that all of the tests are designed to run together as one test suite.
But it introduces problems• Slow feedback loop• Longer build times• Encourages sloppy engineering
Monolith
Looks like a good solution• Testers test, and this is a test• Frees up developers• Clear lines of ownership
Anti-Pattern: When you have testers write automated tests, you are transferring the ownership of quality
But it introduces problems• Need programming skills• Quality is not someone else’s job• Creates ownership issues
Ownership
Looks like a good solution• Can’t get any better than 100%• Everything appears to be tested• The dashboard is all green
Anti-Pattern: You achieve 100% test coverage on your code, ergo you have 100% Quality.
But it introduces problems• Are the tests good tests?• Satisfy the coverage tool • Can lead to assertion free testing
Coverage=
Quality
“I expect a high level of coverage. Sometimes managers require one. There’s a subtle difference.”
“People optimize their performance according to how they’re measured”
“If a part of your test suite is weak in a way that coverage can detect, it’s likely also weak in a way coverage can’t detect.”
-- Brian MarickHow to Misuse Code Coveragehttp://www.exampler.com/testing-com/writings/coverage.pdf