psychology of testing -...
TRANSCRIPT
![Page 1: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/1.jpg)
Psychology of Testing(or why our intuition of testing is wrong)
-- Miško Hevery
![Page 2: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/2.jpg)
Testing is not like frosting
![Page 3: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/3.jpg)
Development Model
TestEngineers
SoftwareEngineers
Automated Tests
QA
Develop
Manual TestsExploratory Tests
Testing magicTools
Testing magicDesign
![Page 4: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/4.jpg)
Micro Development Model
Develop
Test
Check-In
automate
![Page 5: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/5.jpg)
Excuses
Common Misconception
Valid Excuse
No Tests
Legacy code-base
It’s sl
ower
It doesn’t catch bugs It’s boring
Testing is for QA
I write UI
Too many interfaces
“Dirties Design”
Don’tknowhow
Hard to change
![Page 6: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/6.jpg)
How do you write HARD TO TEST
code?
![Page 7: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/7.jpg)
Test Driver Class Under Test
Stimulus
Asserts
![Page 8: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/8.jpg)
Test Driver Class Under Test
![Page 9: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/9.jpg)
Test Driver
Class Under Test
Other Class
Other Class
Other Class
Other Class
Other Class
Other Class
CPU Intensive
Destructive operation
Other Servers
FileSystem
Object Instantiated
Object Passed In
Global Object
Object Lifetime and Calling
![Page 10: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/10.jpg)
Test Driver
Class Under Test
Other Class
Other Class
Other Class
CPU Intensive
Destructive operation
Other Servers
FileSystem
Other Class
Other Class
Other Class
Friendly
Friendly
Friendly
Seam
Object Instantiated
Object Passed In
Global Object
Object Lifetime and Calling
![Page 11: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/11.jpg)
BusinessLogic
Object Graph Construction & Lookup
![Page 12: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/12.jpg)
Test Driver
Class Under Test
Friendly
Friendly
Friendly
Seam
![Page 13: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/13.jpg)
Global State aka Singletons
• API that lies about what it needs
• Spooky action at a distance
![Page 14: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/14.jpg)
Deceptive APItestCharge() { CreditCard cc; cc = new CreditCard(“1234567890121234”); cc.charge(100);}
• At the end of the month I got my Statement!
• I was out $100!
• Spooky action at a distance!
• It never passed in isolation
![Page 15: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/15.jpg)
Deceptive APItestCharge() { CreditCard cc; cc = new CreditCard(“1234567890121234”); cc.charge(100);}
java.lang.NullPointerExceptionat talk3.CreditCard.charge( CreditCard.java:48)
![Page 16: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/16.jpg)
Deceptive APItestCharge() { CreditCardProcessor.init(...); CreditCard cc; cc = new CreditCard(“1234567890121234”); cc.charge(100);}
java.lang.NullPointerExceptionat talk3.CreditCartProcessor.init( CreditCardProcessor.java:146)
![Page 17: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/17.jpg)
Deceptive APItestCharge() { OfflineQueue.start(); CreditCardProcessor.init(...); CreditCard cc; cc = new CreditCard(“1234567890121234”); cc.charge(100);}
java.lang.NullPointerExceptionat talk3.OfflineQueue.start( OfflineQueue.java:16)
![Page 18: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/18.jpg)
Deceptive APItestCharge() { Database.connect(...); OfflineQueue.start(); CreditCardProcessor.init(...); CreditCard cc; cc = new CreditCard(“1234567890121234”); cc.charge(100);}
• CreditCard API lies
• It pretends to not need the CreditCardProcessor even thought in reality it does.
![Page 19: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/19.jpg)
Better APItestCharge() { CreditCard cc; cc = new CreditCard(“12..34”, ccProc); cc.charge(100);}
ccProc = new CCProcessor(queue);queue = new OfflineQueue(db);db = new Database(...);
• Dependency injection enforces the order of initialization at compile time.
![Page 20: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/20.jpg)
Bugs
![Page 21: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/21.jpg)
![Page 22: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/22.jpg)
![Page 23: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/23.jpg)
![Page 24: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/24.jpg)
Bugs
![Page 25: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/25.jpg)
Cost of Fixing the BugProbability of Finding
a Bug
Difficulty of Finding the
Bug
Difficulty of Fixing the
Bug
Logical Bugs
HIGH HARD HARD
Wiring Bugs
MEDIUM MEDIUM MEDIUM
Rendering Bugs
MEDIUM EASY EASY
![Page 26: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/26.jpg)
But my code is different!
![Page 27: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/27.jpg)
I have a Super-Bug!
![Page 28: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/28.jpg)
Testing is like industrialrevolution
![Page 29: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/29.jpg)
(Configuration)Test the whole systempretending to be user
(>10 secs)
(Wiring) Testinteraction/contracts
between classes(<1 sec)
(if)Test individualclasses/methods
in isolation(~1ms)
Different Kinds of TestingEx
ecut
ion
Tim
eFl
akin
ess/
Deb
uggi
ng/M
aint
enan
ce C
ost
End-to-End
Functional
Unit
Soft
war
e En
gine
ers Te
st E
ngin
eers
# of Tests
![Page 30: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/30.jpg)
Dev Model RevisitedEnforcement
VisibilityTools
Education
Develop
Test
Check-In
Review
Testing on the ToiletTech Talks / Blogs1 on 1 TrainingMercenariesImmersion
ToolsAnalysisReports
Games DashboardsScoreboards
Trend Graphs
RoboCopTestabilityCoverage
CI-build
![Page 31: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/31.jpg)
Education: Test Mercenaries
• Internal consulting group specializing in testability refactorings and influencing developers
• 2-3 mercenaries join the group for 3-6 months
![Page 32: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/32.jpg)
Education: Tech Talks
• Weekly series
• Focus on common mistakes which make code hard to test
• new, global-state, singletons, LoD
• Many available on You Tube
![Page 33: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/33.jpg)
Education:
• We try to stick it on everyones monitor
• Set of red-flags to look for
• URL pointing to an explanation of the red-flag
• The reviewer can point the author to the explanation page
![Page 34: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/34.jpg)
Education: Mission Impossible
• Mercenaries in reverse
• Total Immersion
• work with team for a month which “gets it”
• Very good feedback
![Page 35: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/35.jpg)
Education: TotT
• Weekly publications
• Testing Tips (not concepts)
• Captive audience
• Very successful
![Page 36: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/36.jpg)
Education: Blog
• Venue for larger concepts which do not fit to TotT.
• Shared with outside world
![Page 37: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/37.jpg)
Education: TDD University
• Build small project from scratch
• Learn how the code should be like
• Rewrite existing piece of code
• To see how it is different
• Refactor existing code
• To learn how to get from here to there
![Page 38: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/38.jpg)
Tools: Coverage
• Simple but effective
• Just having it installed makes people thing about improving coverage
• Squeaky wheel gets the grease
![Page 39: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/39.jpg)
Tools: Test Farms
• Farms
• Ability to run massive test suites in parallel
• Selenium
• unit-test
• browser matrix
![Page 40: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/40.jpg)
Tools: Testability Report
• Code coverage for testability
• Get high level view about how testable the code is
![Page 41: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/41.jpg)
Tools: Testability Advice
• Personal testability advisor
• Plans to integrate into IDE
• Virtual pairing buddy
![Page 42: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/42.jpg)
Visibility: Games
• Make it into a game
![Page 43: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/43.jpg)
Visibility: Bubbles
• See quality of check-ins over time
• Time, Tests, User, Size
![Page 44: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/44.jpg)
Visibility: Test Pareto
• Test ratios per developer
• Impact per developer
![Page 45: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/45.jpg)
Visibility: Test Trend
• Test focus over time
![Page 46: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/46.jpg)
Visibility: Certification
• Well defined standards
• To be at N level you need to do X, Y, and Z.
• TC level prestige
![Page 47: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/47.jpg)
Enforcement: Tough Love
• Remove manual testing resources
• Provide knowledge and support
![Page 48: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/48.jpg)
Enforcement: RoboCop
• Coverage can not get worse with any one CL
• Testability can not get worse with any one CL
• Plans for arbitrary rule enforcement
![Page 49: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/49.jpg)
Progress
• Too many tests too run
• CI standard
• Everyone agrees tests are good idea
• TC Level
• Dependency Injection - GUICE
• Ratio System vs Unit is improving
![Page 50: Psychology of Testing - misko.hevery.commisko.hevery.com/wp-content/uploads/2008/07/RTAC-Psychology-of … · Psychology of Testing (or why our intuition of testing is wrong)-- Miško](https://reader035.vdocuments.us/reader035/viewer/2022070722/5f01c5907e708231d400f569/html5/thumbnails/50.jpg)
Q&A