analysing test results over time listen beyond the pass/fail · analysing test results over time...
TRANSCRIPT
Listen beyond the pass/fail
Analysing test results over time
Lena Wiberg@LenaPejgan
Lena Wiberg
SCI-FI/Fantasy & Games
Patterns & trends
Visual models
Dev. => Tester => Manager
Awesome new job!
Vision! Reality…
@LenaPejgan
It works fine when I run it locally
@LenaPejgan
The test environment was locked by another
user
@LenaPejgan
We hit a random duality check
@LenaPejgan
It was just an unexpected popup
@LenaPejgan
System/service X had the wrong data/version
@LenaPejgan
Oh, it’s just because of bad data
@LenaPejgan
Run
Analyze
Setup
Fix @LenaPejgan
Identify the problem
@LenaPejgan
NOISE!
@LenaPejgan
@LenaPejgan
FALSE SECURITY
Feedback
Stability
Planning
Investment Is it stable enough to explore?
Did my changes break something?
Is this version ready for release?
Are we spending out money right?
@LenaPejgan
It works fine when I run it
locally
Oh, it’s just because of bad data
The test environment was locked by another user
It was just an unexpected
popup
We hit a random
duality check
The integrating system had the
wrong backup/version
+@LenaPejgan
Operations & Hardware
@LenaPejgan
”Our firewall doesn’t allow that”
”Oh, that’s when we do backups”
”We do patches every 3rd Monday”
Test environments & Test data
@LenaPejgan
”The data was used up by another test”
”Someone else was using the environment”
”Service X was down/had the wrong data”
Processes and communication
@LenaPejgan
”We didn’t have time to update the tests”
”We can only test version X”
”We didn’t know the code has changed”
FIX!…Or an ACTUAL bug!
@LenaPejgan
TEAM AWESOME
Summary•Don’t get (too) caught up in the now •Look past the obvious •Kon Mari those tests! Do they spark joy?•Don’t assume flaky can’t be fixed•Reuse is awesome. Except when it’s not…
@LenaPejgan
