testing observability · 2018-05-04 · white box testing if input == 4 return input else break 44...
TRANSCRIPT
![Page 1: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/1.jpg)
Testing ObservabilityAmy Phillips
![Page 2: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/2.jpg)
Amy Phillips
● Engineering Manager at Moo
● Previously manager and tester at Songkick, The Guardian,
and Yahoo!
● Can be found @amyjph
Testing Observability | Amy Phillips | @amyjph
![Page 3: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/3.jpg)
The early daysTesting Observability | Amy Phillips | @amyjph
![Page 4: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/4.jpg)
Black box testing
Test case Expected result Test result
User saves a value of 4. Retrieve value.
System returns a value of 4 Pass
44
Testing Observability | Amy Phillips | @amyjph
![Page 5: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/5.jpg)
White box testing
if input == 4return inputelsebreak
44 44
Testing Observability | Amy Phillips | @amyjph
Test case Expected result Test result
User saves a value of 4. Retrieve value.
System returns a value of 4 Pass
![Page 6: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/6.jpg)
Releases
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Build new
website
Coding
Testing
Bug fixing
Retesting
UAT
Bug fixing
Retesting
Go Live!
Testing Observability | Amy Phillips | @amyjph
![Page 7: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/7.jpg)
Introducing agile
Mon Tue Wed Thur Fri Mon Tue Wed Thur Fri Mon
Feature 1 Coding Testing
Bug fixingGo Live!
Feature 2 Coding Testing
Feature 3 Coding Testing
Integration testing
Testing Observability | Amy Phillips | @amyjph
![Page 8: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/8.jpg)
Fluid testing
Acceptance testing
Manual Automated
Pairing
Code reviews
Error log reviews
Ad Hoc tools
Testing Observability | Amy Phillips | @amyjph
![Page 9: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/9.jpg)
Continuous deliveryTesting Observability | Amy Phillips | @amyjph
![Page 10: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/10.jpg)
The monolith
Microservices and the impact on testing
Push code Release
Testing Observability | Amy Phillips | @amyjph
![Page 11: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/11.jpg)
The monolith
Microservices and the impact on testing
Testing
Testing
Testing Observability | Amy Phillips | @amyjph
![Page 12: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/12.jpg)
Service
Microservices and the impact on testing
ReleasePush code
Service ReleasePush code
Service ReleasePush code
Testing Observability | Amy Phillips | @amyjph
![Page 13: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/13.jpg)
Service
Microservices and the impact on testing
Push code
Service ReleasePush code
ServicePush code
Testing Observability | Amy Phillips | @amyjph
End to EndTesting
![Page 14: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/14.jpg)
Service 1
Consumer driven contract testing
ReleasePush code
Service 2 ReleasePush code
Test broker e.g. PACT
Test
Test
Testing Observability | Amy Phillips | @amyjph
![Page 15: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/15.jpg)
Node A
Master
Self healing systems
Service 1 Service 2
Node B
Service 3
Testing Observability | Amy Phillips | @amyjph
![Page 16: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/16.jpg)
A quick look at testing
Testing Observability | Amy Phillips | @amyjph
![Page 17: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/17.jpg)
The two sides to testingTesting Observability | Amy Phillips | @amyjph
Have we built the thing right?
Have we broken anything else?
![Page 18: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/18.jpg)
Testing
An investigation conducted to
gather information on the
quality of the system or product.
Testing Observability | Amy Phillips | @amyjph
![Page 19: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/19.jpg)
Quality
A quality system meets the
needs of the user.
Testing Observability | Amy Phillips | @amyjph
![Page 20: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/20.jpg)
Test pyramid
UI
Integration
Unit
Testing Observability | Amy Phillips | @amyjph
![Page 21: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/21.jpg)
Testing dials
Unit testing Integrationtesting
Contracttesting
Manualtesting
Testing Observability | Amy Phillips | @amyjph
![Page 22: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/22.jpg)
Testing that something works can be hard
Push code Test Release
Two problems:
● Process assumes a positive outcome = bug reports or rework
● Forced failure states can be unrealistic = limited value
Testing Observability | Amy Phillips | @amyjph
![Page 23: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/23.jpg)
Observability instead of testing?
Testing Observability | Amy Phillips | @amyjph
![Page 24: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/24.jpg)
Retrospectively adding logging
LoggingService
DB
Testing Observability | Amy Phillips | @amyjph
![Page 25: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/25.jpg)
How will we know when it isn’t working?
Testing Observability | Amy Phillips | @amyjph
![Page 26: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/26.jpg)
How would you test observability?
Monitoring Alerting
Tracing Log aggregation
Testing Observability | Amy Phillips | @amyjph
![Page 27: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/27.jpg)
Chaos monkey
● Weekly scheduled session
● Two “monkeys”
● One “victim”
Testing Observability | Amy Phillips | @amyjph
![Page 28: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/28.jpg)
Make it easy to observe failures
Break
something
Testing Observability | Amy Phillips | @amyjph
Observability
Fix it
Improve
![Page 29: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/29.jpg)
Predictable chimp
● Cheeky
● Multi-talented
● Engaging
● Attention grabbing
● Playful
http://interactivepuppy.com/spinmaster-zoomer-chimp-interactive-monkey-toy-review/
Testing Observability | Amy Phillips | @amyjph
![Page 30: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/30.jpg)
How to design test scenarios
Testing Observability | Amy Phillips | @amyjph
![Page 31: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/31.jpg)
Do you really need a tester?Testing Observability | Amy Phillips | @amyjph
![Page 32: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/32.jpg)
Idea generationTesting Observability | Amy Phillips | @amyjph
![Page 33: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/33.jpg)
Is it worth it?
Test value = risk vs. effort
Testing Observability | Amy Phillips | @amyjph
![Page 34: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/34.jpg)
Make it better
Break
something
Testing Observability | Amy Phillips | @amyjph
Observability
Fix it
Improve
![Page 35: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/35.jpg)
In summary
● Testing is changing as systems change
● Observability may take the place of many testing tasks
● But observability isn’t free
● We should be asking “How will be know when it isn’t
working?”
Testing Observability | Amy Phillips | @amyjph
![Page 36: Testing Observability · 2018-05-04 · White box testing if input == 4 return input else break 44 4 Testing Observability | Amy Phillips | @amyjph Test case Expected result Test](https://reader030.vdocuments.us/reader030/viewer/2022040805/5e426570b1ad1406aa1e4dd5/html5/thumbnails/36.jpg)
ThanksAmy Phillips
@amyjph