![Page 1: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/1.jpg)
Test automation, handcuffsor enforcement?22-06-2016, Pieter WithaarTest automation days 2016
Version 0.7
![Page 2: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/2.jpg)
The Automator
Test automation
Handcuffs or enforcement?
The automator 2
![Page 3: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/3.jpg)
I’m an automator
• I automate everything• Everywhere • Whenever I can
• I like to automate• Manual tasks are preventable
TA Handcuffs or enforcement? 3
![Page 4: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/4.jpg)
I called her Claire
TA Handcuffs or enforcement?
![Page 5: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/5.jpg)
I called her Claire
TA Handcuffs or enforcement?
![Page 6: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/6.jpg)
Automation turned into more automation
TA Handcuffs or enforcement? 6
![Page 7: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/7.jpg)
Is pushing a button that hard?
TA Handcuffs or enforcement? 7
![Page 8: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/8.jpg)
Into the deep
TA Handcuffs or enforcement? 8
![Page 9: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/9.jpg)
Well hello there
This test is currently under construction. Check back soon!
TA Handcuffs or enforcement? 9
![Page 10: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/10.jpg)
I’m a tester
• I like to explore• Find different angles• See all the options• Like problem solving• Design and perform experiments• Discover things I didn’t expect
TA Handcuffs or enforcement? 10
![Page 11: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/11.jpg)
Data trawling method
TA Handcuffs or enforcement? 11
![Page 12: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/12.jpg)
Both the tester and automator
• Create a lot of automated tests• Combined they are good• In solitude it leads to unmaintainable
automation
TA Handcuffs or enforcement? 12
![Page 13: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/13.jpg)
TA Handcuffs or enforcement? 13
![Page 14: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/14.jpg)
Have you ever found yourself in one of these situations?• You didn’t know what the automated test did?• Why they are made or what they are designed
to do?• At the end of delivery still some automated
checks failed, but we still deliver• With every change many automated checks
are failing• After fixing one automated check the next
issue popped up in the following automated check
TA Handcuffs or enforcement? 14
![Page 15: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/15.jpg)
• “1476 out of 1484 tests passed”• “The code coverage is at 88%, but we need
90%”• “The automated suite fails already for a while,
we know the issue”• “When I change the code, many unit test fail”• “Automation always comes after development,
slowing it down”• “The problem with this project is that the
requirements are always changing”
TA Handcuffs or enforcement? 15
![Page 16: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/16.jpg)
Automated test suites
• Tend to slow down agile teams
TA Handcuffs or enforcement? 16
![Page 17: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/17.jpg)
Automated test suites
• Should give fast insight in product properties
TA Handcuffs or enforcement? 17
![Page 18: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/18.jpg)
A good automated test
• As fast as possible• As reliable as possible
- Fails only on its purpose- Can fail (Asserts something)
• Maintainable• Understandable
- Documented
• Controls it’s test context
TA Handcuffs or enforcement? 18
![Page 19: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/19.jpg)
Lessons learned in software development• Make functions easy to understand• Make functions reusable• Structure and allow for refactoring• Document your functions• Test!
TA Handcuffs or enforcement? 19
![Page 20: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/20.jpg)
The roles
• The automator- The one that will automate everything
• The tester- The one that explores everything
• The testautomator- The one that maintains everything
TA Handcuffs or enforcement? 20
![Page 21: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/21.jpg)
We need skills and training
TA Handcuffs or enforcement? 21
![Page 22: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/22.jpg)
The automation minded tester
• The automator- The one that will automate everything
• The tester- The one that explores everything
• The testautomator- The one that maintains everything
• The automation minded tester- The one that gets insight in everything
TA Handcuffs or enforcement? 22
![Page 23: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/23.jpg)
The automated minder tester
• Good documentation practices• Being able to go from a user flow to small
functional decompositions• Developer faced and business faced tests• Stubs, mocks, dependency injection• Being able to decide what to automate• Being able to decide on which level • Being able to distinguish and determine the
purpose of the automation• Data and environment control
TA Handcuffs or enforcement? 23
![Page 24: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/24.jpg)
The automated minder tester 2
• Setup / Teardowns on suites, testruns, tests etc• Test isolation• Assertions and expectations• Deterministic and non-deterministic testing• Polling, callbacks and asynchronous testing• Application domain• Sleeps vs Timeouts / explicit waits• Control time• Test independence• Enforcing the team and getting insight• Test automation reporting
TA Handcuffs or enforcement? 24
![Page 25: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/25.jpg)
Automate on their purpose
• Design test for what their risk that they mitigate- Regression tests focusses on lack of regression- E2E test focusses on E2E use- Multiple browser compatibility test focus on multiple
browser compatibility- High impact feature tests focus on high impact
features- Performance tests focus on performance
• “Do not use regression tests as E2E multiple browser compatibility tests!”
TA Handcuffs or enforcement? 25
![Page 26: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/26.jpg)
Document properly
*Toilet Tubinghttp://www.mindprod.com/jgloss/unmainobfuscation.html 26
/*/* Procedure Name:/*/* Original procedure name:/*/* Author:/*/* Date of creation:/*/* Dates of modification:/*/* Modification authors:/*/* Original file name:/*/* Purpose:/*/* Intent:/*/* Designation:/*/* Classes used:/*
![Page 27: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/27.jpg)
The automation minded tester
TA Specific Programming Testing Requirements Engineering
The automator 27
![Page 28: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/28.jpg)
The test minded automator
TA Specific Programming Testing Requirements Engineering
The automator 28
![Page 29: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/29.jpg)
Questions?
TA Handcuffs or enforcement?
![Page 30: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/30.jpg)
Questions?
TA Handcuffs or enforcement?
![Page 31: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/31.jpg)
Improve Quality Services BV
The automator 31
![Page 32: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/32.jpg)
Improve Quality Services BV
The automator 32
Process ImprovementSkillsTesting- Context driven- Agile- Automation
Quality & Outsourcing (QLM)
Requirements Engineering
Quality in IT- Agile (CAT)- Context driven testing (RST)- Test automation- Requirements Engineering
(IREB, CABA)- ISTQB, TMap- Mobile (CMAP)
![Page 33: Test automation, handcuffs or enforcement? · 2016. 6. 27. · • Stubs, mocks, dependency injection • Being able to decide what to automate • Being able to decide on which level](https://reader036.vdocuments.us/reader036/viewer/2022071100/5fd8a8978ba1f3657427c1ff/html5/thumbnails/33.jpg)
Improve Quality Services BV
The automator 33
www.linkedin.com/company/improve-quality-services
@improveqs
[email protected] +31 (0) 40 202 1803
Prof. Dr. Dorgelolaan 30, 5613 AM EindhovenAmsterdamsestraatweg 55A, 3744 MA Baarn