journey to 1000 tests a day
DESCRIPTION
Automation has come a long way in the last 15 years. This presentation tells the story of my journey from no automation, to running in excess of 1000 tests a day. It shares my insights, failures, successes and lessons learned, so hopefully you can get there too.TRANSCRIPT
Journey to 1000 Tests a DayBruce McLeod
Tuesday, 15 November 11
DISCLAIMER
The views and content presented in this presentationare my own and do not represent the views of any of my employers, past or present.
All content is copyright 2011 Bruce McLeod
Creative commons attribution for images is given at the end of this presentation
Tuesday, 15 November 11
Tuesday, 15 November 11
Every journey has a first stepMy introduction to Test Automation
Tuesday, 15 November 11
Record and playback
Tuesday, 15 November 11
Huge gains compared to manual execution
• Win32 desktop application
• 5 different clients (95, 98 Beta, NT4, NT 351 and Citrix)
• 5 days work for 3 people could be done in 4 hours
• 97% efficiency improvement
Tuesday, 15 November 11
But there were problems
• Tests were not maintainable
• Spent days in maintenance and development
• They were brittle and fragile
• And when I left for my next job ...
Tuesday, 15 November 11
Tuesday, 15 November 11
Tuesday, 15 November 11
KindergartenThe early manual coding years
Tuesday, 15 November 11
Manual coding beginsTuesday, 15 November 11
A 2001 test automation recipe
Ingredients1 eager client 1 test application14 test automators
Tuesday, 15 November 11
A 2001 test automation recipe
MethodStep 1 : Write an automation “framework”Step 2 : Create a gui mapStep 3 : Write a testStep 4 : Repeat step 4 another 200 times
Tuesday, 15 November 11
Lessons learnt
• A Successful project• 200 tests automated in 6 months
• Team was a real dev. team with all the same issues• Versioning• Merging and branching• Coding standards• Individual differences in style and design
Tuesday, 15 November 11
Tuesday, 15 November 11
ChildhoodThe innocent years of keyword testing
Tuesday, 15 November 11
What if we remove the need to code?
Tuesday, 15 November 11
Spreadsheets !!!Tuesday, 15 November 11
Keyword driven testing
Object Action DataTextfield (domain) Enter text <domain>
Textfield (username) Enter text <username>
Textfield (password) Enter text <password>Button (login) Click One left click
lObject - Action - data syntax
lAutomation libraries with excel front end
Tuesday, 15 November 11
Keyword driven testing• Advantages• Reduction in coding effort• “Anyone” can write tests!!
• Challenges• “Anyone” can write tests!!• Each implementation is custom• Maintenance effort moves from code into ...
Tuesday, 15 November 11
… Lots of Spreadsheets !!!Tuesday, 15 November 11
Tuesday, 15 November 11
AdolescenceEmbracing open source
Tuesday, 15 November 11
Manual coding returnsTuesday, 15 November 11
Until …
Tuesday, 15 November 11
Tuesday, 15 November 11
Unexpected benefits
• We could now use 1st generation developer tools• Visual Studio 2005• Version control• C# and .net instead of VBA
Tuesday, 15 November 11
Unexpected benefits
• Testers were now true peers to developers• Same tools• Same development language• Same recruitment process• Object oriented design
Tuesday, 15 November 11
Tuesday, 15 November 11
A discovery & epiphany!The “braidy tester” blog posts
Tuesday, 15 November 11
The “sparkle” stack• Used to test Microsoft Expression• Key principles• Test cases written in customer view• Verification and data separated from tests• Application internals re-created in test code
Tuesday, 15 November 11
Tuesday, 15 November 11
How can I execute 1000 tests a day ?
Tuesday, 15 November 11
Design goals
• Maintainability• Robustness• Performance• Maintainability• Rapid test development
Tuesday, 15 November 11
Maintainability
Tuesday, 15 November 11
Devs change stuffIt’s their job !
Tuesday, 15 November 11
Each action should occur onceand once only
Tuesday, 15 November 11
Each control should be mapped onceand once only
Tuesday, 15 November 11
Abstract the engineYou shouldn’t know what’s “under the bonnet”
Tuesday, 15 November 11
Robustness
Tuesday, 15 November 11
Tests should fail, not your tools.
Tuesday, 15 November 11
Embrace regex
Tuesday, 15 November 11
Leverage your vendor, and the community
Tuesday, 15 November 11
Performance
Tuesday, 15 November 11
CPU should be your bottleneck
Tuesday, 15 November 11
Never wait !Wait, don’t sleep
Tuesday, 15 November 11
Scale-out
Tuesday, 15 November 11
Rapid test development
Tuesday, 15 November 11
K.I.S.S
Tuesday, 15 November 11
Cheat !• Leverage an IDE
•Refactoring• Use code generation•Write less code• Use design patterns
Tuesday, 15 November 11
What about the future?
Tuesday, 15 November 11
Tuesday, 15 November 11
Open source frameworks will consolidateDriven by adoption of webdriver
Tuesday, 15 November 11
More focus on mobile automationiOS, Android, Windows Phone
Tuesday, 15 November 11
Behaviour driven developmentWill become mainstream
Tuesday, 15 November 11
Given I am a logged in userWhen I have a valid credit cardAnd I have 1 click purchasing enabledThen I should see the message “your purchase was successful”And I should be re-directed to the home page.
Tuesday, 15 November 11
Tuesday, 15 November 11
Tuesday, 15 November 11
Automation is development.Hire, plan, develop and execute with that in mind.
Tuesday, 15 November 11
Tuesday, 15 November 11
Image attribution• Creative commons attribution credit is required for the
following users on flickr• seattlemunicipalarchives• mathias-erhart• jaffathecake• 28096801• loty• brilliantmichael
Tuesday, 15 November 11