Download, install, record… we’re automating!??Asking good questions and building good design
to reach reasonable Automation outcomes
Richard Robinson, May 2014
Automation frameworks…. arggghhh!
• So many platforms!
• So many programming languages!
• So many tool vendors!
• So many interfaces!
• So much data!
• So many stakeholders!
• So little time!
Automation framework design principles
• Parameterise your input data
• Modularise your automation blocks
• Setup and tear down maintenance
• Use self-descriptive data (SDD)
• Use a database for scalability
• Manage states
• Expand input stakeholders
• Simple trigger points
• Simple reporting methods
Design-principles-in-a-diagram
Execution UI Output
reports
Automation engine
System under test
Automation databaseInput values lists
Stakeholders
Execution
trigger from
Web UI
Reporting to
Web UI
Input values
from flexible
entry
sourcesDatabase input
values and results
collection
Adding DB
values
Execution of tests
with system
Modularisation
Setup, teardown
Parameterisation
Self-descriptive data
State management
Parameterise - explanation
• Parameterised data is changed each time the test is run
• Values can be randomised
• Known stored valid values can be used (or invalid,
depending on your test)
• Functions can be used to create data depending on
other variables
• Input fields reference the variable, not actual concrete
data
Parameterise - example 1
• For a registration form with Firstname/lastname fields,
you can use a name lookup list using a long static list of
first name of all lengths and punctuation.
• Chandrarajan Sivasaravanamuttu (Sri Lankan Cricket
President)
• Mata'afa Mulinu'u II (Samoan President)
• Haven’T (Actual boys names from 2012)
Parameterise - example 2
• Suburbs in Australia
http://burbs.com.au/australian-postcodes/
Parameterise - example 3
• Credit card numbers used around the world
http://en.wikipedia.org/wiki/Bank_card_number
• Remember to also refer to your institutional whitelist for further
reduce the valid credit card numbers
Parameterise - examples
• Mobile numbers in use in Australia:
http://en.wikipedia.org/wiki/Telephone_numbers_in_Austral
ia
Parameterise - benefits
• A different test is run each time
• Framework is more extensible to other users
• Can scale the input values
• Reduces maintainability
• Get around constraints of the data or function
• Data is realistic, and supports and bugs found
• Can build in safety so that live records, account, or data is not
used
Modularise - explanation
• Modules are building blocks of script or code
• They are like a family tree and any layer can be
executed
• Built once and reused often across scenario tests
Modularise - examples
• Field definitions and reuse - eg. password field is used
on registration and also password reset
• Library use of common functions
• Parts of a scenarios
eg. Log in, register, find random item, checkout,
payment, deregister, change password, update profile
• Full scenarios that can be executed in serial to show
real end-to-end usage of the system
From http://safsdev.sourceforge.net/FRAMESDataDrivenTestAutomationFrameworks.htm
Modularise - breaking it down
Modularise - benefits
• Fewer code blocks to maintain
• Build once, use often, saves time when building
• Non-expert coders can build scenarios using blocks
(premise of BDD)
Setup & teardown - explanation
• Test your framework
• Build quality checks into the framework
• Ensure the system is ready for execution before finding
out from a functional tests
• Check and maintain data
Setup & teardown - examples
• Execution setup script - checking and maintaining
environment variables, 3rd party connectivity, server
availability. eg Ping servers, send test request
• Data setup script - gather record to use, and apply any
initial checks on it
• Reset your data once its been used
Setup & teardown - benefits
• Informed of environmental issues prior to any test is run
• Can be used as an environment monitoring tool
• Allows for better re-use of data and records
Self-descriptive data - explanation
• Self-descriptive data is labelled to identify itself and the
test
• It is constrained by what is acceptable for the field
• Used for quick checks of test results by scanning
sizeable output
Self-descriptive data - examples
• PDF report on screen showing transactions and
balances
• SMS texts being received for all notifications
- matching the action to the message
• Web table data showing groups of results
Self-descriptive data - benefits
• Good for manual output checking from time to time
• Good for simplified results checking, as all tests should fit in same
results class
• Does not require external oracles
• Can generate and verify large amounts of complex
• Reduces time to do any manual verification
• Reduces diagnosis when things go wrong
• Good record keeping practice
Database-driven - explanation
• If your framework is starting to grow and expand, then
consider using a database for:
- input data storage
- parameter storage (eg. environment variables)
- output result storage
• Excel is great for small operations, but at some point,
having excel output files created for all tests becomes
unmanageable
Database-driven - examples
• Environmental variables - server names, IPs,
usernames, passwords, interface values
• Data-set up - time-based variables
• Input data - customer registration, orders, credit cards
Database-driven - benefits
• Scalable
• Efficient processing of data, results, and reports
• Centralised - one collection point for all data and results
• Allows for multiple users
State management - explanation
• Know your record states
• Maintain record states as in-use
• Return records for re-use
State management - examples
• Capture main records in a DB, or file
• Capture defining field values eg. in-use state, customer
cohort, type of payment method, address type, comms
type
State management - benefits
• Efficient for role-based, and state-based systems
• Minimised occurrence of record conflicts
• Allows for multiple instances of execution
• Allows for multiple users to execute
Stakeholder usage - explanation
• Framework used by any stakeholder
• Additional framework components build to support
others
• Access to data inputting, execution, results generation
• Input to methodologies such as BDD, ATDD, or any
keyword-driven framework implementation
Stakeholder usage - examples
• Adding values to data input lists
• UI to add values to input data directly to database
• UI to trigger tests on demand
• Running a demo at a stakeholder meeting
• Obtaining report
Stakeholder usage - benefits
• Buy in from stakeholders, increase in investment
• Credibility and quality for all to see
• More feedback on quality of tests
Simple triggers - explanation
• Automation can be used for many purposes and for may
stakeholders
• If the trigger points are easy to use, then it will get used
more often
Simple triggers - examples
• Web front end page with scenarios and suits available
as a menu to visitors
• Mobile app to trigger tests
Simple triggers - benefits
• Framework more widely used, improving value (“ROI”)
• More usage means more testing (and checking), means
greater chance of finding bugs
Simple reporting - explanation
• Reporting is what stakeholders see and use to
communicate to others
• Publish automation runs to a web GUI
• Allows layers of access to different reports (role-based
login)
Simple reporting - examples
• Report to levels of management, or stakeholder groups
eg. Product owner, development team, managers
• Report to levels of test execution
eg Acceptance test runs, function test runs,
environmental monitoring, data-driven tests, scenarios
Simple reporting - benefits
• Exposes the value of automation to the stakeholders
• Allows for feedback into the quality of the framework
• Allows for higher usage of framework by other
stakeholder groups
Focussing questions… (we are not done yet!)
• Do you agree?
• Learned something new?
• Applicable? How much do I implement the design
principles?
• Are there any more?
• What’s missing?
10th design principle… CONTEXT IS KING!
• What are the outcomes we are trying to realise and
why?
• What problem are we trying to solve?
• What matters and what questions do we ask?
outcomeautomationproblem
context & environment
What outcome are we trying to achieve?
• To automate 100% of the functions
• To gain 100% coverage of a specification
• To reduce the number of manual testers to zero
• To shorten regression cycles to 1 minute
• To deepen exploratory testing to be exhaustive
• To reduce administration overhead to zero
• Monitor all user pathways
• To delivery working code overnight ready for utilisation
What problems are we trying to solve?
• Regression testing takes too long and is too costly
• We cannot afford all these testers, lets get rid of 40%
• We are losing quality from our product, lets automate
• Lets make use of dead time by testing overnight, automatically
• Lets save time by checking the quality of the release prior to
manually testing it
• There are too many bugs found in production, lets increase our
testing coverage by automating
What matters?
Strategy - Cost considerations
• Prototype to get further funding?
• Automation is the saviour of our project, spend, spend,
spend
• Whose vision is it to automate? Do they have a budget?
• Are the automation team in budget?
Strategy- People considerations
• Full time dedicated resources or stealing developer time
• Proven skills or developing them
• Full domain knowledge or getting from business
resources
• Expert automation consultant (selling of implementation)
• Expert automation specialist (coding, implementing)
Strategy - Time considerations
• Big bang approach, or easy onset
• Quick wins, or consistent development
• Milestone based, or open ended outcomes
Strategy - System considerations
• Platform tools, language
• Interfaces - 3rd party vendors, access
• Data sensitivity, record availability
• Environment management and coverage
• Deployment infrastructure and processes
Strategy - Quality considerations
• How to measure?
• Are we actually speeding things up?
• Are we actually reducing any cost?
• What is our maintenance costs?
• When do we know its not working?
• What does ‘good’ look like?
Want a trick to remembering the principles?
The single most important
reason
to reduce cost
or save
and another word for ‘save’ is…
management buy-in to Automation is…
so…
Lets rearrange the principles…
• Setup and tear-down
• Context is king
• Reporting interface
• Input stakeholders
• Modularise
• Parameterise
• Self-descriptive
• State management
• Trigger interface
• Database-driven
And take the first letters together…
and you get…
SCRIMPS STD
(Scrimps Standard)
“A management-supported
automation framework design
standard,
with the sole objective to gain
economical outcomes of the
quality assurance function”
What is your context? Experience reports…
• Setup and tear-down
• Context is king
• Reporting interface
• Input stakeholders
• Modularise
• Parameterise
• Self-descriptive
• State management
• Trigger interface
• Database-driven
Resources
Excellent overview of testing and automation, best next steps to read.
Kaner, 2010, http://kaner.com/pdfs/VISTACONexploratoryTestAutmation.pdf
Models of autoation
McCowatt, http://www.ministryoftesting.com/2014/05/automation-time-change-models-iain-mccowatt/
http://exploringuncertainty.com/blog/archives/1010
Problem, solutions, maintainability,
http://www.ministryoftesting.com/2010/07/an-introduction-to-test-automation-design/
Self-verifying data
Doug Hoffman, 2012, http://www.ssqa-sv.org/presentations/Doug%20Hoffman%20SSQA%2011-13-2012.pdf
Noel Nyman, Microsoft, 2001, http://www.stickyminds.com/sites/default/files/presentation/file/2013/01TAU_T6.pdf
Maintainability
http://dhemery.com/pdf/writing_maintainable_automated_acceptance_tests.pdf
Automation test pyramid
Martin Fowler (2012), http://martinfowler.com/bliki/TestPyramid.html
Framework design
Carl Nagle, ”The test framework should be application-independent"
http://safsdev.sourceforge.net/FRAMESDataDrivenTestAutomationFrameworks.htm
General automation methodology
Borland (2012), https://www.borland.com/_images/Silk-Test_WP_How-to-successfully-automate-the-functional-testing-process_tcm32-205735.pdf
IBM, http://public.dhe.ibm.com/common/ssi/rep_wh/n/GBW03035USEN/GBW03035USEN.PDF
Hybrid approach most popular
Yogi Gupta, general implementation methodology
http://www.slideshare.net/yogindernath/understanding-of-automation-framework-presentation
Resources continued…
IMPLEMENTATION
Good design ideas, technical implementation (2011)
http://technologyandleadership.com/30-feet-view-of-test-automation-framework-with-selenium/
Generic implementation principles
http://www.theeffectiveengineer.com/blog/what-makes-a-good-engineering-culture
Doug Hoffman on Automation
http://www.slideshare.net/Softwarecentral/software-test-automation-beyond-regression-testing
Failure and problems with automation
http://five-ways-to-make-test-automation-fail.pen.io/
Smartbear Guide to converting to automation testing -
desktop http://www2.smartbear.com/rs/smartbear/images/Your%20Guide%20to%20Converting%20to%20Automated%20Testing.pdf
Alan Page - The A Word - desktop https://leanpub.com/TheAWord
Automation Checklist - strategy, business, tests
https://hackpad.com/Tips-to-Approaching-Test-Automation-fDRllKNrRJ2
About the presenter
An independent contract tester who works primarily in
automation, performance and test management roles.
The current President of the Sydney Testers Meetup, and
facilitator of WeekendTesting.com/WTANZ.
The head the facilitation of the Let’s Test Oz and CAST
conferences.
Blogs at www.hellotestworld.com and publish his ideas
to slideshare, and testing magazines such as Testing
Trapeze.
A Black Belt Tester of the Miagi-do school of software
testing, and a founding member of the ISST.
Richard Robinson
@richrichnz
richrob79