winning strategies in test automation

XBOSo&, Inc. All Rights Reserved. 1

Winning Strategies In Test

Automation

XBOSo&

Founded in 2006 Dedicated to software quality

Software QA Consulting Software Testing Offices in San Francisco, Beijing and Amsterdam


House Rules Everyone except the speakers are muted

QuesDons via the gotowebinar control on the right side of your screen or through TwiIer @XBOSo&

QuesDons can be asked throughout the webinar -‐ we’ll try to answer them at the end.

You will receive info on recording a&er the webinar


Meet Our Speakers •  CEO and Founder of XBOSoft •  Relevant specialties and passions

–  Software quality process, evaluation, measurement and improvement

–  Software quality in use / UX design –  Mobile User Experience and

usability –  Cycling and travel


Philip Lew

•  Guest Speaker •  15+ years in Testing and QA management •  International conferences speaker •  STP Community Advisory Board Member •  BBST Instructor •  Passions:

•  Cross-functional team collaboration •  Testers’ skills development •  Quality advocacy

Anna Royzman

Sabrina Gasson •  Marketing Manager of XBOSoft •  Emails you all regularly to join our

industry hot topic webinars •  And invites you all to download our

latest trends in software testing whitepapers.

MY EXPERTISE

•  Manager of so&ware testers and automaDon engineers over variety of projects, in both agile and waterfall

•  Leading AT SIG (special interest group) in Technology department of my organizaDon

•  15+ years experience with home-‐grown test automaDon frameworks, commercial and open source tools


POLL QUESTION

WHAT’S YOUR PRIMARY RESPONSIBILITY:

•  A. MANAGER RESPONSIBLE FOR TESTING •  B. AUTOMATION ENGINEER •  C. TESTER •  D. NONE OF THE ABOVE


DEFINITION OF AUTOMATION

UTILIZATION OF TOOLS, FRAMEWORKS AND SCRIPTS THAT ASSIST WITH TESTING OR CHECKING TASKS


EXAMPLES

•  Unit tesDng frameworks •  Test controllers •  Recording and playback •  Performance tesDng tools

•  Security tesDng tools •  AnalyDcs collectors •  Screen capturing tools •  Code coverage tools

•  Protocol tesDng tools •  Monitoring tools •  Database tesDng tools •  Simulators, log helpers, debuggers

•  Data populaDon scripts: setups, etc.

•  Keystroke recording tools


AUTOMATION IS A HELPER – CreaDng a ‘safety net’ of business-‐criDcal workflows and logic that must work every Dme

– Allowing to check code on various levels: from unit components to distributed systems

– AssisDng humans with humongous or repeatable tasks (stress or load tesDng, system integraDon tesDng, data tesDng, 3rd party cerDficaDon etc.)

– Debugging, analyzing or replicaDng failures – Monitoring running programs and capturing results for later analysis


AUTOMATION IS A CODE

•  It’s created by humans: IT HAS BUGS •  It needs development and debugging Dme •  New/changed funcDonality affects test suite, it requires maintenance to keep up to date

•  It needs skills •  It has upfront costs •  There is always version 2


SKILLS AND RESPONSIBILITIES

•  Toolsmith (A tester? possibly!) •  AutomaDon Engineer (A tester? Not necessarily!)

•  Scripter (A tester? Not necessary, but it’s good to be one)

•  Customer of automaDon (Must be a tesDng expert and has skills in tesDng strategies)


TEST AUTOMATION STRATEGY

•  CriDcal coverage •  Low maintenance: has to be ready for next iteraDon/release with low input

•  Easy to run on demand (one buIon click, ways to select tests for specific scenarios, pre-‐selected suites for different coverage)

•  Fast (mulDple environments, distributed, overnight, etc.)

•  TargeDng for ‘0 tolerance’ – failures are acceptable for bugs in the product only


QUICK POLL

WHAT AUTOMATION TOOLS/FRAMEWORKS DO YOU USE: A.  LICENCED TOOLS B.  OPEN SOURCE C.  HOME-‐GROWN D.  MIX OF THE ABOVE E.  NONE


SELECTING TOOLS AND METHODS

•  What’s your long term and short term goals? •  Build, buy or open source? •  What to look for in a tool •  Who will be supporDng it:

– Do you have specialists? –  Is that their sole responsibility? – What will happen when your company has RIF or specialist leaves?


COMMON MISTAKES TO AVOID •  AutomaDng new versions through GUI (it can work well with

legacy version for tesDng upgrades) •  Hardcoding values •  Trying to test all aIributes all the Dme. The test failing on

insignificant aIribute is noise. All tests failing due to one aIribute is more noise and makes AT unusable.

•  Product code replicas •  Get carried away with cool features of less value •  StarDng automaDon too early or too late •  Using only one tool for everything •  Assuming automaDon replaces human interacDons


MY ADVICE BASED ON MANY YEARS EXPERIENCE:


•  AT is good when is up to date; it should be available when team needs it

•  Cross-‐team integraDon: •  build simulators for majority of tests •  limit integraDon tests to the most criDcal paths.

•  Building a framework for your own product: •  Try to test each layer separately instead of tesDng everything from one layer

•  Try to avoid heavy UI automaDon as much as possible. •  Take responsibility! As a tester, demand that the features important to you are prioriDzed by automaDon engineers

QUICK POLL

ARE YOU WINNING WITH YOUR TEST AUTOMATION: A.  YES B.  NO C.  WE DON’T HAVE IT


RISK MITIGATION HINTS •  If you don’t have a toolsmith in your team, consider lending from other teams or asking programmers

•  Don’t replace exploratory tesDng with AT (it needs human brains)

•  Don’t replace usability tesDng with AT (it needs human emoDons)

•  Always have “plan B” in an emergency such as: •  test server failure •  DB permission errors •  blocking issue takes too long to fix right before release •  no electricity


Post your questions on Twitter and we'll answer them @XBOSoft Join us to keep updated on all our webinars, reports and whitepapers: facebook.com/xbosoft

+xbosoft linkedin.com/company/xbosoft

We post regularly on our blog – check us out! http://xbosoft.com/software-quality-blog/ Why not download our free Whitepapers, available here: http://xbosoft.com/knowledge-center/ You will receive an email with information on slides and recording. Any further queries regarding our services or ideas for future webinars please email us! [email protected]

Q+A

www.xbosoft.com

winning strategies in test automation

Software

recording tools xboso

test automation xboso

xboso quesdons

webinar xboso

travel xboso

amsterdam xboso

open source tools xboso

data tesdng