WRONG REASONS TO AUTOMATE!`Nol Rananand

22 Aug 201412nd TPSE

Some good testing objectives don’t work

for test automation…

2

#1 Find More Bugs

Reasoning• Testing should find bugs. Run more test Find more bugs

Reality• Automation good for regression test. Regression test looks

for unexpected bugs in unchanged software.• Automation free up testers to find more bugs via manual

test.

Better Objective• Automation helps find more

regression bugs

32nd TPSE

#2 Run more testsReasoning

• Test more gives better test coverage

Reality

• Quality of tests that counts, not quantity

• Automating poor tests gives you maintenance overhead with little return. “Automated chaos is just faster chaos” (D. Graham)

Better Objective

• Select set of

valuable

manual tests

to automate

42nd TPSE

#3 Automate 100% of testing

2nd TPSE 5

Reasoning• To measure progress of automation effort

Reality• Some attributes of software , such as usability, are better

tested by human• Another benefit from automation is in form of tool support

Better ObjectiveMeasure how automation provides valuable support to testing• How testers rate support provided by Automation team• # of addn’l verifications made that cannot be done manually

#4 Reduce Testing staff

Reasoning• Automate tests replaces manual tests hence don’t need as many staff.

Can save staff cost.

Reality• More staff actually needed since need staff with script development skills

to build and maintain scripts/framework, analyze results• Automation frees up testers to

contribute at a higher level

Better Objective• Total cost of automation effort is less

than total test effort saved by automation.

62nd TPSE

Next is my own personal experience

Don’t try this at your work!

7

#5 Put testers you have with some coding skills in automation team

Reasoning• What’s the quicker way to start test automation with the

testers you have?

RealityIf testers don’t have sufficient coding skills, you will get• Spaghetti test scripts. Impossible to share & maintain them• Lack of appropriate test automation framework

Better Approach• Have a small “toolsmith” team to work with testers to

develop automate test scripts.

8

#6 Focus on automate E2E test

Reasoning• Convert existing

manual functional test cases to automate scripts

Reality• Such E2E test scripts

are brittle and slow to run

• They don’t help improve quality of development process

Better Objective• Make sure your

automation suite is pyramid shape

2nd TPSE 9

Summary of Key Points

• Quality of tests determines whether or not bugs are found, and has little to do with automation

• Try not to go for 100% automation

• Test automation cannot replace human testers

• Better to have “toolsmiths” develop good test automation framework and leave scripting to testers

• Turn your automation ice-cream cone to pyramid

2nd TPSE 10

Some Good References

1. That’s No Reason to Automate by Dorothy Graham and Mark Fewster: http://www.dorothygraham.co.uk/downloads/generalPdfs/NoReasonAut.pdf

2. Lessons in Test Automation By Elfriede Dustin: http://www.stickyminds.com/sitewide.asp?ObjectId=1802&ObjectType=ART&Function=edetail

3. Agile test automation by James Bach: http://www.satisfice.com/articles/agileauto-paper.pdf

2nd TPSE 11

The End

2nd TPSE 12

To Automators out there,

don’t give up!