1

programs testing programs GUI-based test automation

1st Friday – 02.03.2012

Sascha Bartels

2

1. What‘s „test automation“?

2. Why do you do that?

3. How do I know, that I‘m doing it right?

4. Is our test automation cool?

3 3

01 | WHAT‘S „TEST AUTOMATION“?

4

A user browses with his browser on PokerStrategy.com

5

regression tests before each release of PokerStrategy.com

6 6

02 | WHY DO YOU DO THAT?

7

Possible objectives of test automation:

- find more bugs

- nightly regression tests

- reduce testing staff

- shorter test periods

- test more

Frequent problems of test automation: [1]

1. unrealistic expectations (of the management)

2. poor testing processes („Automating chaos just gives faster chaos“)

... etc.

Let us examine our goals more exactly!

1: Software Test Automation Effective use of test execution tools – Dorothy Graham & Mark Fewster

8

find more bugs

the idea:

manual testing finds bugs

automated tests are faster than manual

tests

so automated tests find more bugs

9

find more bugs

the practice:

- automated tests are repeated tests

- looks for unexpected side effects

- so automated tests can find

more regression bugs

- testers have more time for manual

tests

- poor tests are not getting better by

automating them

10

nightly regression tests

the idea:

There are unused ressources

(like test systems e.g.)

We could run automated tests „while

we sleep“

11

nightly regression tests

the practice:

the correct tests have to be selected

test results need to be analyzed

12

reduce testing staff

the idea:

The automation tool costs money, so we

should be able to save somewhere

else

We want to reduce costs and staff

is expensive

13

reduce testing staff

the practice:

tests have to be automated and maintained

test results must be analysed

test automation can make work easier

for testers

14

shorter test periods

the idea:

reduce deadline pressure

testing is a „bottleneck“, so we will save

time there

15

shorter test periods

the practice:

the goal can be achieved more easily:

run fewer tests, omit long tests,

cut regression testing

Automated tests take time:

creation and maintenance of test

cases

analysis of test results

software quality has the greatest influence on the time needed

for testing

16

test more

the idea:

achieve a better test coverage

testing is good, so more testing

has to be better

17

test more

the practice:

unimportant tests produce more

Maintenance instead of being gainful

some tests require higher

automation effort,

e.g. for technical reasons

18

Possible realistic objectives

- find more regression bugs

- automate the most importants tests

- a useful amount of tests run at night and / or weekend

- relieve testers and support manual testing

It is important to know why you automate, so you know what and how to

automate.

Objectives of testing should not get mixed up with the objectives of

automation.

Test automation is not a tool against bad test practices

That‘s No Reason To Automate! Why Good Objectvies Are Critical to Test Execution Automation – Dorothy Graham and Mark Fewster

19 19

04 | HOW DO I KNOW, THAT I‘M DOING

IT RIGHT?

20

measuring

metrics

Depending on the objective and the initial situation other metrics are

important.

Will this car fit into my garage?

Does this one consume as much petrol as this?

There might be a hole in the tank.

Yes?

21

1. DDP– Defect Detection Percentage

Is the relation between found and currently known bugs.

Example: A release contains 100 new bugs, the test run finds 70 bugs:

DDP: 70%

Weaknesses in this approach:

1. The absolute amount of bugs is unknown.

2. It is unknown at what time a bug has come into the product.

3. Not all bugs are equally critical.

4. Not all bugs are equally difficult to find.

5. No reflection of the absolute count.

It is easy to determine this measure and it can have a high significance.

22

1. ROI – Return on investment for process optimization

current process optimized process

Costs of test execution 10 000 € 20 000€

DDP 70% 90%

found bugs 700 900

bugfixing (100 € per bug) 70 000 € 90 000 €

bugs found after release 300 100

bugfixing after release (1000 € per bug) 300 000 € 100 000 €

total costs 380 000 € 210 000 €

savings: 170 000 € investment: 10 000€ ROI (savings/investment): 1700%

23

1. ROI – Return on investment in test automation

manual testing automated testing

Costs for test case design 6 000 € 6 000 €

costs for automation 0 16 000 €

total one-time expenses 6 000 € 22 000€

costs for test execution 5 000 € 1 000 €

amount of test execution cycles 3 3

costs per release 15 000 € 3 000 €

total costs of 4 releases a year 66 000 € 34 000 €

savings per year: 32 000 € investment: 16 000€ ROI (savings/investment): 200%

24

My test automation is much cooler than

yours!

25

How do you get that idea? - I have a lot of test cases - which run every night for several hours - and provide lots of test results

26

- Well, I have:

- saved more hours of manual testing (effectiveness) - spent less time on test case maintenance (maintainability)

- more bugtickets per fault report (reliability)

- less training time for new employees (usability)

27

Hm… - I need to edit many tests for each release, so that they can be executed - I need 4 hours a day to analyse my logs files - if I find a bug, it is most likely caused by my test environment - my new colleagues need 2 weeks of training before they can help me - and after every release we have to fix many bugs that were missed during testing

28

And then I have: - less effort to find the test case that tests a specific range (flexibility) - fewer test cases blocked by the same software bug (robustness) - less effort to run test cases on different test environments (portability)

29

30 30

05 | IS OUR TEST AUTOMATION COOL?

31

We have 18 test suites for:

- conversion path tests, submit poker accounts, support ticket system,

fraud check, upload images, buy points packages, video section…

- 865 performed tests

- 9 hours test run every night

However, we have two test systems (integration and system test) and almost

all tests run in Internet Explorer and Firefox.

- about 3400 test executions

- 36 hours of test runs in total

- 5 virtual machines for test execution

32

1. maintainability

central master suite with:

- object repository

- procedures for common test steps (e.g. login)

33

2. efficiency

- before the introduction of test automation:

12 hours of manual regression testing for each release and test

environment

- today:

4 hours of manual regression testing for each release and test

environment

3. reliability

solved problems with external dependencies (Facebook, Google ...)

- requests are routed via the host file on an internal web server

- answers with 1x1 pixel images or empty responses to scripts

34

4. flexibility

- a test suite per functional section

- each test suite can be started by pressing a button in Jenkins

- free choice of browser and the test environment

5. usability

- graphical abstraction of test scripts in QF-Test

- using Jython or Groovy scripts (database access, HTML)

6. robustness

- 3 test suites are currently on “failed", there are 2 bugtickets

- a test suite is unfortunately affected by a bug in Firefox

35

7. portability

- choice of the test environment and the browser via parameter

- at the moment QF-Test supports only Internet Explorer and Firefox

36

Any questions?

37

ICANS GmbH

Valentinskamp 18

20354 Hamburg

Germany

Phone: +49 40 22 63 82 9-0

Fax: +49 40 38 67 15 92

Web: www.icans-gmbh.com