automated exploratory tests

32
© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only. Automated Exploratory Tests for Web Application Zbyszek Moćkun

Upload: zbyszek-mockun

Post on 25-May-2015

1.816 views

Category:

Technology


0 download

DESCRIPTION

automated exploratory test tools for web application. Make your test agile. Do not overpay for your automation. Test all aspects (performance, layout, analytic) not only concentrate on one specific thing.

TRANSCRIPT

Page 1: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Automated Exploratory Tests for Web Application

Zbyszek Moćkun

Page 2: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

How Cognifide has developed automated

tests and where we are going

Objective

Page 3: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

In short about process and environment

Introduction

Page 4: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Methodologies, projects …

• Agile: SCRUM, Kanban

• Frequently releases

• No time for manual

regression

• Lot of work at the end of

sprint

• CI Environment

• Web application

• Migration

• Content Management

Systems (CMS)

• Short or medium term

projects

• Automation investment, no

time for money return

Methodologies Projects

Page 5: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Functional tests

Common standards for automation

Page 6: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Test Automation Lifecycle

Run

Manage

Update

Write automated

script

Write TC

Page 7: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Advantages and disadvantages

• Coverage by

automated tests

• No duplications during

regression test

• Functionality is tested

• Time consuming

• Cost

• Invest and wait for return

• Test dependency

• Only coded functionality is

tested

Page 8: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Can you believe your tests?

• Incorrect Xpath (locator) used in tests

• Not all data were verified (unable to write verifications for all

data)

• CSS/images or other files wasn’t load

• Frequently design changes require xpath (locator) update

• Incorrect CSS fixes (designs are not reflected by page

layout)

• Analytics doesn’t work

• Performance decrease

• Few thousands of pages (components variation – can’t test

all of them

Page 9: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Migration (upgrade) projects

• No existing automated test for regression test

• Few thousands of pages

• Quick response needed

• Iterative development

Page 10: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

New approach, theory part

Automated Exploratory Tests

Page 11: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

We need automated tests

because ...

Source: http://www.testfacebook.com

Page 12: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

What should we test?

Two perspectives:

User Client

Page 13: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Aspects

CI Environment

Client

Analytics

User

Layout Functionality Performance

Page 14: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Looking for a symptom

„Symptom is a departure from normal function or feeling

which is noticed by a patient, indicating the presence of

disease or abnormality. A symptom is subjective,

observed by the patient, and not measured.”

http://wikipedia.org

Page 15: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Symptoms

Bug

Layout

Content

HTML code

JavaScript errors

Server response

(404)

Analytics

Validators (W3C)

Link checker

URL strategy

Performance

Page 16: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Oracle Heuristics

It is used by comparing the output(s) of the system under test,

for a given test case input, to the outputs that the oracle

determines that product should have.

− other products (second program that uses a different algorithm to

evaluate the same mathematical expression as the product

under test)

− a consistency oracle that compares the results of one test

execution to another for similarity,

− a model-based oracle that uses the same model to generate and

verify system behavior,

− or a human being's judgment (i.e. does the program "seem" to

the user to do the correct thing?)

Page 17: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Oracle Heuristic for GUI

Page 18: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Reference symptoms

• Previous version of application

• Reference data saved on disc

• Other version of the same application (different servers)

• The same script takes pattern and compare

• One tool for different projects

Layout Content HTML code Performance monitoring

Page 19: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Robots

• Scripts that are looking for symptoms

• The same script for all projects

• Open Source tools

JavaScript errors

Server response (404)

Validators (HTLM, CSS)

Analytics Link checker URL Strategy

Page 20: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Exploratory testing

„Exploratory testing is an approach to software testing

that is concisely described as simultaneous learning, test

design, test execution and test result interpretation”

Cem Kaner

Page 21: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

What are Automated Exploratory

Tests?

Meny different definition:

• Record and play manual exploratory test sessions

• Combined manual and automated tests

• Test steps automation

• Blackbox tests with input/output data generator

• Not at all possible due to human factor

Page 22: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

What are Automated Exploratory

Tests?

„Tests supporting exploratory testing, not necessarily based on

the test scenario, searching areas where there may be errors

or a change in the application, allowing the examination of

applications faster and more deeply or not to test sites

available to man.”

Zbyszek Moćkun

„Automated Tests do not provide complete information. They

provide partial information that might be useful”

Cem Kaner

Page 23: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Automated Exploratory Tests -

advantages

• One tool

− Gather all symptoms

− Use in all projects

− Quick to introduce to project (half a day)

− Resistant to appliciaton changes

• Do not require deep knowledge about application,

support the learning process

• Integrated with Continuous Intergration env

• Strong solution if combined with automated

functional tests

• Migration or upgrade tests

• Iterative development

Page 24: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Track the propagation of

changes in the code - See what pages/components are infected by change

- Positive – false (code change affect all necessary places?)

Page 25: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Practical part

Automated Exploratory Tests

Page 26: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Report – example of use

Page 27: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

URL management

• Sitemap

• Link crawler

• CSV file with url list

• Symptoms as parameter (on, off)

• Comments (bug id, ...)

• Hide element

• Other actions

• Keep in SVN

Page 28: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Pattern management

• Easy to use (overwrite pattern)

• Checkbox (check all)

• Commit to svn

Page 29: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Dynamic content

• Reference symptoms – need processing

• Use regexp

• Hide element

• Write generic function

• Robots – do not need any additional actions

• Dynamic components (counters, date, random numbers)

• Flash/video files

• Embedded content (twitter, facebook,...)

• Require functional test

• Static components – about 75%

Page 30: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Mobile version

• Required (not nice to have), can’t avoid testing

• Mobile pages are more static than desktop version

• Do not use flash

• Not all symptoms are easy to check

Page 31: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Narzędzia

• Selenium II/WebDriver

• Komparatory (ImageMagick, Pretty Diff)

• Walidatory (W3C)

• Performance (JMeter, Proxy)

• Serwery Proxy (BrowserMob)

• Firefox/Firebug (NetExport, ConsoleExport, JS Console

output redirector)

• HAR Viewer (http://www.softwareishard.com/har/viewer/)

Page 32: Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only. © 2010 Cognifide Limited. In commercial confidence only.

Q&A