© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Automated Exploratory Tests

Zbyszek Moćkun

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Automated tests evolution for web application

Aim

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Few words about process and environment

Introduction

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Methodologies, projects, …

• Agile methodologies: Scrum, Kanban

• Release management

• No time for regression tests

• Last sprint cumulation

• Continuous Integration Environment

• Web Application

• Content Management Systems (CMS)

• Small and medium projects

• No time and budget for automation

Methodologies Projects

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Standard approach to automation

Automated Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Automation Tests Lifecycle

Uruchom

Zarządzaj

Ulepszaj

Napisz skrypt automatyczny

Napisz TC

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Adventages and Disadventages

• Coverage

• Lack of duplication during manual test execution

• Functionality

• Writing automated scripts require

• Costs

• Need investments at the begining

• Dependencies between tests

• Only defined functionality is tested

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Automated tests evolution process

Automates Exploratory Tests

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

What should we tests?

Two perspective

User Client

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Aspects

Narzędzie CI

Klient

Statystyki

Użytkownik

Wygląd Funkcjonalność Szybkość

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Looking for a symptom

„Symptom is a departure from normal function or feeling which is noticed by a patient, indicating the presence of disease or

abnormality.”

http://wikipedia.org

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Symptoms

Błąd

Wygląd

Treść

Kod HTML

Błędy JavaScript

Błędna odpowiedź

serwera (404)Analiza

Statystyk Serwisu

Walidatory (W3C)

Obrazki / style

Analiza linków

Strategia URL

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Reference Symptoms

• Previous application data• Saved reference data• Other version of the same application (different server)• Compare data as parameter

Layout Content

HTML code

Performance

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Roboty

• Skrypty, które poszukują symptomów

• Ten sam skrypt dla różnych projektów

• Narzędzia/skrypty

Błędy JScript

Odpowiedź Serwera

(404)

Walidatory (HTLM, CSS)

Analiza Statystyk Serwisu

Analiza linków

Strategia URL

Wczytane obrazki/

style

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Automated Exploratory Tests – new idea

Several different definitions:• Replay exploratory tests executed manually

• Combine manuall and automated functional tests

• Chosen steps automation

• Application Testing with data generator

• There is no ATE, because exploratory tests require human factor

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Automated Exploratory Test – definition

Automated Exploratory Tests supports manual exploratory testing, don’t based on specific test scenario, looking for high risk areas, explore application quicker and deeper, additionally tests area no accessible for testers

„Automated Tests do not provide complete information. They provide partial information that might be useful”

Cem Kaner

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Automated Exploratory Tests- adventages

• One tool− Contains all symptoms

− For all projects

− Quick to introduce into project

− Easy for change management

• Don’t require good knowledge about tested application, speed up learning process

• Continuous Integration

• Strong solution if we combine with standard functional automated tests

• Migration tests

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Automated Exploratory Tests- example

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Open Source Tools

• Selenium II/WebDriver

• Comparators (ImageMagick)

• Validators (W3C)

• Performance (JMeter, Proxy)

• Proxy Servers(BrowserMob)

• Firefox/Firebug (NetExport, ConsoleExport, JS Console output redirector)

• HAR Viewer (http://www.softwareishard.com/har/viewer/)

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Q&A