introduction to automated acceptance testing

copyright (c) 2008 by Micah Martin. All rights reserved.

Introduction to Automated Acceptance

Testing

Micah Martin, 8th Light, [email protected]

mailto:[email protected]



What are Acceptance Tests?



ATs vs UTs

• Written by “Customer”

• Test Features

• Written by developers

• Test Code

Acceptance Tests Unit Tests



Criteria for Completion

• Provided by the “Customer”

• Given to developers

• Specify stories

• Criteria for Done-Done


Communication

• Forces customers, testers, and developers to work together

• creation

• implementation

• execution



FeedbackWhat are Acceptance Tests?

• Are a source of data

• A tool for project management


0102030405060708090

100

1 3 5 7 9 11

Total AT'sFailing AT'sPassing AT's

Data From Acceptance


Acceptance Tests: A Critical Piece of Agile


Analysis

Design

Implementation

DFDERD

DDST

1 May 1 Nov1 Jul 1 Sep

Traditional ApproachAcceptance Tests: A Critical Piece of Agile

Object Mentor


The Agile AproachAcceptance Tests: A Critical Piece of Agile

May 1 Jul 1 Sep 1 Nov 1

Analysis

Design

Implementation


The Agile ApproachAcceptance Tests: A Critical Piece of Agile

IterationFailing Acceptance

Tests

Passing Acceptance

Tests


The Agile ApproachAcceptance Tests: A Critical Piece of Agile

IterationFailing Acceptance

Tests

Passing Acceptance

Tests

TDD


• Every story must have at least one Acceptance Test

• A story is not DONE until it passes it’s Acceptance Tests

Criteria for DONEAcceptance Tests: A Critical Piece of Agile


Manual Acceptance TestsAcceptance Tests: A Critical Piece of Agile


Manual Acceptance TestsAcceptance Tests: A Critical Piece of Agile

Manual Acceptance Tests


Acceptance Tests Are Automated


The Button• How often would

you press it?

• When would you press it?

• Who would press it?

• Testers, Developers, Managers, Customers, Spectators, etc.



Automated Tests Require:

• High level, yet, precise language

• Unambiguous detail

• “Examples”

• Interaction with the system



Criteria for DONE


Criteria for DONE

+


Criteria for DONE

Automated+


Criteria for DONE

Automated

Executable Specification

+


Executable Specification

• A new paradigm for testing

• Puts quality first

• Removes ambiguity from requirements



Who Writes Acceptance Tests?


The Customer• Yeah right! Then who?

• The Customer Role

• Stake holder

• Business Analyst

• Quality Assurance

• Product Owner



Tests Get Technical

• The “Customer” may need technical help to write tests

• Developers are technical

• Pair test authoring



Business Rules Get Fuzzy

• Sometimes developers need help understanding tests

• Customers know business rules

• Pair test implementation



Business Decisions are Hard To Make

• Sometimes, the “customer” will be challenged to make decisions

• Stakeholders can help. Developers can help.

• Do you see a pattern of communication?



Exercise #1


The Login Test

• Write a test plan, in plain text, for the business rules of logging in.

• Web application

• User credentials are stored in relational database

• Successful login redirects to “Welcome” page

Exercise #1


Writing Good Acceptance Tests


Login Test PossibilitiesWriting Good Acceptance Tests

1. Direct browser to URL for login page




1. Enter the username ‘wallace’




1. Enter the username ‘wallace’

Build a Testable Environment First


BOCWriting Good Acceptance Tests

Build

Operate

Check



1. Add some users to the system




3. Enter a value into the username field




3. Enter a value into the username field

Be Specific


Tests are Examples

• Use concrete examples

• Specify concrete behavior

• No ambiguity allowed




1. Insert into User table values (’wallace’, ‘ilikecheeze’)



1. Insert into User table values (’wallace’, ‘ilikecheeze’)2. Open a browser to the URL http://localhost/myapp

http://localhost/myapp






1. Insert into User table values (’wallace’, ‘ilikecheeze’)2. Open a browser to the URL http://localhost/myapp

Avoid Implementation Details






Avoid Implementation Details


System

Database

Tests

DatabaseDatabase

UI


Login Test: Possible Solution

• Add user to system: (’wallace’, ‘ilikecheeze’)

• Process login with username ‘wallace’ and password ‘blah’

• Check login failed

• Process login with username ‘wallace’ and password ‘ilikecheeze’

• Check login succeeded



Tools


Commercial Tools

• WinRunner

• Silk

• Robot

Tools

Are not suitable for Acceptance Testing in an Agile environment


Open Source Options

• FIT

• FitNesse

Tools

Among the few tools that support Test Driven Development


FIT

• Framework for Integrated Tests

• Created by Ward Cunningham

• Open Source

• The most accepted solution for agile acceptance testing

Tools


FitNesse

• Environment build around FIT

• Makes everything easier

• Created by Object Mentor, Inc.

• Open Source

Tools


FIT FitNesse

- Tests written in HTML- Tests are executed on the command line- Tables are executed- Non-table markup is ignored- Tables map to Fixtures- Fixtures are code that is aware of the system- Supplies foundational Fixtures- Implementations ported to many languages

- Stand alone web server- Is a wiki- Tests written in wiki text- Tests are executed from within the wiki- Translates tests into HTML- Uses FIT to execute tests- Supports test suites- Supports variables in tests- Supports test refactoring- Written in Java- Supports FIT implementations in any language


Thinking in Tables


Only Tables ExecuteThinking in Tables

Ignored

Executed


Foundational Table Structure

Thinking in Tables

Name of Fixture

Interaction with Application

• Table structure depends on type of Fixture


3 Foundation Fixtures

• Column Fixture

• Row Fixture

• Action Fixture

Thinking in Tables


Column FixtureThinking in Tables

package eg;// Copyright (c) 2002 Cunningham & Cunningham, Inc.// Released under the terms of the GNU GeneralPublic …

import fit.ColumnFixture;

public class Division extends ColumnFixture { public float numerator; public float denominator; public float quotient() { return numerator / denominator; }}


Row FixtureThinking in Tables

package fitnesse.fixtures;

import fit.RowFixture;

public class EmployeePayRecordsRowFixture extendsRowFixture { public Object[] query() throws Exception { EmployeePayRecord[] records = newEmployeePayRecord[2]; records[0] = new EmployeePayRecord(1, 1000); records[1] = new EmployeePayRecord(2,2000); return records; } public Class getTargetClass() { return EmployeePayRecord.class; }}

public class EmployeePayRecord { public int id; private double salary; public EmployeePayRecord(int id,

double salary){ this.id = id; this.salary = salary; } public double pay() { return salary; }}

Analogous to comparing against rows in a database table


Action Fixture • Think GUI window

Thinking in Tables

Counter WindowCounter:

Counter: 6 Count

public class CountFixture extends Fixture { private int counter = 0; public void count() { counter++; } public int counter() { return counter; }}


Exercise #2


Login With Tables

• Write the Login test using tables

• Keep in mind what we learned from the previous exercise

• BOC

• Examples

• Avoid Implementation Details

Exercise #2


Possible SolutionExercise #2


FitLibrary

• Extension to FIT

• Written by Rick Mugridge

• Adds some handy Fixtures


FitLibrary Fixtures• ArrayFixture for ordered lists

• SetFixture for unordered lists

• Supports

• Graphics

• Tree structures

• Nested Tables

FitLibrary


Text

DoFixtureFitLibrary

• Broken tables

• Highly readable

• Flexibility


Wiki


What is it?• A collaborative web site

• Editable by any

• Created by Ward Cunningham

• Every project should have one

• http://c2.com/wiki

• http://en.wikipedia.com

Wiki

http://c2.com/wiki

http://c2.com/wiki

http://en.wikipedia.com

http://en.wikipedia.com


Creating Tests•Use Wiki syntax to create a page with test tables•Label the page as a Test Page

•Use a page name of the form Test…•Turn on the Test property

•Make sure your Fixtures are in the classpath•Use !path widget

•Mechanics•!path values are concatenated•Java command to start FitServer is executed•Testable HTML is passed to FitServer•FitServer runs the tests•Results are passed back to FitNesse

Wiki


Creating Suites

•Set the Suite property•Create a page with the Suite property•Created test pages inside this page•When the suite is executed, all child test pages will

be included in the suite execution

•Use the !see widget•!see <name of test page>•All “included” tests pages will be included in the

suite execution

Wiki

There are 2 ways to make Suites

Run a Suite by clicking the Suite button


New Advancements


SlimNew Advancements

• New Extension to FitNesse

• Alternative to FIT

• More Flexible Integration

• Written by Uncle Bob


RSpec/CucumberNew Advancements

• BDD version of Acceptance Testing

• Pure Text

• Very Readable (Given, When, Then)

• Dan North, David Chelimsky, Aslak Hellesoy

• Works in Java (JRuby), .NET (Iron Ruby), C (Ruby)


The End.Micah Martin, 8th Light, Inc.

[email protected]

http://www.fitnesse.org http://www.8thlight.com



http://www.objectmentor.com




introduction to automated acceptance testing

Documents