get the balance right: acceptance test driven development, gui automation and exploratory testing

GET THE BALANCE RIGHT

ACCEPTANCE TEST DRIVEN DEVELOPMENT, GUI AUTOMATION AND EXPLORATORY TESTING

ATDD GUI

ET

MICHAEL LARSEN

Senior Tester, Sidereel.com

Chair, Education Special Interest Group, Association For Software Testing

Black Belt, Miagi-do School of Software Testing

Founder/Facilitator, Weekend Testing Americas

TESTHEAD: http://mkltesthead.com

Twitter: @mkltesthead

WHAT’S HAPPENED SINCE 2001?

Eleven years since the Agile Manifesto

Test Driven Development (TDD) has emerged as a standard to guide code development driven by tests.

Acceptance Test Driven Development (ATDD) extends model.

Proliferation of automated testing tools for a variety of platforms and needs.

DO WE STILL NEED TESTERS?

NO: Testing is dead. With the development of these tools, developers now do the testing earlier and up front [Savioa, 2011].

YES: Apply reason and dynamic thought to problems. Thinking, actively engaged testers are needed now more than ever [Tomlinson, 2011].

TDD, WHAT IS IT?

ATDD, WHAT IS IT?

Places the focus on User Stories from the expectations of the customers.  

Is geared towards the members of the team that are not necessarily programmers.

Uses tools that allow a more natural language to present the requirements.

ATDDScenario: Twitter user signs in  Given I connect to twitter from the sign in page  When I log into twitter with good credentials  Then I go to the user private profile page  And I can confirm my twitter credentials

Expressed in Gherkin, the language used with Cucumber.

Tied to actual coded methods to make the process work.

TDD AND ATDD ARE NOT TESTING!

TDD and ATDD are design processes.

Help develop clean, well functioning code.

Help ensure software being developed meets needs of the project

WHY ISN’T THIS TESTING?

Testing:

“The art and the act of asking questions of a product, then developing & devising new and more inventive questions based on the answers we receive.”

WHERE DO TDD AND ATDD EXCEL?

Disciplined approach to developing cleaner software.

Act as a brake on "cowboy code".

New programmers can see code in context with the tests that have been written.  

WHERE DO TDD AND ATDD EXCEL?

TDD and ATDD work well with Continuous Integration.

Keeps issues based on dependencies to a minimum.  

Programmers focus on "just enough design" and "just enough development".

Entire product team does design and development work.

WHERE DO TDD AND ATDD FALL SHORT?

Unrealistic expectations.

Only be as good as the overall skill of the developer(s).

Can be over applied.

WHAT IS FRONT END GUI AUTOMATION?

Proliferation of tools that help simulate the actions of users.

Follow standard workflows

• (logging in, navigating to pages, clicking on links, filling in forms, etc.).

Tools range from lightweight apps like Texter up to full feature development suites (Selenium/WebDriver, FitNesse, TestComplete, etc.).

HOW DOES GUI AUTOMATION DIFFER FROM TDD/ATDD?

Helps with constructing acceptance test cases

Helps to check and demonstrate the acceptance criteria has been met.

GUI AUTOMATION

Remember our original example:

GUI AUTOMATION

Cucumber statements = abstraction of real code.

Map to appropriate tools that make statements work on the application or browser.

GUI AUTOMATION

The code below is Ruby, using Capybara to make calls to the browser:

GUI AUTOMATION

Step Definitions -> Capybara and Ruby

WHERE DOES GUI AUTOMATION EXCEL?

Tremendous blessing when dealing with repetitious steps (set up and take down).

Important part of Acceptance Test verification.

Semi proxy for human interaction.

GUI AUTOMATION DEFICIENCIES

GUI tends to be the most fragile layer of an application.

Tremendous overhead of script maintenance.

Not good at checking context.

WHAT IS EXPLORATORY TESTING?

WHAT IS EXPLORATORY TESTING?

“Scientific thinking in real-time” [Bach, 2012]

Puts test design and test execution together.

The tests that we did before inform the tests that we will do now. Those tests will inform the tests we will perform later.

EXPLORATORY TESTING

Concepts are defined, but they are not completely predefined or run in a rigid sequence.

Allows for a development of test ideas that are interesting.

Example: Automated tests can be made exploratory by randomizing order.

Purpose: Discover where dependencies or state conditions do exist.

EXPLORATORY TESTING VS. AUTOMATED TESTING

Instead of saying "automated testing", I prefer to use the term "computer aided testing".

Exploratory Testing can, and often does, use elements of computer aided testing to help accomplish its goals.

AUTOMATION CAN’S AND CANNOT’S Automation Can: Automation Cannot:

Generate test data to be used in forms or as variable values.

Create curiosity.

Parse the output of a program and use it as input for another program.

Make sapient (actively thinking) decisions.

Create a log of actions and transactions.

Invent a new idea or an approach based on the output of a sequence of tests.

Alert if an assertion is met or not met.

Notice something unexpected (unless we have already defined what is unexpected).

Search for patterns in output, or help to reveal patterns we did not know about.

Make a judgment call as to the value or importance of a piece of functionality.

WHERE DOES EXPLORATORY TESTING EXCEL?

Allows the tester to choose the sequence of steps that can provide answers about the state or condition of a product.

Imagination and creativity open up possibilities.

When there is little in the way of formalized documentation.

EXPLORING REQUIREMENTS

Remember this acceptance test?

Scenario: Twitter user signs in through Twitter

Given I connect to twitter from the sign in page

When I log into twitter with good credentials

Then I go to the user private profile page

And I can confirm my twitter credentials

SAMPLE EXPLORATION QUESTIONS

Is there a login interface?

Does it provide for proper error handling if I can't log in?

Does it give me feedback to let me know that I have successfully logged in?

Can I trick the system into letting me log in with improper credentials?

Does it present me with information that could help me guess a login without actually having one?

Where else could I use this functionality?

EXPLORATORY TESTING IS PART OF TDD/ATDD

Proposing a failing test first, and then creating code that meets the acceptance criteria

Programmer must consider what the application needs to do

Determine which avenue(s) to implement the feature(s).

BENEFITS OF EXPLORATORY TESTING

Walking down various paths

Jumping off point to look in other places

Simple Cucumber statement "And let me see that":

And /^let me see that$/ do

puts "PAUSED - Press Enter to continue: #{$1}\a\a\a"

puts ""

$stdin.gets

End

EXPLORATORY TESTING’S DEFICIENCIES

Less likely to find new revelations by going over the same ground (pesticide paradox)

Not enough time to go through all possibilities.

Sentient humans do it. Sentient humans get bored.

• SBTM techniques can help

CAN WE CREATE A BALANCE?

TDD, ATDD, and front end GUI automation are processes.

Exploratory Testing is a process and a mindset.

Developing and defining acceptance is exploratory.

Front End GUI Automation starts from capturing discoveries made while examining requirements.

THE ROLE OF TESTERS

”Testing is dead" == testing needed to be a part of an entire process of quality and improvement, starting at the earliest stages of design and development.

Exploration, automation, and testing must come earlier in the process, and involve not just testers, but everyone on the product team.

STRIKING A BALANCE

Can we leverage the "testing everywhere" idea, and help develop a balance for all of these goals?

Can we see them as interdependent, and not as separate and standalone activities?

TDD AND ATDD TAKE AN EXPLORATORY MINDSET

• Testing is part of the initial development effort.• Testing comes first. • Putting the emphasis on TDD/ATDD makes sure that

it meets the criteria it is being designed for.

Testers can also help guide and provide consideration for testability in areas the programmers might not consider.

Asking "what if" questions vital at this stage

FOCUS ON DELIVERING BUSINESS VALUE

Question 1: "is what I am doing helping to deliver real value to my customers"?

If we put the business value first, that will guide us in our testing efforts and the approaches that we use [Crispin, 2008].

AUTOMATION DOES NOT HAVE TO BE PERMANENT

Make room for "throwaway automation”; steps that can be used to help explore different avenues.

Freight or passenger trains vs. taxi cabs.

• Freight and passenger trains limited to the rails the trains can ride on.

• Taxi cab can be called to appear anywhere and take us anywhere.

EXPLORATION IS A MINDSET

Exploratory Testing is a way of thinking about questions and answers.

Exploratory Testing does not live outside of automated testing.

Automation and Computer Aided Testing can be central to exploration.

ZEN MIND, BEGINNERS MIND

Domain Expert Level Exploratory Testing

Novice Level Exploratory Testing

They are different.

Source of mental friction is knowing the eccentricities of your application and when they can get in the way of effective testing and exploration.

USE PERSONAS TO VISUALIZE GOALS

Put yourself into the shoes of as many potential customers of our product as I can.

That means understanding the interests, desires and ways of interaction of many different groups of people.

The ways that they interact with the devices they use to discover content/interact is very different.

SESSION BASED TECHNIQUES

• Developing small charters and executing on them in set periods of time.

• Focus testing areas that are most important.

• Focus on a small sub system.

Keep our testing fresh, our eyes and minds focused, work on what's most important.

CONCLUSION

TDD focuses on writing correct and clean code.

ATDD focuses on making sure the functionality being delivered makes good on the goals and promises made to our customers.

Automation happens at many levels.

Exploration happens at all of these levels, and is an approach and a mindset, not a methodology.

Working together, the odds of quality software are much more heavily in our favor.

QUESTIONS

???