collaboration between system stakeholders through cucumbercollaboration between system stakeholders...

Collaboration between system stakeholders

through Cucumber

Roger B. Jones

AGILE Collaboration

• Northrop Grumman Private/Proprietary Level I

This document does not contain technical data controlled under U.S. Export Regulations. Copyright

© 2015 Northrop Grumman Systems Corporation. All rights reserved. Approved for Public Release

#15-1885; Unlimited Distribution

This document does not contain technical data controlled under U.S. Export Regulations. Copyright © 2015 Northrop Grumman Systems Corporation. All rights reserved. Approved for Public Release #15-1885; Unlimited

Distribution

Why Cucumber?

Ever been on this Project?2


Distribution

Why Cucumber?

With this Team?3


Distribution

Why Cucumber?

For this Customer?4


Distribution

Why Cucumber?

And gotten this as your System?5


Distribution

Cucumber, Magic?

6

The Boss will not all of a sudden be able to write Automated Tests

•Tim the Magician chooses


Distribution

What I am here to Talk About

• What is Cucumber?

• How does Cucumber Work?

• Why should Cucumber be Used?

What? How? Why?7


Distribution

What is Cucumber?

• Open Source Testing Tool

• Available at https://cucumber.io

• Supports Behavior-Driven Development (BDD)

• Uses the Gherkin language

• Runs in parallel with a Java application (Eclipse, Maven, Junit and one-jar executable)

Cucumber is Open Source8


Distribution

What is Cucumber?

• Cucumber lets software development teams describe how softwareshould behave in plain text

• The text is written in a business-readable domain-specificlanguage and serves as documentation, automated tests and

development-aid - all rolled into one format

• Cucumber works with Java, Ruby, JavaScript, C++, C# (SpecFlow), PHP (Behat), Python (behave)

• It has been translated to over 60 languages

Cucumber is just Plain Text9


Distribution

What is Cucumber?

• Behavior-Driven Development

• Automated Acceptance Tests

• Ubiquitous Language

• Living Documentation

Cucumber has many Benefits10


Distribution

Cucumber Continuum

11

How does Cucumber Work?


Distribution

The Cucumber Stack – Business Facing

@bunny @daily

Feature: Test the capability for a Trojan bunny to Transition

Knights from Outside to Inside the French Castle

In order to go home to Camelot

As a tired English soldier

I want to end the war quickly

12

Cucumber Starts with Gherkin Plain Text

These Tags Control Test Execution

Each Gherkin begins with the Feature keyword.

Anything can go here, but it should be a Feature Injection:In order to <meet some goal>As a <type of stakeholder>I want <a feature>

A feature injection communicates motivation: Why are we doing this?


Distribution

The Cucumber Stack – Technology Facing

• Cucumber

– Matches the natural language strings

– With a line of code that you want to runWhen the bunny operator transitions the role

| Property | Value | Type |

| MBL | Disabled | string |

– helps out by generating support code snippets

@When("^the bunny operator transitions the role$")

public void the_bunny_operator_transitions_the_role(DataTable arg1)

throws Throwable {

// Express the Regexp above with the code you wish you had

// For automatic conversion, change DataTable to List<YourType>

throw new PendingException();

}

14

Java can read Gherkin

Support code written in Java, Ruby or C++


Distribution

The Cucumber Stack – Technology Facing

Slide 2

• A software test engineer fills in the snippet@When("^the bunny operator transitions the role$")

public void the_bunny_operator_transitions_the_role(DataTable arg1)

throws Throwable {

Map<String, Object> map = JsonAdaptor.convertDataTableToJson(arg3);

String request = JsonUtil.toJson(map);

this.url = "https://localhost:8443/webGUI/jsonapi/";

int responseCode = 0;

responseCode = driver.put(this.url, request);

TestCase.assertTrue("Bad Put Response Code " + responseCode,

(responseCode == 200 || responseCode == 202));

15

A software Test Engineer fills in the Snippet

Cucumber Assert

DataTabletransformed to JavaScript Object Notation

Interface to Web Server


Distribution

Successful Cucumber Run

Cucumber options = features --tags @bunny @daily

Run Time Options Glue [classpath:apps/acceptanceApp]

Feature: Remove Knights from Outside the Castle

!!!!!! Device Test !!!!!!

!!!!!! System State !!!!!!

!!!!!! Role Transition Test !!!!!!

!!!!!! Notification !!!!!!

!!!!!! Alerts !!!!!!




!!!!!! Notification !!!!!!

!!!!!! Alerts !!!!!!

2 scenarios (2 passed)

14 steps (14 passed)

4m33.234s

16

Cucumber Success!

Cucumber reports test time. Long tests do not have to be run daily


Distribution

Unsuccessful Cucumber Run

Cucumber options = features --tags @bunny @daily

Run Time Options Glue [classpath:apps/acceptanceApp]

Feature: Remove Knights from Outside the Castle




!!!!!! Alerts !!!!!!

2 scenarios (0 passed)

14 steps (2 passed, 12 skipped)

0m34.102s

java.io.IOException: HTTP Response 500

at lncs.ui.common.json.JsonDriver.get(JsonDriver.java:146)

at apps.acceptanceApp.CucDeviceTest.the_system_state(CucDeviceTest.java:46)

at (acceptanceApp/transition/transitionMbl.feature:28)

17

Yikes! Someone has Introduced a Regression

Java Test Code failure line number


Distribution

Another Cucumber JVM Scenario Outline

@googleSearch

Feature: Google Searches

In order to learn the sikuli slides interface to cucumber JVM I want to

Use Google to search for different subjects and give different results

Scenario Outline: Search for a few things at www.google.com

Given I have opened google as my search engine

When I enter "<searchString>"

Then I should see "<powerPointFile>" "<helloBlurb>"

Examples:

| searchString powerPointFile | helloBlurb |

| amazon.com | amazon.pptx | Hello Amazon! |

| sikuli slides | sikuliSlides.pptx | Hello Sikuli Slides! |

| cucumber jvm | cucumberJvm.pptx | Hello Cucumber JVM! |

18

Yet Another Cucumber Feature Example

These are variables “<>”

These are variable names

Each line is a record for a scenario instance


Distribution

Why Cucumber?

• Cucumber makes Testing more Efficient and Interesting

• Cucumber is a Communication Tool

• Cucumber is a Collaboration Tool

19

Cucumber’s big secret: the tests and documentation are just a happy side effect


Distribution

Why Cucumber?

• Testing is important because software reliability is defined using testing

• Approximately fifty percent of the software development budget for software projects is spent on testing. 1

• In 2002, AMR Research reports, software defects cost business $60 billion to fix, and 50 percent of software costs were related to finding and fixing those defects. 2

1 - David Crowther, Peter Clarke, Examining Software Testing Tools, Dr. Dobb’s Journal: Software Tools for the Professional Programmer, ISSN# 1044789X, Academic Search Premier, June 2005, Vol. 30, Issue 6.

2 - http://www.cognizant.com/InsightsWhitepapers/Testing_survey.pdf. ©2005, Cognizant Technology Solutions US Corporation, 500 Glenpointe Centre West, Teaneck, NJ 07666, PH: 201-801-0233

20

Lack of Automated Testing is Expensive


Distribution

Testing Using an Ubiquitous Language

21

Cucumber is a Test Tool


Distribution

Cucumber as a Test Tool

• Cucumber tests can run alongside of the Operational System to

provide a predictable and controllable test state

• Operators will not have to input into the System’s GUI, and can

view the results of the test in real time on the System’s GUI. (SikuliSlides)

• Cucumber tests can be used by software developers to providerepetitive input for integration test conditions

22

Cucumber makes Testing more Efficient


Distribution

Cucumber as a Test Tool

• Cucumber tests can be integrated into a Continuous Integration job

stream

• Cucumber can run many of the repetitive and technically boring regression tests that human testers now run

• Freeing human testers to do more interesting work

23

Cucumber makes Testing more Interesting


Distribution

Cucumber is good at Regression Tests

• An example of a repetitive and technically boring regression test is a

menu of fields that control/configure a control menu

• The bunny control menu has ten (10) fields, some are enumerations,

some integers, some octal, some are range, some double, and

some are strings.

• Each time the software is built:

– A human tester enters low, high and sample values into each of the ten fields.

– The human tester presses apply

– And waits for the device to return its state

– The human verifies that the control state is as expected

24

Cucumber can run this Sequence Faster


Distribution

Cucumber Documents the

Testing

• What happens if the driver side wheel inflation is not set to the

same value as the passenger side wheel inflation?

• Were all twenty-five (25) values of the bunny key tested?

• Was this combination of field values used in the test?

• Does our HMI allow octal value 77 to be input as the Primary Bunny Number?

• What happens when a duplicate value is set in the Knight ID List?

• What values were tested for the driver side wheel inflation?

25

Cucumber Documents the Testing


Distribution

A Vision of Automated Testing after each

Continuous Integration (CI) Build

1. A developer delivers code changes through ‘git push’

2. Jenkins (A CI tool) builds the code into the system

3. Jenkins runs unit tests

4. Jenkins installs and starts the following using the latest system build:

a. Two (2) Bunny Simulation Systems

b. Two (2) English to French Forwarding systems

c. Nine (9) English Soldier Simulators

d. Four (4) Faster Coconut Horse Simulators

e. One (1) King system controlling two Knights

5. Jenkins runs automated acceptance tests as defined by Cucumber

6. Jenkins reports success/non-success of automated acceptance tests

26

A Cucumber Testing Vision


Distribution

Cucumber is good at setting up and running

metrics

• Measures of Effectiveness (MOEs) reflect operational performance requirements which are generally derived from the ORD or the CONOPS

• Measures of Performance (MOPs) reflect system requirements. They characterize physical or functional attributes relating to the execution of the mission or function

• Technical Performance Measurements (TPMs) are product metrics that track design progress toward meeting customer performance requirements

• An example of a MOP requirement is:

– The software shall not cause the host to exceed defined CRU maximums for CPU, memory usage, data I/O channel throughput, and data storage

27

Cucumber makes Testing more Rigorous


Distribution

A Metric test that Cucumber can Setup, Run

and Measure

• Wheel Capacity

– Given that the bunny has a fixed amount of wheels

– When an bunny operator configures and activates all the wheels on the bunny

– Then all the wheels on the bunny are configured and activated

– And the bunny system CPU usage does not exceed 50 percent

28

Cucumber makes Testing more Measurable


Distribution

Cucumber Forces Engineers to Work with Others

• Cucumber might seem like a testing tool, but its really a collaboration tool

• If you make a genuine effort to write features that work as documentation for the nontechnical stakeholders on your team

– You’ll find you are forced to talk with them about details that you might never have otherwise made the time to talk about

– Those conversations reveal insights about their understanding of the problem; insights that will help you build a better solution than you would have otherwise

• This is Cucumber’s big secret: the tests and documentation are just a happy side effect; the real value lies in the knowledge you discover during those conversations

The Cucumber for Java Book: Behaviour-Driven Development for Testers and Developers Paperback March 25, 2015, by Seb Rose (Author), Matt Wynne (Author), Aslak Hellesoy (Author), ISBN-13: 978-1941222294 ISBN-10: 1941222293 Edition: 1st, page 126

29

Cucumber is an AGILE Collaboration Tool


Distribution

Cucumber – Single Source of Truth

30

Cucumber Can Change How We Provide Solutions


Distribution

We need Cucumber?

• A software developer is going out for a stroll one evening. His wife

asks him to swing by the store and pick up a gallon of milk, and ifthey had eggs, to get a dozen. He returned with twelve gallons of milk and said "They had eggs."

• There are only two hard things in computer science: cache

invalidation, naming things, and off-by-one errors

• Eight bytes walk into a bar. The bartender asks, “Can I get you

anything?” “Yeah,” reply the bytes. “Make us a double.”

• Why do programmers always mix up Halloween and

Christmas? (BOOL Because = (Oct 31 == Dec 25));

31

Humor is based on Cognitive Dissonance, Systems shouldn’t be!


Distribution

Cucumber Establishes an Ubiquitous Language

32

Cucumber is a Communication Tool


Distribution

Cucumber is Plain Text

• Cucumber is a tool for running automated tests written in plain language

• Because they're written in plain text, they can be read by anyone on the team

• Because they can be read by anyone, you can use them to help

improve communication, collaboration and trust on your team

33

Cucumber is a Collaboration Tool


Distribution

Cucumber Conforms to The Dude’s Law

34

As Why increases and How decreases …

The Dude’s Law:

Value = Why / How


Distribution

Cucumber Improves

Communication- Feature Injection

Exercise

35


Distribution

Top Level Feature Injection

• In order to go home to Penelope

• As a tired Greek soldier

• I want to end the war quickly

– In order to <meet some goal>

– As a <type of stakeholder>

– I want <a feature>

36

The First Principle and Foundation


Distribution

Trojan bunny High Level Features

• These features are too big!

– Attractive to the French

– Mobile

• Pushed by English

• Pushed by French

– Contains soldiers and armaments

– Surprises the French inside the Castle

37

A Feature is the smallest deliverable component of a system


Distribution

Trojan bunny High Level Features

• Use Feature Injections to discover how to break these down to a

lower level

– Attractive to the French

– Mobile

• Pushed by English

• Pushed by French

– Contains soldiers and armaments

– Surprises the French Inside the Castle

38

Write a Feature Injection for One of these Large Features. Winner receives a Prize!


Distribution

Feature Attractive to the French

• In order to <meet some goal>

• As a <type of stakeholder>

• I want <a feature>

39

Write a Feature Injection for one of these Large Features. Winner receives a Prize!


Distribution

Feature Mobile




40



Distribution

Feature Contains Soldiers Armaments




41



Distribution

Surprises the French Inside the Castle




42



Distribution

My ANSWERS

43


Distribution

Feature Attractive to the French

• In order to get the French to move the structure inside the castle

• As a software engineer

• I want the structure to be a bunny form

----------------------------------------------------------

• In order to get the French to move the structure inside the castle

• As a user experience engineer

• I want the structure to be a Jerry Lewis form

44

What was your answer?


Distribution

Feature Mobile

• In order to move the structure to the castle

• As a transportation engineer

• I want the structure to have wheels

---------------------------------------------------

• In order to move the structure inside the castle

• As a French soldier

• I want the structure to weigh less than 2000 lbs.

45



Distribution

Feature Contains Soldiers Armaments

• In order to conquer the French castle

• As a military strategist

• I want the structure to contain soldiers and armaments

----------------------------------------------------

• In order to conquer the French castle

• As a quartermaster

• I want the structure to contain enough soldiers and armaments

46



Distribution

Feature Surprise the French Inside the Castle

• In order to surprise the French

• As a mechanical engineer

• I want a sound deadening interior to reduce any sound to 5 decibels

-----------------------------------------------------

• In order to surprise the French

• As the program manager

• I want to schedule an event to enter the structure before it arrives in

the castle

47



Distribution

Cucumber Sources “Google It”

• The Cucumber for Java Book: Behaviour-Driven Development for Testers and Developers Paperback March 25, 2015, by Seb Rose (Author), Matt Wynne (Author), Aslak Hellesoy (Author), ISBN-13: 978-1941222294 ISBN-10: 1941222293 Edition: 1st

• Cucumber Recipes: Automate Anything with BDD Tools and Techniques Paperback February 22, 2013, by Ian Dees (Author), Matt Wynne (Author), Aslak Hellesoy (Author), ISBN-13: 978-1937785017 ISBN-10: 1937785017 Edition: 1st

• https://cucumber.io/

• http://www.sikulix.com/

• http://slides.sikuli.org/

• https://github.com/ralphkjones/CucumberSikuliSlides

48

For more information “Google It”


Distribution

Cucumber Continuum

49

Cucumber Can Change How We Provide Solutions


Distribution

Questions

50


Distribution

Backup

51


Distribution

Cucumber JVM Execution Sequence

52

Cucumber JVM Runs in Maven

MAVEN reads the Feature files


Distribution

Cucumber JVM Execution Sequence Slide 2

53

Cucumber JVM uses Annotations and Reflection

Application Support Library is the API to the SUT


Distribution

Cucumber JVM Execution Sequence Slide 3

54

Cucumber JVM Test can Run in CI

Cucumber reports the failures

This document does not contain technical data controlled under U.S. Export Regulations. Copyright © 2015 Northrop Grumman Systems Corporation. All rights reserved. Approved for Public Release #15-1885;

Unlimited Distribution

55