Henrik Emilsson, Qamcom Research & Technology

Crafting Our Own Models of Software

Quality

www.eurostarconferences.com

@esconfs #esconfs

Crafting Our Own Models of Software Quality

Henrik Emilsson

EuroSTAR 2012

Qamcom Research & Technology AB | www.qamcom.com | +46 31 721 1730

Abstract

In late 2010, thetesteye.com published a poster called Software Quality

Characteristics which was the result of trying to invent the best model of

quality characteristics. When we started this, there was only one realistic and

thorough model available – Bach’s CRUSSPIC STMPL – but we didn’t think

that this model was perfect for us. Instead we attacked this model and started

to question it in order to come up with a model that we thought were more

true and valid for us and in our context. In this talk I will describe what we did

during this interesting journey; a journey perhaps more important than the

result itself.

Snippet from http://literature.puertoricosupplier.com/023/CN22511.pdf

A short introduction to the poster…

Background & Context

"All models are wrong, but some are useful” (George E.P. Box)

If this is true, why aren't more models challenged? (Including Kaner's and

Bach's!)

You can create a model that is better in your context

Background & Context

The Test Eye – www.thetesteye.com

A collaborative blog about testing

Started in February, 2008, by Henrik Emilsson, Martin Jansson and Rikard Edgren

Later, Torbjörn Ryber and Henrik Andersson joined as contributors

Henrik, Martin & Rikard had been coworkers for more than 6 years

We discussed testing 24/7 (well, something like that…)

The blog became a way of keeping in touch when we split to three different

companies

Background & Context

Since 2007 we have all worked on different companies

Henrik as consultant

Rikard on a software company

Martin on a software/hardware company

We all used different quality models in different ways, e.g.

Henrik used information objectives and quality objectives in his work as a driver for

test planning

Rikard used Bach’s CRUSSPIC STMPL when writing and reviewing test specifications

By using it we identified difficulties and opportunities

The Journey

In 2008 Rikard used Bach’s CRUSSPIC STMPL; especially for detailed

functionality

It is a great model that inspires you to think for yourself

It’s not complete

Not too specific

Vague

Apply it in your context

Some areas were elaborated; some were on a too high level for us

Other models were too focused on metrics

Insights

We wanted more elaborations, Bach’s model vs. our model:

The Journey

Rikard started to extend Bach’s model – an extension written on paper

He then asked for help on extending the model;

Henrik and Martin joined

Rikard wanted to have the model on one page;

so Henrik and Martin did their best to extend it to two pages…

STMPL was not included from the start

Insights

Limit your model

Forced to focus on importance

Need to decide between merging or separating

We limited our model to one page, then when we discovered enough

important things that it became extended to two pages

We played with different dimensions in order to not break the boundary:

Font and font size

Formatting

Sentence lengths

Margins

By limiting the model and doing the above, we also thought more of the

meaning by examining synonyms

The Journey

Henrik added a new category “Charisma”

(This is sometimes referred to as Desirability but we did not know of that)

STMPL went back in after some hot discussions

Installability became IT-bility because there are more important things than

installation; we thought of quality characteristics an IT-department might value

The Journey

During the work with the model, we did a mix of

reading

thinking

using

discussing

Insights

Mix theoretical and empirical research

Theoretical research

Investigate previous models

Learn what is written about a subject e.g. usability

Look up definitions and the meaning of words and expressions

Empirical research

During your regular work, use what you learn and try to formulate quality

characteristics

Invent something by experiencing for real

If you find something important, see if it fits somewhere in the model

(e.g. Clarity came from health care project)

A mix of these methods over time was very good

The Journey

During the work with the model we used it with clients in their context

Revealed new categories

Improved existing categories

We discovered new categories by really thinking about the existing ones

applied in our context

Our Mnemonic became CRUCSPIC STMP

Scalability Under Performance

Localization Under Portability

Charisma has its own mnemonic: U SPACEHEADS

Insights

Understanding in your own context

Understanding in your own language

Nationality

Context (business language etc)

Understanding by applying it

There is always a better version of the model; not more true, but more useful

In order to be useful you need to be generic but specific in wordings

The Journey

In the middle of 2010, the model felt good enough

We wanted to incorporate James Bach and present this as an extended

CRUSSPIC STMPL

However, we decided to create our own model as much as possible;

we decided to rephrase all items on the list

Insights

It was actually good that we decided to create our own

This way we got rid of legacy words that we hadn’t written; instead we had to

come up with our own

It is important how you phrase things; in version 1.1 of the model, more items

will be phrased as questions

By putting your own words into it, you understand it better

Result

We published the model as a poster in November, 2010:

“With all due respect, this is the announcement of the perhaps most powerful

public two-page document in the history of software testing.”

Then it was remarkably quiet…

However, the poster is downloaded 100-500 times per month

Result

We often use the model in discussions with clients and prospect

It is a good way of letting the customer say, with their own words, what they

really value

Henrik has used them in a university course “Software Test Design” and in a

higher vocational study course “Manual Testing in Practice”

We continuously work with improving the model

Insights

Having all these in the back of your head when testing will make you a better

tester

You don’t have to remember everything;

you will recognize violations when you see them

Result

Comments on Charisma can be boiled down to:

“I like Charisma, but how do you test for it?”

We test for Charisma all the time…

You can do it

Use proxies to interpret the result

Compare with Usability

Not long ago Usability Testing was a branch of CHI

How do you test for usability without setting up a usability test lab and equip it with

eye-tracking machines etc?

"Everyone" agree that Charisma is important for the success of products, so

how come we don't test for it?

Usage

Test idea triggers

Define quality objectives

Write specifications

Review specifications

Evaluate test results

Report test results

Classify bugs

Talk about software testing

Learn about software testing

Identify risks

The Future

1.1 is coming up pretty soon

We are currently working on new two-page poster on sources for test ideas

And more to come…

Insights

The recipe for crafting our own models

1) This model can be improved

2) It is worth being improved

3) We are capable of doing it

4) Put in a lot of work

Questions

Software Quality Characteristics

http://thetesteye.com/posters/TheTestEye_SoftwareQualityCharacteristics.pdf

Blog

www.thetesteye.com

Contact

Email henrik.emilsson@qamcom.se

Twitter @henrikemilsson

Qamcom Research & Technology AB | www.qamcom.com | +46 31 721 1730