7/14/2014

1

Testing Unexplained (The Importance of Tacit Knowledge in Testing and How to Develop It)

James Bach @jamesmarcusbach james@satisfice.com

Consulting Software Testing, Satisfice, Inc.

Tacit and Explicit Knowledge

EXPLICIT means it can be represented completely in the form of a string of bits: words, pictures, even actions can be explicit. (software is explicit)

TACIT means it is not manifested in a form that can be equated to a string of bits: it is unspoken, unwritten, unpictured. • Relational Tacit Knowledge is tacit by convenience. • Somatic Tacit Knowledge is tacit in your body. • Social Tacit Knowledge is tacit in a community. (see Collins, Tacit and Explicit Knowledge)

7/14/2014

2

Examples of the Need for Tacit Knowledge

• You consider how users will interact with and adjust to a product.

• You consider what a specification and product was intended to say or do, not just what they literally say and do.

• You notice what the specification and product strangely omits.

• You focus on business risks even when no one tells you what they are.

• You notice changes in the product’s behavior over time—none of which are

failures in and of themselves—you form conjectures of why that happens, and you connect those conjectures with an evolving

understanding of plausible error and failure patterns.

• You use tools, you notice when those tools misbehave, and you make

adjustments or workarounds to get the job done.

• You report your test results based partly on how you think your clients will react. You anticipate their questions based on your insight about

how they will understand you.

“2+2=“

7/14/2014

3

“2+2=“

What is the Point? You have MANY expectations about how a calculator should work or might work…

(you acquire them automatically)

…but you aren’t aware of many of them...

(they are encoded or generated in your mind, but not in words or pictures)

…and that means no explicit test procedure can ever duplicate the value of tacit knowledge of a skilled human tester.

We need a process that respects tacit knowledge.

7/14/2014

4

Fact Gathering

Result Checking

Tester Thinking

(theorizing about meaning

of results)

Tester Thinking

(failure modeling and test design)

Tester Thinking

Tester Thinking

Tester Thinking

Tester Thinking

Tester Thinking

Tester Thinking

Tester Thinking Tester

Thinking

Tester Thinking

Tester Thinking

Tester Thinking

Tester Thinking

Tester Thinking

Tester Thinking

If testing is an intellectual process that requires deep mental

engagement, personal motivation, and constant learning to do well,

how do we manage the situation in order to foster that?

7/14/2014

5

Managing Tacit Knowledge

10

Heuristics bring useful structure to problem-solving skill.

• adjective:

“serving to discover.”

• noun:

“a fallible method for solving a problem or

making a decision.”

“The engineering method is the use of heuristics

to cause the best change in a poorly understood situation within the available resources.”

-- Billy Vaughan Koen, Discussion of The Method

7/14/2014

6

11

-Idea -Idea

…

Heuristics They are applied, not followed.

1. Do this 2. Then do this 3. Then do this 4. Then do this 5. And then this…

This…

…not this.

The skilled tester remains in control of

the process.

“2+2=“

7/14/2014

7

Tacit Explicit O

ther

Peo

ple

Te

ster

Your Feelings &

Mental Models

Shared Artifacts (specs, tools, etc.)

Stakeholders’ Feelings &

Mental Models

Inference

Observable Consistencies

Reference Conference

Experience

Oracles From the Inside Out

Consistency (“this agrees with that”) an important theme in oracles

• Familiarity: The system is not consistent with the pattern of any familiar problem.

• Explainability: The system is consistent with our ability to explain it.

• World: The system is consistent with objects and states, in the world, that it represents.

• History: The present version of the system is consistent with past versions of it.

• Image: The system is consistent with an image that the organization wants to project.

• Comparable Products: The system is consistent with comparable systems.

• Claims: The system is consistent with what important people say it’s supposed to be.

• Users’ Expectations: The system is consistent with what users want.

• Product: Each element of the system is consistent with comparable elements in the same system.

• Purpose: The system is consistent with its purposes, both explicit and implicit.

• Statutes & Standards: The system is consistent with applicable laws and standards.

7/14/2014

8

A General Statement to Protect Role of Tacit Knowledge/Skill

A system testing event in Malmo, Sweden. Maria Kedemo, manager, is serving food.

7/14/2014

9

7/14/2014

10

Three Roles of Testing

Test Leads/Managers

Cultivate and support

Helpers

Cultivate and supervise

Heuristics and Tools

Use

Use

Clients and Context

Respond to

Respond to

“I am ABLE to make decisions that hurt people (or me).”

“I want to AVOID hurting people (including me).”

“I KNOW how my decisions might hurt people (or me).”

7/14/2014

11

Excellent tacit knowledge can be systematically cultivated.

7/14/2014

12

The Tester’s Sociologist

Sociologist Harry Collins studies how scientists actually work, and what aspects of their work can be made explicit.