Jonathon Wright Email: jlwright@iee.org Twitter: @Jonathon_Wright Julie Gardiner Email: julie@qstc.co.uk Twitter: @CheekyTester

White Box e.g.

• Statement

• Decision

• Branch

Condition

Combination

• etc.

Black Box e.g.

• Classification Trees

• Pairwise

• Boundary Value

• etc.

• Heuristics

• Exploratory

• Defect Based

• Fault Attacks

20 “things”

tests

our tests have covered 10/20 “things” = 50% “thing” coverage

“things” can be: lines of code = statements

decision outcomes =

decisions

branches

modules

menu options

functions

…

component

level

integration level

system level

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

# Declare variables. integer Counter, Quantity float Amount, Price # Program begins here. MAIN Amount = 0.0 Read Quantity # Determine total cost. FOR Counter = 1 to Quantity Read Price Amount = Amount + Price ENDFOR Print “Total Cost is “ & Amount ENDMAIN

1

2

3

4

5

6

7

Declaration section

(declares variables)

Executable section

(comprises executable

statements)

Comments

Labels (optional)

1 Line numbers

1 Statement numbers

Example control flow chart

Make a list of items to sell For each item in the list Calculate cost of postage Write a description of this item If a photograph is needed Photograph this item Transfer photograph to PC Endif Submit details of item to eBay Endfor

End

end of IF

sequential

statements

single statement

decision

statement (FOR)

sequential

statement

decision

statement (IF)

sequential

statements

end of FOR

end of program

executable statements

− a measurement technique

− percentage of executable statements exercised by a test suite number of statements exercised

total number of statements

− example: − program has 10 statements

− tests exercise 8 statements

statement coverage =

=

Statement coverage

is normally measured

by a software tool

8/10 = 80%

1. statement

2. statement

3. IF….

4. then 6. else

5. … 7. …

8. ENDIF

9. statement

10. statement

test

5

2

1

3

4

Example of statement coverage

read(n)

IF n > 6 THEN

n = n * 2

ENDIF

print n

1

2

3

4

5

Statement

numbers

5

2

1

3

4

True

False

Test Path Statement

Case Taken Coverage

A 1, 2, 3, 4, 5 100%

1

2

3

4

5

Test

Case Input Expected

Output

A 7 14

True

False IF

Decision coverage

is normally measured

by a software tool

½ = 50%

= 1. statement

2. statement

3. IF….

4. then 6. else

5. … 7. …

8. ENDIF

9. statement

10. statement

test

B 1, 2, 4, 5 False 50% 80%

Example of decision coverage

read(n)

IF n > 6 THEN

n = n * 2

ENDIF

print n

1

2

3

4

5

Statement

numbers

B 3 3

5

1

3

4

True

False

Test

Case Input

Expected

Output

A 7 14

Test Path Decision Decision Statement

Case Taken Outcome Coverage Coverage

A 1, 2, 3, 4, 5 True 50% 100%

True

2 False

5

1

3

4

True

False 2

False

1

2

3

4

5

Read A

IF A > 0 THEN

Print “A positive”

ENDIF

Print “A positive”

FALSE

Print

ENDIF

TRUE A>0

Read Read A

IF A > 0 THEN

ENDIF

2

1

IF Age < 18 THEN

Print “No alcohol”

ELSE

Print “Have an alcoholic drink”

ENDIF

Print “Have an alcoholic drink”

FALSE

no alcohol

ENDIF

TRUE A<18

IF Age < 18 THEN

ENDIF

Print “No alcohol”

ELSE

alcohol

2

2

Read stock level for item

IF Item in stock THEN

Get from stores

ELSE

Order from supplier

Print “Item back-ordered”

ENDIF

Print “Item processed”

Print “Item processed”

FALSE

Get

ENDIF

TRUE In?

Read stock level for item

ENDIF

Get from stores

ELSE

Order from supplier

IF Item in stock THEN

Read

Print

Print

Print “Item back-ordered”

Order

Draw the

diagram

in your

notes

2

2

TRUE

Get

ENDWHILE

Check

Read order line

WHILE more items ordered DO

Check availability

Get from stores

ENDWHILE

Print “Finished processing”

Print “Finished processing”

1 1

Read order line

WHILE more items ordered DO

Check availability

Get from stores

ENDWHILE

FALSE

Read

Print

?

Read Questions

Result = 0

Right = 0

WHILE more Questions

IF Answer = Correct THEN

Right = Right + 1

ENDIF

END WHILE

Result = (Right / Questions)

IF Result > 60% THEN

Print "pass"

ELSE

Print "fail”

ENDIF

Pseudo-code:

2 2 end if

end if

do

if r=r+1

Init

if

result

pass

fail

end

while

Control flow graph structures

Graph notation as used in Bill Hetzel’s “The Complete Guide to Software Testing”

Node:

a single statement or

sequence of sequential

statements (that are not

decisions or branches).

Branch or Edge :

a logical link between

two nodes, indicates the

next statement(s) that

can be executed.

Decision Statement

or Branching

Statement

IF condition THEN

do this

ENDIF

IF condition THEN

do this

ELSE

do that

ENDIF

WHILE condition

do this

ENDWHILE

FOR condition

do this

ENDFOR

DO

this

UNTIL condition

Statement(s) executed

zero or more times

Statement(s) executed

one or more times

Producing a control flow graph

− identify sequential statements − combine a series into one node

− draw one branch coming out from each node

− add a node for each decision statement − draw two branches coming out from each decision

node

− add a node for the end of a decision structure − i.e. where lines join back together

− complete graph with an end node − do not leave a branch pointing into space!

Example control flow graph

Make a list of items to sell For each item in the list Calculate cost of postage Write a description of this item If a photograph is needed Photograph this item Transfer photograph to PC Endif Submit details of item to eBay Endfor

End

end of IF

sequential

statements

single statement

decision

statement (FOR)

sequential

statement

decision

statement (IF)

sequential

statements

end of FOR

end of program

ENDIF

print b

b = a * 2

read(a)

IF a > 6 THEN b = a * 2

read(a)

IF a > 6 THEN

ENDIF

print b

b = a * 2

read(a)

IF a > 6 THEN

ENDIF

print b

read(a)

IF a > 6 THEN

branch coverage counts

all branches

decision coverage counts only

conditional branches

Branch/decision coverage differences

Basic blocks

TRUE

FALSE

Conditional

branch

Unconditional

branch

ENDIF

print b

b = a * 2

1 7 14

Test case Input

Expected output

2 4 ?

1 67%

2 B1 -> B3 33%

Test case 2

could reveal

a fault (‘b’

has not been

defined)

B1 -> B2 B2 -> B3

Test case Path

Branch coverage

Decision coverage

50%

50%

Totals: 100% 100%

read(a)

IF a > 6 THEN

b = a * 2

ENDIF

print b

B1

B3

B2

TRUE

FALSE

number of boolean operand values executed

total number of boolean operand values

IF (a < b) AND (c > d) AND (count < 100) THEN

=

Operand

values

Condition coverage does not guarantee decision coverage

number of boolean operand value combinations executed

total number of boolean operand value combinations

Multiple Condition coverage guarantees decision coverage

IF (a < b) AND (c > d) AND (count < 100) THEN

=

IF (a < b) AND (c > d) AND (count < 100) THEN

number of boolean operand value (independently affect the decision)

total number of boolean operands

=

number of executed LCSAJ’s

total number of LCSAJ’s =

Result = 0

Right = 0

DO While more questions

IF Key = Answer then

Incr Right

END DO

Result = (Right / Questions) x 100

IF Result > 50 Then

Print "pass"

ELSE

Print "fail"

END

LCSAJ's

From To Target

1 3 7

1 4 6

1 6 3

3 3 7

3 4 6

3 6 3

6 6 3

7 8 10

7 10 12

10 12 end

12 12 end

etc.

1

2

3

4

5

6

7

8

9

10

11

12

? ?

1 2

? ?

1 2 3 1 2 ?

?

1 3 2 4

?

4 5 6 7 8 ….

for as many times as it

is possible to go round

the loop (this can be

unlimited, i.e. infinite)

2 1 3

All Paths

L

B

S

BCC

MCDC

BC

Where:

All Paths = Path Coverage

L = LCSAJ

B = Branch/Decision Coverage

S = Statement Coverage

BCC = Branch Condition

Combination Coverage

MCDC = Modified Condition

Decision Coverage

BC = Branch Condition

Coverage

Source: BS 7925-2

structural coverage can measure completeness or

incompleteness of a set of specification or

experienced based techniques

code coverage tools capture the structural coverage achieved by the tests

any gaps to achieve 100% are identified

very useful for demonstrating compliance to standards

especially safety-critical systems

Manual Testing

Layer Diagram

UML Modeling

Load Testing Web Testing

Test Case Management IntelliTrace™

Architecture Explorer Fast Fwd for Manual Test

Logical Class Designer

Cloud Development

Office Development

Windows Development

New WPF Editor Customizable IDE

Multi-core Development Silverlight Tools

Web Development

SharePoint Development

Generate from Usage

Static Code Analysis

Database Deployment

Code Metrics

Database Unit Testing Test Data Generation

Test Impact Analysis UI Test Automation

Code Coverage

Performance Profiling

Database Change Mgmt.

Visual Studio – Products

Fully supported platform

Partial solution. Further work

required to complete

Best efforts. May work with

known issues. No major

ongoing investment

Currently no support but on

the roadmap for future

releases

Currently no support and

none planned for now.

Platform Native CodedUI 3rd Party Notes

IE7/8/9+ HTML/AJAX

ASP.NET MVC 3/4

WPF 3.5+

Windows Win32

Windows Forms 2.0+

FF3 – HTML/AJAX

Citrix/Terminal Services

SharePoint

Silverlight 3.0/4.0

Java/Java Applets

Office Client Apps

IE 6 (Legacy)

Chrome/Opera/Safari

Flash/AIR

SAP

generalist specialist

manual

testing

some

scripting

creates scripts

to set up lab,

create data

scripting skills

some

coding skills

strong coding

develops fully

automated

testing

procedures

expert

coding skills

black box testing

white box testing

API testing

coded UI test

unit testing web performance test

load test

test runner

test case management

virtual lab management

data diagnostic adapters (video, intelliTrace, action log, event log etc.)

team foundation server with reporting

(defects, requirements, user stories, code coverage, source control & build information)

generalist specialist

CodeCoverage.wrf

Maintainability Index

Cyclomatic Complexity

Depth of Inheritance

Class Coupling

Lines of Code

CodeCoverageHighlighting.wrf

TestImpact.wrf

StaticCodeAnalysis.wrf

GivenWhenThen.wrf

TDD

BDD

UI

Automation

Manual

(Assisted)

bla

ck b

ox te

stin

g

wh

ite

bo

x te

stin

g

AP

I te

stin

g

component level

ATDD

Automation

Fakes & Stubs

system level

integration level

Dynamic Diagnostic Adapters (BPM Assist, IntelliTrace, Amber (Action & Event Logging))

A1

B1

Home

Add.Book

Shopping.Cart

Search.Book

Submit.Order

D1

C1

B2B4

C3B3C2

B5

D3C4 D2

D4 D5

C5

= High

= Medium

= Low

Test Control Variable

Confidence Level

www.microsoft.com/visualstudio/11/

www.microsoft.com/visualstudio/11/en-US/downloads#tfs-group

www.windows.microsoft.com/en-US/windows-8/download/

www.specflow.org