announcements topic: code inspection and testing group...

32
2 Analysis of Software Artifacts – Aldrich © 2009 Announcements Second assignment came out Monday evening Topic: Code Inspection and Testing Find defects in Hnefetafl rules written by your classmates Compare inspection, coverage testing, random testing, and black-box testing Group assignment Due Wednesday, January 28, at 10:30am

Upload: others

Post on 10-Oct-2019

3 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

2Analysis of Software Artifacts –

Aldrich ©

2009

Announcem

ents

•Second a

ssig

nm

ent

cam

e o

ut

Monday e

venin

g�

Topic

: Code I

nspection a

nd T

esting

•Fin

d d

efe

cts

in H

nefe

tafl

rule

s w

ritt

en b

y y

our

cla

ssm

ate

s•

Com

pare

inspection, covera

ge t

esting, ra

ndom

testing, and b

lack-b

ox t

esting

�G

roup a

ssig

nm

ent

�D

ue W

ednesday, Ja

nuary

28, at

10:3

0am

Page 2: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

3

Testing

©2009 by Jonathan Aldrich

Portions ©

2007 by W

illiam L Scherlis

used by permission

No part may be copied or used

without written permission.

Prim

ary source: Kaner, Falk, Nguyen.

Testing Computer Software (2nd Edition).

Jonathan Aldrich

Assis

tant

Pro

fessor

Institu

te for

Soft

ware

Researc

h

School of Com

pute

r Scie

nce

Carn

egie

Mellon U

niv

ers

ity

jonath

an.a

ldrich@

cs.c

mu.e

du

+1 4

12 2

68 7

278

Anal

ysis

of S

oftw

are

Artif

acts

Anal

ysis

of S

oftw

are

Artif

acts

Anal

ysis

of S

oftw

are

Artif

acts

Anal

ysis

of S

oftw

are

Artif

acts

Janu

ary

2009

Page 3: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

4Analysis of Software Artifacts –

Aldrich ©

2009

Testing –

The Big Questions

1.What is testing?

�And w

hy d

o w

e t

est?

2.To w

hat standard do w

e test?

�Specific

ation o

f behavio

r and q

uality

att

ribute

s

3.How do w

e select a set of good tests?

�Functional (b

lack-b

ox)

testing

�Str

uctu

ral (w

hite-b

ox)

testing

4.How do w

e assess our test suites?

�Covera

ge, M

uta

tion, Captu

re/R

ecaptu

re…

5.What are effective testing practices?

�Levels

of str

uctu

re:

unit, in

tegra

tion, syste

m…

�D

esig

n for

testing

�H

ow

does t

esting inte

gra

te into

lifecycle

and m

etr

ics?

6.What are the lim

its of testing?

•W

hat

are

com

ple

menta

ry a

ppro

aches?

•Inspections

•Static and dynamic analysis

Page 4: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

5Analysis of Software Artifacts –

Aldrich ©

2009

Equiv

ale

nce, Boundary

and R

obustn

ess E

xam

ple

•Exam

ple

: Tic

-Tac-T

oe g

am

e�

Tw

o p

layers

,X

and O

, ta

ke t

urn

s m

ark

ing t

he s

paces in a

3 g

rid, w

ith

Xgoin

g first.

The p

layer

who s

ucceeds in p

lacin

g t

hre

e r

espective

mark

s

in a

horizonta

l, v

ert

ical or

dia

gonal ro

w w

ins t

he g

am

e.

•Equiv

ale

nce c

lasses?

•Boundary

valu

es?

•Robustn

ess t

est

cases?

Page 5: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

6Analysis of Software Artifacts –

Aldrich ©

2009

Pro

tocol Testing

•Object protocols

�D

evelo

p t

est

cases t

hat

involv

e r

epre

senta

tive s

equence o

f opera

tions

on o

bje

cts

•Exam

ple

: D

ictionary

str

uctu

re

•Cre

ate

, AddEntr

y*, Lookup, M

odifyEntr

y*, D

ele

teEntr

y, Lookup, D

estr

oy

•Exam

ple

: IO

Str

eam

•O

pen, Read, Read, Clo

se, Read, O

pen, W

rite

, Read, Clo

se, Clo

se

�Test

concurr

ent

access fro

m m

ultip

le t

hre

ads

•Exam

ple

: FIF

O q

ueue for

events

, lo

ggin

g, etc

.

Create

Put

Put

Get

Get

Put Get Get

Put Put

Get

•Appro

ach

�D

evelo

p r

epre

senta

tive s

equences –

based o

n u

se c

ases, scenarios,

pro

file

s

�Random

ly g

enera

te c

all s

equences

•Exam

ple

: Account

•O

pen, D

eposit, W

ithdra

w, W

ithdra

w, D

eposit, Q

uery

, W

ithdra

w, Clo

se

�Covera

ge:

Conceptu

al sta

tes

•Als

o u

sefu

l fo

r pro

tocol in

tera

ctions w

ithin

dis

trib

ute

d d

esig

ns

Page 6: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

7Analysis of Software Artifacts –

Aldrich ©

2009

Random

Testing

•Random

ly g

enera

te p

rogra

m input

and e

xecute

test

case

•Challenges

�H

ow

to g

enera

te t

he input?

•To a

ssess q

uality

, w

ant

repre

senta

tive input

•To fin

d d

efe

cts

, w

ant

good input

covera

ge

�H

ow

to c

heck if th

e input

is v

alid?

•Check t

he p

reconditio

n•

If p

reconditio

n fals

e, th

row

out

test

or

use a

s r

obustn

ess t

est

�H

ow

to c

heck t

he o

utp

ut?

•N

eed a

n oracle t

hat

can t

ell if th

e a

nsw

er

is r

ight

•Essentially t

he p

ostc

onditio

n–

but

may n

ot

alw

ays b

e form

ally

specifie

d•

May b

e e

asie

r to

check a

n a

nsw

er

than t

o c

om

pute

it

•M

ay h

ave a

refe

rence im

ple

menta

tion

•M

ay log info

rmation a

nd c

heck m

anually

•M

ay c

heck o

nly

cert

ain

pro

pert

ies o

f th

e o

utp

ut

•(p

art

ial) p

ost-

conditio

n•

lack o

f exceptions t

hro

wn

Page 7: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

8Analysis of Software Artifacts –

Aldrich ©

2009

Checkpoin

ts:

Loggin

g, Assert

ions, and B

reakpoin

ts

•U

se “

checkpoin

ts”

in c

ode

�Access t

o inte

rmedia

te v

alu

es

�Enable

checks during

execution

Three approaches

•Loggin

g�

Cre

ate

a log r

ecord

of in

tern

al events

�Tools

to s

upport

•java.util.L

ogging

•or

g.ap

ache.lo

g4j

�Log r

ecord

s c

an b

e a

naly

zed for

patt

ern

s o

f events

•Lis

tener

events

•Pro

tocol events

•Etc.

•Assert

ions

�Logic

al sta

tem

ents

explicitly

checked d

uring t

est

runs

�(N

o s

ide e

ffects

on p

rogra

m v

ariable

s)

�Check d

ata

inte

grity

•Absence o

f null p

oin

ter

•Arr

ay b

ounds

•Etc.

•Bre

akpoin

ts�

Pro

vid

e inte

ractive a

ccess t

o inte

rmedia

te s

tate

when a

conditio

n is r

ais

ed

Page 8: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

9Analysis of Software Artifacts –

Aldrich ©

2009

Assert

ions a

nd D

ata

Inte

grity

Exam

ple

•W

hat

are

som

e d

ata

inte

grity

conditio

ns o

n a

Tic

-Tac-T

oe

gam

e p

ositio

n?

Page 9: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

10

Analysis of Software Artifacts –

Aldrich ©

2009

Test

Covera

ge

•Analy

sis

: the systematic examination of a software artifact

to determine its properties

�W

e c

annot

test

all inputs

, so h

ow

can w

e b

e s

yste

matic?

•Black box t

esting

�Syste

matically t

est

diffe

rent

part

s o

f th

e input

dom

ain

•“d

om

ain

covera

ge”

�Test

thro

ugh t

he p

ublic A

PI

•fo

cuses a

ttention o

n c

lient-

vis

ible

behavio

r

�N

o v

isib

ility into

the c

ode inte

rnals

–a “

bla

ck”

box

•White box t

esting

�Syste

matically t

est

diffe

rent

ele

ments

in t

he c

ode

•“c

ode c

overa

ge”

�Can t

est

inte

rnal ele

ments

directly –

a “

white”

box

•G

ood for

quality

att

ribute

s lik

e r

obustn

ess

�Takes a

dvanta

ge o

f desig

n info

rmation a

nd c

ode s

tructu

re –

a

“white”

box

•“g

lass b

ox”

may b

e b

ett

er

term

inolo

gy

Page 10: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

11

Analysis of Software Artifacts –

Aldrich ©

2009

White B

ox:

Sta

tem

ent

Covera

ge

•Statement coverage

�W

hat

port

ion o

f pro

gra

m s

tate

ments

(nodes)

are

touched b

y t

est

cases

•Advanta

ges

�Test

suite s

ize lin

ear

in s

ize o

f code

�Covera

ge e

asily a

ssessed

•Is

sues

�D

ead c

ode is n

ot

reached

�M

ay r

equire s

om

e s

ophis

tication t

osele

ct

input

sets

(M

cCabe b

asis

path

s)

�Fault-t

ole

rant

err

or-

handling c

ode

may b

e d

ifficult t

o “

touch”

�M

etr

ic:

Could

cre

ate

incentive t

o

remove

err

or

handle

rs!

Page 11: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

12

Analysis of Software Artifacts –

Aldrich ©

2009

White B

ox:

Sta

tem

ent

Covera

ge

•Statement coverage

�W

hat

port

ion o

f pro

gra

m s

tate

ments

(nodes)

are

touched b

y t

est

cases

•Advanta

ges

�Test

suite s

ize lin

ear

in s

ize o

f code

�Covera

ge e

asily a

ssessed

•Is

sues

�D

ead c

ode is n

ot

reached

�M

ay r

equire s

om

e s

ophis

tication t

osele

ct

input

sets

(M

cCabe b

asis

path

s)

�Fault-t

ole

rant

err

or-

handling c

ode

may b

e d

ifficult t

o “

touch”

�M

etr

ic:

Could

cre

ate

incentive t

o

remove

err

or

handle

rs!

Page 12: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

13

Analysis of Software Artifacts –

Aldrich ©

2009

White B

ox:

Sta

tem

ent

Covera

ge

•Statement coverage

�W

hat

port

ion o

f pro

gra

m s

tate

ments

(nodes)

are

touched b

y t

est

cases

•Advanta

ges

�Test

suite s

ize lin

ear

in s

ize o

f code

�Covera

ge e

asily a

ssessed

•Is

sues

�D

ead c

ode is n

ot

reached

�M

ay r

equire s

om

e s

ophis

tication t

osele

ct

input

sets

(M

cCabe b

asis

path

s)

�Fault-t

ole

rant

err

or-

handling c

ode

may b

e d

ifficult t

o “

touch”

�M

etr

ic:

Could

cre

ate

incentive t

o

remove

err

or

handle

rs!

Page 13: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

14

Analysis of Software Artifacts –

Aldrich ©

2009

White B

ox:

Sta

tem

ent

Covera

ge

•Statement coverage

�W

hat

port

ion o

f pro

gra

m s

tate

ments

(nodes)

are

touched b

y t

est

cases

•Advanta

ges

�Test

suite s

ize lin

ear

in s

ize o

f code

�Covera

ge e

asily a

ssessed

•Is

sues

�D

ead c

ode is n

ot

reached

�M

ay r

equire s

om

e s

ophis

tication t

osele

ct

input

sets

(M

cCabe b

asis

path

s)

�Fault-t

ole

rant

err

or-

handling c

ode

may b

e d

ifficult t

o “

touch”

�M

etr

ic:

Could

cre

ate

incentive t

o

remove

err

or

handle

rs!

Page 14: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

15

Analysis of Software Artifacts –

Aldrich ©

2009

White B

ox:

Sta

tem

ent

Covera

ge

•Statement coverage

�W

hat

port

ion o

f pro

gra

m s

tate

ments

(nodes)

are

touched b

y t

est

cases

•Advanta

ges

�Test

suite s

ize lin

ear

in s

ize o

f code

�Covera

ge e

asily a

ssessed

•Is

sues

�D

ead c

ode is n

ot

reached

�M

ay r

equire s

om

e s

ophis

tication t

osele

ct

input

sets

(M

cCabe b

asis

path

s)

�Fault-t

ole

rant

err

or-

handling c

ode

may b

e d

ifficult t

o “

touch”

�M

etr

ic:

Could

cre

ate

incentive t

o

remove

err

or

handle

rs!

Page 15: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

16

Analysis of Software Artifacts –

Aldrich ©

2009

White B

ox:

Sta

tem

ent

Covera

ge

•Statement coverage

�W

hat

port

ion o

f pro

gra

m s

tate

ments

(nodes)

are

touched b

y t

est

cases

•Advanta

ges

�Test

suite s

ize lin

ear

in s

ize o

f code

�Covera

ge e

asily a

ssessed

•Is

sues

�D

ead c

ode is n

ot

reached

�M

ay r

equire s

om

e s

ophis

tication t

osele

ct

input

sets

(M

cCabe b

asis

path

s)

�Fault-t

ole

rant

err

or-

handling c

ode

may b

e d

ifficult t

o “

touch”

�M

etr

ic:

Could

cre

ate

incentive t

o

remove

err

or

handle

rs!

Page 16: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

17

Analysis of Software Artifacts –

Aldrich ©

2009

White B

ox:

Sta

tem

ent

Covera

ge

•Statement coverage

�W

hat

port

ion o

f pro

gra

m s

tate

ments

(nodes)

are

touched b

y t

est

cases

•Advanta

ges

�Test

suite s

ize lin

ear

in s

ize o

f code

�Covera

ge e

asily a

ssessed

•Is

sues

�D

ead c

ode is n

ot

reached

�M

ay r

equire s

om

e s

ophis

tication t

osele

ct

input

sets

(M

cCabe b

asis

path

s)

�Fault-t

ole

rant

err

or-

handling c

ode

may b

e d

ifficult t

o “

touch”

�M

etr

ic:

Could

cre

ate

incentive t

o

remove

err

or

handle

rs!

Page 17: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

18

Analysis of Software Artifacts –

Aldrich ©

2009

White B

ox:

Bra

nch C

overa

ge

•Branch coverage

�W

hat

port

ion o

f conditio

n b

ranches a

recovere

d b

y t

est

cases?

�Or: W

hat

port

ion o

f re

lational expre

ssio

ns

and v

alu

es a

re c

overe

d b

y t

est

cases?

•Conditio

n t

esting (

Tai)

�Multiconditioncoverage

–all b

oole

an

com

bin

ations o

f te

sts

are

covere

d

•Advanta

ges

�Test

suite s

ize a

nd c

onte

nt

derived

from

str

uctu

re o

f boole

an e

xpre

ssio

ns

�Covera

ge e

asily a

ssessed

•Is

sues

�D

ead c

ode is n

ot

reached

�Fault-t

ole

rant

err

or-

handling c

ode

may b

e d

ifficult t

o “

touch”

Page 18: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

19

Analysis of Software Artifacts –

Aldrich ©

2009

White B

ox:

Bra

nch C

overa

ge

•Branch coverage

�W

hat

port

ion o

f conditio

n b

ranches a

recovere

d b

y t

est

cases?

�Or: W

hat

port

ion o

f re

lational expre

ssio

ns

and v

alu

es a

re c

overe

d b

y t

est

cases?

•Conditio

n t

esting (

Tai)

�Multiconditioncoverage

–all b

oole

an

com

bin

ations o

f te

sts

are

covere

d

•Advanta

ges

�Test

suite s

ize a

nd c

onte

nt

derived

from

str

uctu

re o

f boole

an e

xpre

ssio

ns

�Covera

ge e

asily a

ssessed

•Is

sues

�D

ead c

ode is n

ot

reached

�Fault-t

ole

rant

err

or-

handling c

ode

may b

e d

ifficult t

o “

touch”

Page 19: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

20

Analysis of Software Artifacts –

Aldrich ©

2009

White B

ox:

Bra

nch C

overa

ge

•Branch coverage

�W

hat

port

ion o

f conditio

n b

ranches a

recovere

d b

y t

est

cases?

�Or: W

hat

port

ion o

f re

lational expre

ssio

ns

and v

alu

es a

re c

overe

d b

y t

est

cases?

•Conditio

n t

esting (

Tai)

�Multiconditioncoverage

–all b

oole

an

com

bin

ations o

f te

sts

are

covere

d

•Advanta

ges

�Test

suite s

ize a

nd c

onte

nt

derived

from

str

uctu

re o

f boole

an e

xpre

ssio

ns

�Covera

ge e

asily a

ssessed

•Is

sues

�D

ead c

ode is n

ot

reached

�Fault-t

ole

rant

err

or-

handling c

ode

may b

e d

ifficult t

o “

touch”

Page 20: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

21

Analysis of Software Artifacts –

Aldrich ©

2009

White B

ox:

Bra

nch C

overa

ge

•Branch coverage

�W

hat

port

ion o

f conditio

n b

ranches a

recovere

d b

y t

est

cases?

�Or: W

hat

port

ion o

f re

lational expre

ssio

ns

and v

alu

es a

re c

overe

d b

y t

est

cases?

•Conditio

n t

esting (

Tai)

�Multiconditioncoverage

–all b

oole

an

com

bin

ations o

f te

sts

are

covere

d

•Advanta

ges

�Test

suite s

ize a

nd c

onte

nt

derived

from

str

uctu

re o

f boole

an e

xpre

ssio

ns

�Covera

ge e

asily a

ssessed

•Is

sues

�D

ead c

ode is n

ot

reached

�Fault-t

ole

rant

err

or-

handling c

ode

may b

e d

ifficult t

o “

touch”

Page 21: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

22

Analysis of Software Artifacts –

Aldrich ©

2009

White B

ox:

Path

Covera

ge

•Path coverage

�W

hat

port

ion o

f all p

ossib

le p

ath

s t

hro

ugh

the p

rogra

m a

re c

overe

d b

y t

ests

?�

Loop t

esting:

Consid

er

repre

senta

tive a

nd

edge c

ases:

•Zero

, one, tw

o ite

rations

•If

there

is a

bound n

: n-1

, n, n+

1 ite

rations

•N

este

d loops/c

onditio

nals

fro

m insid

e o

ut

•Advanta

ges

�Bett

er

covera

ge o

f lo

gic

al flow

s

•D

isadvanta

ges

�N

ot

all p

ath

s a

re p

ossib

le, or

necessary

•W

hat

are

the significant

path

s?

�Com

bin

ato

rial explo

sio

n in c

ases u

nle

ss

care

ful choic

es a

re m

ade

•E.g

., s

equence o

f nif

tests

can y

ield

up t

o 2

^n p

ossib

le p

ath

s

�Assum

ption t

hat

pro

gra

m s

tructu

reis

basic

ally s

ound

Page 22: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

23

Analysis of Software Artifacts –

Aldrich ©

2009

White B

ox:

Path

Covera

ge

•Path coverage

�W

hat

port

ion o

f all p

ossib

le p

ath

s t

hro

ugh

the p

rogra

m a

re c

overe

d b

y t

ests

?�

Loop t

esting:

Consid

er

repre

senta

tive a

nd

edge c

ases:

•Zero

, one, tw

o ite

rations

•If

there

is a

bound n

: n-1

, n, n+

1 ite

rations

•N

este

d loops/c

onditio

nals

fro

m insid

e o

ut

•Advanta

ges

�Bett

er

covera

ge o

f lo

gic

al flow

s

•D

isadvanta

ges

�N

ot

all p

ath

s a

re p

ossib

le, or

necessary

•W

hat

are

the significant

path

s?

�Com

bin

ato

rial explo

sio

n in c

ases u

nle

ss

care

ful choic

es a

re m

ade

•E.g

., s

equence o

f nif

tests

can y

ield

up t

o 2

^n p

ossib

le p

ath

s

�Assum

ption t

hat

pro

gra

m s

tructu

reis

basic

ally s

ound

Page 23: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

24

Analysis of Software Artifacts –

Aldrich ©

2009

White B

ox:

Path

Covera

ge

•Path coverage

�W

hat

port

ion o

f all p

ossib

le p

ath

s t

hro

ugh

the p

rogra

m a

re c

overe

d b

y t

ests

?�

Loop t

esting:

Consid

er

repre

senta

tive a

nd

edge c

ases:

•Zero

, one, tw

o ite

rations

•If

there

is a

bound n

: n-1

, n, n+

1 ite

rations

•N

este

d loops/c

onditio

nals

fro

m insid

e o

ut

•Advanta

ges

�Bett

er

covera

ge o

f lo

gic

al flow

s

•D

isadvanta

ges

�N

ot

all p

ath

s a

re p

ossib

le, or

necessary

•W

hat

are

the significant

path

s?

�Com

bin

ato

rial explo

sio

n in c

ases u

nle

ss

care

ful choic

es a

re m

ade

•E.g

., s

equence o

f nif

tests

can y

ield

up t

o 2

^n p

ossib

le p

ath

s

�Assum

ption t

hat

pro

gra

m s

tructu

reis

basic

ally s

ound

Page 24: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

25

Analysis of Software Artifacts –

Aldrich ©

2009

White B

ox:

Path

Covera

ge

•Path coverage

�W

hat

port

ion o

f all p

ossib

le p

ath

s t

hro

ugh

the p

rogra

m a

re c

overe

d b

y t

ests

?�

Loop t

esting:

Consid

er

repre

senta

tive a

nd

edge c

ases:

•Zero

, one, tw

o ite

rations

•If

there

is a

bound n

: n-1

, n, n+

1 ite

rations

•N

este

d loops/c

onditio

nals

fro

m insid

e o

ut

•Advanta

ges

�Bett

er

covera

ge o

f lo

gic

al flow

s

•D

isadvanta

ges

�N

ot

all p

ath

s a

re p

ossib

le, or

necessary

•W

hat

are

the significant

path

s?

�Com

bin

ato

rial explo

sio

n in c

ases u

nle

ss

care

ful choic

es a

re m

ade

•E.g

., s

equence o

f nif

tests

can y

ield

up t

o 2

^n p

ossib

le p

ath

s

�Assum

ption t

hat

pro

gra

m s

tructu

reis

basic

ally s

ound

Page 25: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

26

Analysis of Software Artifacts –

Aldrich ©

2009

White B

ox:

Path

Covera

ge

•Path coverage

�W

hat

port

ion o

f all p

ossib

le p

ath

s t

hro

ugh

the p

rogra

m a

re c

overe

d b

y t

ests

?�

Loop t

esting:

Consid

er

repre

senta

tive a

nd

edge c

ases:

•Zero

, one, tw

o ite

rations

•If

there

is a

bound n

: n-1

, n, n+

1 ite

rations

•N

este

d loops/c

onditio

nals

fro

m insid

e o

ut

•Advanta

ges

�Bett

er

covera

ge o

f lo

gic

al flow

s

•D

isadvanta

ges

�N

ot

all p

ath

s a

re p

ossib

le, or

necessary

•W

hat

are

the significant

path

s?

�Com

bin

ato

rial explo

sio

n in c

ases u

nle

ss

care

ful choic

es a

re m

ade

•E.g

., s

equence o

f nif

tests

can y

ield

up t

o 2

^n p

ossib

le p

ath

s

�Assum

ption t

hat

pro

gra

m s

tructu

reis

basic

ally s

ound

Page 26: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

27

Analysis of Software Artifacts –

Aldrich ©

2009

White B

ox:

Assessin

g s

tructu

ral covera

ge

•Covera

ge a

ssessm

ent

tools

�Tra

ck e

xecution o

f code b

y t

est

cases

�Techniq

ues

•M

odifie

d r

untim

e e

nvironm

ent

(e.g

., s

pecia

l JV

M)

•Sourc

e c

ode t

ransfo

rmation

•Count

vis

its t

o s

tate

ments

�D

evelo

p r

eport

s w

ith r

espect

to s

pecific

covera

ge c

rite

ria

•Exam

ple

: EclE

mm

a–

Eclipse p

lugin

for

JUnit t

est

covera

ge

Page 27: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

28

Analysis of Software Artifacts –

Aldrich ©

2009

EclE

mm

ain

Eclipse

•Bre

akdow

n b

y p

ackage,

cla

ss, and m

eth

od

•Covera

ge

�Cla

sses

�M

eth

ods

�Sta

tem

ents

�In

str

uctions

•G

raphic

al and n

um

erical

pre

senta

tion

Page 28: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

29

Analysis of Software Artifacts –

Aldrich ©

2009

Clo

ver

in E

clipse

•Covera

ge r

eport

in e

ditor

win

dow

�re

d:

not

covere

d�

yellow

: covere

d o

nce

�gre

en:

covere

d m

ultip

le t

imes

Page 29: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

30

Analysis of Software Artifacts –

Aldrich ©

2009

Benefits

of W

hite-B

ox

•Tool support

can m

easure

covera

ge

�H

elp

s t

o e

valu

ate

test

suite (

care

ful!)

�Can fin

d u

nte

ste

d c

ode

•Can t

est

pro

gra

m o

ne p

art

at

a t

ime

•Can c

onsid

er

code-r

ela

ted b

oundary

conditio

ns

�If

conditio

ns

�Boundaries o

f fu

nction input/

outp

ut

ranges

•e.g

. sw

itch b

etw

een a

lgorith

ms a

t data

siz

e=

100

•Can fin

d late

nt

faults

�Cannot

trig

ger

a failure

in p

rogra

m, but

can b

e found b

y a

unit t

est

Page 30: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

31

Analysis of Software Artifacts –

Aldrich ©

2009

White B

ox:

Lim

itations

•Is

it

possib

le t

o a

chie

ve 1

00%

covera

ge?

•Can y

ou t

hin

k o

f a p

rogra

m t

hat

has a

defe

ct,

even t

hough

it p

asses a

test

suite w

ith 1

00%

covera

ge?

Page 31: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

32

Analysis of Software Artifacts –

Aldrich ©

2009

White B

ox:

Lim

itations

•Is

it

possib

le t

o a

chie

ve 1

00%

covera

ge?

•Can y

ou t

hin

k o

f a p

rogra

m t

hat

has a

defe

ct,

even t

hough

it p

asses a

test

suite w

ith 1

00%

covera

ge?

•Exclu

siv

e focus o

n c

overa

ge focus m

isses im

port

ant

bugs

�M

issin

g c

ode

�In

corr

ect

boundary

valu

es

�Tim

ing p

roble

ms

�Configura

tion issues

�D

ata

/mem

ory

corr

uption b

ugs

�U

sability p

roble

ms

�Custo

mer

requirem

ents

issues

•Covera

ge is n

ot

a g

ood a

dequacy c

rite

rion

�In

ste

ad, use t

o fin

d p

laces w

here

testing is inadequate

Page 32: Announcements Topic: Code Inspection and Testing Group ...aldrich/courses/654-sp09/slides/4-testing.pdf · Announcements •Second assignment came out Monday evening Analysis of Software

33

Analysis of Software Artifacts –

Aldrich ©

2009

Testing e

xam

ple

•Equiv

ale

nce c

lasses?

•Boundary

valu

es?

•Robustn

ess t

ests

?

•H

ow

to a

chie

ve line coverage?public static int binsrch

(int[] a, int ke

y) {

int low

= 0

;int high= a.leng

th -

1;

while (true

) {

if ( low

> high) re

turn

-(low

+1);

int mid

= (low+high) / 2;

if ( a[m

id] < key ) low

= m

id+ 1

;else if ( a[m

id] > key ) high= m

id-

1;

else re

turn

mid;

}}