announcements topic: code inspection and testing group...
TRANSCRIPT
2Analysis of Software Artifacts –
Aldrich ©
2009
Announcem
ents
•Second a
ssig
nm
ent
cam
e o
ut
Monday e
venin
g�
Topic
: Code I
nspection a
nd T
esting
•Fin
d d
efe
cts
in H
nefe
tafl
rule
s w
ritt
en b
y y
our
cla
ssm
ate
s•
Com
pare
inspection, covera
ge t
esting, ra
ndom
testing, and b
lack-b
ox t
esting
�G
roup a
ssig
nm
ent
�D
ue W
ednesday, Ja
nuary
28, at
10:3
0am
3
Testing
©2009 by Jonathan Aldrich
Portions ©
2007 by W
illiam L Scherlis
used by permission
No part may be copied or used
without written permission.
Prim
ary source: Kaner, Falk, Nguyen.
Testing Computer Software (2nd Edition).
Jonathan Aldrich
Assis
tant
Pro
fessor
Institu
te for
Soft
ware
Researc
h
School of Com
pute
r Scie
nce
Carn
egie
Mellon U
niv
ers
ity
jonath
an.a
ldrich@
cs.c
mu.e
du
+1 4
12 2
68 7
278
Anal
ysis
of S
oftw
are
Artif
acts
Anal
ysis
of S
oftw
are
Artif
acts
Anal
ysis
of S
oftw
are
Artif
acts
Anal
ysis
of S
oftw
are
Artif
acts
Janu
ary
2009
4Analysis of Software Artifacts –
Aldrich ©
2009
Testing –
The Big Questions
1.What is testing?
�And w
hy d
o w
e t
est?
2.To w
hat standard do w
e test?
�Specific
ation o
f behavio
r and q
uality
att
ribute
s
3.How do w
e select a set of good tests?
�Functional (b
lack-b
ox)
testing
�Str
uctu
ral (w
hite-b
ox)
testing
4.How do w
e assess our test suites?
�Covera
ge, M
uta
tion, Captu
re/R
ecaptu
re…
5.What are effective testing practices?
�Levels
of str
uctu
re:
unit, in
tegra
tion, syste
m…
�D
esig
n for
testing
�H
ow
does t
esting inte
gra
te into
lifecycle
and m
etr
ics?
6.What are the lim
its of testing?
•W
hat
are
com
ple
menta
ry a
ppro
aches?
•Inspections
•Static and dynamic analysis
5Analysis of Software Artifacts –
Aldrich ©
2009
Equiv
ale
nce, Boundary
and R
obustn
ess E
xam
ple
•Exam
ple
: Tic
-Tac-T
oe g
am
e�
Tw
o p
layers
,X
and O
, ta
ke t
urn
s m
ark
ing t
he s
paces in a
3×
3 g
rid, w
ith
Xgoin
g first.
The p
layer
who s
ucceeds in p
lacin
g t
hre
e r
espective
mark
s
in a
horizonta
l, v
ert
ical or
dia
gonal ro
w w
ins t
he g
am
e.
•Equiv
ale
nce c
lasses?
•Boundary
valu
es?
•Robustn
ess t
est
cases?
6Analysis of Software Artifacts –
Aldrich ©
2009
Pro
tocol Testing
•Object protocols
�D
evelo
p t
est
cases t
hat
involv
e r
epre
senta
tive s
equence o
f opera
tions
on o
bje
cts
•Exam
ple
: D
ictionary
str
uctu
re
•Cre
ate
, AddEntr
y*, Lookup, M
odifyEntr
y*, D
ele
teEntr
y, Lookup, D
estr
oy
•Exam
ple
: IO
Str
eam
•O
pen, Read, Read, Clo
se, Read, O
pen, W
rite
, Read, Clo
se, Clo
se
�Test
concurr
ent
access fro
m m
ultip
le t
hre
ads
•Exam
ple
: FIF
O q
ueue for
events
, lo
ggin
g, etc
.
Create
Put
Put
Get
Get
Put Get Get
Put Put
Get
•Appro
ach
�D
evelo
p r
epre
senta
tive s
equences –
based o
n u
se c
ases, scenarios,
pro
file
s
�Random
ly g
enera
te c
all s
equences
•Exam
ple
: Account
•O
pen, D
eposit, W
ithdra
w, W
ithdra
w, D
eposit, Q
uery
, W
ithdra
w, Clo
se
�Covera
ge:
Conceptu
al sta
tes
•Als
o u
sefu
l fo
r pro
tocol in
tera
ctions w
ithin
dis
trib
ute
d d
esig
ns
7Analysis of Software Artifacts –
Aldrich ©
2009
Random
Testing
•Random
ly g
enera
te p
rogra
m input
and e
xecute
test
case
•Challenges
�H
ow
to g
enera
te t
he input?
•To a
ssess q
uality
, w
ant
repre
senta
tive input
•To fin
d d
efe
cts
, w
ant
good input
covera
ge
�H
ow
to c
heck if th
e input
is v
alid?
•Check t
he p
reconditio
n•
If p
reconditio
n fals
e, th
row
out
test
or
use a
s r
obustn
ess t
est
�H
ow
to c
heck t
he o
utp
ut?
•N
eed a
n oracle t
hat
can t
ell if th
e a
nsw
er
is r
ight
•Essentially t
he p
ostc
onditio
n–
but
may n
ot
alw
ays b
e form
ally
specifie
d•
May b
e e
asie
r to
check a
n a
nsw
er
than t
o c
om
pute
it
•M
ay h
ave a
refe
rence im
ple
menta
tion
•M
ay log info
rmation a
nd c
heck m
anually
•M
ay c
heck o
nly
cert
ain
pro
pert
ies o
f th
e o
utp
ut
•(p
art
ial) p
ost-
conditio
n•
lack o
f exceptions t
hro
wn
8Analysis of Software Artifacts –
Aldrich ©
2009
Checkpoin
ts:
Loggin
g, Assert
ions, and B
reakpoin
ts
•U
se “
checkpoin
ts”
in c
ode
�Access t
o inte
rmedia
te v
alu
es
�Enable
checks during
execution
Three approaches
•Loggin
g�
Cre
ate
a log r
ecord
of in
tern
al events
�Tools
to s
upport
•java.util.L
ogging
•or
g.ap
ache.lo
g4j
�Log r
ecord
s c
an b
e a
naly
zed for
patt
ern
s o
f events
•Lis
tener
events
•Pro
tocol events
•Etc.
•Assert
ions
�Logic
al sta
tem
ents
explicitly
checked d
uring t
est
runs
�(N
o s
ide e
ffects
on p
rogra
m v
ariable
s)
�Check d
ata
inte
grity
•Absence o
f null p
oin
ter
•Arr
ay b
ounds
•Etc.
•Bre
akpoin
ts�
Pro
vid
e inte
ractive a
ccess t
o inte
rmedia
te s
tate
when a
conditio
n is r
ais
ed
9Analysis of Software Artifacts –
Aldrich ©
2009
Assert
ions a
nd D
ata
Inte
grity
Exam
ple
•W
hat
are
som
e d
ata
inte
grity
conditio
ns o
n a
Tic
-Tac-T
oe
gam
e p
ositio
n?
10
Analysis of Software Artifacts –
Aldrich ©
2009
Test
Covera
ge
•Analy
sis
: the systematic examination of a software artifact
to determine its properties
�W
e c
annot
test
all inputs
, so h
ow
can w
e b
e s
yste
matic?
•Black box t
esting
�Syste
matically t
est
diffe
rent
part
s o
f th
e input
dom
ain
•“d
om
ain
covera
ge”
�Test
thro
ugh t
he p
ublic A
PI
•fo
cuses a
ttention o
n c
lient-
vis
ible
behavio
r
�N
o v
isib
ility into
the c
ode inte
rnals
–a “
bla
ck”
box
•White box t
esting
�Syste
matically t
est
diffe
rent
ele
ments
in t
he c
ode
•“c
ode c
overa
ge”
�Can t
est
inte
rnal ele
ments
directly –
a “
white”
box
•G
ood for
quality
att
ribute
s lik
e r
obustn
ess
�Takes a
dvanta
ge o
f desig
n info
rmation a
nd c
ode s
tructu
re –
a
“white”
box
•“g
lass b
ox”
may b
e b
ett
er
term
inolo
gy
11
Analysis of Software Artifacts –
Aldrich ©
2009
White B
ox:
Sta
tem
ent
Covera
ge
•Statement coverage
�W
hat
port
ion o
f pro
gra
m s
tate
ments
(nodes)
are
touched b
y t
est
cases
•Advanta
ges
�Test
suite s
ize lin
ear
in s
ize o
f code
�Covera
ge e
asily a
ssessed
•Is
sues
�D
ead c
ode is n
ot
reached
�M
ay r
equire s
om
e s
ophis
tication t
osele
ct
input
sets
(M
cCabe b
asis
path
s)
�Fault-t
ole
rant
err
or-
handling c
ode
may b
e d
ifficult t
o “
touch”
�M
etr
ic:
Could
cre
ate
incentive t
o
remove
err
or
handle
rs!
12
Analysis of Software Artifacts –
Aldrich ©
2009
White B
ox:
Sta
tem
ent
Covera
ge
•Statement coverage
�W
hat
port
ion o
f pro
gra
m s
tate
ments
(nodes)
are
touched b
y t
est
cases
•Advanta
ges
�Test
suite s
ize lin
ear
in s
ize o
f code
�Covera
ge e
asily a
ssessed
•Is
sues
�D
ead c
ode is n
ot
reached
�M
ay r
equire s
om
e s
ophis
tication t
osele
ct
input
sets
(M
cCabe b
asis
path
s)
�Fault-t
ole
rant
err
or-
handling c
ode
may b
e d
ifficult t
o “
touch”
�M
etr
ic:
Could
cre
ate
incentive t
o
remove
err
or
handle
rs!
13
Analysis of Software Artifacts –
Aldrich ©
2009
White B
ox:
Sta
tem
ent
Covera
ge
•Statement coverage
�W
hat
port
ion o
f pro
gra
m s
tate
ments
(nodes)
are
touched b
y t
est
cases
•Advanta
ges
�Test
suite s
ize lin
ear
in s
ize o
f code
�Covera
ge e
asily a
ssessed
•Is
sues
�D
ead c
ode is n
ot
reached
�M
ay r
equire s
om
e s
ophis
tication t
osele
ct
input
sets
(M
cCabe b
asis
path
s)
�Fault-t
ole
rant
err
or-
handling c
ode
may b
e d
ifficult t
o “
touch”
�M
etr
ic:
Could
cre
ate
incentive t
o
remove
err
or
handle
rs!
14
Analysis of Software Artifacts –
Aldrich ©
2009
White B
ox:
Sta
tem
ent
Covera
ge
•Statement coverage
�W
hat
port
ion o
f pro
gra
m s
tate
ments
(nodes)
are
touched b
y t
est
cases
•Advanta
ges
�Test
suite s
ize lin
ear
in s
ize o
f code
�Covera
ge e
asily a
ssessed
•Is
sues
�D
ead c
ode is n
ot
reached
�M
ay r
equire s
om
e s
ophis
tication t
osele
ct
input
sets
(M
cCabe b
asis
path
s)
�Fault-t
ole
rant
err
or-
handling c
ode
may b
e d
ifficult t
o “
touch”
�M
etr
ic:
Could
cre
ate
incentive t
o
remove
err
or
handle
rs!
15
Analysis of Software Artifacts –
Aldrich ©
2009
White B
ox:
Sta
tem
ent
Covera
ge
•Statement coverage
�W
hat
port
ion o
f pro
gra
m s
tate
ments
(nodes)
are
touched b
y t
est
cases
•Advanta
ges
�Test
suite s
ize lin
ear
in s
ize o
f code
�Covera
ge e
asily a
ssessed
•Is
sues
�D
ead c
ode is n
ot
reached
�M
ay r
equire s
om
e s
ophis
tication t
osele
ct
input
sets
(M
cCabe b
asis
path
s)
�Fault-t
ole
rant
err
or-
handling c
ode
may b
e d
ifficult t
o “
touch”
�M
etr
ic:
Could
cre
ate
incentive t
o
remove
err
or
handle
rs!
16
Analysis of Software Artifacts –
Aldrich ©
2009
White B
ox:
Sta
tem
ent
Covera
ge
•Statement coverage
�W
hat
port
ion o
f pro
gra
m s
tate
ments
(nodes)
are
touched b
y t
est
cases
•Advanta
ges
�Test
suite s
ize lin
ear
in s
ize o
f code
�Covera
ge e
asily a
ssessed
•Is
sues
�D
ead c
ode is n
ot
reached
�M
ay r
equire s
om
e s
ophis
tication t
osele
ct
input
sets
(M
cCabe b
asis
path
s)
�Fault-t
ole
rant
err
or-
handling c
ode
may b
e d
ifficult t
o “
touch”
�M
etr
ic:
Could
cre
ate
incentive t
o
remove
err
or
handle
rs!
17
Analysis of Software Artifacts –
Aldrich ©
2009
White B
ox:
Sta
tem
ent
Covera
ge
•Statement coverage
�W
hat
port
ion o
f pro
gra
m s
tate
ments
(nodes)
are
touched b
y t
est
cases
•Advanta
ges
�Test
suite s
ize lin
ear
in s
ize o
f code
�Covera
ge e
asily a
ssessed
•Is
sues
�D
ead c
ode is n
ot
reached
�M
ay r
equire s
om
e s
ophis
tication t
osele
ct
input
sets
(M
cCabe b
asis
path
s)
�Fault-t
ole
rant
err
or-
handling c
ode
may b
e d
ifficult t
o “
touch”
�M
etr
ic:
Could
cre
ate
incentive t
o
remove
err
or
handle
rs!
18
Analysis of Software Artifacts –
Aldrich ©
2009
White B
ox:
Bra
nch C
overa
ge
•Branch coverage
�W
hat
port
ion o
f conditio
n b
ranches a
recovere
d b
y t
est
cases?
�Or: W
hat
port
ion o
f re
lational expre
ssio
ns
and v
alu
es a
re c
overe
d b
y t
est
cases?
•Conditio
n t
esting (
Tai)
�Multiconditioncoverage
–all b
oole
an
com
bin
ations o
f te
sts
are
covere
d
•Advanta
ges
�Test
suite s
ize a
nd c
onte
nt
derived
from
str
uctu
re o
f boole
an e
xpre
ssio
ns
�Covera
ge e
asily a
ssessed
•Is
sues
�D
ead c
ode is n
ot
reached
�Fault-t
ole
rant
err
or-
handling c
ode
may b
e d
ifficult t
o “
touch”
19
Analysis of Software Artifacts –
Aldrich ©
2009
White B
ox:
Bra
nch C
overa
ge
•Branch coverage
�W
hat
port
ion o
f conditio
n b
ranches a
recovere
d b
y t
est
cases?
�Or: W
hat
port
ion o
f re
lational expre
ssio
ns
and v
alu
es a
re c
overe
d b
y t
est
cases?
•Conditio
n t
esting (
Tai)
�Multiconditioncoverage
–all b
oole
an
com
bin
ations o
f te
sts
are
covere
d
•Advanta
ges
�Test
suite s
ize a
nd c
onte
nt
derived
from
str
uctu
re o
f boole
an e
xpre
ssio
ns
�Covera
ge e
asily a
ssessed
•Is
sues
�D
ead c
ode is n
ot
reached
�Fault-t
ole
rant
err
or-
handling c
ode
may b
e d
ifficult t
o “
touch”
20
Analysis of Software Artifacts –
Aldrich ©
2009
White B
ox:
Bra
nch C
overa
ge
•Branch coverage
�W
hat
port
ion o
f conditio
n b
ranches a
recovere
d b
y t
est
cases?
�Or: W
hat
port
ion o
f re
lational expre
ssio
ns
and v
alu
es a
re c
overe
d b
y t
est
cases?
•Conditio
n t
esting (
Tai)
�Multiconditioncoverage
–all b
oole
an
com
bin
ations o
f te
sts
are
covere
d
•Advanta
ges
�Test
suite s
ize a
nd c
onte
nt
derived
from
str
uctu
re o
f boole
an e
xpre
ssio
ns
�Covera
ge e
asily a
ssessed
•Is
sues
�D
ead c
ode is n
ot
reached
�Fault-t
ole
rant
err
or-
handling c
ode
may b
e d
ifficult t
o “
touch”
21
Analysis of Software Artifacts –
Aldrich ©
2009
White B
ox:
Bra
nch C
overa
ge
•Branch coverage
�W
hat
port
ion o
f conditio
n b
ranches a
recovere
d b
y t
est
cases?
�Or: W
hat
port
ion o
f re
lational expre
ssio
ns
and v
alu
es a
re c
overe
d b
y t
est
cases?
•Conditio
n t
esting (
Tai)
�Multiconditioncoverage
–all b
oole
an
com
bin
ations o
f te
sts
are
covere
d
•Advanta
ges
�Test
suite s
ize a
nd c
onte
nt
derived
from
str
uctu
re o
f boole
an e
xpre
ssio
ns
�Covera
ge e
asily a
ssessed
•Is
sues
�D
ead c
ode is n
ot
reached
�Fault-t
ole
rant
err
or-
handling c
ode
may b
e d
ifficult t
o “
touch”
22
Analysis of Software Artifacts –
Aldrich ©
2009
White B
ox:
Path
Covera
ge
•Path coverage
�W
hat
port
ion o
f all p
ossib
le p
ath
s t
hro
ugh
the p
rogra
m a
re c
overe
d b
y t
ests
?�
Loop t
esting:
Consid
er
repre
senta
tive a
nd
edge c
ases:
•Zero
, one, tw
o ite
rations
•If
there
is a
bound n
: n-1
, n, n+
1 ite
rations
•N
este
d loops/c
onditio
nals
fro
m insid
e o
ut
•Advanta
ges
�Bett
er
covera
ge o
f lo
gic
al flow
s
•D
isadvanta
ges
�N
ot
all p
ath
s a
re p
ossib
le, or
necessary
•W
hat
are
the significant
path
s?
�Com
bin
ato
rial explo
sio
n in c
ases u
nle
ss
care
ful choic
es a
re m
ade
•E.g
., s
equence o
f nif
tests
can y
ield
up t
o 2
^n p
ossib
le p
ath
s
�Assum
ption t
hat
pro
gra
m s
tructu
reis
basic
ally s
ound
23
Analysis of Software Artifacts –
Aldrich ©
2009
White B
ox:
Path
Covera
ge
•Path coverage
�W
hat
port
ion o
f all p
ossib
le p
ath
s t
hro
ugh
the p
rogra
m a
re c
overe
d b
y t
ests
?�
Loop t
esting:
Consid
er
repre
senta
tive a
nd
edge c
ases:
•Zero
, one, tw
o ite
rations
•If
there
is a
bound n
: n-1
, n, n+
1 ite
rations
•N
este
d loops/c
onditio
nals
fro
m insid
e o
ut
•Advanta
ges
�Bett
er
covera
ge o
f lo
gic
al flow
s
•D
isadvanta
ges
�N
ot
all p
ath
s a
re p
ossib
le, or
necessary
•W
hat
are
the significant
path
s?
�Com
bin
ato
rial explo
sio
n in c
ases u
nle
ss
care
ful choic
es a
re m
ade
•E.g
., s
equence o
f nif
tests
can y
ield
up t
o 2
^n p
ossib
le p
ath
s
�Assum
ption t
hat
pro
gra
m s
tructu
reis
basic
ally s
ound
24
Analysis of Software Artifacts –
Aldrich ©
2009
White B
ox:
Path
Covera
ge
•Path coverage
�W
hat
port
ion o
f all p
ossib
le p
ath
s t
hro
ugh
the p
rogra
m a
re c
overe
d b
y t
ests
?�
Loop t
esting:
Consid
er
repre
senta
tive a
nd
edge c
ases:
•Zero
, one, tw
o ite
rations
•If
there
is a
bound n
: n-1
, n, n+
1 ite
rations
•N
este
d loops/c
onditio
nals
fro
m insid
e o
ut
•Advanta
ges
�Bett
er
covera
ge o
f lo
gic
al flow
s
•D
isadvanta
ges
�N
ot
all p
ath
s a
re p
ossib
le, or
necessary
•W
hat
are
the significant
path
s?
�Com
bin
ato
rial explo
sio
n in c
ases u
nle
ss
care
ful choic
es a
re m
ade
•E.g
., s
equence o
f nif
tests
can y
ield
up t
o 2
^n p
ossib
le p
ath
s
�Assum
ption t
hat
pro
gra
m s
tructu
reis
basic
ally s
ound
25
Analysis of Software Artifacts –
Aldrich ©
2009
White B
ox:
Path
Covera
ge
•Path coverage
�W
hat
port
ion o
f all p
ossib
le p
ath
s t
hro
ugh
the p
rogra
m a
re c
overe
d b
y t
ests
?�
Loop t
esting:
Consid
er
repre
senta
tive a
nd
edge c
ases:
•Zero
, one, tw
o ite
rations
•If
there
is a
bound n
: n-1
, n, n+
1 ite
rations
•N
este
d loops/c
onditio
nals
fro
m insid
e o
ut
•Advanta
ges
�Bett
er
covera
ge o
f lo
gic
al flow
s
•D
isadvanta
ges
�N
ot
all p
ath
s a
re p
ossib
le, or
necessary
•W
hat
are
the significant
path
s?
�Com
bin
ato
rial explo
sio
n in c
ases u
nle
ss
care
ful choic
es a
re m
ade
•E.g
., s
equence o
f nif
tests
can y
ield
up t
o 2
^n p
ossib
le p
ath
s
�Assum
ption t
hat
pro
gra
m s
tructu
reis
basic
ally s
ound
26
Analysis of Software Artifacts –
Aldrich ©
2009
White B
ox:
Path
Covera
ge
•Path coverage
�W
hat
port
ion o
f all p
ossib
le p
ath
s t
hro
ugh
the p
rogra
m a
re c
overe
d b
y t
ests
?�
Loop t
esting:
Consid
er
repre
senta
tive a
nd
edge c
ases:
•Zero
, one, tw
o ite
rations
•If
there
is a
bound n
: n-1
, n, n+
1 ite
rations
•N
este
d loops/c
onditio
nals
fro
m insid
e o
ut
•Advanta
ges
�Bett
er
covera
ge o
f lo
gic
al flow
s
•D
isadvanta
ges
�N
ot
all p
ath
s a
re p
ossib
le, or
necessary
•W
hat
are
the significant
path
s?
�Com
bin
ato
rial explo
sio
n in c
ases u
nle
ss
care
ful choic
es a
re m
ade
•E.g
., s
equence o
f nif
tests
can y
ield
up t
o 2
^n p
ossib
le p
ath
s
�Assum
ption t
hat
pro
gra
m s
tructu
reis
basic
ally s
ound
27
Analysis of Software Artifacts –
Aldrich ©
2009
White B
ox:
Assessin
g s
tructu
ral covera
ge
•Covera
ge a
ssessm
ent
tools
�Tra
ck e
xecution o
f code b
y t
est
cases
�Techniq
ues
•M
odifie
d r
untim
e e
nvironm
ent
(e.g
., s
pecia
l JV
M)
•Sourc
e c
ode t
ransfo
rmation
•Count
vis
its t
o s
tate
ments
�D
evelo
p r
eport
s w
ith r
espect
to s
pecific
covera
ge c
rite
ria
•Exam
ple
: EclE
mm
a–
Eclipse p
lugin
for
JUnit t
est
covera
ge
28
Analysis of Software Artifacts –
Aldrich ©
2009
EclE
mm
ain
Eclipse
•Bre
akdow
n b
y p
ackage,
cla
ss, and m
eth
od
•Covera
ge
�Cla
sses
�M
eth
ods
�Sta
tem
ents
�In
str
uctions
•G
raphic
al and n
um
erical
pre
senta
tion
29
Analysis of Software Artifacts –
Aldrich ©
2009
Clo
ver
in E
clipse
•Covera
ge r
eport
in e
ditor
win
dow
�re
d:
not
covere
d�
yellow
: covere
d o
nce
�gre
en:
covere
d m
ultip
le t
imes
30
Analysis of Software Artifacts –
Aldrich ©
2009
Benefits
of W
hite-B
ox
•Tool support
can m
easure
covera
ge
�H
elp
s t
o e
valu
ate
test
suite (
care
ful!)
�Can fin
d u
nte
ste
d c
ode
•Can t
est
pro
gra
m o
ne p
art
at
a t
ime
•Can c
onsid
er
code-r
ela
ted b
oundary
conditio
ns
�If
conditio
ns
�Boundaries o
f fu
nction input/
outp
ut
ranges
•e.g
. sw
itch b
etw
een a
lgorith
ms a
t data
siz
e=
100
•Can fin
d late
nt
faults
�Cannot
trig
ger
a failure
in p
rogra
m, but
can b
e found b
y a
unit t
est
31
Analysis of Software Artifacts –
Aldrich ©
2009
White B
ox:
Lim
itations
•Is
it
possib
le t
o a
chie
ve 1
00%
covera
ge?
•Can y
ou t
hin
k o
f a p
rogra
m t
hat
has a
defe
ct,
even t
hough
it p
asses a
test
suite w
ith 1
00%
covera
ge?
32
Analysis of Software Artifacts –
Aldrich ©
2009
White B
ox:
Lim
itations
•Is
it
possib
le t
o a
chie
ve 1
00%
covera
ge?
•Can y
ou t
hin
k o
f a p
rogra
m t
hat
has a
defe
ct,
even t
hough
it p
asses a
test
suite w
ith 1
00%
covera
ge?
•Exclu
siv
e focus o
n c
overa
ge focus m
isses im
port
ant
bugs
�M
issin
g c
ode
�In
corr
ect
boundary
valu
es
�Tim
ing p
roble
ms
�Configura
tion issues
�D
ata
/mem
ory
corr
uption b
ugs
�U
sability p
roble
ms
�Custo
mer
requirem
ents
issues
•Covera
ge is n
ot
a g
ood a
dequacy c
rite
rion
�In
ste
ad, use t
o fin
d p
laces w
here
testing is inadequate
33
Analysis of Software Artifacts –
Aldrich ©
2009
Testing e
xam
ple
•Equiv
ale
nce c
lasses?
•Boundary
valu
es?
•Robustn
ess t
ests
?
•H
ow
to a
chie
ve line coverage?public static int binsrch
(int[] a, int ke
y) {
int low
= 0
;int high= a.leng
th -
1;
while (true
) {
if ( low
> high) re
turn
-(low
+1);
int mid
= (low+high) / 2;
if ( a[m
id] < key ) low
= m
id+ 1
;else if ( a[m
id] > key ) high= m
id-
1;
else re
turn
mid;
}}