d, 2004. spain), january 13r roc analysis icos y ... · 1 classifier evaluation in data mining: roc...
TRANSCRIPT
1
Cla
ssif
ier
Evalu
ati
on
in D
ata
Min
ing
:
RO
C A
naly
sis
José
Her
nánd
ez-O
rallo
Alb
acet
e(S
pain
), Ja
nuar
y 13
rd, 2
004.
Dpt
o. d
e Si
stem
as In
form
átic
os y
Com
puta
ción
,U
nive
rsid
ad P
olité
cnic
a de
Val
enci
a,
jora
llo@
dsic
.upv
.es
2
Out
line
•In
trodu
ctio
n. T
he C
lass
ifica
tion
Task
and
its
Eva
luat
ion
•S
kew
-sen
sitiv
e E
valu
atio
n
•R
OC
Ana
lysi
s fo
r Cris
p C
lass
ifier
s
•R
OC
Ana
lysi
s fo
r Sof
t Cla
ssifi
ers
•Th
e A
UC
Met
ric: t
he A
rea
Und
er th
e R
OC
Cur
ve
•R
elat
ion
betw
een
AU
C a
nd E
rror
. Thr
esho
ld c
hoic
e.
•A
pplic
atio
ns
•E
xten
sion
to M
ore
than
Tw
o C
lass
es
•C
oncl
usio
ns
3
Intro
duct
ion.
The
Cla
ssifi
catio
nTa
skan
dits
Eva
luat
ion
•C
lass
ifica
tion.
–O
ne o
f the
mos
t im
porta
nt ta
sks
in d
ata
min
ing.
–
Goa
l: to
obt
ain
a m
odel
, pat
tern
or f
unct
ion
that
tells
be
twee
n tw
o or
mor
e ex
clus
ive
clas
ses.
•C
lass
ifica
tion
Eva
luat
ion.
–Tr
aditi
onal
Mea
sure
for E
valu
atin
g C
lass
ifier
s:•
Err
or (a
lso
inve
rsel
y ac
cura
cy):
perc
enta
ge o
f bad
ly
clas
sifie
d in
stan
ces
(wrt.
the
test
set
or u
sing
cro
ss-
valid
atio
n / b
oots
trapp
ing)
.
4
Intro
duct
ion.
The
Cla
ssifi
catio
nTa
skan
dits
Eva
luat
ion
•A
cla
ssifi
erpr
ovid
essu
ppor
tin
deci
sion
mak
ing
(bet
wee
ndi
ffere
ntac
tions
).
Are
we
still
mak
ing
deci
sion
sin
an
unsc
ient
ific
way
?
•Th
is q
uest
ion
is ra
ised
by:
–Sw
ets,
J.A
., D
awes
, R.M
., &
Mon
ahan
, J. (
2000
). “B
ette
r dec
isio
ns th
roug
h sc
ienc
e”Sc
ient
ific
Amer
ican ,
283
, 82-
87.
5
Ske
w-s
ensi
tive
Eva
luat
ion
•D
istri
butio
n-se
nsiti
ve E
valu
atio
n:–
The
clas
ses
of th
e pr
oble
m d
o no
t hav
e th
e sa
me
dist
ribut
ion
(they
are
not
wel
l bal
ance
d, e
.g. 5
0% e
ach
of th
em)
•E
xam
ple:
–W
e ha
ve s
ever
al c
lass
ifier
s c 1
, c2,
c 3th
at p
redi
ct w
heth
er a
re
frige
ratio
n va
lve
has
to b
e op
ened
or c
lose
d in
a n
ucle
ar
plan
t.–
In o
rder
to e
valu
ate
the
clas
sifie
rs w
e us
e a
data
set o
btai
ned
durin
g la
st m
onth
, whe
re a
n op
erat
or h
as b
een
deci
ding
to
open
or t
o cl
ose
the
valv
e.•
100,
000
exam
ples
, fro
m w
hich
99,
500
are
of c
lass
“Clo
se”a
nd
500
are
of c
lass
“Ope
n”.
–Le
t’s s
ay th
at c
lass
ifier
c2
alw
ays
pred
ict “
Clo
se”(
trivi
al
clas
sifie
r).
•E
rror
of c
2: 0.
5%.
Isth
isa
good
clas
sifie
r?
6
Ske
w-s
ensi
tive
Eva
luat
ion
•C
onfu
sion
/con
tinge
ncy
Mat
rix(e
.g. f
orth
ete
stse
t):Ac
tual
TNFN
CLO
SE (N
)FP
TPO
PEN
(P)
clos
e(n)
open
(p)
Dia
gona
l of
hits
Pred
icte
d
•Fr
omhe
re, s
ever
alm
etric
sha
vebe
ende
fined
:–
Pr(
P|p
) ≈Tr
ue P
ositi
ve R
ate:
TP
R =
TP
/ (T
P +
FN
). (“r
ecal
l”ot
”sen
sitiv
ity”o
t “po
sitiv
e ac
cura
cy”).
–P
r(N|p
) ≈Fa
lse
Neg
ativ
e R
ate:
FN
R =
FN
/ (T
P +
FN
). (“p
ositi
ve e
rror”)
–P
r(N|n
) ≈Tr
ue N
egat
ive
Rat
e: T
NR
= T
N /
(TN
+ F
P).
(”spe
cific
ity”o
t ”ne
gativ
e ac
cura
cy”).
–P
r(P
|n) ≈
Fals
e P
ositi
ve R
ate:
FP
R =
FP
/ (T
N +
FP
). (“
nega
tive
erro
r”)–
Pr(
p|P
) ≈P
ositi
ve P
redi
ctiv
e V
alue
: PP
V =
TP
/ (T
P +
FP
). (”p
reci
sion
”).
–P
r(n|N
) ≈N
egat
ive
Pre
dict
ive
Val
ue: N
PV
= T
N /
(TN
+ F
N).
–M
acro
-ave
rage
= A
VG
(TP
R, T
NR
). (T
hem
ean
can
be a
rithm
etic
, geo
met
ricor
othe
r)
–B
RE
AK
-EV
EN
=(P
reci
sion
+ R
ecal
l) / 2
= (P
PV
+ T
PR
) / 2
–F-
ME
AS
UR
E=
(Pre
cisi
on* R
ecal
l) / B
RE
AK
-EV
EN
= 2
*PP
V*T
PR
/ (P
PV
+ T
PR
)
7
•E
xam
ple:
(te
stda
tase
twith
100,
000
inst
ance
s)
9900
020
0C
LOSE
500
300
OPE
Ncl
ose
open
c 1
Actu
al
Pred
.94
100
100
CLO
SE54
0040
0O
PEN
clos
eop
enc 3
Actu
al
9950
050
0C
LOSE
00
OPE
Ncl
ose
open
c 2
Actu
al
Ske
w-s
ensi
tive
Eva
luat
ion
ERR
OR
: 0,7
%
TPR
= 30
0 / 5
00 =
60%
FNR
=20
0 / 5
00 =
40%
TNR
= 99
000
/ 995
00 =
99,
5%FP
R=
500
/ 995
00 =
0.5
%PP
V= 3
00 /
800
= 37
.5%
NPV
= 99
000
/ 992
00 =
99.
8%
Mac
roav
g= (6
0 +
99.5
) / 2
=79
.75%
ERR
OR
: 0,5
%
TPR
= 0
/ 500
= 0
%FN
R=
500
/ 500
= 1
00%
TNR
= 99
500
/ 995
00 =
100
%FP
R=
0 / 9
9500
= 0
%PP
V= 0
/ 0
= IN
DEF
INID
ON
PV=
9950
0 / 1
0000
= 9
9.5%
Mac
roav
g= (0
+ 1
00 )
/ 2 =
50%
ERR
OR
: 5,5
%
TPR
= 40
0 / 5
00 =
80%
FNR
=10
0 / 5
00 =
20%
TNR
=94
100
/ 995
00 =
94.
6%FP
R=
5400
/ 99
500
= 5.
4%PP
V= 4
00 /
5800
= 6
.9%
NPV
= 94
100
/ 942
00 =
99.
9%
Mac
roav
g= (8
0 +
94.6
) / 2
=87
.3%
SpecificitySensitivity Recall Precision
Whi
chcl
assi
fieri
sbe
st?
8
Ske
w-s
ensi
tive
Eva
luat
ion
•C
ost-s
ensi
tive
Eva
luat
ion:
–In
man
yci
rcum
stan
ces,
not
allt
heer
rors
prod
uced
by a
pr
edic
tive
mod
elha
veth
esa
me
cons
eque
nces
:•
Exa
mpl
e: k
eepi
nga
valv
ecl
osed
in a
nuc
lear
pla
ntw
hen
itsh
ould
be o
pen,
can
pro
voke
anex
plos
ion,
whi
leop
enin
ga
valv
ew
hen
itsh
ould
be c
lose
d, c
an p
rovo
kea
stop
.
–C
ostm
atrix
:
–Th
eim
porta
ntth
ing
isno
tto
obta
inth
e“c
lass
ifier
” with
few
er
erro
rs b
ut th
e on
e w
ith lo
wes
t cos
t.–
From
this
mat
rix, w
eca
n co
mpu
te th
eco
stof
a cl
assi
fier.
•C
lass
ifier
sar
e ev
alua
ted
with
thes
eco
sts.
•Th
ecl
assi
fierw
ithlo
wes
tcos
tis
chos
en.
Actu
al
020
00€
CLO
SE10
0€0
OPE
Ncl
ose
open
Pred
icte
d
9
Ske
w-s
ensi
tive
Eva
luat
ion
•E
xam
ples
: Actu
alAc
tual
Actu
alC
onfu
sion
Mat
rices 0
2000
€C
LOSE
100€
0O
PEN
clos
eop
en
Actu
al
Pred
icte
d
9900
020
0C
LOSE
500
300
OPE
Ncl
ose
open
c 1
Pred
9410
010
0C
LOSE
5400
400
OPE
Ncl
ose
open
c 3
9950
050
0C
LOSE
00
OPE
Ncl
ose
open
c 2
Cos
tM
atrix
0€40
0,000
€C
LOSE
50,00
0€0€
OPE
Ncl
ose
open
c 1
0€20
0,000
€C
LOSE
540,0
00€
0€O
PEN
clos
eop
enc 3
0€1,0
00,00
0€C
LOSE
0€0€
OPE
Ncl
ose
open
c
Res
ultin
gM
atric
es
2
TOTA
L C
OST
: 450
,000
€TO
TAL
CO
ST: 1
,000
,000
€TO
TAL
CO
ST: 7
40,0
00€
10
Ske
w-s
ensi
tive
Eva
luat
ion
•W
hata
ffect
sth
efin
al c
ost?
–Fo
rtw
ocl
asse
s. It
depe
nds
ona
cont
exto
rske
w:
•Th
eco
stof
fals
epo
sitiv
esan
dfa
lse
nega
tives
: FP
cost
and
FNco
st•
The
perc
enta
geof
exam
ples
ofth
ene
gativ
ecl
ass
wrt.
the
exam
ples
ofth
epo
sitiv
ecl
ass.
(Neg
/ Pos
).•
We
com
pute
: (fo
rthe
prev
ious
exam
ples
)
20120
0010
0=
=FN
cost
FPco
st19
950
099
500=
=Po
sN
eg1
199
9.95
20sl
ope=
×=
–Fo
rtw
ocl
asse
s, th
eva
lue
“slo
pe” i
s su
ffici
ent t
o de
term
ine
whi
ch c
lass
ifier
is b
est.
Cla
ssif.
1: F
NR
= 40
%, F
PR=
0.5%
Cos
t per
uni
t =
1 x
0.40
+ 9
.95
x 0.
005
= 0.
45
Cla
ssif.
2: F
NR
= 10
0%, F
PR=
0%C
ost p
er u
nit =
1
x 1
+ 9.
95x
0 =
1
Cla
ssif.
3: F
NR
= 20
%, F
PR=
5.4%
Cos
t per
uni
t =
1 x
0.20
+ 9
.95
x 0.
054
= 0.
74
11
RO
C A
naly
sis
ofC
risp
Cla
ssifi
ers
•Th
e cl
assi
fier w
ith lo
wes
t err
or ra
te is
freq
uent
ly n
ot th
e be
st c
lass
ifier
.•
The
cont
exto
r ske
w(th
e cl
ass
dist
ribut
ion
and
the
cost
sof
eac
h er
ror)
det
erm
ine
the
good
ness
of
clas
sifie
rs.
•P
RO
BLE
M:
–In
man
y ci
rcum
stan
ces,
unt
il th
e ap
plic
atio
n tim
e, w
e do
not
kn
ow th
e cl
ass
dist
ribut
ion
and/
or it
is d
iffic
ult t
o es
timat
e th
e co
st m
atrix
. E.g
. a s
pam
filte
r.–
But
mod
els
are
lear
ned
befo
re.
•R
OC
(Rec
eive
r Ope
ratin
g C
hara
cter
istic
) Ana
lysi
s.–
Firs
t use
d in
the
WW
2 to
eva
luat
e ra
dars
, lat
er w
as u
sed
to s
tudy
the
resp
onse
of t
rans
isto
rs, a
nd fu
rther
dev
elop
ed fo
r med
ical
dia
gnos
is
appl
icat
ions
from
the
seve
ntie
s. It
beg
ins
to b
e kn
own
in d
ata
min
ing
in th
e la
te n
inet
ies.
12
RO
C A
naly
sis
ofC
risp
Cla
ssifi
ers
•Th
eR
OC
Spa
ce–
Eac
hco
lum
nof
the
conf
usio
nm
atrix
isno
rmal
ised
: TP
R, F
NR
TN
R, F
PR
.Ac
tual
Actu
al
ROC
Spac
e
0,00
0
0,20
0
0,40
0
0,60
0
0,80
0
1,00
0
0,00
00,
200
0,40
00,
600
0,80
01,
000
Fals
e Po
sitiv
es
True Positives
8750
010
0C
LOSE
1200
040
0O
PEN
clos
eop
en
0.879
0.2C
LOSE
0.121
0.8O
PEN
clos
eop
en
TPR
= 40
0 / 5
00 =
80%
FNR
=10
0 / 5
00 =
20%
TNR
= 87
500
/ 995
00 =
87.
9%FP
R=
1200
0 / 9
9500
= 1
2.1%
Pred
Pred
13
RO
C A
naly
sis
ofC
risp
Cla
ssifi
ers
•R
OC
spa
ce: g
ood
and
bad
clas
sifie
rs.
01
1 0FP
R
TPR •
Goo
dcl
assi
fier.
–H
igh
TPR
.–
Low
FPR
.
01
1 0FP
R
TPR
01
1 0FP
R
TPR •
Bad
clas
sifie
r.–
Low
TPR
.–
Hig
hFP
R.
•B
adcl
assi
fier
(rea
l pic
ture
).
14
RO
C A
naly
sis
ofC
risp
Cla
ssifi
ers
•Th
eR
OC
Cur
ve R
OC
. “C
ontin
uity
”.R
OC
dia
gram
01
1 0FP
R
TPR
We
can
cons
truct
any
“in
term
edia
te” c
lass
ifier
just
ra
ndom
ly w
eigh
ting
both
cl
assi
fiers
(giv
ing
mor
e or
less
w
eigh
t to
one
or th
e ot
her).
This
cre
ates
a “c
ontin
uum
” of
clas
sifie
rs b
etw
een
any
two
clas
sifie
rs.
Giv
en tw
o cl
assi
fiers
:
15
RO
C A
naly
sis
ofC
risp
Cla
ssifi
ers
•R
OC
Cur
ve. C
onst
ruct
ion.
RO
C d
iagr
am
01
1 0FP
R
TPR
We
cons
truct
the
conv
ex h
ull o
f th
eir p
oint
s (F
PR,T
PR) a
s w
ell a
s th
e tw
o tri
vial
cla
ssifi
ers
(0,0
) and
(1
,1).
The
clas
sifie
rs b
elow
the
RO
C
curv
e ar
e di
scar
ded.
The
best
cla
ssifi
er (f
rom
thos
e re
mai
ning
) will
be s
elec
ted
in
appl
icat
ion
time…
Giv
en s
ever
al c
lass
ifier
s:
We
can
disc
ard
thos
e w
hich
are
bel
ow b
ecau
se th
ere
is n
o co
mbi
natio
n of
cla
ss
dist
ribut
ion
/ cos
t mat
rix fo
r whi
ch th
ey c
ould
be
optim
al.
The
diag
onal
sh
ows
the
wor
st
situ
atio
n po
ssib
le.
16
RO
C A
naly
sis
ofC
risp
Cla
ssifi
ers
•In
the
cont
exto
fapp
licat
ion,
we
choo
seth
eop
timal
clas
sifie
rfro
mth
ose
kept
. Exa
mpl
e1:
0%20%
40%
60%
80%
100%
0%20
%40
%60
%80
%10
0%
fals
e po
sitiv
e ra
te
true positive rate
FPco
stFN
cost=
1 2
Neg Po
s=
4
slop
e=
42=
2
Con
text
(ske
w):
17
RO
C A
naly
sis
ofC
risp
Cla
ssifi
ers
•In
the
cont
exto
fapp
licat
ion,
we
choo
seth
eop
timal
clas
sifie
rfro
mth
ose
kept
. Exa
mpl
e2:
0%20%
40%
60%
80%
100%
0%20
%40
%60
%80
%10
0%
fals
e po
sitiv
e ra
te
true positive rate
FPco
stFN
cost=
1 8
Neg Po
s=
4
slop
e=
48=
.5
Con
text
(ske
w):
18
RO
C A
naly
sis
ofC
risp
Cla
ssifi
ers
•W
hath
ave
we
lear
ned
from
this
?–
The
optim
ality
ofa
clas
sifie
rdep
ends
onth
ecl
ass
dist
ribut
ion
and
the
erro
r cos
ts.
–Fr
omth
isco
ntex
t/ s
kew
we
can
obta
inth
e“s
lope
”, w
hich
cha
ract
eris
es th
is c
onte
xt.
•If
we
know
this
cont
ext,
we
can
sele
ctth
ebe
st c
lass
ifier
, m
ultip
lyin
gth
eco
nfus
ion
mat
rixan
dth
eco
stm
atrix
.•
Ifw
edo
n’t k
now
this
con
text
in th
e le
arni
ng s
tage
, by
usin
g R
OC
ana
lysi
s w
e ca
n ch
oose
a s
ubse
t of c
lass
ifier
s, fr
om
whi
ch th
e op
timal
cla
ssifi
er w
ill b
e se
lect
ed w
hen
the
cont
ext
is k
now
n. Can
we
gofu
rther
than
this
?
19
RO
C A
naly
sis
ofS
oftC
lass
ifier
s
•C
risp
and
Sof
tCla
ssifi
ers:
–A
“cris
p” c
lass
ifier
pre
dict
s a
clas
s be
twee
n a
set o
f pos
sibl
e cl
asse
s.–
A “s
oft”
clas
sifie
r (pr
obab
ilistic
) pre
dict
s a
clas
s, b
ut
acco
mpa
nies
eac
h pr
edic
tion
with
an
estim
atio
n of
the
relia
bilit
y (c
onfid
ence
) of e
ach
pred
ictio
n.•
Mos
tlea
rnin
gm
etho
dsca
n be
ada
pted
toge
nera
teth
isco
nfid
ence
valu
e.
•A
spe
cial
kind
ofso
ftcl
assi
fieri
sa
clas
spr
obab
ility
estim
ator
.–
Inst
ead
ofpr
edic
ting
“a”,
“b” o
r “c”
, it g
ives
a p
roba
bilit
y es
timat
ion
for “
a”, “
b” o
r “c”
, i.e
., “p
a”, “
p b” a
nd “p
c”. E
xam
ple:
•C
lass
ifier
1: p
a=0.
2, p
b=0.
5 an
dp c
=0.
3.•
Cla
ssifi
er2:
pa=
0.3,
pb=
0.4
and
p c=
0.3.
–B
oth
pred
ictb
, but
clas
sifie
r1 is
“sur
er”.
20
RO
C A
naly
sis
ofS
oftC
lass
ifier
s
•“R
anke
rs”:
–W
hene
verw
eha
vea
prob
abili
tyes
timat
orfo
ra tw
o-cl
ass
prob
lem
:•
p a=
x, th
enp b
= 1 −
x.–
Itis
only
nece
ssar
yto
spec
ifyth
epr
obab
ility
ofon
ecl
ass.
–Le
t’s c
all o
ne c
lass
0 (n
eg) a
nd th
e ot
her c
lass
1 (p
os).
–A
rank
eris
a so
ftcl
assi
fiert
hatg
ives
a va
lue
betw
een
0 an
d1
ofth
epr
obab
ility
ofcl
ass
1. T
his
valu
eis
also
calle
d“s
core
” and
de
term
ines
whe
ther
the
pred
ictio
n is
clo
ser t
o cl
ass
0 or
cla
ss 1
.–
Exa
mpl
es:
•P
roba
bilit
yof
a cu
stom
erbu
ying
a pr
oduc
t.•
Pro
babi
lity
ofa
mes
sage
bein
gsp
am.
•...
21
RO
C A
naly
sis
ofS
oftC
lass
ifier
s
•R
OC
Cur
ve o
fa S
oftC
lass
ifier
:–
A s
oftc
lass
ifier
can
be c
onve
rted
into
a cr
isp
clas
sifie
rusi
nga
thre
shol
d.•
Exa
mpl
e: “i
f sco
re >
0.7
then
cla
ss A
, oth
erw
ise
clas
s B
”.•
With
diffe
rent
thre
shol
ds, w
eha
vedi
ffere
ntcl
assi
fiers
, gi
ving
mor
e or
less
rele
vanc
eto
each
ofth
ecl
asse
s(w
ithou
tnee
dof
over
sam
plin
gan
dun
ders
ampl
ing)
.–
We
can
cons
ider
each
thre
shol
das
a d
iffer
ent
clas
sifie
rand
draw
them
in th
eR
OC
spa
ce. T
his
gene
rate
sa
curv
e…
We
have
a “c
urve
” for
just
one
sof
t cla
ssifi
er
•Th
iscu
rve
islik
ea
stai
r(th
eco
nvex
hull
isno
tdon
e ge
nera
lly).
22
RO
C A
naly
sis
ofS
oftC
lass
ifier
sA
ctua
l Cla
ss
•R
OC
Cur
ve o
fa S
oftC
lass
ifier
:–
Exa
mpl
e:n n n n n n n n n n n n n n n n n n n n
Pred
icte
dC
lass p p p p p p p p p p p p p p p p p p p p
p n n n n n
n n n
n n n
n n n
n n n
n n
p p n n n n
n n n
n n n
n n n
n n n
n n
...
© T
omFa
wce
tt
23
RO
C A
naly
sis
ofS
oftC
lass
ifier
s
•R
OC
Cur
ve o
fa S
oftC
lass
ifier
:
© T
omFa
wce
tt
24
RO
C A
naly
sis
ofS
oftC
lass
ifier
s
•R
OC
Ana
lysi
sof
seve
ral“
soft”
cla
ssifi
ers:
In th
iszo
neth
ebe
st
clas
sifie
ris
“inst
s”
In th
iszo
neth
ebe
st
clas
sifie
ris“
inst
s2”
•W
em
ustp
rese
rve
the
clas
sifie
rsth
atha
veat
leas
tone
“bes
t zon
e” a
nd
then
beh
ave
in th
e sa
me
way
as
we
did
for c
risp
clas
sifie
rs.
© R
ober
tHol
te
25
The
“AU
C” M
etric
: the
Are
a U
nder
the
RO
C C
urve
RO
C c
urve
0,00
0
0,20
0
0,40
0
0,60
0
0,80
0
1,00
0
0,00
00,
200
0,40
00,
600
0,80
01,
000
Fals
e Po
sitiv
es
True Positives
•W
hati
fwe
wan
tto
sele
ctju
ston
ecl
assi
fier?
–Th
ecl
assi
fierw
ithgr
eate
stAr
eaU
nder
the
RO
C C
urve
(A
UC
) is
chos
en.
AU
C
–Fo
rcris
pcl
assi
fiers
itis
equi
vale
ntto
the
mac
roav
erag
e.
Alte
rnat
ive
toer
ror f
orev
alua
ting
clas
sifie
rs•
A d
ata
min
ing
/ lea
rnin
gm
etho
dw
illbe
bet
teri
fit
gene
rate
scl
assi
fiers
with
high
AUC
.
26
The
“AU
C” M
etric
: the
Are
a U
nder
the
RO
C C
urve
•W
hati
fwe
wan
tto
sele
ctju
ston
eso
ftcl
assi
fier?
–Th
ecl
assi
fierw
ithgr
eate
stAr
eaU
nder
the
RO
C C
urve
(A
UC
) is
chos
en.
In th
isca
se w
ese
lect
B.
© T
omFa
wce
tt
But
fort
he“s
oft”
case
we
have
sur
pris
es…
27
The
“AU
C” M
etric
: the
Are
a U
nder
the
RO
C C
urve
•Th
e A
UC
and
the
Wilc
oxon
-Man
n-W
hitn
ey (W
MW
) (W
ilcox
on19
45) (
Man
n &
Whi
tney
194
7) s
tatis
tic a
re e
quiv
alen
t .–
The
WM
W te
st is
use
ful t
o de
term
ine
whe
ther
one
of t
wo
rand
om
varia
bles
is s
toch
astic
ally
gre
ater
than
the
othe
r.P[
X>Y]
–If
we
choo
se X
as th
e ex
ampl
es o
f one
cla
ss a
nd Y
as th
e ex
ampl
es o
f th
e ot
her c
lass
, and
the
valu
es X
and
Yas
the
estim
ated
sco
reby
the
clas
sifie
r, w
e ha
ve th
at th
e A
UC
is e
quiv
alen
t to
the
WM
W s
tatis
tic.
Sho
uld
I be
happ
yof
this
?Th
eA
UC
real
lyes
timat
esth
epr
obab
ility
that
, ifw
ech
oose
anex
ampl
eof
clas
s1
and
anex
ampl
eof
clas
s0,
the
clas
sifie
rwill
give
mor
e sc
ore
toth
efir
ston
eth
anto
the
seco
ndon
e.
(Car
e! T
his
does
notm
ean
that
itcl
assi
fies
wel
lbot
hex
ampl
es).
But:
•Th
ere
does
exis
ta th
resh
old
from
whi
chit
can
clas
sify
both
corre
ctly
.•
Itca
nnot
clas
sify
both
wro
ngly
, for
any
thre
shol
dw
hats
oeve
r.
28
The
“AU
C” M
etric
: the
Are
a U
nder
the
RO
C C
urve
•Th
e A
UC
met
ric fo
r sof
t cla
ssife
rsor
rank
ers.
–E
valu
ates
how
wel
l a c
lass
ifier
per
form
s a
rank
ing
of it
s pr
edic
tions
.or
, in
othe
r wor
ds,
–E
valu
ates
how
wel
l a c
lass
ifier
is a
ble
to s
ort i
ts p
redi
ctio
ns
acco
rdin
g to
the
conf
iden
ce it
ass
igns
to th
em.
–Th
e “r
anki
ngs”
of p
redi
ctio
ns a
re fu
ndam
enta
l in
man
y ap
plic
atio
ns:
•Fr
aud
dete
ctio
n.•
Mai
ling
cam
paig
n de
sign
.•
Spa
m fi
lterin
g.•
Failu
re d
etec
tion,
med
ical
dia
gnos
is.
•…
–A
nd m
any
othe
r dat
a m
inin
g m
etho
ds:
•C
ombi
natio
n of
cla
ssifi
ers.
•C
olla
bora
tive
Met
hods
. Rec
omm
ende
r sys
tem
s…
29
Rel
atio
nbe
twee
nA
UC
and
erro
r. Th
resh
old
choi
ce
•W
eha
vese
enth
atA
UC
isa
bette
reva
luat
ion
mea
sure
than
erro
r rat
e. B
ut, w
hich
rela
tion
isth
ere
betw
een
both
?–
Logi
cally
, an
AU
C c
lose
rto
1 w
illgi
vean
erro
r rat
ecl
ose
to0.
01
1 0FP
R
TPR
•W
hate
vers
kew
we
have
, a lo
wer
ror i
sex
pect
ed.
9950
050
0C
LOSE
00
OPE
NC
lose
open
c 2
01
1 0FP
R
TPR
–B
utan
erro
r rat
ecl
oser
to0
does
note
nsur
ean
AU
C c
lose
to1.
•Le
t’s re
call
the
nucl
ear p
lant
exa
mpl
e:
AUC
= 0
.5(M
inim
um p
ossi
ble
valu
e)M
acro
avg=
AU
C =
(0
+ 1
00 )
/ 2 =
50%
ERR
OR
: 0.
005
30
Rel
atio
nbe
twee
nA
UC
and
erro
r. Th
resh
old
choi
ce
•M
any
lear
ning
met
hods
are
soft
by d
efin
ition
and
are
conv
erte
din
tocr
isp
forp
erfo
rmin
gth
ecl
assi
ficat
ion.
–Fo
rexa
mpl
e, a
Naï
veB
ayes
cla
ssifi
er, f
ora
two-
clas
spr
oble
m(a
and
b), e
stim
ates
two
prob
abili
ties:
P(a|
x) a
ndP(
b|x)
.–
The
clas
sific
atio
nru
leis
the
follo
win
gon
e:
–Th
isru
le“w
aste
s” a
Bay
esia
n cl
assi
fier.
But
, whi
ch o
ther
rule
co
uld
be u
sed?
(Lac
hich
e &
Fla
ch 2
003)
•Fi
rst,
we
conv
ertt
hepr
obab
ilitie
sin
tosc
ores
, as
follo
ws.
s(x)
= P
(a|x
) / P
(b|x
)
IfP(
a|x)
> P
(b|x
) the
ncl
ass
aO
ther
wis
ecl
ase
b
31
Rel
atio
nbe
twee
nA
UC
and
erro
r. Th
resh
old
choi
ce
•N
ow, s
impl
y, le
t’s d
o R
OC
ana
lysi
s.–
We
draw
the
RO
C c
urve
usi
ngth
esc
ore.
–W
eco
mpu
te th
esl
ope
from
the
a pr
iori
clas
sdi
strib
utio
n(o
rthe
dist
ribut
ion
inap
plic
atio
ntim
e, a
ndco
sts
ifw
eha
veth
em).
–W
ech
oose
the
thre
shol
dbe
twee
nbo
thcl
asse
sfo
rs(
x).
© N
icol
asLa
chic
he&
Pet
erFl
ach
Cho
ice
usin
g ru
le
P(a|
x) >
P(b
|x)
Cho
ice
usin
g R
OC
ana
lysi
s an
d or
igin
al
dist
ribut
ion
Slop
egi
ven
by
the
orig
inal
di
strib
utio
n(tr
ain
orte
st).
32
Rel
atio
nbe
twee
nA
UC
and
erro
r. Th
resh
old
choi
ce
•E
xam
ple
ofre
sults
(acc
urac
y) fo
r25
data
sets
:(L
achi
che
& F
lach
2003
)
–R
esul
tsar
e im
prov
edby
ju
stch
angi
ngth
ede
cisi
onru
leof
the
Naï
veB
ayes
C
lass
ifier
.
© N
icol
asLa
chic
he&
Pet
erFl
ach
33
Rel
atio
nbe
twee
nA
UC
and
erro
r. Th
resh
old
choi
ce
•In
the
othe
r way
roun
d, it
wou
ld b
e in
tere
stin
g to
co
nver
t dat
a m
inin
g m
etho
ds th
at o
btai
n cr
isp
clas
sifie
rs in
to s
oft c
lass
ifier
s.–
This
wou
ld a
llow
its
use
as ra
nker
s, fo
r com
bina
tion,
...
–Th
is w
ould
mak
e it
poss
ible
to c
hoos
e a
bette
r thr
esho
ld a
nd
impr
ove
resu
lts.
•M
any
clas
sica
l met
hods
hav
e be
en re
desi
gned
and
re-
thou
ght.
For i
nsta
nce,
for d
ecis
ion
trees
:–
Dec
isio
n tre
es a
re le
arne
d us
ing
AU
C a
s sp
littin
g cr
iterio
n (F
erri
et a
l. 20
02).
–Th
e le
af p
roba
bilit
ies
are
smoo
thed
(usi
ng L
apla
ce c
orre
ctio
n or
oth
er m
ore
soph
istic
ated
sm
ooth
ing
met
hods
) and
pru
ning
is
show
n to
be
coun
terp
rodu
ctiv
e to
obt
ain
good
AU
C m
easu
res
(Pro
vost
& D
omin
gos
2003
) (Li
ng &
Yan
2003
) (Fe
rriet
al.
2003
a).
34
App
licat
ions
•An
exam
ple:
“mai
l cam
paig
n de
sign
”:–
A co
mpa
nyw
ants
tom
ake
a m
aile
dof
fert
oin
crea
seth
esa
les
ofon
eof
itspr
oduc
ts. I
n ca
se o
fpos
itive
repl
ycu
stom
ers
buy
prod
ucts
with
a m
ean
valu
eof
100€
. If 5
5% a
re p
rodu
ctio
n co
sts
(fix
and
varia
ble)
, we
have
that
for e
ach
posi
tive
repl
y th
ere
is a
m
ean
gain
of 4
5€.
–Ea
chpo
st s
enth
as a
cos
tof1
€ (m
ail,
leaf
let)
and
the
who
le o
f th
e ca
mpa
ign
(inde
p. o
f the
num
ber o
f pos
tings
) wou
ld h
ave
a ba
se c
ost o
f 20,
000€
.–
With
1,00
0,00
0 cu
stom
ers,
forw
hich
, with
a fir
stsa
mpl
eof
1,00
0 cu
stom
ers,
we
have
estim
ated
that
1% re
plie
s(b
uys)
...
How
shou
ldw
eac
t?
35
App
licat
ions
•“m
ail c
ampa
ign
desi
gn”:
–Le
t’s c
alcu
late
the
cost
s / b
enef
its.
•C
OST
S (a
ssum
ing
that
we
send
the
cam
paig
nto
allc
usto
mer
s):
–20
,000
€ de
sign
of t
he c
ampa
ign.
–1,
000,
000
x 1€
= 1
,000
,000
€.–
TOTA
L: 1
,020
,000
ۥ
BEN
EFIT
S–
1% o
frep
lies
from
1,00
0,00
0 ar
e 10
,000
repl
ies,
45€
eac
h on
e.–
TOTA
L: 4
50,0
00€
–M
ore
cost
sth
anbe
nefit
sC
ance
lled
cam
paig
n.
Hav
ew
edo
ne w
ell?
36
•“m
ail c
ampa
ign
desi
gn”:
–Le
tus
train
a so
ftcl
assi
fierw
ithth
esa
mpl
ean
dle
tus
draw
itsR
OC
cur
ve (w
itha
prev
ious
lyre
mov
ed p
artf
orte
stse
t).
App
licat
ions
Cos
t Mat
rixBu
ys
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,91
00,
10,
20,
30,
40,
50,
60,
70,
80,
91
-44€
1€SÍ
-45€
0€N
Osí
no
Sent
But t
he c
onfu
sion
mat
rix h
as
an im
poss
ible
cell:
“Not
sen
t an
d bo
ught
”
•Ad
ditio
nally
, the
clas
sifie
ris
notv
ery
good
(clo
sert
oth
edi
agon
al)
tokn
ow w
hich
cust
omer
sw
illbu
yan
dw
hich
cust
omer
sw
on’t.
•Ev
enth
ough
, we
can
do m
any
thin
gs...
37
App
licat
ions
•“m
ail c
ampa
ign
desi
gn”:
–W
eca
n us
e th
ecl
assi
fiert
ode
term
ine
who
tose
ndth
eof
fer.
Cos
tofC
ampa
ign
20,0
00 --
> 20
,000
100,
000
x 1
--> 1
00,0
00
Tota
l:12
0,00
0
Ben
ef. C
ampa
ign
3,00
0 x
45 --
> 13
5,00
0
Net
Ben
ef.:
1
5,00
0
Cos
tofC
ampa
ign
20,0
00 --
> 20
,000
200,
000
x 1
--> 1
00,0
00
Tota
l:22
0,00
0
Ben
ef. C
ampa
ign
5,00
0 x
45 --
> 22
5,00
0
Net
Ben
ef.:
5
,000
38
App
licat
ions
•“m
ail c
ampa
ign
desi
gn”:
–G
raph
show
ing
the
bene
fitfo
rthr
eedi
ffere
ntca
mpa
igns
...
39
Ext
ensi
onto
Mor
e th
anTw
oC
lass
es
•Th
ete
chni
ques
pres
ente
dhe
reha
vebe
enillu
stra
ted
for
two
clas
ses:
–Th
eev
alua
tion
ofm
ultic
lass
clas
sifie
rsba
sed
onco
sts
can
be
perfo
rmed
equa
lly.
•Ex
ampl
e:C
OST
actu
al
low
m
ediu
mhi
gh
low
0€
5€
2€
m
ediu
m20
0€
-200
0€
10€
pr
edic
ted
high
10
€ 1€
-1
5€
To
tal c
ost:
-297
87€
ERR
OR
actu
al
low
m
ediu
mhi
gh
low
20
0
13
med
ium
5 15
4
pr
edic
ted
high
4
7 60
40
Ext
ensi
onto
Mor
e th
anTw
oC
lass
es
•R
OC
ana
lysi
s, th
ough
, is
not e
asily
ext
ensi
ble:
–G
iven
ncl
asse
s, th
ere
is a
nx
(n−1
) dim
ensi
onal
spa
ce.
•C
alcu
latin
g th
e co
nvex
hul
l im
prac
tical
.•
The
“con
text
” now
is n
ot d
eter
min
ed b
y a
valu
e (s
lope
), bu
t nx
(n−1
) −1
val
ues.
–
Ther
e ha
ve b
een
appr
oxim
atio
ns (f
or th
ree
clas
ses,
Mos
sman
199
9)or
ef
forts
to ta
ckle
the
gene
ral p
robl
em (S
riniv
asan
1999
) (Fe
rriet
al.
2003
b).
∑∑
=<
=−
=c i
c
ij
jH
Tj
iAU
Cc
cAU
C1
,1)
,()1
(1
•Th
e AU
C m
easu
re h
as b
een
exte
nded
.–
All-p
air e
xten
sion
(Han
d &
Till
2001
).
–“o
ne v
s. re
st” e
xten
sion
(Faw
cett)
–O
ther
ext
ensi
ons
(Yan
et a
l. 20
03) (
AUC
*, Ti
ng 2
002)
.
41
Con
clus
ions
•R
OC
Ana
lysi
s:
–H
ighl
ight
s th
e fa
ct th
at th
e ev
alua
tion
of c
lass
ifier
s go
es m
uch
beyo
nd th
e es
timat
ion
of th
e er
ror r
ate.
–M
akes
it p
ossi
ble
to w
ork
with
cost
s an
d di
strib
utio
ns, o
r with
out
this
info
rmat
ion,
impr
ovin
g th
e ge
nera
tion,
sel
ectio
n an
d ap
plic
atio
n of
cla
ssifi
ers.
–P
rovi
des
a se
t of v
arie
d m
etric
s an
d te
chni
ques
to e
valu
ate
clas
sifie
rs d
epen
ding
on
the
task
: min
imis
eer
ror,
min
imis
eco
st,
impr
ove
a ra
nkin
g, e
tc.
•It
is a
hot
rese
arch
are
a of
wid
e ap
plic
abili
ty in
dat
a m
inin
g an
d m
achi
ne le
arni
ng.
42
Som
eR
efer
ence
s
•Br
adle
y, A
.P. (
1997
) “Th
e us
e of
the
area
und
er th
e R
OC
cur
ve in
the
eval
uatio
n of
mac
hine
lear
ning
al
gorit
hms”
Patte
rn R
ecog
nitio
n, 3
0(7)
, 114
5-11
59.
•Eg
an, J
.P. (
1975
). Si
gnal
Det
ectio
n Th
eory
and
RO
C An
alys
is. S
erie
s in
Cog
nitio
n an
d Pe
rcep
tion.
Ac
adem
ic P
ress
, New
Yor
k.•
Faw
cett,
T.(2
001)
. “U
sing
rule
set
s to
max
imiz
e R
OC
per
form
ance
”.In
Proc
eedi
ngs
of th
e IE
EE In
tern
atio
nal
Con
fere
nce
on D
ata
Min
ing
(ICD
M-2
001)
, pp.
131-
138.
•Fa
wce
tt, T
., &
Prov
ost,
F. (1
997)
. “Ad
aptiv
e fra
ud d
etec
tion”
. Dat
a M
inin
g an
d Kn
owle
dge
Disc
over
y,
1(3)
,291
-316
.•
Faw
cett,
T. (2
003)
. “R
OC
gra
phs:
Not
es a
nd p
ract
ical
con
side
ratio
ns fo
r dat
a m
inin
g re
sear
cher
s”Te
ch
repo
rt H
PL-2
003-
4. H
P La
bora
torie
s, P
aloA
lto, C
A, U
SA. A
vaila
ble:
ht
tp://
ww
w.p
url.o
rg/n
et/tf
awce
tt/pa
pers
/HPL
-200
3-4.
pdf.
•Fe
rri, C
., Fl
ach,
P.;
Her
nánd
ez-O
rallo
, J. (
2002
). “L
earn
ing
Dec
isio
n Tr
ees
usin
g th
e Ar
ea U
nder
the
RO
C
Cur
ve”,
inC
. Sam
mut
; A. H
offm
an (e
ds.)
“The
200
2 In
tern
atio
nal C
onfe
renc
e on
Mac
hine
Lea
rnin
g”(IC
ML2
002)
, IO
S Pr
ess,
Mor
gan
Kauf
man
n Pu
blis
hers
, pp.
139
-146
.•
Ferri
, C.;
Flac
h, P
.A.;
Her
nánd
ez-O
rallo
, J. (
2003
a) "I
mpr
ovin
g th
e AU
C o
f Pro
babi
listic
Est
imat
ion
Tree
s".
Euro
pean
Con
fere
nce
on M
achi
ne L
earn
ing,
EC
ML
2003
: 121
-132
•Fe
rri, C
.; H
erná
ndez
-Ora
llo, J
.; Sa
lido,
M.A
. (20
03b)
"Vol
ume
unde
r the
RO
C S
urfa
ce fo
r Mul
ti-cl
ass
Prob
lem
s". E
urop
ean
Con
fere
nce
on M
achi
ne L
earn
ing,
EC
ML
2003
: 108
-120
•Fl
ach,
P.;
Bloc
keel
, H.;
Ferri
, C.;
Her
nánd
ez-O
rallo
, J.;
Stru
yf, J
. (20
03) “
Dec
isio
n Su
ppor
t for
Dat
a M
inin
g:
Intro
duct
ion
to R
OC
ana
lysi
s an
d its
app
licat
ions
”in
Data
Min
ing
and
Decis
ion
Supp
ort:
Inte
grat
ion
and
Colla
bora
tion ,
Klu
wer
Aca
dem
ic P
ublis
hers
, Bos
ton,
200
3.•
Flac
h, P
.A. "
The
Geo
met
ryof
RO
C S
pace
: Und
erst
andi
ngM
achi
neLe
arni
ngM
etric
sth
roug
hR
OC
Is
omet
rics"
. (20
03) I
nter
natio
nal C
onfe
renc
e on
Mac
hine
Lea
rnin
g, IC
ML
2003
: 194
-201
•Fü
rnkr
anz,
J.;
Flac
h, P
.A.:
AnAn
alys
isof
Rul
eEv
alua
tion
Met
rics.
(200
3) In
tern
atio
nal C
onfe
renc
e on
M
achi
ne L
earn
ing,
ICM
L 20
03: 2
02-2
09•
Han
d, D
.J.,
& Ti
ll, R
.J. (
2001
). “A
Sim
ple
Gen
eral
isat
ion
of th
e Ar
ea U
nder
the
RO
C C
urve
for M
ultip
le C
lass
C
lass
ifica
tion
Prob
lem
s”, M
achi
ne L
earn
ing,
45,
pp.
171
-186
.•
Han
ley,
J.A
. , &
McN
eil,
B.J.
(198
2). “
The
mea
ning
and
use
of t
he a
rea
unde
r a re
ceiv
er o
pera
ting
char
acte
ristic
(RO
C) c
urve
”. Ra
diol
ogy,
143,
29-3
6.
43
Som
eR
efer
ence
s
•La
chic
he, N
. & F
lach
, P.A
. (20
03).
“Impr
ovin
g Ac
cura
cy a
nd C
ost o
f Tw
o-cl
ass
and
Mul
ti-cl
ass
Prob
abilis
tic
Cla
ssifi
ers
Usi
ng R
OC
Cur
ves”
. Int
erna
tiona
l Con
fere
nce
on M
achi
ne L
earn
ing,
ICM
L 20
03: 4
16-4
23•
Lane
, T. (
2000
). “E
xten
sion
s of
RO
C a
naly
sis
to m
ulti-
clas
s do
mai
ns”.
In D
iette
rich,
T.,
Mar
gine
antu
, D.,
Prov
ost,
F., &
Tur
ney,
P. (
Eds.
), IC
ML-
2000
Wor
ksho
p on
Cos
t-Sen
sitiv
e Le
arni
ng•
Ling
, C.X
.; Ya
n, R
.J. (
2003
) “D
ecis
ion
Tree
with
Bet
ter R
anki
ng”T
he 2
003
Inte
rnat
iona
l Con
fere
nce
on
Mac
hine
Lea
rnin
g (IC
ML2
003)
, IO
S Pr
ess,
Mor
gan
Kauf
man
n Pu
blis
hers
, to
appe
ar.
•M
ann,
H. B
. & W
hitn
ey, D
. R. (
1947
). "O
n a
test
whe
ther
one
of t
wo
rand
om v
aria
bles
is s
toch
astic
ally
larg
er
than
the
othe
r". A
nn. M
ath.
Sta
tist.,
18,
pp.
50-
60.
•M
ossm
an,D
.(199
9). “
Thre
e-w
ay R
OC
s”. M
edica
l Dec
ision
Mak
ing,
19,7
8-89
.•
Prov
ost,
F., &
Dom
ingo
s, P
. (20
03).
“Tre
e In
duct
ion
for P
roba
bilit
y-ba
sed
Ran
king
”, M
achi
ne L
earn
ing
52:3
(in
pre
ss),
2003
.•
Srin
ivas
an, A
. (19
99) “
Not
e on
the
Loca
tion
of O
ptim
al C
lass
ifier
s in
N-d
imen
sion
al R
OC
Spa
ce”T
echn
ical
R
epor
t PR
G-T
R-2
-99,
Oxf
ord
Uni
vers
ity C
ompu
ting
Labo
rato
ry, O
xfor
d.•
Swet
s, J
.A. (
1988
). “M
easu
ring
the
accu
racy
of d
iagn
ostic
sys
tem
s”. S
cienc
e, 2
40,1
285-
1293
.•
Swet
s, J
.A.,
Daw
es, R
.M.,
& M
onah
an, J
. (20
00).
“Bet
ter d
ecis
ions
thro
ugh
scie
nce”
. Sci
entif
ic A
mer
ican
, 28
3, 8
2-87
. http
://w
ww
.psy
chol
ogic
alsc
ienc
e.or
g/pd
f/psp
i/sci
am.p
df.
•Ti
ng, K
aiM
ing
(200
2). “
Issu
es in
Cla
ssifi
er E
valu
atio
n us
ing
Opt
imal
Cos
t Cur
ves”
The
Pro
ceed
ings
of t
he
Inte
rnat
iona
l Con
fere
nce
on M
achi
ne L
earn
ing"
Inte
rnat
iona
l Con
fere
nce
on M
achi
ne L
earn
ing,
ICM
L 20
02,
pp. 6
42-6
49.
•Tu
rney
, P. (
2000
) “Ty
pes
of C
osti
n In
duct
ive
Con
cept
Lea
rnin
g”Pr
ocee
ding
s W
orks
hop
on C
ost-S
ensi
tive
Lear
ning
at t
he S
even
teen
th In
tern
atio
nal C
onfe
renc
e on
Mac
hine
Lea
rnin
g (W
CSL
at I
CM
L-20
00),
15-2
1.•
Wei
ss, G
. and
Pro
vost
, F. “
The
Effe
ct o
f Cla
ss D
istri
butio
n on
Cla
ssifi
er L
earn
ing:
An
Empi
rical
Stu
dy”
Tech
nica
l Rep
ort M
L-TR
-44,
Dep
artm
ent o
f Com
pute
r Sci
ence
, Rut
gers
Uni
vers
ity, 2
001.
•
Wilc
oxon
, F. (
1945
). "In
divi
dual
com
paris
ons
by ra
nkin
g m
etho
ds".
Biom
etric
s, 1
, pp.
80-
83.
•Ya
n, L
., D
odie
r, R
., M
ozer
, M. C
., &
Wol
niew
icz,
R. (
2003
). "O
ptim
izin
g cl
assi
fier p
erfo
rman
ce v
ia th
e W
ilcox
on-M
ann-
Whi
tney
sta
tistic
. In
The
Pro
ceed
ings
of t
he In
tern
atio
nal C
onfe
renc
e on
Mac
hine
Lea
rnin
g"
Inte
rnat
iona
l Con
fere
nce
on M
achi
ne L
earn
ing,
ICM
L (p
p. 8
48-8
55).
http
://w
ww
.cs.
colo
rado
.edu
/~m
ozer
/pap
ers/
•Zw
eig,
M.H
.; C
ampb
ell,
G. (
1993
) “R
ecei
ver-o
pera
ting
char
acte
ristic
(RO
C) p
lots
: a fu
ndam
enta
l eva
luat
ion
tool
in c
linic
alm
edic
ine”
, Clin
. Che
m, 1
993;
39:
561
-77.
44
Som
eW
ebsi
tes
toK
now
Mor
e
•To
mFa
wce
tt’s
page
on
RO
C A
naly
sis:
http
://w
ww
.hpl
.hp.
com
/per
sona
l/Tom
_Faw
cett/
RO
CC
H/
•R
OC
Ana
lysi
sSo
ftwar
eht
tp://
epiw
eb.m
asse
y.ac
.nz/
RO
C_a
naly
sis_
softw
are.
htm
http
://cs
.bris
.ac.
uk/~
farra
nd/ro
con/
•R
OC
Ana
lysi
s Ex
tens
ive
Bibl
iogr
aphy
:ht
tp://
splw
eb.b
wh.
harv
ard.
edu:
8000
/pag
es/p
pl/z
ou/ro
c.ht
ml
•1s
t Wor
ksho
p on
“RO
C A
naly
sis
in A
I”, V
alen
cia,
22/
23 A
ugus
t 20
04 (w
ithin
EC
AI’2
004)
http
://w
ww
.dsi
c.up
v.es
/~fli
p/R
OC
AI20
04/