la go: a for statistical - université de...
TRANSCRIPT
'&
$%
LAGO:AComputationallyEfficientMethod
forStatistica
lDetection
Mu
Zhu
Univ
ersity
ofW
ate
rlo
o
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-1
-
'&
$%
Acknowledgment
•Co-authors:
–WanhuaSu;
–HughA.Chipman.
•Resea
rchsupport:
–NSERC;
–MIT
ACS;
–CFI;
–Acadia
Centre
forMathem
atica
lModellin
gandComputatio
n.
•Others:
Willia
mJ.Welch
,R.Way
neOldford,Jerry
F.Law
less,
Mary
Thompson,S.Young.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-2
-
'&
$%
Agenda
1.Thesta
tisticaldetectio
nproblem
.
2.Avera
geprecisio
n.
3.Drugdiscov
eryandhighthroughputscreen
ing.
4.LAGO.
5.Radialbasis
functio
n(R
BF)netw
orks.
6.Support
vecto
rmachines
(SVMs).
7.Resu
lts.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-3
-
'&
$%
TheDetectionProblemH
its: h(t)
Detected
t
Relevant:
π
Collection
100%
Fig
ure
1:Illu
stratio
nofatypica
ldetectio
nopera
tion.A
smallfra
ction
πoftheentire
collectio
nCis
ofinterest
(relevant).
Analgorith
mdetects
afra
ctiontfro
mC,outofwhich
h(t)
isreleva
nt.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-4
-
'&
$%
TheTypicalParadigm
New
Data
Results
Ranking
Training D
ata
Model Supervised L
earning
Fig
ure
2:Illu
stratio
nofthetypica
lmodellin
gandpred
ictionprocess.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-5
-
'&
$%
TheHitCurve
0.00.2
0.40.6
0.81.0
0.00 0.01 0.02 0.03 0.04 0.05
t
h(t)
hA (t)
hB (t)
hR (t)
hP (t)
Fig
ure
3:Illu
stratio
nofsomehitcu
rves.
Note
thathA(t)
andhB(t)
cross
each
other;
hP(t)
isanidealcu
rveproduced
byaperfect
algorith
m;hR(t)
corresp
ondsto
thecase
ofrandom
detectio
n.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-6
-
'&
$%
TheAveragePrecisio
n
Let
h(t)
bethehit
curve;
let
r(t)=
h(t)
πand
p(t)
=h(t)
t.
Then
,
Avera
gePrecisio
n=
∫
p(t)d
r(t).(1)
Inpractice,
h(t)
takes
values
only
atafinite
number
ofpointsti=i/n,
i=
1,2,...,n
.Hen
ce,theinteg
ral(1)is
replaced
with
afinite
sum
∫
p(t)d
r(t)=
n∑i=1
p(ti )∆
r(ti )
(2)
where
∆r(t
i )=r(t
i )−r(t
i−1 ).
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-7
-
'&
$%
ASimpleExample
Algorith
mA
Algorith
mB
Item(i)
Hit
p(ti )
∆r(t
i )Hit
p(ti )
∆r(t
i )
11
1/1
1/3
11/1
1/3
21
2/2
1/3
01/2
0
30
2/3
00
1/3
0
41
3/4
1/3
12/4
1/3
50
3/5
01
3/5
1/3
AP(A
)=
5∑i=1
p(ti )∆
r(ti )
=
(
11+
22+
34
)
×13≈
0.92.
AP(B
)=
5∑i=1
p(ti )∆
r(ti )
=
(
11+
24+
35
)
×13=
0.70.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-8
-
'&
$%
HighThroughputScreening(HTS)
HT
S
Com
pounds
chemistry
library
Y
p 1
X XC
omputational
Chem
ical
Fig
ure
4:Illu
stratio
nofthehighthroughputscreen
ingprocess.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-9
-
'&
$%
DrugDiscoveryData
Orig
inaldata
from
Natio
nalCancer
Institu
te(N
CI)
with
pred
ictors
calcu
lated
byGlaxoSmith
Klin
e,Inc.
1.n=
29,812chem
icalcompounds,
ofwhich
only
608are
activ
e
against
theHIV
viru
s.
2.d=
6chem
ometric
descrip
tors
ofthemolecu
larstru
cture,
know
nas
BCUT
numbers.
3.Usin
gstra
tified
samplin
g,randomly
split
ofthedata
toproduce
a
trainingset
andatest
set(ea
chwith
n=
14,906and304activ
e
compounds).
4.Tuningparameters
selectedusin
g5-fo
ldcro
ss-valid
atio
nonthe
trainingset,
andcompare
perfo
rmance
onthetest
set.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-10
-
'&
$%
RankingFunctions
1.Given
avecto
rofpred
ictors
x,theposterio
rprobability
g(x)≡P(y
=1|x)=
π1 p1 (x
)
π1 p1 (x
)+π0 p0 (x
)(3)
isarguably
agoodrankingfunctio
n,i.e.,
itemswith
ahigh
probability
ofbein
greleva
ntshould
beranked
first.
2.Asfarasrankingis
concern
ed,allmonotonic
transfo
rmatio
nsofg
are
clearly
equiva
lent,so
itsuffices
tofocu
sontheratio
functio
n
f(x)=
p1 (x
)
p0 (x
)(4)
since
thefunctio
ngis
oftheform
g(x)=
af(x)
af(x)+
1
forsomeconsta
ntanotdep
endingon
x,which
isamonotonic
transfo
rmatio
noff.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-11
-
'&
$%
TwoAssumptions
A1.Forallpractica
lpurposes
theden
sityfunctio
np1 (x
)canbeassu
med
tohavebounded
localsupport,
possib
lyover
anumber
of
disco
nnected
regions,Sγ⊂
Rd,γ=
1,2,...,Γ
,in
which
case
the
support
ofp1canbewritten
as
S=
Γ⋃
γ=1
Sγ⊂
Rd.
A2.Forevery
observa
tion
xi∈C1 ,
there
are
atlea
stacerta
innumber
of
observa
tions,
saym,fro
mC0in
itsim
med
iate
localneig
hborhood;
moreov
er,theden
sityfunctio
np0 (x
)in
thatneig
hborhoodcanbe
assu
med
toberela
tively
flatin
compariso
nwith
p1 (x
).
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-12
-
'&
$%
Inordertobuildapredictive
modelfor
statisticaldetection
problems,
itsufficesto
☞estim
atep
1 (x)alon
eand
☞adjustp
1 (x)locally
dependingon
p0 (x)nearb
y.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-13
-
'&
$%
Assumex∈
Risascalar.
⇓
Generalize
tox∈
Rdfor
d>1.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-14
-
'&
$%
Step1:Estim
atingp
1
1.Use
anadaptiv
ebandwidth
kern
elestim
ator:
p1 (x
)=
1n1
∑
yi=1
K(x;x
i ,ri ).
(5)
2.Foreach
xi∈C1 ,
choose
riadaptiv
elyto
betheavera
gedista
nce
betw
eenxiandits
K-nearest
neig
hbors
from
C0 ,
i.e.,
ri=
1K
∑
wj∈N(x
i,K) |x
i−wj |.
(6)
Thenotatio
nN(x
i ,K)is
used
torefer
totheset
thatcontainsthe
K-nearest
class-0
neig
hbors
ofxi .
Thenumber
Kis
atuning
parameter
tobeselected
empirica
lly,e.g
.,with
cross-va
lidatio
n.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-15
-
'&
$%
OriginalInspiration
Fig
ure
5:Theancien
tChinese
gameofGoisagamein
which
each
play
er
triesto
claim
asmanyterrito
riesaspossib
leontheboard.Im
agetaken
from
http
://go.arad.ro
/Intro
ducere.h
tml.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-16
-
'&
$%
Step2:LocalAdjustmentofp
1
1.View
thekern
elden
sityestim
ate
(5)asamixture
andadjust
each
mixture
component(cen
teredatxi )
acco
rdingly.
2.Estim
ate
p0locally
aroundevery
xi∈C1 ,
sayp0 (x
;xi ),
anddivideit
intoK(x;x
i ,ri ).
Assu
mptio
nA2im
plies
thatwecansim
ply
estimate
p0 (x
;xi )
locally
asaconsta
nt,
sayci .
Hen
ce,weobtain
f(x)=
1n1
∑
yi=1
K(x;x
i ,ri )
ci
(7)
asanestim
ate
oftherankingfunctio
nf(x).
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-17
-
'&
$%
AnIdealizedSituation
Instea
dofsay
ingp0 (x
;xi )≈ci ,weshallexplicitly
assu
methat,
forevery
xi∈C1 ,
there
exist
i.i.d.observa
tionsw1 ,w
2 ,...,wm
from
C0thatcanbe
taken
tobeunifo
rmly
distrib
uted
ontheinterva
l[x
i−
1/2ci ,x
i+
1/2ci ].
Theorem
1Let
x0be
afixed
observa
tionfro
mcla
ss1.Suppose
w1 ,w
2 ,...,wmare
i.i.d.observa
tionsfro
mcla
ss0thatare
uniform
ly
distribu
tedaroundx0 ,sayontheinterva
l[x0−
1/2c0 ,x
0+
1/2c0 ].
Ifr0
istheavera
gedista
nce
between
x0andits
Knearest
neigh
bors
from
class
0(K
<m),then
wehave
E(r0 )
=K
+1
4(m
+1)c0.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-18
-
'&
$%
ImplicationsoftheTheorem
•Canassu
methere
are
atlea
stm
observa
tionsfro
mC0distrib
uted
approxim
ately
unifo
rmly
aroundevery
xi∈C1 .
•ForK
<m,canconclu
deriisapproxim
ately
proportio
nalto
1/ci .
•Since
riis
alrea
dycomputed
,there
isnoneed
toestim
ate
ci ;we
simply
use
f(x)=
1n1
∑
yi=1
riK(x;x
i ,ri ).
(8)
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-19
-
'&
$%
AShortSummary
1.Estim
atio
nofp1 :
p1 (x
)=
1n1
∑
yi=1
K(x;x
i ,ri ).
2.Adjustm
entofp1acco
rdingto
p0nearby:
f(x)=
1n1
∑
yi=1
K(x;x
i ,ri )
ci
=⇒
f(x)=
1n1
∑
yi=1
riK(x;x
i ,ri ).
⇓
LAGO
=“Locally
Adjusted
GO-kern
elden
sityestim
ator.”
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-20
-
'&
$%
Extensionto
Rd
1.Forevery
trainingobserva
tionin
class
1,xi∈C1 ,
compute
aspecifi
c
bandwidth
vecto
rri=
(ri1 ,r
i2 ,...,rid )
T,where
rij
istheavera
ge
dista
nce
betw
eenxiandits
K-nearest
class-0
neig
hbors
inthejth
dim
ensio
n.
2.Forevery
new
observa
tion
x=
(x1 ,x
2 ,...,xd )
Twhere
apred
ictionis
required
,sco
reandrank
xacco
rdingto:
f(x)=
1n1
∑
yi=1
{
d∏
j=1
rij K
(xj ;x
ij ,rij )
}
,(9)
which
uses
theNaiveBayes
prin
ciple
(Hastie
etal.2001,Sectio
n
6.6.3).
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-21
-
'&
$%
SomeKernelFunctions
Gau
ssianT
riang
ular
Un
iform
f(u)∝
exp
(
−u22
)
f(u)=
1−|u|
|u|≤
1
f(u)=
1
|u|≤
1
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-22
-
'&
$%
RadialBasisFunctionNetworks
Aradialbasis
functio
n(R
BF)netw
ork
hastheform
:
f(x)=
n∑i=1
βi K
(x;µ
i ,ri ),
(10)
where
K(x;µ,r)
isakern
elfunctio
ncen
teredatlocatio
nµ
with
radius
(orbandwidth)vecto
rr=
(r1 ,r
2 ,...,rd )
T.Clea
rly,in
order
toconstru
ct
anRBFnetw
ork
wemust
specify
thecen
tersµiandtheradii
rifor
i=
1,2,...,n
.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-23
-
'&
$%
GeneralParameterization
p1
p1
p0
p0 ’
f1
f1
f1 ’
f1 ’
β−effect: height
α−
effect: radius
Fig
ure
6:Illu
stratio
n.Left:
Den
sityfunctio
nsp0andp1 .
Right:
The
ratio
functio
nf(x).
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-24
-
'&
$%
Parameterizin
gtheα-andβ-Effects
•Takeakern
elfunctio
nbelo
ngingto
alocatio
n-sca
lefamily
:
1ri K
(
x−xi
ri
)
.
Canexplicitly
parameterize
theα-andβ-eff
ectsasfollow
s:
rβ′
i
1αri K
(
x−xi
αri
)
∝rβ′−1
iK
(
x−xi
αri
)
≡rβiK
(
x−xi
αri
)
.
•In
constru
ctingtheLAGO
model,
wehavein
effect
argued
that
β=
0(orβ′=
1).
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-25
-
'&
$%
TheLAGOModel
•Forrela
tively
largeK,canusually
obtain
amodel
with
very
simila
r
perfo
rmance
bysettin
gα>
1andusin
gamuch
smaller
K.
•Hen
cebykeep
ingα,canrestrict
ourselv
esto
amuch
narrow
errange
when
selectingK
bycro
ss-valid
atio
n.
•Thefinalform
oftheLAGO
model
is:
f(x)=
1n1
∑
yi=1
{
d∏
j=1
rij K
(xj ;x
ij ,αrij )
}
,(11)
with
twotuningparameters,
Kandα.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-26
-
'&
$%
SeparatingHyperplanes
•Given
xi∈
Rd,
ahyperp
lanein
Rdis
characterized
by
f(x)=βTx+β0=
0.
•Given
yi∈{−1,+
1}(tw
ocla
sses),ahyperp
laneis
asep
aratin
g
hyperp
laneifthere
exists
c>
0such
that
yi (β
Txi+β0 )≥c∀i.
•A
hyperp
lanecanberep
arameterized
bysca
ling,e.g
.,
βTx+β0=
0is
thesameas
s(βTx+β0 )
=0.
•A
separatin
ghyperp
lanesatisfy
ing
yi (β
Txi+β0 )≥
1∀i
(i.e.,sca
ledso
thatc=
1)is
sometim
escalled
acanonica
lsep
aratin
g
hyperp
lane(C
ristianiniandShaw
e-Tay
lor2000).
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-27
-
'&
$%
SeparatingHyperplanesandMargins
Margin (W
orse)
Margin (B
etter)
Fig
ure
7:Twosep
aratin
ghyperp
lanes,
onewith
alarger
margin
thanthe
other.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-28
-
'&
$%
TheSupportVectorMachine
•It
canbecalcu
lated
thatacanonica
lsep
aratin
ghyperp
lanehas
margin
equalto
1
‖β‖.
•Thesupport
vecto
rmachine(SVM)findsa“best”
(maxim
al
margin)canonica
lsep
aratin
ghyperp
laneto
separate
thetw
ocla
sses
(labelled
+1and−1)bysolving
min
12‖β‖2+γ
n∑i=1
ξi
s.t.ξi≥
0and
yi (β
Txi+β0 )≥
1−ξi∀i.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-29
-
'&
$%
ASVMforUnbalancedClasses
Let
w0andw1becla
ssweig
hts;
exten
dtheoptim
izatio
nproblem
tobe:
min
12‖β‖2+γ1
∑
yi=1
ξi+γ0
∑
yi=0
ξi
s.t.ξi≥
0and
yi (β
Txi+β0 )≥
1−ξi∀i,
where
γ0=γw0andγ1=γw1 .
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-30
-
'&
$%
SVM:Characterizin
gtheSolution
•Thesolutio
nforβ
ischaracterized
by
β=∑
i∈SV
αi yi x
i ,
where
αi≥
0(i
=1,2,...,n
)are
solutio
nsto
thedualoptim
izatio
n
problem
andSV,theset
of“support
vecto
rs”with
αi>
0strictly
positiv
e.
•This
meanstheresu
ltinghyperp
lanecanbewritten
as
f(x)=βTx+β0=∑
i∈SV
αi yi x
Tix+β0=
0.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-31
-
'&
$%
SVMsandRBFNetworks
•Canrep
lace
theinner
product
xTixwith
akern
elfunctio
nK(x;x
i )
toget
anonlin
eardecisio
nboundary:
f(x)=∑
i∈SV
αi yi K
(x;x
i )+β0=
0.
Theboundary
islin
earin
thespace
ofh(x)where
h(·)
issuch
that
K(u
;v)=〈h(u
),h(v
)〉is
theinner
product
inthespace
ofh(x).
•Hen
ceSVM
canbeview
edasanautomatic
way
ofconstru
ctingan
RBFnetw
ork
(Scholkopfet
al.1997).
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-32
-
'&
$%
PerformanceResults:
DrugDiscoveryData
Index of Split
12
34
0.18 0.20 0.22 0.24 0.26G
aussianT
riangularU
niformK
NN
SV
MA
SV
M
Averag
e Precisio
n
Fig
ure
8:Theavera
geprecisio
nofallalgorith
mseva
luated
onthetest
data.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-33
-
'&
$%
PerformanceResults:
ANOVASet-up
Let
µK,µ
S,µ
A,µ
U,µ
TandµG
betheavera
geresu
ltofK-N
N,SVM,
ASVM,andLAGO
usin
gtheunifo
rmkern
el,thetria
ngularkern
eland
theGaussia
nkern
el,resp
ectively.
Contra
stExpressio
nEstim
ate
Cntr1
µT−µG
0.0027
Cntr2
µG−µA
0.0339
Cntr3
µA−µS
0.0230
Cntr4
µS−
(µK+µU)/2
0.0157
Cntr5
µU−µK
0.0014
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-34
-
'&
$%
PerformanceResults:
ANOVASummary
Source
SS(×
10−4)
df
MS(×
10−4)
F0
P-V
alue
Meth
ods
233.504
546.701
64.307
<0.0001
Cntr1
0.140
10.140
0.193
0.6664
Cntr2
22.916
122.916
31.556
<0.0001
Cntr3
10.534
110.534
14.505
0.0017
Cntr4
6.531
16.531
8.994
0.0090
Cntr5
0.036
10.036
0.050
0.8258
Splits
18.877
36.292
8.664
0.0014
Erro
r10.893
15
0.726
Total
263.274
23
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-35
-
'&
$%
HitCurves:DrugDiscoveryData
0100
200300
400500
0 20 40 60 80 100 120
Total N
umber D
etected: n
Actual Hits: h(n)
Gaussian
Triangular
Uniform
KN
NS
VM
AS
VM
Sp
lit 1
0100
200300
400500
0 20 40 60 80 100 120
Total N
umber D
etected: n
Actual Hits: h(n)
Gaussian
Triangular
Uniform
KN
NS
VM
AS
VM
Sp
lit 2
0100
200300
400500
0 20 40 60 80 100 120
Total N
umber D
etected: n
Actual Hits: h(n)
Gaussian
Triangular
Uniform
KN
NS
VM
AS
VM
Sp
lit 3
0100
200300
400500
0 20 40 60 80 100 120
Total N
umber D
etected: n
Actual Hits: h(n)
Gaussian
Triangular
Uniform
KN
NS
VM
AS
VM
Sp
lit 4
Fig
ure
9:Only
theinitia
lpart
ofthecu
rves
(upto
n=
500)are
show
n.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-36
-
'&
$%
MainConclusions
☞
(Tria
ngle
LAGO∼
Gaussia
nLAGO)Â
ÂASVMÂ
SVMÂ
Â(U
nifo
rmLAGO∼
KNN).
☞
Computatio
nally,
ASVM
isextrem
elyexpen
sive.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-37
-
'&
$%
TheNumberofSVs
SVM
ASVM
C0
C1
C0
C1
Split
111531
294
5927
291
Split
211419
303
3472
284
Split
311556
293
11706
290
Split
41863
293
6755
281
TotalPossib
le14602
304
14602
304
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-38
-
'&
$%
Empirica
lEvidence:β=0
Co
nto
ur o
f AP
: Sp
lit 1
α
β
01
23
45
−1.0 −0.5 0.0 0.5 1.0
Co
nto
ur o
f AP
: Sp
lit 2
α
β
01
23
45
−1.0 −0.5 0.0 0.5 1.0
Co
nto
ur o
f AP
: Sp
lit 3
α
β
01
23
45
−1.0 −0.5 0.0 0.5 1.0
Co
nto
ur o
f AP
: Sp
lit 4
α
β
01
23
45
−1.0 −0.5 0.0 0.5 1.0
Fig
ure
10:Choosin
gαandβ(w
hile
fixingK
=5)usin
g5-fo
ldCV.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-39
-
'&
$%
Refe
rence
s
Cristia
nini,N.andShaw
e-Tay
lor,
J.(2000).AnIntrod
uctio
nto
Support
Vecto
rMachines
andOther
Kern
el-based
LearningMeth
ods.
Cambrid
geUniversity
Press.
Hastie,
T.J.,Tibshira
ni,R.J.,andFried
man,J.H.(2001).The
Elem
entsofStatistica
lLearning:
Data-M
ining,Inferen
ceand
Pred
iction.Sprin
ger-V
erlag.
Scholkopf,B.,Sung,K.K.,Burges,
C.J.C.,Giro
si,F.,Niyogi,P.,
Poggio,T.,andVapnik,V.(1997).
Comparin
gsupport
vecto
r
machines
with
gaussia
nkern
elsto
radialbasis
functio
ncla
ssifiers.
IEEETransactio
nsonSign
alProcessin
g,45(11),2758–2765.
Zhu,M.,Su,W
.,andChipman,H.A.(2005).
LAGO:A
computatio
nally
efficien
tapproach
forsta
tisticaldetectio
n.WorkingPaper
2005-01,
Dep
artm
entofStatistics
andActu
aria
lScien
ce,University
of
Waterlo
o.Toappearin
Tech
nometrics.
LA
GO
Copyrig
ht
c©2003–2005
by
Mu
Zhu
-40
-