![Page 1: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/1.jpg)
Point Distribution Models
Jan Kybic
winter semester 2007
![Page 2: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/2.jpg)
Point distribution models
(Cootes et al., 1992)
I Shape description techniques
I A family of shapes = mean + eigenvectors (eigenshapes)
I Shapes described by points
![Page 3: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/3.jpg)
Point distribution model procedure
Input:
I M training samples
I N points each
xi = (x i1, y
i1, x
i2, y
i2, . . . , x
iN , y i
N)T
Procedure:
I Rigidly align all shapes
I Calculate the mean and the covariance matrix
I PCA (eigen analysis) — find principal modes
![Page 4: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/4.jpg)
Rigid alignment
before alignment after alignment
![Page 5: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/5.jpg)
Aligning two shapes
x(1) = (x(1)1 , y
(1)1 , x
(1)2 , y
(1)2 , . . . , x
(1)N , y
(1)N )T
x(2) = (x(2)1 , y
(2)1 , x
(2)2 , y
(2)2 , . . . , x
(2)N , y
(2)N )T
Find a transformation (rotation, translation, scaling) of x(2)
T (x(2)) = s R
[x
(2)i
y(2)i
]+
[txty
]=
[x
(2)i s cos θ − y
(2)i s sin θ
x(2)i s sin θ + y
(2)i s cos θ
]+
[txty
]such that a sum of squared distances is minimized
E =M∑i=1
wi
∥∥∥∥∥ s
[cos θ − sin θsin θ cos θ
][x
(2)i
y(2)i
]+
[txty
]−
[x
(1)i
y(1)i
] ∥∥∥∥∥2
![Page 6: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/6.jpg)
Aligning two shapes
E =M∑i=1
wi
∥∥∥∥∥ s
[cos θ − sin θsin θ cos θ
][x
(2)i
y(2)i
]+
[txty
]−
[x
(1)i
y(1)i
] ∥∥∥∥∥2
Minimize E (θ, s, tx , ty ) as minθ mins,tx ,ty Eθ(s, tx , ty )
I Inner minimization wrt s, tx , ty
∂E
∂tx= 0 ,
∂E
∂ty= 0 ,
∂E
∂s= 0 ,
I Outer minimization wrt θOne dimensional functional minimization, e.g. Brent’s routineor golden section search.
![Page 7: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/7.jpg)
Aligning two shapes
I Inner minimization wrt s, tx , ty
∂E
∂tx= 0 ,
∂E
∂ty= 0 ,
∂E
∂s= 0 ,
leads to linear equations:
s∑M
i=1 wi q( yi ,−xi , θ)− N tx = −∑M
i=1 wi x′i
s∑M
i=1 wi q(−xi ,−yi , θ)− N ty = −∑M
i=1 wi y′i
s∑M
i=1 w2i
(q2(yi ,−xi , θ) + q2(xi , yi , θ)
)− tx
∑Mi=1 wi q(yi ,−xi , θ)
− ty∑M
i=1 wi q(−xi ,−yi , θ)
= −M∑i=1
wi x′i q(yi ,−xi , θ) +
M∑i=1
wi y′i q(xi ,−yi , θ)
where q(a, b, θ) = a sin θ + b cos θ.
I Outer minimization wrt θOne dimensional functional minimization, e.g. Brent’s routineor golden section search.
![Page 8: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/8.jpg)
Aligning two shapes
I Inner minimization wrt s, tx , tyI Outer minimization wrt θ
One dimensional functional minimization, e.g. Brent’s routineor golden section search.
![Page 9: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/9.jpg)
Aligning all training shapes
Before alignment After alignment
![Page 10: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/10.jpg)
Aligning all training shapes
I Align each xi with x1, for i = 2, 3, . . . ,M, obtaining{x1, x2, x3, . . . , xM}.
I Calculate the mean x =[x1, y1, x2, y2, . . . , xN , yN
]of the
aligned shapes {x1, x2, x3, . . . , xM}.
xj =1
M
M∑i=1
x ij and yj =
1
M
M∑i=1
y ij .
I Align the mean shape x with x1. (Necessary for convergence.)
I Align x2, x3, . . . , xM to the adjusted mean.
I Repeat until convergence.
![Page 11: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/11.jpg)
Aligning all training shapes
I Align each xi with x1, for i = 2, 3, . . . ,M, obtaining{x1, x2, x3, . . . , xM}.
I Calculate the mean x =[x1, y1, x2, y2, . . . , xN , yN
]of the
aligned shapes {x1, x2, x3, . . . , xM}.I Align the mean shape x with x1. (Necessary for convergence.)
I Align x2, x3, . . . , xM to the adjusted mean.
I Repeat until convergence.
![Page 12: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/12.jpg)
Aligning all training shapes
I Align each xi with x1, for i = 2, 3, . . . ,M, obtaining{x1, x2, x3, . . . , xM}.
I Calculate the mean x =[x1, y1, x2, y2, . . . , xN , yN
]of the
aligned shapes {x1, x2, x3, . . . , xM}.I Align the mean shape x with x1. (Necessary for convergence.)
I Align x2, x3, . . . , xM to the adjusted mean.
I Repeat until convergence.
![Page 13: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/13.jpg)
Aligning all training shapes
I Align each xi with x1, for i = 2, 3, . . . ,M, obtaining{x1, x2, x3, . . . , xM}.
I Calculate the mean x =[x1, y1, x2, y2, . . . , xN , yN
]of the
aligned shapes {x1, x2, x3, . . . , xM}.I Align the mean shape x with x1. (Necessary for convergence.)
I Align x2, x3, . . . , xM to the adjusted mean.
I Repeat until convergence.
We have obtained M (mutually aligned) boundaries x1, x2, . . . , xM
and the mean x.
![Page 14: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/14.jpg)
Deriving the model
We have M boundaries x1, x2, . . . , xM and the mean x.
I Variation from the mean for each training shape
δxi = xi − x .
I Covariance matrix S (2N × 2N)
S =1
M
M∑i=1
δxi (δxi )T
I Principal component analysis
![Page 15: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/15.jpg)
Deriving the model
We have M boundaries x1, x2, . . . , xM and the mean x.
I Variation from the mean for each training shape
δxi = xi − x .
I Covariance matrix S (2N × 2N)
S =1
M
M∑i=1
δxi (δxi )T
I Principal component analysis
![Page 16: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/16.jpg)
Principal component analysis
I Eigen decomposition
Spi = λipi
P =[p1p2p3 . . .p2N
]We know eigenvalues λi are real because S is symmetric,positive definite. Eigenvectors (principal components) pi areorthogonal, so P is a basis and any vector x can berepresented as
x = x + Pb
I Order eigenvectors pi and eigenvalues λi such thatλ1 ≥ λ2 ≥ λ3 ≥ . . . λ2N . Most changes are then described bythe first few eigenvectors.
I Consider only K largest eigenvalues.
Approximation x ≈ x + PK bK
with PK =[p1p2p3 . . .pK
]bt =
[b1, b2, . . . , bK
]T
Choose the smallest K , such that∑K
i=1 λi ≥ α∑N
i=1 λi .
![Page 17: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/17.jpg)
Principal component analysis
I Eigen decomposition
Spi = λipi
P =[p1p2p3 . . .p2N
]We know eigenvalues λi are real because S is symmetric,positive definite. Eigenvectors (principal components) pi areorthogonal, so P is a basis and any vector x can berepresented as
x = x + Pb
I Order eigenvectors pi and eigenvalues λi such thatλ1 ≥ λ2 ≥ λ3 ≥ . . . λ2N . Most changes are then described bythe first few eigenvectors.
I Consider only K largest eigenvalues.
Approximation x ≈ x + PK bK
with PK =[p1p2p3 . . .pK
]bt =
[b1, b2, . . . , bK
]T
Choose the smallest K , such that∑K
i=1 λi ≥ α∑N
i=1 λi .
![Page 18: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/18.jpg)
Principal component analysis
I Eigen decomposition
Spi = λipi
P =[p1p2p3 . . .p2N
]x = x + Pb
I Order eigenvectors pi and eigenvalues λi such thatλ1 ≥ λ2 ≥ λ3 ≥ . . . λ2N . Most changes are then described bythe first few eigenvectors.
I Consider only K largest eigenvalues.
Approximation x ≈ x + PK bK
with PK =[p1p2p3 . . .pK
]bt =
[b1, b2, . . . , bK
]T
Choose the smallest K , such that∑K
i=1 λi ≥ α∑N
i=1 λi .
![Page 19: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/19.jpg)
Point distribution model
I Input: M non-aligned boundaries x1, x2, . . . , xM .
I Output: mean x and reduced eigvector matrix PK
I New shape generation:
x = x + PK bK
For “well-behaved’ shapes
−3√
λi ≤ bi ≤ 3√
λi
![Page 20: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/20.jpg)
Point distribution model
I Input: M non-aligned boundaries x1, x2, . . . , xM .
I Output: mean x and reduced eigvector matrix PK
I New shape generation:
x = x + PK bK
For “well-behaved’ shapes
−3√
λi ≤ bi ≤ 3√
λi
![Page 21: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/21.jpg)
PDM example
Before alignment After alignment, mean shape
![Page 22: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/22.jpg)
PDM example
First mode Second mode
The mean shape is in red, the shape corresponding to −3√
λ inblue and the shape corresponding to +3
√λ in green.
![Page 23: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/23.jpg)
Active shape models
PDM Image to fit
Fit a learned point distribution model (PDM) to a given image.
![Page 24: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/24.jpg)
Pose and shape parameters
I Point distribution model (PDM) consists ofI mean pI eigenvectors P
I Fitted model given by:I pose parameters: θ, s, tx , tyI shape parameters: b
p = Pb + p
p =[p1 p2 . . . pN
]p =
[x1 y1 x2 y2 . . . xN yN
][xi
yi
]= s
[cos θ − sin θsin θ − cos θ
] [xi
yi
]+
[txty
]p =
[x1 y1 x2 y2 . . . xN yN
]
![Page 25: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/25.jpg)
Pose and shape parameters
I Point distribution model (PDM) consists ofI mean pI eigenvectors P
I Fitted model given by:I pose parameters: θ, s, tx , tyI shape parameters: b
p = Pb + p
p =[p1 p2 . . . pN
]p =
[x1 y1 x2 y2 . . . xN yN
][xi
yi
]= s
[cos θ − sin θsin θ − cos θ
] [xi
yi
]+
[txty
]p =
[x1 y1 x2 y2 . . . xN yN
]
![Page 26: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/26.jpg)
Fitting
p = Ts,θ,tx ,ty (p) = s Qθp + rtx ,ty where p = Pb + p
I Calculate an edge map of the image
I For each landmark pi we find a line normal to the shape contour.
I New position p′i is the maximum of the edge map on the line. (Ifmaximum too weak, no change.)
I Adjust pose parameters θ, s, tx , ty by the alignment algorithm.
I Adjust shape parameters b as follows:
p′i = T−1(p′i )
b′ = P−1(p′i − p) = PT (p′i − p)
I Repeat until convergence
![Page 27: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/27.jpg)
Fitting
I Calculate an edge map of the image
I For each landmark pi we find a line normal to the shape contour.
I New position p′i is the maximum of the edge map on the line. (Ifmaximum too weak, no change.)
50 100 150 200 250 300 350
50
100
150
200
250
300
350
I Adjust pose parameters θ, s, tx , ty by the alignment algorithm.
I Adjust shape parameters b as follows:
p′i = T−1(p′i )
b′ = P−1(p′i − p) = PT (p′i − p)
I Repeat until convergence
![Page 28: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/28.jpg)
Fitting
p = Ts,θ,tx ,ty (p) = s Qθp + rtx ,ty where p = Pb + p
I Calculate an edge map of the image
I For each landmark pi we find a line normal to the shape contour.
I New position p′i is the maximum of the edge map on the line. (Ifmaximum too weak, no change.)
I Adjust pose parameters θ, s, tx , ty by the alignment algorithm.
I Adjust shape parameters b as follows:
p′i = T−1(p′i )
b′ = P−1(p′i − p) = PT (p′i − p)
I Repeat until convergence
![Page 29: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/29.jpg)
Fitting
p = Ts,θ,tx ,ty (p) = s Qθp + rtx ,ty where p = Pb + p
I Calculate an edge map of the image
I For each landmark pi we find a line normal to the shape contour.
I New position p′i is the maximum of the edge map on the line. (Ifmaximum too weak, no change.)
I Adjust pose parameters θ, s, tx , ty by the alignment algorithm.
I Adjust shape parameters b as follows:
p′i = T−1(p′i )
b′ = P−1(p′i − p) = PT (p′i − p)
I Repeat until convergence
![Page 30: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/30.jpg)
Fitting
p = Ts,θ,tx ,ty (p) = s Qθp + rtx ,ty where p = Pb + p
I Calculate an edge map of the image
I For each landmark pi we find a line normal to the shape contour.
I New position p′i is the maximum of the edge map on the line. (Ifmaximum too weak, no change.)
I Adjust pose parameters θ, s, tx , ty by the alignment algorithm.
I Adjust shape parameters b as follows:
p′i = T−1(p′i )
b′ = P−1(p′i − p) = PT (p′i − p)
I Repeat until convergence
![Page 31: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/31.jpg)
Example
50 100 150 200 250 300 350
50
100
150
200
250
300
350
Hand image Edge map + initial shape.
The image is smoothed and a gradient magnitude image calculated ineach color channel. The edge map is a maximum over the three colorchannels, thresholded to obtain a clean background.
![Page 32: Point Distribution Models - CMPcmp.felk.cvut.cz/.../slidy/pointdistributionmodels.pdfPoint distribution models (Cootes et al., 1992) I Shape description techniques I A family of shapes](https://reader033.vdocuments.us/reader033/viewer/2022060317/5f0c61e87e708231d435201d/html5/thumbnails/32.jpg)
Example
50 100 150 200 250 300 350
50
100
150
200
250
300
350
50 100 150 200 250 300 350
50
100
150
200
250
300
350
First iteration Final postition