Point Distribution Models
Jan Kybic
winter semester 2007
Point distribution models
(Cootes et al., 1992)
I Shape description techniques
I A family of shapes = mean + eigenvectors (eigenshapes)
I Shapes described by points
Point distribution model procedure
Input:
I M training samples
I N points each
xi = (x i1, y
i1, x
i2, y
i2, . . . , x
iN , y i
N)T
Procedure:
I Rigidly align all shapes
I Calculate the mean and the covariance matrix
I PCA (eigen analysis) — find principal modes
Rigid alignment
before alignment after alignment
Aligning two shapes
x(1) = (x(1)1 , y
(1)1 , x
(1)2 , y
(1)2 , . . . , x
(1)N , y
(1)N )T
x(2) = (x(2)1 , y
(2)1 , x
(2)2 , y
(2)2 , . . . , x
(2)N , y
(2)N )T
Find a transformation (rotation, translation, scaling) of x(2)
T (x(2)) = s R
[x
(2)i
y(2)i
]+
[txty
]=
[x
(2)i s cos θ − y
(2)i s sin θ
x(2)i s sin θ + y
(2)i s cos θ
]+
[txty
]such that a sum of squared distances is minimized
E =M∑i=1
wi
∥∥∥∥∥ s
[cos θ − sin θsin θ cos θ
][x
(2)i
y(2)i
]+
[txty
]−
[x
(1)i
y(1)i
] ∥∥∥∥∥2
Aligning two shapes
E =M∑i=1
wi
∥∥∥∥∥ s
[cos θ − sin θsin θ cos θ
][x
(2)i
y(2)i
]+
[txty
]−
[x
(1)i
y(1)i
] ∥∥∥∥∥2
Minimize E (θ, s, tx , ty ) as minθ mins,tx ,ty Eθ(s, tx , ty )
I Inner minimization wrt s, tx , ty
∂E
∂tx= 0 ,
∂E
∂ty= 0 ,
∂E
∂s= 0 ,
I Outer minimization wrt θOne dimensional functional minimization, e.g. Brent’s routineor golden section search.
Aligning two shapes
I Inner minimization wrt s, tx , ty
∂E
∂tx= 0 ,
∂E
∂ty= 0 ,
∂E
∂s= 0 ,
leads to linear equations:
s∑M
i=1 wi q( yi ,−xi , θ)− N tx = −∑M
i=1 wi x′i
s∑M
i=1 wi q(−xi ,−yi , θ)− N ty = −∑M
i=1 wi y′i
s∑M
i=1 w2i
(q2(yi ,−xi , θ) + q2(xi , yi , θ)
)− tx
∑Mi=1 wi q(yi ,−xi , θ)
− ty∑M
i=1 wi q(−xi ,−yi , θ)
= −M∑i=1
wi x′i q(yi ,−xi , θ) +
M∑i=1
wi y′i q(xi ,−yi , θ)
where q(a, b, θ) = a sin θ + b cos θ.
I Outer minimization wrt θOne dimensional functional minimization, e.g. Brent’s routineor golden section search.
Aligning two shapes
I Inner minimization wrt s, tx , tyI Outer minimization wrt θ
One dimensional functional minimization, e.g. Brent’s routineor golden section search.
Aligning all training shapes
Before alignment After alignment
Aligning all training shapes
I Align each xi with x1, for i = 2, 3, . . . ,M, obtaining{x1, x2, x3, . . . , xM}.
I Calculate the mean x =[x1, y1, x2, y2, . . . , xN , yN
]of the
aligned shapes {x1, x2, x3, . . . , xM}.
xj =1
M
M∑i=1
x ij and yj =
1
M
M∑i=1
y ij .
I Align the mean shape x with x1. (Necessary for convergence.)
I Align x2, x3, . . . , xM to the adjusted mean.
I Repeat until convergence.
Aligning all training shapes
I Align each xi with x1, for i = 2, 3, . . . ,M, obtaining{x1, x2, x3, . . . , xM}.
I Calculate the mean x =[x1, y1, x2, y2, . . . , xN , yN
]of the
aligned shapes {x1, x2, x3, . . . , xM}.I Align the mean shape x with x1. (Necessary for convergence.)
I Align x2, x3, . . . , xM to the adjusted mean.
I Repeat until convergence.
Aligning all training shapes
I Align each xi with x1, for i = 2, 3, . . . ,M, obtaining{x1, x2, x3, . . . , xM}.
I Calculate the mean x =[x1, y1, x2, y2, . . . , xN , yN
]of the
aligned shapes {x1, x2, x3, . . . , xM}.I Align the mean shape x with x1. (Necessary for convergence.)
I Align x2, x3, . . . , xM to the adjusted mean.
I Repeat until convergence.
Aligning all training shapes
I Align each xi with x1, for i = 2, 3, . . . ,M, obtaining{x1, x2, x3, . . . , xM}.
I Calculate the mean x =[x1, y1, x2, y2, . . . , xN , yN
]of the
aligned shapes {x1, x2, x3, . . . , xM}.I Align the mean shape x with x1. (Necessary for convergence.)
I Align x2, x3, . . . , xM to the adjusted mean.
I Repeat until convergence.
We have obtained M (mutually aligned) boundaries x1, x2, . . . , xM
and the mean x.
Deriving the model
We have M boundaries x1, x2, . . . , xM and the mean x.
I Variation from the mean for each training shape
δxi = xi − x .
I Covariance matrix S (2N × 2N)
S =1
M
M∑i=1
δxi (δxi )T
I Principal component analysis
Deriving the model
We have M boundaries x1, x2, . . . , xM and the mean x.
I Variation from the mean for each training shape
δxi = xi − x .
I Covariance matrix S (2N × 2N)
S =1
M
M∑i=1
δxi (δxi )T
I Principal component analysis
Principal component analysis
I Eigen decomposition
Spi = λipi
P =[p1p2p3 . . .p2N
]We know eigenvalues λi are real because S is symmetric,positive definite. Eigenvectors (principal components) pi areorthogonal, so P is a basis and any vector x can berepresented as
x = x + Pb
I Order eigenvectors pi and eigenvalues λi such thatλ1 ≥ λ2 ≥ λ3 ≥ . . . λ2N . Most changes are then described bythe first few eigenvectors.
I Consider only K largest eigenvalues.
Approximation x ≈ x + PK bK
with PK =[p1p2p3 . . .pK
]bt =
[b1, b2, . . . , bK
]T
Choose the smallest K , such that∑K
i=1 λi ≥ α∑N
i=1 λi .
Principal component analysis
I Eigen decomposition
Spi = λipi
P =[p1p2p3 . . .p2N
]We know eigenvalues λi are real because S is symmetric,positive definite. Eigenvectors (principal components) pi areorthogonal, so P is a basis and any vector x can berepresented as
x = x + Pb
I Order eigenvectors pi and eigenvalues λi such thatλ1 ≥ λ2 ≥ λ3 ≥ . . . λ2N . Most changes are then described bythe first few eigenvectors.
I Consider only K largest eigenvalues.
Approximation x ≈ x + PK bK
with PK =[p1p2p3 . . .pK
]bt =
[b1, b2, . . . , bK
]T
Choose the smallest K , such that∑K
i=1 λi ≥ α∑N
i=1 λi .
Principal component analysis
I Eigen decomposition
Spi = λipi
P =[p1p2p3 . . .p2N
]x = x + Pb
I Order eigenvectors pi and eigenvalues λi such thatλ1 ≥ λ2 ≥ λ3 ≥ . . . λ2N . Most changes are then described bythe first few eigenvectors.
I Consider only K largest eigenvalues.
Approximation x ≈ x + PK bK
with PK =[p1p2p3 . . .pK
]bt =
[b1, b2, . . . , bK
]T
Choose the smallest K , such that∑K
i=1 λi ≥ α∑N
i=1 λi .
Point distribution model
I Input: M non-aligned boundaries x1, x2, . . . , xM .
I Output: mean x and reduced eigvector matrix PK
I New shape generation:
x = x + PK bK
For “well-behaved’ shapes
−3√
λi ≤ bi ≤ 3√
λi
Point distribution model
I Input: M non-aligned boundaries x1, x2, . . . , xM .
I Output: mean x and reduced eigvector matrix PK
I New shape generation:
x = x + PK bK
For “well-behaved’ shapes
−3√
λi ≤ bi ≤ 3√
λi
PDM example
Before alignment After alignment, mean shape
PDM example
First mode Second mode
The mean shape is in red, the shape corresponding to −3√
λ inblue and the shape corresponding to +3
√λ in green.
Active shape models
PDM Image to fit
Fit a learned point distribution model (PDM) to a given image.
Pose and shape parameters
I Point distribution model (PDM) consists ofI mean pI eigenvectors P
I Fitted model given by:I pose parameters: θ, s, tx , tyI shape parameters: b
p = Pb + p
p =[p1 p2 . . . pN
]p =
[x1 y1 x2 y2 . . . xN yN
][xi
yi
]= s
[cos θ − sin θsin θ − cos θ
] [xi
yi
]+
[txty
]p =
[x1 y1 x2 y2 . . . xN yN
]
Pose and shape parameters
I Point distribution model (PDM) consists ofI mean pI eigenvectors P
I Fitted model given by:I pose parameters: θ, s, tx , tyI shape parameters: b
p = Pb + p
p =[p1 p2 . . . pN
]p =
[x1 y1 x2 y2 . . . xN yN
][xi
yi
]= s
[cos θ − sin θsin θ − cos θ
] [xi
yi
]+
[txty
]p =
[x1 y1 x2 y2 . . . xN yN
]
Fitting
p = Ts,θ,tx ,ty (p) = s Qθp + rtx ,ty where p = Pb + p
I Calculate an edge map of the image
I For each landmark pi we find a line normal to the shape contour.
I New position p′i is the maximum of the edge map on the line. (Ifmaximum too weak, no change.)
I Adjust pose parameters θ, s, tx , ty by the alignment algorithm.
I Adjust shape parameters b as follows:
p′i = T−1(p′i )
b′ = P−1(p′i − p) = PT (p′i − p)
I Repeat until convergence
Fitting
I Calculate an edge map of the image
I For each landmark pi we find a line normal to the shape contour.
I New position p′i is the maximum of the edge map on the line. (Ifmaximum too weak, no change.)
50 100 150 200 250 300 350
50
100
150
200
250
300
350
I Adjust pose parameters θ, s, tx , ty by the alignment algorithm.
I Adjust shape parameters b as follows:
p′i = T−1(p′i )
b′ = P−1(p′i − p) = PT (p′i − p)
I Repeat until convergence
Fitting
p = Ts,θ,tx ,ty (p) = s Qθp + rtx ,ty where p = Pb + p
I Calculate an edge map of the image
I For each landmark pi we find a line normal to the shape contour.
I New position p′i is the maximum of the edge map on the line. (Ifmaximum too weak, no change.)
I Adjust pose parameters θ, s, tx , ty by the alignment algorithm.
I Adjust shape parameters b as follows:
p′i = T−1(p′i )
b′ = P−1(p′i − p) = PT (p′i − p)
I Repeat until convergence
Fitting
p = Ts,θ,tx ,ty (p) = s Qθp + rtx ,ty where p = Pb + p
I Calculate an edge map of the image
I For each landmark pi we find a line normal to the shape contour.
I New position p′i is the maximum of the edge map on the line. (Ifmaximum too weak, no change.)
I Adjust pose parameters θ, s, tx , ty by the alignment algorithm.
I Adjust shape parameters b as follows:
p′i = T−1(p′i )
b′ = P−1(p′i − p) = PT (p′i − p)
I Repeat until convergence
Fitting
p = Ts,θ,tx ,ty (p) = s Qθp + rtx ,ty where p = Pb + p
I Calculate an edge map of the image
I For each landmark pi we find a line normal to the shape contour.
I New position p′i is the maximum of the edge map on the line. (Ifmaximum too weak, no change.)
I Adjust pose parameters θ, s, tx , ty by the alignment algorithm.
I Adjust shape parameters b as follows:
p′i = T−1(p′i )
b′ = P−1(p′i − p) = PT (p′i − p)
I Repeat until convergence
Example
50 100 150 200 250 300 350
50
100
150
200
250
300
350
Hand image Edge map + initial shape.
The image is smoothed and a gradient magnitude image calculated ineach color channel. The edge map is a maximum over the three colorchannels, thresholded to obtain a clean background.
Example
50 100 150 200 250 300 350
50
100
150
200
250
300
350
50 100 150 200 250 300 350
50
100
150
200
250
300
350
First iteration Final postition