Download - Image warping Li Zhang CS559
![Page 1: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/1.jpg)
Image warpingLi ZhangCS559
Slides stolen from Prof Yungyu Chuanghttp://www.csie.ntu.edu.tw/~cyy/courses/vfx/07spring/overview/
![Page 2: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/2.jpg)
What is an image
• We can think of an image as a function, f: R2R:– f(x, y) gives the intensity at position (x, y) – defined over a rectangle, with a finite range:
• f: [a,b]x[c,d] [0,1]
• A color image
( , )
( , ) ( , )
( , )
r x y
f x y g x y
b x y
x
y
f
![Page 3: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/3.jpg)
A digital image
• We usually operate on digital (discrete) images:– Sample the 2D space on a regular grid– Quantize each sample (round to nearest
integer)
• If our samples are D apart, we can write this as:
f[i ,j] = Quantize{ f(i D, j D) }• The image can now be represented as a
matrix of integer values
![Page 4: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/4.jpg)
Image warping
hf
g
Change pixels locations to create a new image:f(x) = g(h(x))
h([x,y])=[x,y/2]
![Page 5: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/5.jpg)
Parametric (global) warping
translation rotation aspect
affineperspective
cylindrical
Examples of parametric warps:
![Page 6: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/6.jpg)
Nonparametric (local) warping
Original Warped
![Page 7: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/7.jpg)
Parametric (global) warping
• Transformation T is a coordinate-changing machine: p’ = T(p)
• What does it mean that T is global?– can be described by just a few numbers
(parameters)– the parameters are the same for any point p
• Represent T as a matrix: p’ = M*p
T
p = (x,y) p’ = (x’,y’)
y
x
y
xM
'
'
![Page 8: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/8.jpg)
Scaling
• Scaling a coordinate means multiplying each of its components by a scalar
• Uniform scaling means this scalar is the same for all components:
2
f g
y
x
y
x
2
2
'
'
y
x
![Page 9: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/9.jpg)
• Non-uniform scaling: different scalars per component:
Scaling
x 2,y 0.5
y
x
y
x
5.0
2
'
'
'
'
y
xg
y
xf
![Page 10: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/10.jpg)
Scaling
• Scaling operation:
• Or, in matrix form:
byy
axx
'
'
y
x
b
a
y
x
0
0
'
'
scaling matrix S
What’s inverse of S?
![Page 11: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/11.jpg)
2-D Rotation
(x, y)
(x’, y’)
x’ = x cos() - y sin()y’ = x sin() + y cos()
![Page 12: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/12.jpg)
2-D Rotation
• This is easy to capture in matrix form:
• Even though sin() and cos() are nonlinear to ,– x’ is a linear combination of x and y– y’ is a linear combination of x and y
• What is the inverse transformation?– Rotation by –– For rotation matrices, det(R) = 1 so
y
x
y
x
cossin
sincos
'
'
TRR 1
R
![Page 13: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/13.jpg)
2x2 Matrices
• What types of transformations can be represented with a 2x2 matrix?
2D Identity?
yyxx
''
yx
yx
1001
''
2D Scale around (0,0)?
ysy
xsx
y
x
*'
*'
y
x
s
s
y
x
y
x
0
0
'
'
![Page 14: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/14.jpg)
2x2 Matrices
• What types of transformations can be represented with a 2x2 matrix?
2D Rotate around (0,0)?
yxy
yxx
*cos*sin'
*sin*cos'
y
x
y
x
cossin
sincos
'
'
2D Shear?
yxshy
yshxx
y
x
*'
*'
y
x
sh
sh
y
x
y
x
1
1
'
'
![Page 15: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/15.jpg)
2x2 Matrices
• What types of transformations can be represented with a 2x2 matrix?
2D Mirror about Y axis?
yyxx
''
yx
yx
1001
''
2D Mirror over (0,0)?
yyxx
''
yx
yx
1001
''
![Page 16: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/16.jpg)
All 2D Linear Transformations
• Linear transformations are combinations of …– Scale,– Rotation,– Shear, and– Mirror
• Properties of linear transformations:– Origin maps to origin– Lines map to lines– Parallel lines remain parallel– Ratios are preserved– Closed under composition
y
x
dc
ba
y
x
'
'
![Page 17: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/17.jpg)
2x2 Matrices
• What types of transformations can not be represented with a 2x2 matrix?
2D Translation?
y
x
tyy
txx
'
'
Only linear 2D transformations can be represented with a 2x2 matrix
NO!
![Page 18: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/18.jpg)
Translation
• Example of translation
11100
10
01
1
'
'
y
x
y
x
ty
tx
y
x
t
t
y
x
tx = 2ty = 1
Homogeneous Coordinates
![Page 19: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/19.jpg)
Affine Transformations
• Affine transformations are combinations of …– Linear transformations, and– Translations
• Properties of affine transformations:– Origin does not necessarily map to origin– Lines map to lines– Parallel lines remain parallel– Ratios are preserved– Closed under composition– Models change of basis
11001
'
'
y
x
fed
cba
y
x
![Page 20: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/20.jpg)
Projective Transformations
• Projective transformations …– Affine transformations, and– Projective warps
• Properties of projective transformations:– Origin does not necessarily map to origin– Lines map to lines– Parallel lines do not necessarily remain parallel– Ratios are not preserved– Closed under composition– Models change of basis
'/'
'/'
wy
wx
1'
'
'
y
x
ihg
fed
cba
w
y
x
Very Useful In Texture Mapping!
![Page 21: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/21.jpg)
Image warping
• Given a coordinate transform x’ = T(x) and a source image I(x), how do we compute a transformed image I’(x’) = I(T(x))?
I(x) I’(x’)
x x’
T(x)
![Page 22: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/22.jpg)
Forward warping
• Send each pixel I(x) to its corresponding location x’ = T(x) in I’(x’)
I(x) I’(x’)
x x’
T(x)
![Page 23: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/23.jpg)
Forward warping
fwarp(I, I’, T)
{
for (y=0; y<I.height; y++)
for (x=0; x<I.width; x++) {
(x’,y’)=T(x,y);
I’(x’,y’)=I(x,y);
}
} I I’
x
x’
T
![Page 24: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/24.jpg)
Forward warping
• Send each pixel I(x) to its corresponding location x’ = T(x) in I’(x’)
f(x) g(x’)x x’
h(x)
• What if pixel lands “between” two pixels?• Will be there holes?• Answer: add “contribution” to several pixels, n
ormalize later (splatting)
![Page 25: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/25.jpg)
Forward warping
fwarp(I, I’, T)
{
for (y=0; y<I.height; y++)
for (x=0; x<I.width; x++) {
(x’,y’)=T(x,y);
Splatting(I’,x’,y’,I(x,y),kernel);
}
} I I’
x
x’
T
![Page 26: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/26.jpg)
Inverse warping
• Get each pixel I’(x’) from its corresponding location x = T-1(x’) in I(x)
I(x) I’(x’)
x x’
T-1(x’)
![Page 27: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/27.jpg)
Inverse warping
iwarp(I, I’, T)
{
for (y=0; y<I’.height; y++)
for (x=0; x<I’.width; x++) {
(x,y)=T-1(x’,y’);
I’(x’,y’)=I(x,y);
}
} I I’
xx’
T-1
![Page 28: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/28.jpg)
Inverse warping
• Get each pixel I’(x’) from its corresponding location x = T-1(x’) in I(x)
• What if pixel comes from “between” two pixels?• Answer: resample color value from interpolate
d (prefiltered) source image
f(x) g(x’)x x’
![Page 29: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/29.jpg)
Inverse warping
iwarp(I, I’, T)
{
for (y=0; y<I’.height; y++)
for (x=0; x<I’.width; x++) {
(x,y)=T-1(x’,y’);
I’(x’,y’)=Reconstruct(I,x,y,kernel);
}
} I I’
xx’
T-1
![Page 30: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/30.jpg)
Samplingband limited
![Page 31: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/31.jpg)
Reconstruction
The reconstructed function is obtained by interpolating among the samples in some manner
![Page 32: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/32.jpg)
Reconstruction
• Reconstruction generates an approximation to the original function. Error is called aliasing.
sample position
sample valuesampling reconstruction
![Page 33: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/33.jpg)
Reconstruction
• Computed weighted sum of pixel neighborhood; output is weighted average of input, where weights are normalized values of filter kernel k
width
d
color=0;weights=0;for all q’s dist < width d = dist(p, q); w = kernel(d); color += w*q.color; weights += w;p.Color = color/weights;
p
q
i i
i ii
qk
qfqkpf
)(
)()()(
![Page 34: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/34.jpg)
Reconstruction (interpolation)
• Possible reconstruction filters (kernels):– nearest neighbor– bilinear– bicubic – sinc (optimal reconstruction)
![Page 35: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/35.jpg)
Bilinear interpolation (triangle filter)• A simple method for resampling images
![Page 36: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/36.jpg)
Non-parametric image warping
![Page 37: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/37.jpg)
Non-parametric image warping
• Specify a more detailed warp function• Splines, meshes, optical flow (per-pixel motio
n)
![Page 38: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/38.jpg)
Non-parametric image warping
• Mappings implied by correspondences• Inverse warping
P’?
![Page 39: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/39.jpg)
Non-parametric image warping
'''' CwBwAwP CBA
Barycentric coordinateCwBwAwP CBA
![Page 40: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/40.jpg)
Barycentric coordinates
1321
332211
ttt
AtAtAtP
![Page 41: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/41.jpg)
Non-parametric image warping
P’
'''' CwBwAwP CBA
Barycentric coordinateCwBwAwP CBA
P
![Page 42: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/42.jpg)
Non-parametric image warping
radial basis function
2
)( rer
)log()( 2 rrr
Gaussian
thin platespline
i
iX XPkK
Pi
)'(1
![Page 43: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/43.jpg)
Demo
• http://www.colonize.com/warp/warp04-2.php
• Warping is a useful operation for mosaics, video matching, view interpolation and so on.
![Page 44: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/44.jpg)
Image morphing
![Page 45: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/45.jpg)
Image morphing
• The goal is to synthesize a fluid transformation from one image to another.
image #1 image #2dissolving
• Cross dissolving is a common transition between cuts, but it is not good for morphing because of the ghosting effects.
![Page 46: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/46.jpg)
Image morphing
• Why ghosting?• Morphing = warping + cross-dissolving
shape(geometric)
color(photometric)
![Page 47: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/47.jpg)
morphing
cross-dissolving
Image morphing
image #1 image #2
warp warp
![Page 48: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/48.jpg)
Morphing sequence
![Page 49: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/49.jpg)
Multi-source morphing
![Page 50: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/50.jpg)
Multi-source morphing
![Page 51: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/51.jpg)
Face averaging by morphing
average faces
![Page 52: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/52.jpg)
The average face
• http://www.uni-regensburg.de/Fakultaeten/phil_Fak_II/Psychologie/Psy_II/beautycheck/english/index.htm
![Page 53: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/53.jpg)
Image morphing
create a morphing sequence: for each time t1. Create an intermediate warping field (by
interpolation)2. Warp both images towards it3. Cross-dissolve the colors in the newly warped
images
t=0 t=1t=0.33
![Page 54: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/54.jpg)
Warp specification (mesh warping)
• How can we specify the warp?1. Specify corresponding spline control points
interpolate to a complete warping function
easy to implement, but may not be expressive enough
![Page 55: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/55.jpg)
Warp specification
• How can we specify the warp2. Specify corresponding points
• interpolate to a complete warping function
![Page 56: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/56.jpg)
Solution: convert to mesh warping
1. Define a triangular mesh over the points– Same mesh in both images!– Now we have triangle-to-triangle correspondences
2. Warp each triangle separately from source to destination– How do we warp a triangle?– 3 points = affine warp!– Just like texture mapping
![Page 57: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/57.jpg)
Warp specification (field warping)
• How can we specify the warp?3. Specify corresponding vectors
• interpolate to a complete warping function• The Beier & Neely Algorithm
![Page 58: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/58.jpg)
Beier&Neely (SIGGRAPH 1992)• Single line-pair PQ to P’Q’:
![Page 59: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/59.jpg)
Algorithm (single line-pair)
• For each X in the destination image:1. Find the corresponding u,v2. Find X’ in the source image for that u,v3. destinationImage(X) = sourceImage(X’)
• Examples:
Affine transformation
![Page 60: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/60.jpg)
Multiple Lines
length = length of the line segment, dist = distance to line segmentThe influence of a, p, b. The same as the average of Xi’
iii XXD '
![Page 61: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/61.jpg)
Full Algorithm
![Page 62: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/62.jpg)
Resulting warp
![Page 63: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/63.jpg)
Comparison to mesh morphing
• Pros: more expressive • Cons: speed and control
![Page 64: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/64.jpg)
Warp interpolation
• How do we create an intermediate warp at time t?– linear interpolation for line end-points– But, a line rotating 180 degrees will become 0
length in the middle– One solution is to interpolate line mid-point
and orientation angle
t=0
t=1
![Page 65: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/65.jpg)
Animation
![Page 66: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/66.jpg)
Animated sequences
• Specify keyframes and interpolate the lines for the inbetween frames
• Require a lot of tweaking
![Page 67: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/67.jpg)
Results
Michael Jackson’s MTV “Black or White”http://www.michaeljackson.com/quicktime_blackorwhite.html
![Page 68: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/68.jpg)
Problem with morphing
• So far, we have performed linear interpolation of feature point positions
• But what happens if we try to morph between two views of the same object?
![Page 69: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/69.jpg)
View morphing
• Seitz & Dyerhttp://www.cs.washington.edu/homes/seitz/vmorph/vmorph.
htm • Interpolation consistent with 3D view
interpolation
![Page 70: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/70.jpg)
Main trick
• Prewarp with a homography to "pre-align" images
• So that the two views are parallel– Because linear
interpolation works when views are parallel
![Page 71: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/71.jpg)
prewarp prewarp
morph morph
homographies
input inputoutput
![Page 72: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/72.jpg)
![Page 73: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/73.jpg)
![Page 74: Image warping Li Zhang CS559](https://reader035.vdocuments.us/reader035/viewer/2022062520/5681581e550346895dc5842b/html5/thumbnails/74.jpg)