sampling resampling warping morphing
TRANSCRIPT
Sampling
Resampling
Warping
Morphing
Dr. Shai Avidan
Faculty of Engineering
Tel-Aviv University
Slide Credits (Partial List)
• Alyosha Efros
• Steve Seitz
• Rick Szeliski
• Bill Freeman
• Fredo Durand
Sampling/Resampling/Warping/Morphing/Retargeting
• Sampling
– Given a continuous signal g(x) and its sampled counterpart g_s(x), are the
samples of g_s(x) sufficient to exactly describe g(x)?
– If so, how can g(x) be reconstructed from g_s(x)?
• Resampling
– Image resampling is the process of transforming a sampled image from one
coordinate system to another
• Warping
– Is a geometric transformation that redefines the spatial relationship between
points in the image
• Morphing
– Is the process of blending two images
• Retargeting
– Is a content-aware resizing operation
Sampled representations• How to store and compute with continuous functions?
• Common scheme for representation: samples
– write down the function’s values at many points
[FvD
FH
fig.1
4.1
4b
/ W
olb
erg
]
Reconstruction
• Making samples back into a continuous function
– for output (need realizable method)
– for analysis or processing (need mathematical method)
– amounts to “guessing” what the function did in between
[FvD
FH
fig.1
4.1
4b
/ W
olb
erg
]
1D Example: Audio
low high
frequencies
Sampling in digital audio• Recording: sound to analog to samples to disc
• Playback: disc to samples to analog to sound again
– how can we be sure we are filling in the gaps correctly?
Sampling and Reconstruction
• Simple example: a sine wave
Undersampling
• What if we “missed” things between the samples?
• Simple example: undersampling a sine wave
– unsurprising result: information is lost
Undersampling
• What if we “missed” things between the samples?
• Simple example: undersampling a sine wave
– unsurprising result: information is lost
– surprising result: indistinguishable from lower frequency
Undersampling• What if we “missed” things between the samples?
• Simple example: undersampling a sine wave
– unsurprising result: information is lost
– surprising result: indistinguishable from lower frequency
– also was always indistinguishable from higher frequencies
– aliasing: signals “traveling in disguise” as other frequencies
Aliasing in video
Slide by Steve Seitz
Aliasing in images
What’s happening?Input signal:
x = 0:.05:5; imagesc(sin((2.^x).*x))
Plot as image:
Alias!
Not enough samples
Antialiasing
• What can we do about aliasing?
• Sample more often
– Join the Mega-Pixel craze of the photo industry
– But this can’t go on forever
• Make the signal less “wiggly”
– Get rid of some high frequencies
– Will loose information
– But it’s better than aliasing
Preventing aliasing• Introduce lowpass filters:
– remove high frequencies leaving only safe, low frequencies
– choose lowest frequency in reconstruction (disambiguate)
Sampling and Reconstruction
Image AcquisitionImaging
subsystem
*h(x,y)
Sampling
Subsytem
*s(x,y)Quantizer
f(x,y) g(x,y) gs(x,y) gd(x,y)
Digitized
Image
Scene Image Sampled
Image
g(x,y)=f(x,y)*h(x,y)The scene is mapped to the image via the
point spread function of the camera h(x,y)
gs(x,y)=g(x,y)s(x,y)The continuous image g(x,y) then enters
A sampling subsystem 2D comb function
s(x,y) given by: � � � �� ��
���
�
���
���m n
nymxyxs ,, �
Therefore, gs(x,y) is now a discrete-continuous
image with intensity values defined only over
integral indices of x and y
After sampling, the quantizer takes the continuous
Intensity values and quantizes them
(either uniform or non-uniform)
Sampling (I)
We start with continuous signal f(x). Due to the psf
We obtain a degraded, band-limited output g(x).
The frequency content of g(x) is given by its spectrum,
G(f), as determined by the Fourier transform:� � � �
� � � �fxicfxe
dxexgfG
fxi
fxi
���
�
2sin2cos2
2
�
�
��
��
Where x represents the spatial position and f denotes
Spatial frequency. Since g(x) does not have any
frequencies beyond fmax, we say that it is band-limited.
|G(f)|
-fmax fmax
Sampling (II)
The continuous output g(x) is then digitized by an ideal
Impulse sampler, the comb function, to get the sampled
signal gs(x). The ideal 1-D sampler is given by:
� � � ���
���
��n
snTxxs �
where \delta is the impulse function and T_s is the sampling
period. The running index n is used with \delta to define the
impulse train of the comb function. We now have:
� � � � � �xsxgxg s �
Sampling (III)Taking the Fourier transform of gs(x) yields � � � � � �
� � � �
� ��
��
���
�
���
��
��
��
���
�
n
ss
n
ss
s
nffGf
nffffG
fSfGfG
*
*
�
Where fs is the sampling frequency and * denotes convolution.
We make use of the following properties of the Fourier Transform
1. Multiplication in the spatial domain corresponds to convolution in the
frequency domain.
2. The Fourier transform on an impulse train is itself an impulse train.
3. The spectrum of a signal sampled with frequency fs (Ts=1/fs) yields the
original spectrum replicated in the frequency domain with period fs|Gs(f)|
-fmax fmax-fs fs
Sampling (IV)
The above result reveals that the sampling operation has left the
original input spectrum intact, merely replicating it periodically in
the frequency domain with a spacing of f_s.
This allows us to rewrite G_s(f) as a sum of two terms, the low
frequency (baseband) and high frequency components. The
baseband specturm is exactly G(f), and the high frequency
components, G_high(f), consist of the remaining replicated
versions of G(f).
G_s(f) = G(f) + G_high(f)
Exact signal reconstruction from sampled data requires us to
discard the replicated spectra G_high(f), leaving only G(f), the
spectrum of the signal we seek to recover.
Reconstruction Conditions
The only provision for exact reconstruction is that G(f) be
undistorted due to overlap with G_high(f). Two conditions
must hold for this to be true:
1. The signal must be bandlimited. This avoids spectra with
infinite extent that are impossible to replicate without
overlap. (In most imaging systems, the psf ensures this
condition is satisfied in practice).
2. The sampling frequency f_s must be greater than twice the
maximum frequency f_max, present in the signal. This
minimum sampling frequency, known as the Nyquist rate, is
the minimum distance between spectra copies, each with
bandwidth f_max.
f_s>f_nyquist=2*f_max
The Ideal Low-Pass Filter
Given that it is theoretically possible to perform reconstruction,
how may it be done? The act of reconstruction requires us to
completely suppress G_high(f). This is done by multiplying
G_s(g) with H(f) given as:
� ����
�
��
max
max
0
1
ff
fffH
|Gs(f)|
-fmax fmax-fs fs
H(f)
Sinc Function
In the spatial domain, the ideal low-pass filter is derived by
computing the inverse Fourier transform of H(f). This yields the
sinc function defined as:
Since multiplication in the frequency domain is identical to
convolution in the spatial domain, then we can convolve the
sample signal g_s(x) with sinc(x) to obtain the continuous g(x).
The problem is that sinc(x) requires infinite support which is
impossible in practice.
� � � �x
xx
��sin
sinc �
� � � � � � � � � � ��� dxgxgxxg ss ��� �
��sinc*sinc
Ringing effect
When using a truncated sinc function for reconstruction we get
ringing. These artifacts, known as the Gibbs phenomenon, are
the overshoots and undershoots caused by reconstructing a
signal with truncated frequency terms.
Approximations to the ideal low pass filter balance between
tampering frequencies below f_max and not suppressing fully
frequencies above it.
Nonideal Reconstruction
Since H(f), the ideal filter cannot be computed in practice, we
resort to nonideal filters.�This bring up the problem of how to
assess the quality of a filter
|G(f)|
-fmax fmax-fs fs
Hr(f)
H_r(f) attenuate some frequencies that should remain intact and pass and
pass some of the high frquency that should be suppressed.
This leads to the problem of aliasing:
aliasing: signals “traveling in disguise” as other frequencies high
frequencies disguised as low frequencies
Aliasingaliasing: signals “traveling in disguise” as other
frequencies
It can be solved in one of two ways:
1. Increasing sampling rate (to reach Nyquist rate)
2. Applying low pass filter (to bandlimit the signal)
Aliasing in video
Slide by Steve Seitz
Image Resampling
So, we have a discrete image and we want to warp it to another discrete
image. The two coordinate systems are related by a spatial warping
function. This permits the output image to be generated by the following
straightforward procedure. First, the inverse mapping function is applied to
the output sampling grid, projecting it onto the input. The result is a
resampling grid, specifying the locations at which the input is to be
resampled. Then, the input image is sampled at these points and the
values are assigned to their respective output pixels.
The problem is that resampling grid does not coincide with the input
sampling grid.
The solution is to reconstruct the input image to a continuous image and
then do the sampling.
Ideal Image Resampling
There are four basic elements to ideal image resampling: reconstruction,
warping, prefiltering and sampling.
The Bible
• http://www-2.cs.cmu.edu/~ph/texfund/texfund.pdf
Resampling2D texture space
2D screen space
warp due to perspective
Notations
• Input signal f(u)
• Forward mapping (texture-to-screen)
x=m(u)
• Output signal g(x)
f(u)
g(x)
m
Resampling
• What do we need to do?
f(u)
g(x)
m
Resampling
1. Reconstruct the continuous signal from the
discrete input signal
2. Warp the domain of the continuous signal
3. Prefilter the warped continuous signal
4. Sample this signal
f(u)
g(x)
m
Resampling
Resampling: progression• Discrete input texture f(u) for integer u
• Reconstructed input texture
fc(u) = f(u) * r(u) = � _{k \in Z} f(k) r(u-k)
• Warped texture gc(x)=fc(m-1(x))
• Band-limited output g’c(x) = gc(x) * h(x) = \Integral gc(t) h(x-t) dt
• Discrete output g(x) = g’c(x) s(x)
f(u)
g(x)
m
Resampling
discrete input
warp
warped inputreconstructed input
reconstruct
continuous output
prefilter
source space
resampling with
prefiltering
[Heckbert 89]
discrete output
sample
destination space
Put it together
• Discrete input texture f(u) for integer u
• Reconstructed input texture fc(u) = f(u) - r(u) = �_k f(k) r(u-k)
• Warped texture gc(x)=fc(m-1(x))
• Band-limited output g’c(x)=gc(x)*h(x) = S gc(t) h(x-t) dt
• Discrete output g(x) = g’c(x)- s(x)
• g(x) = gc’(x)
= s gc(t) h(x-t) dt
= s fc(m-1(t)) h(x-t) dt
= s h(x-t) å f(k) r(m-1(t)-k) dt= � _k f(k) � (x, k)
• Where �(x,k) = s h(x – t) r (m-1(t) –k) dt
• Ignoring normalization
• The image space resampling filter combines a warped reconstruction
filter and a low-pass filter
• The signal x, in the output image is warped to the input image through
m^{-1}. Since it falls on non-integer pixel value we use the
reconstruction filter r to interpolate the discrete input signal f, and to
make sure the result conforms to the Nyquist rate we bandlimit it with
the function h
Resampling – convolution view
� �� �
i
ii xhxmrxfxg )())(()()( 1
resampling filter
Resampling
reconstruction kernels
reconstructed input
position
color
irregular spacing
Resampling
Source Space
3. Filter
Destination Space2. W
arp
Destination Space
4.
Sam
ple
Destination Space
Resampling
low-pass filter convolution
resampling filters
sum of resampling filters
warped reconstruction
kernel
Source Space
3. Filter
Destination Space2. W
arp
Destination Space
4.
Sam
ple
Destination Space
• Ignoring normalization
• The image space resampling filter combines a warped reconstruction filter and a low-pass filter
• This is great, but how do we warp reconstruction filters?
Resampling – convolution view
� �� �
i
ii xhxmrxfxg )())(()()( 1
resampling filter
Resampling
• Use local affine approximation of warp
• Elliptical Gaussian kernels [Heckbert 89]
– Closed under affine mappings and convolution
� �� �
i
iii xhxmrcxg )())(~()(1
��i
ii xGc )(
Gaussian resampling kernel
(EWA resampling kernel)
Resampling filter
• Depends on local warp
• For perspective,
approximated by local affine at
center of kernel
• Not bad approximation
because filter small at
periphery
Resampling Filter
warped recon-
struction kernel
low-pass
filter
resampling
filterminification
magnification
EWA resampling
Image Quality Comparison
• Trilinear mipmapping
EWA trilinear mipmapping
Minification/Magnification
In the special case of magnification, we may ignore the prefilter altogether, treating it instead as an impulse function. This is due to the fact that no high frequencies are introduced into the output upon magnification. So we only have to do reconstruction (i.e., interpolation).
Conversely, minifcation introduces high frequencies and does not require any reconstruction (i.e., interpolation) filter and treat it simply as an impulse function. Therefore,
1) For magnification we only need to interpolate
2) For minification we need to low-pass filter and then sub-sample.
For magnification, the shape of the reconstruction filter does not change in
response to the mapping function. In minifcation the reconstruction filter is
replaced by a prefilter. Unlike reconstruction filters, though, the actual
shape must be scaled by an amount linearly related to the minifcation
factor. As the input in increasingly decimated, the prefilter must become
broader and shorter. It becomes broader in order to average more
neighboring pixel together, thereby further bandlimiting the input. Since
larger neighborhood are used to compute each output pixel, the
normalized weights applied to the input decrease to reflect the diminishing
impact of each input sample. As a result, the prefilter grows shorter.
InterpolationInterpolation is the process of determining the values of a function at positions lying between its samples. I achieves this process by fitting a continuous function through the discrete input samples.
For equally spaced data, interpolation can be expressed as:
Where h is the interpolation kernel weighted by coefficients c_k and applied to K data sample x_k
� � � ���
�
��1
0
K
k
kk xxhcxf
Interpolation KernelsNearest neighbor method is given by the following interpolating polynomial:
It can be achieved by convolving the image with a one-pixel width rectangle in the spatial domain. The interpolation kernel for the nearest neighbor algorithm is defined as
Due to the prominent side lobes and infinite extent, a sinc function makes a poor low-pass filter.
� � � �22
11 �� ���
�� kkkk
k
xxx
xxxfxf
� ����
�
���
x
xxh
5.00
5.001
Interpolation KernelsLinear interpolation method is given by the following interpolating polynomial:
In the spatial domain, linear interpolation is equivalent to convolving the sampled input with the following interpolation kernel (tent filter)
� � � � � �
� � � �01
01
00
10
01101001
polynomialinginterpolatfollowing thegivesThis
11 :solvingby determinedare, where,
ffxx
xxfxf
xxaaffaaaxaxf
����
!!"
#
��
��
��
��
����
� ����
�
����
x
xxxh
10
101
Image Morphing History
Morphing is turning one image into another through a seamless transition
Michael Jackson’s “Black or White”
Cross-fading