sampling, filtering, reconstruction
TRANSCRIPT
1
Sampling, Filtering, Reconstruction
Image Manipulation and Computational Photography
CS294-69 Fall 2011
Maneesh Agrawala
[Some slides from James Hays , Derek Hoiem, Alexei Efros and Fredo Durand]
Sampling and Reconstruction
2
Sampled Representations How to store and compute with continuous f(x)’s?
Common scheme for representation: samples • write down the function’s values at many points
© 2006 Steve Marschner
[FvD
FH !
g.14
.14b
/ W
olbe
rg]
Reconstruction Making samples back into a continuous f(x)
• for output (need realizable method) • for analysis or processing (need mathematical method) • amounts to “guessing” what the function did in between
© 2006 Steve Marschner
[FvD
FH !
g.14
.14b
/ W
olbe
rg]
3
Sampling and Reconstruction Simple example: a sine wave
© 2006 Steve Marschner
Undersampling What if we “missed” things between the samples? Simple example: undersampling a sine wave
• unsurprising result: information is lost
© 2006 Steve Marschner
4
Undersampling What if we “missed” things between the samples? Simple example: undersampling a sine wave
• unsurprising result: information is lost • surprising result: indistinguishable from lower frequency
© 2006 Steve Marschner
Undersampling What if we “missed” things between the samples? Simple example: undersampling a sine wave
• unsurprising result: information is lost • surprising result: indistinguishable from lower frequency • also was always indistinguishable from higher frequencies • aliasing: signals “traveling in disguise” as other frequencies
© 2006 Steve Marschner
5
Aliasing in Video
Slide by Steve Seitz
Aliasing in images
6
What’s Happening? Input signal:
x = 0:.05:5; imagesc(sin((2.^x).*x))
Plot as image:
Alias! Not enough samples
Antialiasing What can we do about aliasing? Sample more often
• Join the Mega-Pixel craze of the photo industry • But this can’t go on forever
Make the signal less “wiggly”
• Get rid of some high frequencies • Will loose information • But it looks better than aliasing
7
Preventing Aliasing Introduce lowpass !lters:
• remove high frequencies leaving only safe, low frequencies • choose lowest frequency in reconstruction (disambiguate)
© 2006 Steve Marschner
Linear Filtering: A Key Idea Transformations on signals; e.g.:
• bass/treble controls on stereo • blurring/sharpening operations in image editing • smoothing/noise reduction in tracking
Key properties
• linearity: !lter(f + g) = !lter(f) + !lter(g) • shift invariance: behavior invariant to shifting the input
– delaying an audio signal – sliding an image around
Can be modeled mathematically by convolution
© 2006 Steve Marschner
8
Moving Average Idea: de!ne new function by averaging over sliding window
a simple example to start off: smoothing
© 2006 Steve Marschner
Weighted Moving Average Can add weights to our moving average Weights […, 0, 1, 1, 1, 1, 1, 0, …] / 5
© 2006 Steve Marschner
9
Weighted Moving Average bell curve (gaussian-like) weights […, 1, 4, 6, 4, 1, …]
© 2006 Steve Marschner
Moving Average In 2D What are the weights H?
© 2006 Steve Marschner • 20
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Slide by Steve Seitz
10
Cross-Correlation Filtering Equation with averaging window size (2k+1)x(2k+1):
© 2006 Steve Marschner
Allow different weights for different neighboring pixels:
This is called a cross-correlation operation and written:
H is called the “!lter,” “kernel,” or “mask.”
Slide by Steve Seitz
Gaussian Filtering Gaussian kernel: less weight further from window center H[u,v] is an approximation of a continuous Gaussian:
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
1 2 1
2 4 2
1 2 1
Slide by Steve Seitz
11
Mean vs. Gaussian filtering
Slide by Steve Seitz
Convolution Cross-correlation: A convolution operation is a cross-correlation where the !lter is #ipped
both horizontally and vertically before being applied to the image: It is written: Suppose H is a Gaussian or mean kernel. How does convolution differ
from cross-correlation?
Slide by Steve Seitz
12
Convolution is Nice! Notation: Convolution is a multiplication-like operation
• commutative • associative • distributes over addition • scalars factor out • identity: unit impulse e = […, 0, 0, 1, 0, 0, …]
Conceptually no distinction between !lter and signal Usefulness of associativity
• often apply several !lters one after another: (((a * b1) * b2) * b3) • this is equivalent to applying one !lter: a * (b1 * b2 * b3)
© 2006 Steve Marschner
13
14
15
Tricks with Convolutions
10 20 30 40 50 60
10
20
30
40
50
60
=
UCB UCB UCB
16
Yucky details What about near the image edge?
• the !lter window falls off the edge of the image • need to extrapolate • methods:
– clip !lter (black) – wrap around – copy edge – re#ect across edge – vary !lter near edge
© 2006 Steve Marschner • 43
17
Image Half-Sizing
This image is too big to !t on the screen. How can we reduce it? How to generate a half- sized version?
Sub-sampling
Throw away every other row and column to create a 1/2 size image
- called image sub-sampling
1/4
1/8
Slide by Steve Seitz
18
Sub-sampling
1/4 (2x zoom) 1/8 (4x zoom)
Aliasing! What do we do?
1/2
Slide by Steve Seitz
Downsampling
original image
directly downsampled
19
Pre-!ltering, Reconstruction, Resampling
original image
downsampled reconstruction
reconstructed signal
convolve with Gaussian
Gaussian (lowpass) pre-!ltering
G 1/4
G 1/8
Gaussian 1/2
Solution: !lter the image, then subsample • Filter size should double for each ½ size reduction. Why?
Slide by Steve Seitz
20
Subsampling with Gaussian pre-!ltering
G 1/4 G 1/8 Gaussian 1/2
Slide by Steve Seitz
Compare with...
1/4 (2x zoom) 1/8 (4x zoom) 1/2
Slide by Steve Seitz
21
Resampling to Enlarge
downsampled reconstruction
reconstructed image
convolve with box !lter reconstructed signal
??
upsampled reconstruction
??
Gaussian (lowpass) pre-!ltering
G 1/4
G 1/8
Gaussian 1/2
Solution: !lter the image, then subsample • Filter size should double for each ½ size reduction. Why?
Slide by Steve Seitz
22
Jean Baptiste Joseph Fourier (1768-1830) Had crazy idea (1807):
Any periodic function can be rewritten as a weighted sum of sines and cosines of different frequencies.
Don’t believe it? • Neither did Lagrange,
Laplace, Poisson and other big wigs
• Not translated into English until 1878!
But it’s true! • called Fourier Series
A Sum of Sines
)+φωxAsin(Our building block:
Add enough of them to get any signal f(x) you want!
How many degrees of freedom?
What does each control?
Which one encodes the coarse vs. !ne structure of the signal?
23
For every ω from 0 to inf, F(ω) holds the amplitude A and phase φ of the corresponding sine
How can F hold both? Complex number trick!
Fourier Transform To understand frequency ω let’s reparametrize the signal by ω:
)+φωxAsin(
f(x) F(ω) Fourier Transform
F(ω) f(x) Inverse Fourier Transform
)()()( ωωω iIRF +=
22 )()( ωω IRA +±=)()(tan 1
ωω
φRI−=
We can always go back:
Time and Frequency example : g(t) = sin(2pf t) + (1/3)sin(2p(3f) t)
24
Time and Frequency example : g(t) = sin(2pf t) + (1/3)sin(2p(3f) t)
= +
Frequency Spectra example : g(t) = sin(2pf t) + (1/3)sin(2p(3f) t)
= +
25
Frequency Spectra Usually, frequency is more interesting than the phase
= +
=
Frequency Spectra
26
= +
=
Frequency Spectra
= +
=
Frequency Spectra
27
= +
=
Frequency Spectra
= +
=
Frequency Spectra
28
= 1
1 sin(2 )kA kt
kπ
∞
=∑
Frequency Spectra
Frequency Spectra
29
Extension to 2D
in Matlab, check out: imagesc(log(abs(fftshift(fft2(im)))));
Man-made Scene
30
Can Change Spectrum, then Reconstruct
Low and High Pass !ltering
31
The Convolution Theorem The greatest thing since sliced (banana) bread!
The Fourier transform of the convolution of two functions is the product of their Fourier transforms
The inverse Fourier transform of the product of two Fourier transforms is the convolution of the two inverse Fourier transforms
Convolution in spatial domain is equivalent to multiplication in frequency domain!
]F[]F[]F[ hghg =∗
][F][F][F 111 hggh −−− ∗=
2D Convolution Theorem Example
*
f(x,y)
h(x,y)
g(x,y)
|F(sx,sy)|
|H(sx,sy)|
|G(sx,sy)|
32
Fourier Transform Pairs
Low-pass, Band-pass, High-pass !lters
Low-pass:
High-pass / band-pass:
33
Edges in Images
What Does Blurring Remove?
original
34
smoothed (5x5 Gaussian)
What Does Blurring Remove?
High-Pass Filter
smoothed – original
35
Unsharp Masking
200 400 600 800
100200300400500
- =
=+ α
Hybrid Images [Oliva, Torralba & Schyns 05]
Assignment 0: Implement Hybrid Images
36
Hybrid Images [Oliva, Torralba & Schyns 05]
Assignment 0: Implement Hybrid Images
Image Pyramids
Known as a Gaussian Pyramid [Burt and Adelson, 1983] • In computer graphics, a mip map [Williams, 1983] • A precursor to wavelet transform
Slide by Steve Seitz
37
A bar in the big images is a hair on the zebra’s nose; in smaller images, a stripe; in the smallest, the animal’s nose
Figure from David Forsyth
Gaussian Pyramid Construction
!lter mask
Repeat • Filter • Subsample
Until minimum resolution reached • can specify desired number of levels (e.g., 3-level pyramid)
The whole pyramid is only 4/3 the size of the original image! Slide by Steve Seitz
38
Band-pass !ltering
Laplacian Pyramid (subband images) Created from Gaussian pyramid by subtraction
Gaussian Pyramid (low-pass images)
Laplacian Pyramid
How can we reconstruct (collapse) this pyramid into the original image?
Need this!
Original image
39
Pyramid Based Graphics Laplacian pyramid introduced in classic paper: [Burt & Adelson 83]
Applications summarized in 84 paper • Image interpolation
• Merging images into mosaics
• Creating realistic shadows and shading • Generation of natural looking textures using fractals • Real-time animation of fractals
[Ogden, Adelson, Bergen & Burt 84]
Assignment 0: Extra Credit
Image Gradient The gradient of an image:
The gradient points in the direction of most rapid change in intensity
The gradient direction is given by:
• how does this relate to the direction of the edge?
The edge strength is given by the gradient magnitude
40
Effects of Noise Consider a single row or column of the image
• Plotting intensity as a function of position gives a signal
Th
Where is the edge?
How to compute a derivative?
Where is the edge?
Solution: Smooth First
Th
Look for peaks in
41
Derivative Theorem of Convolution This saves us one operation:
Laplacian of Gaussian
Consider
Laplacian of Gaussian operator
Where is the edge? Zero-crossings of bottom graph
42
2D Edge Detection Filters
is the Laplacian operator:
Laplacian of Gaussian
Gaussian derivative of Gaussian