sampling and reconstruction - cornell university · 2016-02-18 · cornell cs5625 spring 2016...
Post on 11-Jun-2020
1 Views
Preview:
TRANSCRIPT
Lecture 6 • Cornell CS5625 Spring 2016
Sampling and reconstruction
CS 5625 Lecture 6
1
Lecture 6 • Cornell CS5625 Spring 2016
Sampled representations
• How to store and compute with continuous functions?
• Common scheme for representation: samples write down the function’s values at many points
[FvD
FH fi
g.14
.14b
/ W
olbe
rg]
2
Lecture 6 • Cornell CS5625 Spring 2016
Reconstruction
• Making samples back into a continuous function for output (need realizable method)
for analysis or processing (need mathematical method)
amounts to “guessing” what the function did in between
[FvD
FH fi
g.14
.14b
/ W
olbe
rg]
3
Lecture 6 • Cornell CS5625 Spring 2016
Filtering
• Processing done on a function can be executed in continuous form (e.g. analog circuit)
but can also be executed using sampled representation
• Simple example: smoothing by averaging
4
Lecture 6 • Cornell CS5625 Spring 2016
Roots of sampling
• Nyquist 1928; Shannon 1949 famous results in information theory
• 1940s: first practical uses in telecommunications
• 1960s: first digital audio systems
• 1970s: commercialization of digital audio
• 1982: introduction of the Compact Disc the first high-profile consumer application
• This is why all the terminology has a communications or audio “flavor”
early applications are 1D; for us 2D (images) is important
5
Lecture 6 • Cornell CS5625 Spring 2016
Sampling in digital audio
• Recording: sound to analog to samples to disc
• Playback: disc to samples to analog to sound again how can we be sure we are filling in the gaps correctly?
6
Lecture 6 • Cornell CS5625 Spring 2016
Undersampling
• What if we “missed” things between the samples?
• Simple example: undersampling a sine wave unsurprising result: information is lost
surprising result: indistinguishable from lower frequency
also was always indistinguishable from higher frequencies
aliasing: signals “traveling in disguise” as other frequencies
7
© 2016 Steve Marschner • Cornell CS5625 Spring 2016
Sneak preview
• Sampling creates copies of the signal at higher frequencies
• Aliasing is these frequencies leaking into the reconstructed signal
– frequency fs – x shows up as frequency x
• The solution is filtering– during sampling, filter to keep the high frequencies out so
they don’t create aliases at the lower frequencies– during reconstruction, again filter high frequencies to avoid
including high-frequency aliases in the output.
Lecture 6 • Cornell CS5625 Spring 2016
Preventing aliasing
• Introduce lowpass filters: remove high frequencies leaving only safe, low frequencies
choose lowest frequency in reconstruction (disambiguate)
9
Lecture 6 • Cornell CS5625 Spring 2016
Linear filtering: a key idea
• Transformations on signals; e.g.: bass/treble controls on stereo
blurring/sharpening operations in image editing
smoothing/noise reduction in tracking
• Key properties linearity: filter(f + g) = filter(f) + filter(g)
shift invariance: behavior invariant to shifting the input • delaying an audio signal • sliding an image around
• Can be modeled mathematically by convolution
10
Lecture 6 • Cornell CS5625 Spring 2016
Convolution warm-up
• basic idea: define a new function by averaging over a sliding window
• a simple example to start off: smoothing
11
Lecture 6 • Cornell CS5625 Spring 2016
Convolution warm-up
• Same moving average operation, expressed mathematically:
12
c[i] =1
2r + 1
i+rX
j=i�r
a[j].
Lecture 6 • Cornell CS5625 Spring 2016
Discrete convolution
• Simple averaging:
every sample gets the same weight
• Convolution: same idea but with weighted average
each sample gets its own weight (normally zero far away)
• This is all convolution is: it is a moving weighted average
13
c[i] =1
2r + 1
i+rX
j=i�r
a[j].
ii
ii
ii
ii
9.2. Convolution 191
Σ
0 0 1 4 6 4 1 0 0 ……
a
b =
a b
× 116
Figure 9.4. Computing one value in the discrete convolution of a sequence a with a filter b
that has support five samples wide. Each sample in a ? b is an average of nearby samplesin a, weighted by the values of b.
time instead of equally weighting all samples within a distance r, we use a secondsequence b to give a weight to each sample (Figure 9.4). The value b[i� j] givesthe weight for the sample at position j, which is at a distance i � j from theindex i where we are evaluating the convolution. Here is the definition of (a ? b),expressed as a formula:
(a ? b)[i] =X
j
a[j]b[i� j]. (9.2)
By omitting bounds on j, we indicate that this sum runs over all integers (thatis, from �1 to +1). Figure 9.4 illustrates how one output sample is com-puted, using the example of b = 1
16
[. . . , 0, 1, 4, 6, 4, 1, 0, . . .]—that is, b[0] = 6
16
,a[±1] =
4
16
, etc.In graphics, one of the two functions will usually have finite support (as does
the example in Figure 9.4), which means that it is non-zero only over a finiteinterval of argument values. If we assume that b has finite support, there is someradius r such that b[k] = 0 whenever |k| > r. In that case, we can write the sum
Lecture 6 • Cornell CS5625 Spring 2016
Filters
• Sequence of weights b is called a filter
• Filter is nonzero over its region of supportusually centered on zero: support radius r
• Filter is normalized so that it sums to 1.0 this makes for a weighted average, not just any
old weighted sum
• Most filters are symmetric about 0 since for images we usually want to treat
left and right the same
a box filter
14
Lecture 6 • Cornell CS5625 Spring 2016
Convolution and filtering
• Can express sliding average as convolution with a box filter
• bbox = […, 0, 1, 1, 1, 1, 1, 0, …]/5
15
Lecture 6 • Cornell CS5625 Spring 2016
ii
ii
ii
ii
9.2. Convolution 193
1
1
00.6
15/ 1
5/
15/
0 0 0 0 0 0 0 0 .2 .2 .2 .2 .2 0
0 0 0 .2 .2 .2 0
a[j]
b[i – j]
a[ j]b[i – j]
b[i – j]
a[ j]b[i – j]
(a b)[i]
×
=
×
==
×ΣΣ
Σ
0
0
0 0
0 6–7
j
i
i – j i – j
i – j
Figure 9.6. Discrete convolution of a box function with a step function.
What is the result of convolving a and b? At a particular index i, as shown inFigure ??, the result is the average of the step function over the range from i� 2
to i + 2. If i < �2, we are averaging all zeros and the result is zero. If i � 2,we are averaging all ones and the result is one. In between there are i + 3 ones,resulting in the value i+3
5
. The output is a linear ramp that goes from 0 to 1 overfive samples: 1
5
[. . . , 0, 0, 1, 2, 3, 4, 5, 5, . . . ]. ⇤
Properties of Convolution
The way we’ve written it so far, convolution seems like an asymmetric operation:a is the sequence we’re smoothing, and b provides the weights. But one of the niceproperties of convolution is that it actually doesn’t make any difference which iswhich: the filter and the signal are interchangeable. To see this, just rethink thesum in Equation 9.2 with the indices counting from the origin of the filter b, rather
Example: box and step
16
Lecture 6 • Cornell CS5625 Spring 2016
Convolution and filtering
• Convolution applies with any sequence of weights
• Example: bell curve (gaussian-like) […, 1, 4, 6, 4, 1, …]/16
17
Lecture 6 • Cornell CS5625 Spring 2016
Discrete convolution
• Notation:
• Convolution is a multiplication-like operation commutative
associative
distributes over addition
scalars factor out
identity: unit impulse e = […, 0, 0, 1, 0, 0, …]
• Conceptually no distinction between filter and signal
18
Lecture 6 • Cornell CS5625 Spring 2016
Discrete filtering in 2D
• Same equation, one more index
now the filter is a rectangle you slide around over a grid of numbers
• Commonly applied to images blurring (using box, using gaussian, …)
sharpening (impulse minus blur)
• Usefulness of associativity often apply several filters one after another: (((a * b1) * b2) * b3)
this is equivalent to applying one filter: a * (b1 * b2 * b3)
19
Lecture 6 • Cornell CS5625 Spring 2016
original | box blur sharpened | gaussian blur[Philip Greenspun]
20
Lecture 6 • Cornell CS5625 Spring 2016
Optimization: separable filters
• basic alg. is O(r2): large filters get expensive fast!
• definition: a2(x,y) is separable if it can be written as:
this is a useful property for filters because it allows factoring:
21
Lecture 6 • Cornell CS5625 Spring 2016
Separable filtering
first, convolve with this
second, convolve with this
22
= *
Lecture 6 • Cornell CS5625 Spring 2016
Continuous convolution: warm-up
• Can apply sliding-window average to a continuous function just as well
output is continuous
integration replaces summation
23
Lecture 6 • Cornell CS5625 Spring 2016
Continuous convolution
• Sliding average expressed mathematically:
note difference in normalization (only for box)
• Convolution just adds weights
weighting is now by a function
weighted integral is like weighted average
again bounds are set by support of f(x)
24
Lecture 6 • Cornell CS5625 Spring 2016
One more convolution
• Continuous–discrete convolution
input: a sequence and a continuous function
output: a continuous function
used for reconstruction and resampling
25
Lecture 6 • Cornell CS5625 Spring 2016
Continuous-discrete convolution
26
Lecture 6 • Cornell CS5625 Spring 2016
Resampling
• Changing the sample rate in images, this is enlarging and reducing
• Creating more samples: increasing the sample rate
“upsampling”
“enlarging”
• Ending up with fewer samples: decreasing the sample rate
“downsampling”
“reducing”
27
Lecture 6 • Cornell CS5625 Spring 2016
Resampling
• Reconstruction creates a continuous function forget its origins, go ahead and sample it
28
Lecture 6 • Cornell CS5625 Spring 2016
Cont.–disc. convolution in 2D
• same convolution—just two variables now
loop over nearby pixels, average using filter weight
looks like discrete filter,but offsets are not integersand filter is continuous
remember placement of filterrelative to grid is variable
29
Lecture 6 • Cornell CS5625 Spring 2016
Cont.–disc. convolution in 2D
30
Lecture 6 • Cornell CS5625 Spring 2016
Separable filters for resampling
• just as in filtering, separable filters are useful separability in this context is a statement about a continuous filter, rather than a
discrete one:
• resample in two passes, one resampling each row and one resampling each column
• intermediate storage required: product of one dimension of source and the other dimension of destination
31
Lecture 6 • Cornell CS5625 Spring 2016
two-stage resampling using aseparable filter
[Phi
lip G
reen
spun
]
32
Lecture 6 • Cornell CS5625 Spring 2016
A gallery of filters
• Box filter Simple and cheap
• Tent filter Linear interpolation
• Gaussian filter Very smooth antialiasing filter
• B-spline cubic Very smooth
• Catmull-rom cubic Interpolating
• Mitchell-Netravali cubic Good for image upsampling
33
Lecture 6 • Cornell CS5625 Spring 2016
Box filter
34
Lecture 6 • Cornell CS5625 Spring 2016
Tent filter
35
Lecture 6 • Cornell CS5625 Spring 2016
Gaussian filter
36
Lecture 6 • Cornell CS5625 Spring 2016
B-Spline cubic
37
fB(x) =16
�⌅⇤
⌅⇥
�3(1� |x|)3 + 3(1� |x|)2 + 3(1� |x|) + 1 �1 ⇥ x ⇥ 1,
(2� |x|)3 1 ⇥ |x| ⇥ 2,
0 otherwise.
Lecture 6 • Cornell CS5625 Spring 2016
Catmull-Rom cubic
38
fC(x) =12
�⌅⇤
⌅⇥
�3(1� |x|)3 + 4(1� |x|)2 + (1� |x|) �1 ⇥ x ⇥ 1,
(2� |x|)3 � (2� |x|)2 1 ⇥ |x| ⇥ 2,
0 otherwise.
Lecture 6 • Cornell CS5625 Spring 2016
Michell-Netravali cubic
39
fM (x) =13fB(x) +
23fC(x)
=118
�⌅⇤
⌅⇥
�21(1� |x|)3 + 27(1� |x|)2 + 9(1� |x|) + 1 �1 ⇥ x ⇥ 1,
7(2� |x|)3 � 6(2� |x|)2 1 ⇥ |x| ⇥ 2,
0 otherwise.
Lecture 6 • Cornell CS5625 Spring 2016
Lanczos
40
0
1
-2 -1 0 1 2
0
1
-3 -2 -1 0 1 2 3
fL2(x) =
(sinc(x) sinc(x/2) |x| < 2
0 otherwise
fL3(x) =
(sinc(x) sinc(x/3) |x| < 3
0 otherwise
sinc(x) =sin(⇡x)
⇡x
Lecture 6 • Cornell CS5625 Spring 2016
Effects of reconstruction filters
• For some filters, the reconstruction process winds up implementing a simple algorithm
• Box filter (radius 0.5): nearest neighbor sampling box always catches exactly one input point
it is the input point nearest the output point
so output[i, j] = input[round(x(i)), round(y(j))] x(i) computes the position of the output coordinate i on the input grid
• Tent filter (radius 1): linear interpolation tent catches exactly 2 input points
weights are a and (1 – a)
result is straight-line interpolation from one point to the next
41
Lecture 6 • Cornell CS5625 Spring 2016
Properties of filters
• Degree of continuity
• Impulse response
• Interpolating or no
• Ringing, or overshoot
interpolating filter used for reconstruction
42
Lecture 6 • Cornell CS5625 Spring 2016
Ringing, overshoot, ripples
• Overshoot caused by
negative filtervalues
• Ripples constant in,
non-const. out
ripple free when:
43
Lecture 6 • Cornell CS5625 Spring 2016
Constructing 2D filters
• Separable filters (most common approach)
44
Lecture 6 • Cornell CS5625 Spring 2016
Yucky details
• What about near the edge? the filter window falls off the edge of the image
need to extrapolate
methods: • clip filter (black) • wrap around • copy edge • reflect across edge • vary filter near edge
45
[Phi
lip G
reen
spun
]
Lecture 6 • Cornell CS5625 Spring 2016
Reducing and enlarging
• Very common operation devices have differing resolutions
applications have different memory/quality tradeoffs
• Also very commonly done poorly
• Simple approach: drop/replicate pixels
• Correct approach: use resampling
46
Lecture 6 • Cornell CS5625 Spring 2016
Resampling example
47
Lecture 6 • Cornell CS5625 Spring 2016
Reducing and enlarging
• Very common operation devices have differing resolutions
applications have different memory/quality tradeoffs
• Also very commonly done poorly
• Simple approach: drop/replicate pixels
• Correct approach: use resampling
48
Lecture 6 • Cornell CS5625 Spring 2016
1000 pixel width [Philip Greenspun]
49
Lecture 6 • Cornell CS5625 Spring 2016
250 pixel width
by dropping pixels gaussian filter
[Philip Greenspun]
50
Lecture 6 • Cornell CS5625 Spring 2016
4000 pixel width
box reconstruction filter bicubic reconstruction filter
[Phi
lip G
reen
spun
]
51
Lecture 6 • Cornell CS5625 Spring 2016
Types of artifacts
• Garden variety what we saw in this natural image
fine features become jagged or sparkle
• Moiré patterns
52
Lecture 6 • Cornell CS5625 Spring 2016
600ppi scan of a color halftone image
[Hea
rn &
Bak
er c
over
]
53
Lecture 6 • Cornell CS5625 Spring 2016
downsampling a high resolution scan
by dropping pixels gaussian filter
[Hea
rn &
Bak
er c
over
]
54
Lecture 6 • Cornell CS5625 Spring 2016
Types of artifacts
• Garden variety what we saw in this natural image
fine features become jagged or sparkle
• Moiré patterns caused by repetitive patterns in input
produce large-scale artifacts; highly visible
• These artifacts are aliasing just like in the audio example earlier
• How do I know what filter is best at preventing aliasing? practical answer: experience
theoretical answer: there is another layer of cool math behind all this • based on Fourier transforms • provides much insight into aliasing, filtering, sampling, and reconstruction
55
top related