![Page 1: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/1.jpg)
Guided Image Filtering
Kaiming He
Jian Sun
Xiaoou Tang
The Chinese University of Hong Kong
Microsoft Research Asia
The Chinese University of Hong Kong
![Page 2: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/2.jpg)
Introduction
• Edge-preserving filtering
– An important topic in computer vision
• Denoising, image smoothing/sharpening, texture decomposition, HDR
compression, image abstraction, optical flow estimation, image super-
resolution, feature smoothing…
– Existing methods
• Weighted Least Square [Lagendijk et al. 1988]
• Anisotropic diffusion [Perona and Malik 1990]
• Bilateral filter [Aurich and Weule 95], [Tomasi and Manduchi 98]
• Digital TV (Total Variation) filter [Chan et al. 2001]
![Page 3: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/3.jpg)
Introduction
• Bilateral filter
bilateral
W=GsGr
output q
j
iNj
iji ppWq
)(
)(
input p
range Gr(pi-pj)
spatial Gs(xi-xj)
![Page 4: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/4.jpg)
Introduction
• Joint bilateral filter [Petschnigg et al. 2004]
input p
range Gr(Ii-Ij)
spatial Gs(xi-xj)
bilateral
W=GsGr
guide I
j
iNj
iji pIWq
)(
)(
bilateral filter: I=p
E.g. p: noisy / chrominance channel
I: flash / luminance channel
output q
![Page 5: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/5.jpg)
Introduction
• Advantages of bilateral filtering
– Preserve edges in the smoothing process
– Simple and intuitive
– Non-iterative
![Page 6: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/6.jpg)
Introduction
• Problems in bilateral filtering
– Complexity
• Brute-force: O(r2)
• Distributive histogram: O(logr) [Weiss 06]
• Bilateral grid: band-dependent [Paris and Durand 06], [Chen et al. 07]
• Integral histogram: O(1) [Porikli 08], [Yang et al. 09]
Approximate
(quantized)
![Page 7: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/7.jpg)
Introduction
• Problems in bilateral filtering
– Complexity
– Gradient distortion
• Preserves edges,
but not gradients
Example: detail enhancement
input enhanced
gradient
reversal
gradient
reversal
![Page 8: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/8.jpg)
Introduction
• Our target - to design a new filter
– Edge-preserving filtering
– Non-iterative
– O(1) time, fast and non-approximate
– No gradient distortion
Advantages of bilateral filter
Overcome bilateral filter’s
problems
![Page 9: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/9.jpg)
Guided filter
guide I
input p
22
),()(min apbaI
i
iiba
Iapb
I
pIa
)var(
),cov(
Linear regression
iii npq
ni - noise / texture
Bilateral/joint bilateral filter does
not have this linear model
output q
baIq ii
ii Iaq
![Page 10: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/10.jpg)
• Extend to the entire image
– In all local windows ωk ,compute
the linear coefficients
– Compute the average of akIi+bk in
all ωk that covers pixel qi
Guided filter Definition
iii
ik
kiki
bIa
bIaqk
|
)(1
kkk
k
kk
Iapb
I
pIa
)(var
),(cov
ω2
ω3
ω1
qi
![Page 11: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/11.jpg)
ω2
ω3
ω1
• Parameters
– Window radius r
– Regularization ε
Guided filter Definition
kkk
k
kk
Iapb
I
pIa
)(var
),(cov
2r
qi iii
ik
kiki
bIa
bIaqk
|
)(1
![Page 12: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/12.jpg)
pb
a
0
),cov(
)var(
pI
I
guide I
)var(I
input p
pbIaq ii
output q
Guided filter: smoothing
Iapb
I
pIa
)var(
),cov(
r : determines
band-width
(like σs in BF)
a cascade of
mean filters
![Page 13: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/13.jpg)
Guided filter: edge-preserving
bIaq ii
guide I output q
iI iq
baIIaq iii
)var(
),cov(
I
pIa
ε : degree of
edge-preserving
(like σr in BF)
![Page 14: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/14.jpg)
Example – edge-preserving smoothing
r=4, ε=0.12
σs=4, σr=0.1
r=4, ε=0.22
σs=4, σr=0.2
r=4, ε=0.42
σs=4, σr=0.4
guided
filter
bilateral
filter
(let I=p)
input &
guide
![Page 15: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/15.jpg)
• Our target - to design a new filter
– Edge-preserving filtering
– Non-iterative
– O(1) time, fast and non-approximate
– No gradient distortion
Advantages of bilateral filter
Overcome bilateral filter’s
problems
![Page 16: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/16.jpg)
• mean, var, cov in all local windows
• Integral images [Franklin 1984]
– O(1) time – independent of r
– Non-approximate
Complexity Definition
kkk
k
kk
Iapb
I
pIa
)(var
),(cov
iiii bIaq
O(1) bilateral
(32-bin, 40ms/M) [Porikli 08]
O(1) bilateral
(64-bin, 80ms/M)
O(1) guided
(exact, 80ms/M)
![Page 17: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/17.jpg)
Gradient Preserving
input filtered
enhanced
(detail * 5 + input) gradient
reversal
bilateral filter guided filter
Iaq
detail
(input - filtered)
large
fluctuation
![Page 18: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/18.jpg)
input (I=p) bilateral filter
σs=16, σr=0.1
guided filter
r=16, ε=0.12
Example – detail enhancement
gradient
reversal
bilateral filter guided filter
![Page 19: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/19.jpg)
input (I=p) bilateral filter
σs=16, σr=0.1
guided filter
r=16, ε=0.12
Example – detail enhancement
gradient
reversal
bilateral filter guided filter
![Page 20: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/20.jpg)
Example – HDR compression
input HDR
bilateral filter
σs=15, σr=0.12
guided filter
r=15, ε=0.122
gradient
reversal
bilateral filter guided filter
keep
anti-aliased
![Page 21: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/21.jpg)
Example – flash/no-flash denoising
input p
(no-flash)
joint bilateral filter
σs=8, σr=0.02
guide I
(flash)
guided filter
r=8, ε=0.022
joint bilateral guided filter
gradient
reversal
![Page 22: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/22.jpg)
• Applications: feathering/matting, haze removal
Beyond smoothing
output q
input p
guide I
Iaq
very small ε
preserve most
gradients
![Page 23: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/23.jpg)
Example – feathering
guide I
(size 3000x2000)
![Page 24: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/24.jpg)
Example – feathering
filter input p (binary segmentation)
![Page 25: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/25.jpg)
Example – feathering
filter output q (alpha matte)
![Page 26: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/26.jpg)
Example – feathering
guide I filter input p filter output q
0.3s
image size 6M
matting Laplacian
[Levin et al. 06]
2 min
![Page 27: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/27.jpg)
Example – haze removal
guide I filter input p
(dark channel prior
[He et al. 09])
filter output q
![Page 28: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/28.jpg)
Example – haze removal
guide I guided filter
(<0.1s, 600x400p)
global optimization
(10s)
![Page 29: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/29.jpg)
• “What is an edge” – inherently ambiguous, context-dependent
Limitation
Guided filter
r=16, ε=0.42
Bilateral filter
σs=16, σr=0.4
Input
halo halo
stronger
texture
weaker
edge
![Page 30: Guided Image Filtering - Kaiming Hekaiminghe.com/cvpr09/eccv10ppt.pdf · Introduction •Edge-preserving filtering –An important topic in computer vision •Denoising, image smoothing/sharpening,](https://reader033.vdocuments.us/reader033/viewer/2022060608/605e7abae7143b0fba6bad0d/html5/thumbnails/30.jpg)
Conclusion
• We go from “BF” to “GF”
– Edge-preserving filtering
– Non-iterative
– O(1) time, fast, accurate
– Gradient preserving
– More generic than “smoothing”
Thank you!