![Page 1: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/1.jpg)
![Page 2: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/2.jpg)
A Gentle Introductionto Bilateral Filteringand its Applications
A Gentle Introductionto Bilateral Filteringand its Applications
08/10: Applications: Advanced uses of Bilateral Filters
Jack Tumblin – EECS, Northwestern University
![Page 3: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/3.jpg)
Advanced Uses of Bilateral FiltersAdvanced Uses of Bilateral Filters
![Page 4: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/4.jpg)
Advanced Uses for BilateralAdvanced Uses for Bilateral
A few clever, exemplary applications…
• Flash/No Flash Image Merge(Petschnigg2004,Eisenman2004)
• Tone Management (Bae 2006)
• Exposure Correction (Bennett2006)(See also: Bennett 2007 Multispectral Bilateral Video Fusion, IEEE Trans. On Img
Proc)
Many more, many new ones… – 6 new SIGGRAPH 2007 papers!
![Page 5: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/5.jpg)
Flash / No-Flash Photo Improvement(Petschnigg04) (Eisemann04)
Flash / No-Flash Photo Improvement(Petschnigg04) (Eisemann04)
Merge best features: warm, cozy candle light (no-flash) low-noise, detailed flash image
![Page 6: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/6.jpg)
‘Joint Bilateral’ or ‘Cross Bilateral’ (2004)‘Joint Bilateral’ or ‘Cross Bilateral’ (2004)
Bilateral two kinds of weights,
Cross Bilateral Filter (CBF):
get them from two kinds of images.
• Spatial smoothing of pixels in image A, with
• WEIGHTED by intensity similarities in image B:
![Page 7: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/7.jpg)
‘Cross’ or ‘Joint’ Bilateral Idea:‘Cross’ or ‘Joint’ Bilateral Idea:
Noisy but Strong…
Noisy and Weak…
Range filter preserves signalRange filter preserves signal
Use stronger signal’s range Use stronger signal’s range filter weights…filter weights…
![Page 8: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/8.jpg)
‘Joint’ or ‘Cross’ Bilateral Filter (CBF)‘Joint’ or ‘Cross’ Bilateral Filter (CBF)
• Enhanced ability to find weak details in noise (B’s weights preserve similar edges in A)
• Useful Residues for ‘Detail Transfer’
– CBF(A,B) to remove A’s noisy details
– CBF(B,A) to remove B’s less-noisy details;
– add to CBF(A,B) for clean, detailed, sharp image
(See the papers for details)
![Page 9: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/9.jpg)
‘Joint’ or ‘Cross’ Bilateral Filter (CBF)‘Joint’ or ‘Cross’ Bilateral Filter (CBF)
• Enhanced ability to find weak details in noise(B’s weights preserve similar edges in A)
![Page 10: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/10.jpg)
OverviewOverview
Remove noise + details from image A,
Keep as image A Lighting
-----------------------
Obtain noise-free details from image B,
Discard Image B Lighting Result
No-flash
Basic approach of both flash/noflash papers
![Page 11: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/11.jpg)
Petschnigg: Detail Transfer ResultsPetschnigg: Detail Transfer Results
• Lamp made of hay:
No Flash Flash Detail Transfer
![Page 12: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/12.jpg)
Petschnigg:Petschnigg:
• Flash
![Page 13: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/13.jpg)
Petschnigg:Petschnigg:
• No Flash,
![Page 14: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/14.jpg)
Petschnigg:Petschnigg:
• Result
![Page 15: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/15.jpg)
ApproachesApproaches -- Main IdeaMain Idea
![Page 16: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/16.jpg)
Petschnigg04, Eisemann04 FeaturesPetschnigg04, Eisemann04 Features
Eisemann 2004:Eisemann 2004:
--included image registration,--included image registration,--used lower-noise flash image for color, and --used lower-noise flash image for color, and --compensates for flash shadows--compensates for flash shadows
Petschnigg 2004:Petschnigg 2004:
--included explicit color-balance & red-eye--included explicit color-balance & red-eye--interpolated ‘continuously variable’ flash,--interpolated ‘continuously variable’ flash,--Compensates for flash specularities--Compensates for flash specularities
![Page 17: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/17.jpg)
Tonal Management(Bae et al., SIGGRAPH 2006)Tonal Management(Bae et al., SIGGRAPH 2006)
Cross bilateral, residues visually compelling image decompositions.
• Explore: adjust component contrast, find visually pleasing transfer functions, etc.
• Stylize: finds transfer functions that match histograms of preferred artists,
• ‘Textureness’; local measure of textural richness; can use this to guide local mods to match artist’s
![Page 18: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/18.jpg)
Tone Mgmt. Examples:Tone Mgmt. Examples:
Original
![Page 19: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/19.jpg)
Tone Mgmt. Examples:Tone Mgmt. Examples:
‘Bright and Sharp’
![Page 20: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/20.jpg)
Tone Mgmt. Examples:Tone Mgmt. Examples:
Gray anddetailed
![Page 21: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/21.jpg)
Tone Mgmt. Examples:Tone Mgmt. Examples:
Smooth andgrainy
![Page 22: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/22.jpg)
Tone Management ExamplesTone Management Examples
Source
![Page 23: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/23.jpg)
Tone Management (Bae06) Tone Management (Bae06)
‘Textured
-ness’
Metric:
(shows
highest
Contrast-
adjusted
texture)
![Page 24: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/24.jpg)
Reference ModelReference ModelModel: Ansel Adams
![Page 25: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/25.jpg)
ResultsResultsInput with auto-levels
![Page 26: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/26.jpg)
ResultsResults• Direct Histogram Transfer (dull)
![Page 27: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/27.jpg)
ResultsResults• Best…
![Page 28: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/28.jpg)
Video Enhancement Using Per Pixel Exposures (Bennett, 06)
Video Enhancement Using Per Pixel Exposures (Bennett, 06)
From this video:
ASTA: Adaptive SSpatio-
TTemporal Accumulation Filter
![Page 29: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/29.jpg)
VIDEOVIDEO
![Page 30: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/30.jpg)
• Raw Video Frame:(from FIFO center)
• Histogram stretching;(estimate gain for each pixel)
• ‘Mostly Temporal’ Bilateral Filter:
– Average recent similar values,
– Reject outliers (avoids ‘ghosting’), spatial avg as needed
– Tone Mapping
The Process for One FrameThe Process for One Frame
![Page 31: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/31.jpg)
The Process for One FrameThe Process for One Frame
• Raw Video Frame:(from FIFO center)
• Histogram stretching;(estimate gain for each pixel)
• ‘Mostly Temporal’ Bilateral Filter:
– Average recent similar values,
– Reject outliers (avoids ‘ghosting’), spatial avg as needed
– Tone Mapping
![Page 32: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/32.jpg)
The Process for One FrameThe Process for One Frame
• Raw Video Frame:(from FIFO center)
• Histogram stretching;(estimate gain for each pixel)
• ‘Mostly Temporal’ Bilateral Filter:
– Average recent similar values,
– Reject outliers (avoids ‘ghosting’), spatial avg as needed
– Tone Mapping
(color: # avg’ pixels)
![Page 33: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/33.jpg)
The Process for One FrameThe Process for One Frame
• Raw Video Frame:(from FIFO center)
• Histogram stretching;(estimate gain for each pixel)
• ‘Mostly Temporal’ Bilateral Filter:
– Average recent similar values,
– Reject outliers (avoids ‘ghosting’), spatial avg as needed
– Tone Mapping
![Page 34: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/34.jpg)
Bilateral Filter Variant: Mostly TemporalBilateral Filter Variant: Mostly Temporal
• FIFO for Histogram-stretched video
– Carry gain estimate for each pixel;
– Use future as well as previous values;
• Expanded Bilateral Filter Methods:
– Static scene? Temporal-only avg. works well
– Motion? Bilateral rejects outliers: no ghosts!
• Generalize: ‘Dissimilarity’ (not just || Ip – Iq ||2)
• Voting: spatial filter de-noises motion
![Page 35: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/35.jpg)
Multispectral Bilateral Video Fusion(Bennett,07)
Multispectral Bilateral Video Fusion(Bennett,07)
• Result:
– Produces watchable result from unwatchable input
– VERYVERY robust; accepts almost any dark video;
– Exploits temporal coherence to emulateLow-light HDR video, without special equipment
![Page 36: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/36.jpg)
ConclusionsConclusions
• Bilateral Filter easily adapted, customized to broad class of problems
• One tool among many for complex problems
• Useful in for any task that needsRobust, reliable smoothing with outlier rejection
![Page 37: A Gentle Introduction to Bilateral Filtering and its Applications 08/10: Applications: Advanced uses of Bilateral Filters Jack Tumblin – EECS, Northwestern](https://reader036.vdocuments.us/reader036/viewer/2022062408/56649de45503460f94ada9dd/html5/thumbnails/37.jpg)