wien stereo-based image and video analysis for multimedia applications application:...

WIEN

Stereo-based Image and Video Stereo-based Image and Video Analysis for Multimedia ApplicationsAnalysis for Multimedia Applications

Application:Application: “Computer-generated Stereoscopic Paintings“ “Computer-generated Stereoscopic Paintings“

M. Gelautz, E. Stavrakis, and M. BleyerM. Gelautz, E. Stavrakis, and M. Bleyer

Interactive Media Systems GroupInteractive Media Systems Group

Technical University Vienna, A-1040 Vienna, AustriaTechnical University Vienna, A-1040 Vienna, Austria

Email: [email protected]: [email protected]

ISPRS 2004

WIEN

Stereo Analysis for Multimedia Applications

Artistic Rendering of Stereo ViewsArtistic Rendering of Stereo Views

Our goal is to provide stereo views of Our goal is to provide stereo views of real scenes a hand-painted appearance.real scenes a hand-painted appearance.

The work combines photogrammetric The work combines photogrammetric techniques (stereo) with computer techniques (stereo) with computer graphic algorithms.graphic algorithms.

Available painterly rendering algorithms Available painterly rendering algorithms have been designed for single images – have been designed for single images – painterly rendering of stereo views painterly rendering of stereo views presents a new topic of research! presents a new topic of research!

ISPRS 2004

WIEN


Traditional Stereoscopic PaintingTraditional Stereoscopic Painting

Fig1. Example of a stereoscopic painting by Salvador DalFig1. Example of a stereoscopic painting by Salvador Dalíí (“The sleeping smoker”, 1972/73)..

ISPRS 2004

WIEN


Motivation for Computer-Motivation for Computer-generated Stereo Paintinggenerated Stereo Painting The manual creation of stereo paintings is a The manual creation of stereo paintings is a

labor intensive task.labor intensive task. The artist needs to reproduce the same The artist needs to reproduce the same

composition twice from different viewpoints.composition twice from different viewpoints. Some painters used stereo photography to Some painters used stereo photography to

base their compositions on.base their compositions on. The excessive effort associated with the The excessive effort associated with the

manual creation of stereoscopic paintings can manual creation of stereoscopic paintings can be reduced by computer vision techniques.be reduced by computer vision techniques.

ISPRS 2004

WIEN


Single Image Painterly RenderingSingle Image Painterly RenderingHertzmann, A., 1998: “Painterly rendering with Hertzmann, A., 1998: “Painterly rendering with curved brush strokes of multiple sizes”curved brush strokes of multiple sizes”

(a) Original image(a) Original image (b) Rendered image (b) Rendered image (impressionist style)(impressionist style)

paint spillingpaint spilling

ISPRS 2004

WIEN


Requirements of Stereoscopic Requirements of Stereoscopic Painterly RenderingPainterly Rendering Observation:Observation: Individual painting of the left and Individual painting of the left and

right stereo image will usually not produce right stereo image will usually not produce satisfactory results.satisfactory results.

Requirements:Requirements: To preserve coherence between the brush To preserve coherence between the brush

strokes of the two images.strokes of the two images. To preserve the depth discontinuities in To preserve the depth discontinuities in

order to prevent „paint spilling“.order to prevent „paint spilling“. To deal with occlusions.To deal with occlusions.

Solution:Solution: We incorporate a stereo-derived We incorporate a stereo-derived depth map into the painting algorithm.depth map into the painting algorithm.

ISPRS 2004

WIEN


Stereo Matching AlgorithmStereo Matching Algorithm ““A Layered Stereo Matching Algorithm Using Color A Layered Stereo Matching Algorithm Using Color

Segmentation and Global Visibility Constraints“Segmentation and Global Visibility Constraints“ Key features:Key features:

Color segmentation of the reference image.Color segmentation of the reference image. Color segments are clustered into more robust Color segments are clustered into more robust

depth layers (mean shift clustering).depth layers (mean shift clustering). Iterative refinement of the solution based on a cost Iterative refinement of the solution based on a cost

function (greedy search algorithm).function (greedy search algorithm). Results:Results: 2nd rank (among 30 algorithms) on the 2nd rank (among 30 algorithms) on the

Middlebury Stereo Evaluation Website Middlebury Stereo Evaluation Website (http://www.middlebury.edu/stereo)(http://www.middlebury.edu/stereo)

ISPRS 2004

WIEN


Clustering of Color SegmentsClustering of Color Segmentsinto Layersinto Layers

Fitting of planar Fitting of planar models (depth layers) models (depth layers) to segments (top) and to segments (top) and layers (bottom) ->layers (bottom) ->layer fitting is more layer fitting is more robust than segment robust than segment fitting!fitting!

ISPRS 2004

WIEN


Stereo Matching ResultsStereo Matching Results

Evaluation: 0.2% Evaluation: 0.2% „bad“ pixels (i.e., „bad“ pixels (i.e., unoccluded pixels unoccluded pixels whose absolute whose absolute disparity error is disparity error is greater than 1)greater than 1)

SawtoothSawtooth stereo pair stereo pair (Middlebury website)(Middlebury website)

Left imageLeft image Right imageRight image

Stereo disparitiesStereo disparities Ground truthGround truth

ISPRS 2004

WIEN


Stereo Painting Algorithm (1)Stereo Painting Algorithm (1) Modify Hertzmann‘s single image painting Modify Hertzmann‘s single image painting

algorithm to prevent brush strokes from algorithm to prevent brush strokes from crossing depth discontinuities.crossing depth discontinuities.

Paint the reference image using the modified Paint the reference image using the modified single image painting algorithm.single image painting algorithm.

Paint (dilated) occluded regions of the second Paint (dilated) occluded regions of the second image.image.

Project paint from the reference image to non-Project paint from the reference image to non-occluded regions of the second image. As a occluded regions of the second image. As a consequence, we avoid repainting large parts consequence, we avoid repainting large parts of the second image.of the second image.

ISPRS 2004

WIEN


Stereo Painting Algorithm (2)Stereo Painting Algorithm (2)

Original imageOriginal image Stereo disparitiesStereo disparities

Occlusion mapOcclusion map Occlusion paintOcclusion paint

ISPRS 2004

WIEN


Stereo Painting Result (1)Stereo Painting Result (1)

Single image Single image painting resultspainting results

Stereo image Stereo image painting resultspainting results

Left imageLeft image

Left imageLeft image Right imageRight image

Right imageRight image

ISPRS 2004

WIEN



Original Original stereo pairstereo pair

Painted Painted stereo pairstereo pair

ISPRS 2004

WIEN



Creating the Creating the paintings…paintings…

ISPRS 2004

WIEN


Summary and OutlookSummary and Outlook We have presentedWe have presented

a new stereo matching algorithm that a new stereo matching algorithm that relies on clustering color-segmented relies on clustering color-segmented regions into depth layers.regions into depth layers.

an algorithm for stereoscopic painterly an algorithm for stereoscopic painterly rendering based on real image pairs.rendering based on real image pairs.

We plan to use the stereo-derived depth We plan to use the stereo-derived depth maps to enrich the computer-generated maps to enrich the computer-generated paintings with further depth cues (e.g., paintings with further depth cues (e.g., different levels of detail).different levels of detail).

ISPRS 2004

WIEN


AcknowledgementsAcknowledgements

This work was supported by the Austrian This work was supported by the Austrian Science Fund (FWF) under project Science Fund (FWF) under project P15663.P15663.

We wish to thank “Verwertungsgesell-We wish to thank “Verwertungsgesell-schaft Bildender Künstler (VBK)“, Austria, schaft Bildender Künstler (VBK)“, Austria, for permission to reproduce „The for permission to reproduce „The sleeping smoker“.sleeping smoker“.

wien stereo-based image and video analysis for multimedia applications application:...

Documents

stereo analysis

stereo image

stereo photography

stereobased image

stereoderived depth

stereo views of real

video analysis

reference image