video matting from depth maps jonathan finger oliver wang university of california, santa cruz...
TRANSCRIPT
-
Video Matting from Depth MapsJonathan FingerOliver Wang
University of California, Santa Cruz
{jfinger, owang}@soe.ucsc.edu
-
MotivationGiven a video, replace the background with something differentIsolate the find foreground in each frameGiven a video, replace the background with something differentIsolate the find foreground in each frame
-
Our MethodUse a depth camera to automate foreground extractionUse Bayesian mattingImprove the matting algorithm to get more realistic video
-
The Matting ProblemSeparation of a foreground image from a background image
-
The Easy DirectionBackground(known)Foreground(known)Composite(unknown)2 knowns, 1 unknown
-
The Hard DirectionBackground(unknown)Foreground(unknown)Composite(known)1 known, 2 unknowns
-
The Matting ProblemActually there is another unknownRepresents areas that are a combination of foreground and background
-
The Matting Problem=1=0.5=0Foreground
-
The Matting ProblemHow do we isolate the foreground?Use an alpha maskAlpha MaskAn image who's color represents foreground and background
-
The Matting Problemoriginalalpha mask
-
The Masking ProblemBasic pipelineOriginal compositeAlpha maskIsolated foregroundNew backgroundNew composite
-
The Masking ProblemBut, how do you get an alpha mask?
-
Previous WorkBlue Screen MattingPetro Vlahos (1964)Hollywood Special Effects pioneer
Can isolate the foreground if the background is a constant color
-
Previous WorkBackground is known so it is easy to make a mask
-
The Matting ProblemHow can this be done with an unknown background?
Use a general matting algorithminput: original composite + trimapoutput: alpha mask
-
TrimapsA three color image (usually drawn by hand)Black = 100% backgroundWhite = 100% foregroundGray = unknown
-
TrimapsThe matting algorithm fills in the gray area with estimated alpha values
-
Matting AlgorithmsThe matting equationFor each 2D location in the image,there is a given composite pixel C
We are to find F, B, and at each pixel whereC = F + (1 - )B
-
Bayesian MattingOriginal compositeTrimapForeground estimationBackground estimationAlpha mask
-
Matting Algorithmsalpha maskbackground removedclose upKnockoutRuzon and TomasiBayesian
-
Problem with Bayesian MattingThese all require a manual trimapOur goal is to do this with videoWe do not want to make trimaps by hand
-
Previous WorkDefocus Video Matting(McGuire)Two camerasone focused on the backgroundone focused on the foreground
-
Previous WorkA trimap can be generated from the defocused foregroundHowever, apertures have to be very specific and can be thrown off by lightingAlso requires texture in the scene
-
Previous WorkBayesian Matting Using Learned Image Priors(Apostoloff, Fitzgibbon)Sequences of frames can be compared in order to find movement
-
Previous Workassumptionsforeground is movingnothing else is moving
-
Our ContributionAutomatically generated trimapsDoes not depend on lighting, texture or movementImproved Bayesian Matting using depth informationHella Trimaps
-
OverviewLow res depthOriginal compositeHigh res depthTrimapNew compositeNew backgroundAlpha maskSupersampleBayesian mattingCompose
-
Our MethodCanesta depth cameraUses infrared lasers to detect distances from the camera
-
Our MethodCanesta takes 64x64 resolution imageOptical images are 640x640 or more
-
Trimap OverviewTo get a trimap Upsample depth image to resolution of optical image Threshold to separate into two colors Erode/dilate to create a gray border around the foreground
-
UpsamplingUse Qing's supersampled depth methodUse edge cues from high resolution color imageCan increase the depth resolution to up to 100X
-
ThresholdingAssumptionForeground is in front of backgroundThreshold on a distance planeDone once for entire animation
-
Erode/DilateGrow unknown area around edges
-
Improved Bayesian MattingBayesian matting is ill defined when the foreground and background are similar colorsOriginal imageAlpha mask
-
Improved Bayesian MattingUse depth information in Bayesian Matting optimization stepOriginal imageBayesian mattingDepth map
-
Improved Bayesian MattingBayesian MattingImproved Bayesian Method
-
Resultsvideo
-
ConclusionVideo matting can be done without the user having to manually tweak any individual framesWe were able to improve Bayesian Matting using depth information