advanced topics in computer vision spring 2006 video segmentation tal kramer, shai bagon video...
Post on 20-Jan-2016
221 views
TRANSCRIPT
![Page 1: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/1.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Video Segmentation
April 30th, 2006
![Page 2: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/2.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Introduction
• Recognition and Segmentation• Min Cut Max Flow• Single Image Methods
– GrabCut– Lazy Snapping– …
![Page 3: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/3.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Lazy Snapping
• Interactive User Interface
![Page 4: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/4.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Lazy Snapping
• Energy minimization
![Page 5: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/5.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Lazy Snapping
• Energy minimization
![Page 6: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/6.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Lazy Snapping
• Boundary overriding
![Page 7: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/7.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Lazy Snapping
• Boundary overriding
![Page 8: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/8.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Motivation
• Obvious Next Step• Video Cut & Paste• Video Manipulation and Editing
![Page 9: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/9.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Introduction
• Frame by Frame– Time Consuming and Tedious
• Error With Simple Methods– Fast motions– Deforming silhouettes – Changing topologies
![Page 10: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/10.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Introduction
• Two Papers– Video Object Cut
and Paste– Video Cutout
![Page 11: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/11.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Video Object Cut and Paste
Yin Li, Jian Sun, Heung-Yeung Shum
![Page 12: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/12.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Overview
![Page 13: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/13.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Pre-segmentation
• Pre-Segmentation to All Frames
![Page 14: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/14.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Key Frames
• Picking Key Frames.
![Page 15: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/15.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Key Frames
• User Fore/Background Segmentation
![Page 16: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/16.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
3D Graph Cut Segmentation
• 3D Graph – G=(V,A)
• Labeling– Foreground = 1 – Background = 0
• Volume Between Successive Key Frames
![Page 17: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/17.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
3D Graph Construction
• 2 Kinds of Arcs:– AI – Intra
Frames (BLUE)
– AT – Inter Frame (RED)
![Page 18: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/18.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
3D Graph Construction
• Minimizing Equation:
• E1 – Global Color Models
• E2 – Penalizing Spatially
• E3 – Penalizing Temporally
![Page 19: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/19.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Likelihood Energy
• GMMs Decide Label• In Key Frames:
![Page 20: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/20.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
GMM• Gaussian Mixture
Model
• Distance is Measured By:
![Page 21: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/21.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Prior Energies
• E2, E3 Are the Same
• Distance of Adjacent Regions.
• β = (2 E (||cr – cs||2 ))-1
![Page 22: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/22.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Prior Energies
• λ1 = 24
• λ2 = 12
![Page 23: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/23.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
3D Graph Segmentation
![Page 24: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/24.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Errors
• Global Colors• Similarity to
Background• Thin Areas
![Page 25: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/25.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Error Overriding
• Video tubes• Manual corrections
![Page 26: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/26.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Video Tubes
• Local Color Models• Put Two Windows• Tracking Algorithm• Key Frames to
Solve
W1
WT
![Page 27: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/27.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Fixing graph cut segmentation
• Minimizing:
![Page 28: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/28.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Overriding Brush
• Fixing Boundary Manually
![Page 29: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/29.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Manual error overriding
![Page 30: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/30.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Soften hard segmentation
Matting
![Page 31: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/31.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Coherent Matting
• Boundary is not 0/1• Prevent Bolting Pixels• Smooth Paste
![Page 32: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/32.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Coherent Matting
![Page 33: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/33.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Example
![Page 34: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/34.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Example
![Page 35: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/35.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Example
![Page 36: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/36.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Example
![Page 37: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/37.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Video CutOut
J. WANG, P. BHAT, A. COLBURN, M. AGRAWALA, M. COHEN. Interactive Video Cutout. ACM Trans. on Graphics
(Proc. of SIGGAPH2005), 2005
![Page 38: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/38.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Video Cutout introduction
What’s new?• Different user interface• 3D graph formation• Refinement mechanism
![Page 39: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/39.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
System overview
![Page 40: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/40.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
3D Graph construction
• Hierarchical graph nodes:1. Frame by frame mean shift
segmentation2. Aggregating segments across
frames
![Page 41: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/41.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Pixel 26-neighborhood induce links• Lower level links induce higher level
link
3D Graph construction
![Page 42: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/42.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Stroking foreground and background over the 3D spatio-temporal volume
• Not segmenting any frame
User Interface
![Page 43: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/43.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Graph construction– User input propagates upward – Min cut uses yellow nodes
3D Min cut/Max flow
![Page 44: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/44.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
3D Min cut/Max flow
• Weights / Energy function– The energy function:
– Data term: color similarity to F/B model– Link term: cut likelihood
1,
, , , , , ,i i i i j i ji nghbrs i j
E D x c L x x c c
![Page 45: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/45.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
3D Min cut/Max flow
• Terms in energy function
Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
![Page 46: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/46.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Data weight
• User input generates color model (GMM)
• Infinite weight preserves marked pixels
• Data weight = abiding to F/B color model
Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
![Page 47: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/47.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Data weight
White – high probability ForegroundBlack – Low probability Foreground
Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
![Page 48: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/48.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Data weight
White – high probability BackgroundBlack – Low probability Background
Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
![Page 49: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/49.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Strong gradients segment border
• Link cost encourage cut at edges
Link weight Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
![Page 50: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/50.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Link weight
White – low cut probabilityBlack – high cut probability
Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
![Page 51: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/51.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Pixel span: (xo, yo, t)t>0
Data weight Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
![Page 52: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/52.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Data weight Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
• Local background model• Assuming camera is stabilized, video is
registered• Extracting “clean plate” • Weight per pixel span
![Page 53: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/53.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• d(zi) = minimum color distance {“clean plate”, B marked pixel}.
• “Clean plate” cannot be always trusted
• Weight:
Data weight Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
1 100
2 1N
NumFramese
![Page 54: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/54.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Data weight
White – high probability BackgroundBlack – Low probability Background
Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
![Page 55: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/55.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Link span: links between two adjacent pixel spans
Link weight Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
![Page 56: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/56.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Strong edges exists within segment
• Small change over time• Local temporal link cost penalize
strong temporal gradient
Link weight Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
![Page 57: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/57.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Link weight Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
![Page 58: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/58.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Link weight
White – low cut probabilityBlack – high cut probability
Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
![Page 59: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/59.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Energy function
Graph Cut Energy function
L LinkDF ForegroundDB Background
DB,L Pix. history DB,G Color DF,G Color LL Local temporal LG Gradient
3D Min cut/Max flow
1,
, , , , , ,i i i i j i ji nghbrs i j
E D x c L x x c c
λ2
λ1
λ3
![Page 60: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/60.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Iterative process
• The user refines the cut• Adds F/B strokes• Graph is re-computed
Nth iterationN+1th iteration
![Page 61: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/61.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Post Processing
![Page 62: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/62.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Post processing
• Binary cut obtained• Edges need refinement
![Page 63: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/63.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• A pixel-level min cut around edges• Color model obtained form
boundary• Uniform edge cost = small
cut
Refinement
![Page 64: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/64.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
![Page 65: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/65.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Matting
• Soften hard segmentation• Evaluate α Channel
![Page 66: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/66.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Matting
• Refinement fixed boundary locally
• Global 3D mesh• α Channel along
mesh normals
![Page 67: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/67.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Results
![Page 68: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/68.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Results
![Page 69: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/69.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Performance
![Page 70: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/70.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• Pros– Online 3D min cut – Spatio temporal smooth cut
• Cons– Does not handle shadows– Ignore motion blur (LPF to avoid
temporal aliasing)– Cannot separate translucent objects
Summary
![Page 71: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/71.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
ComparisonVideo object cut
and pasteVideo Cutout
Features
•Graph nodes
•UI
2D segmentation
Frame base interface
3D segmentation in 2 stages
spatial-temporal manipulation
Performance
•Preprocessing•Artist time•Post processing•Total
4-5 min25 min
?30 min
25 min10 sec per Min cut
30 min60 min
![Page 72: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/72.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
Questions?
![Page 73: Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649d4b5503460f94a29787/html5/thumbnails/73.jpg)
Video SegmentationTal Kramer, Shai Bagon
Advanced Topics in Computer Vision Spring 2006
• The total energy:
• Foreground and background terms:
• Background terms:
• Link terms:
Energy function
1,
, , , , , ,i i i i j i ji nghbrs i j
E D x c L x x c c
;B Fi i
B F B F
D DD x B D x F
D D D D
2 , 2 ,1B i B L B GD x B D D
3 31i j G LL x x L L 3 0.3
1 100
2 1N
NumFramese