![Page 1: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/1.jpg)
Background Subtraction and Matting
15-463: Computational PhotographyAlexei Efros, CMU, Fall 2006
© Yuri Bonder
![Page 2: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/2.jpg)
Vote Today!
![Page 3: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/3.jpg)
…if you don’t vote
© sma1
![Page 4: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/4.jpg)
Projects 3 highlights
Stephen Lin
![Page 5: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/5.jpg)
Chaz PRatt
ken2
![Page 6: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/6.jpg)
Severin Hacker
Wei-Chen Chiu
![Page 7: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/7.jpg)
Scott Shih-yu Wang
![Page 8: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/8.jpg)
Faustinus Kevin Gozali
![Page 9: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/9.jpg)
Charudatta Phatak
![Page 10: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/10.jpg)
Stephen Lin
![Page 11: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/11.jpg)
Project 4 highlights
Wei-Chen Chiu
![Page 12: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/12.jpg)
sma1
![Page 13: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/13.jpg)
sma1
![Page 14: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/14.jpg)
Project 4 highlights
Charudatta Phatak
![Page 15: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/15.jpg)
Midterm on Thursday!• 1.5 hours long• Closed book, closed notes, closed laptops• But can have a cheat sheet (2 pages, both sides)
• Will cover all material up to last week!
![Page 16: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/16.jpg)
Midterm Review• Cameras
• Pin-hole model, camera “knobs”, perspective projection, other projections, etc.
• Capturing & modeling Light• Light perception, color, plenoptic function, Lumigraph/Lightfields
• Image Processing• Point processing, histograms, filtering, correlation, convolution, 2D
Fourier transform, low-pass/band-pass/high-pass filtering, edge detection, Gaussian and laplacian pyramids, blending, etc.
• Image Warping and Morphing• 2D parametric transformations, homogeneous coordinates,
degrees of freedom, forward/inverse warping, morphing, face modeling, PCA, etc.
• Mosaicing• Homographies, planar mosaics, cylindrical/spherical mosaics,
degrees of freedom, direct alignment (optical flow), image features, RANSAC, etc.
![Page 17: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/17.jpg)
“Smoke” (1996), the “photo album scene”
![Page 18: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/18.jpg)
Moving in TimeMoving only in time, while not moving in space, has
many advantages• No need to find correspondences• Can look at how each ray changes over time• In science, always good to change just one variable at a time
This approach has always interested artists (e.g. Monet)
Modern surveillance video camera is a great source of information• There are now many such WebCams now, some running for
several years!
![Page 19: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/19.jpg)
Image Stack
As can look at video data as a spatio-temporal volume• If camera is stationary, each line through time corresponds
to a single ray in space• We can look at how each ray behaves • What are interesting things to ask?
t0
255time
![Page 20: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/20.jpg)
Example
![Page 21: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/21.jpg)
Getting the right pixels
Average image
Median Image
![Page 22: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/22.jpg)
Webcams
Lots of cool potential projects • PCA, weather morphing, weather extrapolation, etc.
http://sv.berkeley.edu/view/
![Page 23: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/23.jpg)
Input Video
![Page 24: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/24.jpg)
Average Image
What is happening?
![Page 25: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/25.jpg)
Figure-centric Representation
![Page 26: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/26.jpg)
Context-based Image Correction
Input sequence
3 closestframes
median images
![Page 27: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/27.jpg)
Average/Median ImageWhat can we do with this?
![Page 28: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/28.jpg)
Background Subtraction
--
==
![Page 29: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/29.jpg)
Crowd Synthesis (with Pooja Nath)
1. Do background subtraction in each frame2. Find and record “blobs”3. For synthesis, randomly sample the blobs, taking
care not to overlap them
![Page 30: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/30.jpg)
Background SubtractionA largely unsolved problem…
Estimatedbackground
Difference Image
ThresholdedForegroundon blue
One videoframe
![Page 31: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/31.jpg)
How does Superman fly?
Super-human powers?ORImage Matting and Compositing?
![Page 32: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/32.jpg)
Image Compositing
![Page 33: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/33.jpg)
Compositing Procedure1. Extract Sprites (e.g using Intelligent Scissors in Photoshop)
Composite by David Dewey
2. Blend them into the composite (in the right order)
![Page 34: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/34.jpg)
Compositing: Two Issues
Semi-transparent objects
Pixels too large
![Page 35: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/35.jpg)
Solution: alpha channelAdd one more channel:
• Image(R,G,B,alpha) Sprite!
Encodes transparency (or pixel coverage):• Alpha = 1: opaque object (complete coverage)• Alpha = 0: transparent object (no coverage)• 0<Alpha<1: semi-transparent (partial coverage)
Example: alpha = 0.7
Partial coverage or semi-transparency
![Page 36: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/36.jpg)
Multiple Alpha BlendingSo far we assumed that one image (background)
is opaque. If blending semi-transparent sprites (the “A over
B” operation):
Icomp = αaIa + (1-αa)αbIbαcomp = αa + (1-αa)αb
Note: sometimes alpha is premultiplied: im(αR,αG,αB,α):
Icomp = Ia + (1-αa)Ib(same for alpha!)
![Page 37: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/37.jpg)
“Pulling a Matte”Problem Definition:
• The separation of an image C into– A foreground object image Co, – a background image Cb, – and an alpha matte α
• Co and α can then be used to composite the foreground object into a different image
Hard problem• Even if alpha is binary, this is hard to do automatically
(background subtraction problem)• For movies/TV, manual segmentation of each frame is
infeasible • Need to make a simplifying assumption…
![Page 38: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/38.jpg)
Blue Screen
![Page 39: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/39.jpg)
Blue Screen mattingMost common form of matting in TV studios & movies
Petros Vlahos invented blue screen matting in the 50s. His Ultimatte® is still the most popular equipment. He won an Oscar for lifetime achievement.
A form of background subtraction:• Need a known background• Compute alpha as SSD(C,Cb) > threshold
– Or use Vlahos’ formula: α = 1-p1(B-p2G)• Hope that foreground object doesn’t look like background
– no blue ties!• Why blue?• Why uniform?
![Page 40: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/40.jpg)
The Ultimatte
p1 and p2
![Page 41: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/41.jpg)
Blue screen for superman?
![Page 42: Background Subtraction and Matting - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2006_fall/www/...As can look at video data as a spatio-temporal volume • If camera is stationary,](https://reader035.vdocuments.us/reader035/viewer/2022071213/602d95b1f16e17588f43455d/html5/thumbnails/42.jpg)
to be continued…