![Page 1: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/1.jpg)
Fast High Quality Image and Video Background Removal with CUDA Timo Stich | NVIDIA
![Page 2: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/2.jpg)
Motivation
Take a Picture Image Segmentation
(This Talk)
Play Golf in Ireland!
Photo Credits: Golf Girl, Bob Cotter; Cliffs of Moher, Peter Gorman, CC 2.0
![Page 3: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/3.jpg)
Outline
Interactive Segmentation
Probabilistic Color Models
— Gaussian Mixture Model
— RGB Histogram
Edge Aware Segmentation
— Graph Cut (NPP Primitive)
Summary
![Page 4: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/4.jpg)
Interactive Segmentation
User draws strokes to
highlight FG and BG
Intelligent computation of
segmentation
User corrects/refines with
additional strokes until
satisfied
![Page 5: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/5.jpg)
PROBABILISTIC COLOR MODELS
![Page 6: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/6.jpg)
Color Models from User Strokes
Strokes define sets of pixels
that are FG and BG
Anchor the solution
(spatial constraint)
Sample of the Color
Distributions
(color constraint)
![Page 7: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/7.jpg)
Gaussian Mixture Model
N Gaussian blobs for each FG/BG
— Very Accurate
— Initialization Non-Trivial
— Picking N wrong can lead to Overfitting
Implementation Details
— Compute Largest Eigenvector for Clustering
— Reductions to compute the Means and Deviations
— Code available in ―GrabCut‖ SDK Sample
![Page 8: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/8.jpg)
RGB Histogram Model
32x32x32 RGB Histogram for each FG/BG
— Less Accurate
— Simple Implementation
— Very Robust
Implementation Details
— Accumulate Votes per Warp (__ballot, __popc)
— GMEM Atomics
— Multiple Histograms, Reduce in second Kernel
![Page 9: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/9.jpg)
Color Model Comparison
2.8 MP,
GTX 680
Accuracy Robustness Simplicity Performance
GMM + 0 - 16 ms
Histogram 0 + + 4.7 ms
*
GMM (N=4) Histogram (32^3)
![Page 10: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/10.jpg)
EDGE AWARE SEGMENTATION
![Page 11: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/11.jpg)
Computing the Segmentation
What we have:
— Probabilities for colors to belong to FG / BG
— Some pixels classified as FG / BG by the user
— A lot of pixels where we don’t know the class
What we want:
— Classify all pixels as either FG or BG
![Page 12: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/12.jpg)
Naïve Approach
Classify according to Max Probability
— L(x) : max(PFG(C(x)), PBG(C(x)))
Result:
— Very noisy
— Not High Quality
![Page 13: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/13.jpg)
Edge Aware Segmentation
Problem with Naïve:
— Each pixel for itself
— No context information
Intuition:
— Segmentation boundaries should be at image edges
Solution: Decision depends also on neighborhood labels
— Similar Color - Strong Coupling
— Different Color - Weak Coupling
![Page 14: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/14.jpg)
Graph Cut Segmentation
F
B
Capacity ~ P(x=FG)
Capacity ~ P(x=BG)
Capacity ~ Neighbor Coupling
![Page 15: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/15.jpg)
Graph Cut Segmentation
Minimal Cut <-> Edge Aware Boundary
F
B Cost is Sum of all
Cut Edges
![Page 16: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/16.jpg)
NPP Graph Cut Primitive
nppiGraphcut(8)_32(f/s)8u
Computes Minimum Cut on Regular Graphs
— One array for edges capacities for each direction (left,
right, top bottom + diagonal dirs for 8 nbhd)
— One array for connections to Source and Sink
Terminals = Source – Sink
Returns 0/1 in 8u array -> BG/FG classification
![Page 17: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/17.jpg)
Graph Cut Performance
2.8 MP Quadro
60001,2
GTX 6801,2 Core i7 @
3 GHz1,3
4 Nbhd 47 ms 32 ms 460 ms
8 Nbhd 110 ms 72 ms 843 ms
• Result:
– Noise free result
– Segmentation boundaries align
well with image edges
1) Includes time for graph setup
2) NPP 5.0 (10-40% performance boost over 4.1)
3) Maxflow v3.01, Boykov et al, http://vision.csd.uwo.ca/code/maxflow-v3.01.zip
![Page 18: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/18.jpg)
Update the Color Models
Optionally use computed segmentation to
improve the Color Models
User Strokes only User Strokes + Computed Segmentation
![Page 19: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/19.jpg)
Video Segmentation
Initialize the Color Model once
Compute Segmentation for each Frame
![Page 20: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/20.jpg)
Summary
Interactive Tool
Probabilistic Color Model
Edge Aware Segmentation
Code: GrabCut SDK Sample (under CUDALibraries)
![Page 21: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/21.jpg)
BACKUP SLIDES
![Page 22: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/22.jpg)
4/8 Neighborhood Comparison
4 Neighborhood 8 Neighborhood
![Page 23: Fast High Quality Image and Video Segmentation · Graph Cut Performance 2.8 MP 1,2Quadro 60001,2 GTX 680 Core i7 @ 3 GHz1,3 4 Nbhd 47 ms 32 ms 460 ms 8 Nbhd 110 ms 72 ms 843 ms •Result:](https://reader035.vdocuments.us/reader035/viewer/2022071003/5fbfe79879c7583fd32d13f7/html5/thumbnails/23.jpg)
CUDA Warp Vote Accumulation
while(1) {
*smem_warpMaster = laneIdx;
warpMaster = *smem_warpMaster;
if( myKey == smem_keys[warpMaster] ) {
ballot = __ballot(1);
myBallot = laneIdx == warpMaster ?
ballot : 0;
break;
}
myVote = __popc(myBallot);