graph based segmentation - university of...
TRANSCRIPT
![Page 1: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/1.jpg)
Graph‐based Segmentation
Computer VisionCS 543 / ECE 549
University of Illinois
Derek Hoiem
02/25/10
![Page 2: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/2.jpg)
Last class
• Gestalt cues and principles of organization
• Mean‐shift segmentation– Good general‐purpose segmentation method
– Generally useful clustering, tracking technique
• Watershed segmentation– Good for hierarchical segmentation
– Use in combination with boundary prediction
![Page 3: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/3.jpg)
Today’s class
• Treating the image as a graph–Normalized cuts segmentation
–MRFs Graph cuts segmentation
• Recap
• Go over HW2 instructions
![Page 4: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/4.jpg)
i
Images as graphs
• Fully‐connected graph– node for every pixel
– link between every pair of pixels, p,q
– similarity wij for each link
j
wijc
Source: Seitz
![Page 5: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/5.jpg)
Similarity matrix
Increasing sigma
![Page 6: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/6.jpg)
Segmentation by Graph Cuts
• Break Graph into Segments– Delete links that cross between segments– Easiest to break links that have low cost (low similarity)
• similar pixels should be in the same segments• dissimilar pixels should be in different segments
w
A B C
Source: Seitz
![Page 7: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/7.jpg)
Cuts in a graph
• Link Cut– set of links whose removal makes a graph disconnected– cost of a cut:
A B
One idea: Find minimum cut• gives you a segmentation• fast algorithms exist for doing this
Source: Seitz
![Page 8: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/8.jpg)
But min cut is not always the best cut...
![Page 9: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/9.jpg)
Cuts in a graph
A B
Normalized Cut• a cut penalizes large segments• fix by normalizing for size of segments
• volume(A) = sum of costs of all edges that touch A
Source: Seitz
![Page 10: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/10.jpg)
Recursive normalized cuts1. Given an image or image sequence, set up a weighted
graph: G=(V, E)– Vertex for each pixel– Edge weight for nearby pairs of pixels
2. Solve for eigenvectors with the smallest eigenvalues: (D − W)y = λDy
– Use the eigenvector with the second smallest eigenvalueto bipartition the graph
– Note: this is an approximation
4. Recursively repartition the segmented parts if necessary
http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdfDetails:
![Page 11: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/11.jpg)
Normalized cuts results
![Page 12: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/12.jpg)
Normalized cuts: Pro and con• Pros
– Generic framework, can be used with many different features and affinity formulations
– Provides regular segments
• Cons– Need to chose number of segments
– High storage requirement and time complexity
– Bias towards partitioning into equal segments
• Usage– Use for oversegmentation when you want
regular segments
![Page 13: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/13.jpg)
Graph cuts segmentation
![Page 14: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/14.jpg)
Markov Random Fields
∑∑∈
=edgesji
jii
i datayydataydataEnergy,
21 ),;,(),;(),;( θψθψθy
Node yi: pixel label
Edge: constrained pairs
Cost to assign a label to each pixel
Cost to assign a pair of labels to connected pixels
![Page 15: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/15.jpg)
Markov Random Fields
• Example: “label smoothing” gridUnary potential
0 10 0 K1 K 0
Pairwise Potential
0: -logP(yi = 0 ; data)1: -logP(yi = 1 ; data)
∑∑∈
=edgesji
jii
i datayydataydataEnergy,
21 ),;,(),;(),;( θψθψθy
![Page 16: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/16.jpg)
Solving MRFs with graph cuts
∑∑∈
=edgesji
jii
i datayydataydataEnergy,
21 ),;,(),;(),;( θψθψθy
Source (Label 0)
Sink (Label 1)
Cost to assign to 0
Cost to assign to 1
Cost to split nodes
![Page 17: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/17.jpg)
Solving MRFs with graph cuts
∑∑∈
=edgesji
jii
i datayydataydataEnergy,
21 ),;,(),;(),;( θψθψθy
Source (Label 0)
Sink (Label 1)
Cost to assign to 0
Cost to assign to 1
Cost to split nodes
![Page 18: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/18.jpg)
Grab cuts and graph cuts
User Input
Result
Magic Wand(198?)
Intelligent ScissorsMortensen and Barrett (1995)
GrabCut
Regions Boundary Regions & Boundary
Source: Rother
![Page 19: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/19.jpg)
Colour Model
Gaussian Mixture Model (typically 5-8 components)
Foreground &Background
Background
Foreground
BackgroundG
R
G
RIterated graph cut
Source: Rother
![Page 20: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/20.jpg)
Graph cutsBoykov and Jolly (2001)
ImageMin Cut
Cut: separating source and sink; Energy: collection of edges
Min Cut: Global minimal enegry in polynomial time
Foreground (source)
Background(sink)
Source: Rother
![Page 21: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/21.jpg)
Graph cuts segmentation
1. Define graph – usually 4‐connected or 8‐connected
2. Define unary potentials– Color histogram or mixture of Gaussians for
background and foreground
3. Define pairwise potentials
4. Apply graph cuts5. Return to 2, using current labels to compute
foreground, background models
⎪⎭
⎪⎬⎫
⎪⎩
⎪⎨⎧ −−
+= 2
2
21 2)()(
exp),(_σ
ycxckkyxpotentialedge
⎟⎟⎠
⎞⎜⎜⎝
⎛−=
));(());((
log)(_background
foreground
xcPxcP
xpotentialunaryθθ
![Page 22: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/22.jpg)
Moderately straightforward examples
… GrabCut completes automaticallyGrabCut – Interactive Foreground Extraction 10
![Page 23: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/23.jpg)
Difficult Examples
Camouflage & Low Contrast Harder CaseFine structure
Initial Rectangle
InitialResult
GrabCut – Interactive Foreground Extraction 11
![Page 24: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/24.jpg)
Using graph cuts for recognition
TextonBoost (Shotton et al. 2009 IJCV)
![Page 25: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/25.jpg)
Using graph cuts for recognition
TextonBoost (Shotton et al. 2009 IJCV)
Unary Potentials
Alpha Expansion Graph Cuts
![Page 26: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/26.jpg)
Limits of graph cuts
• Associative: edge potentials penalize different labels
• If not associative, can sometimes clip potentials
• Approximate for multilabel– Alpha‐expansion or alpha‐beta swaps
Must satisfy
![Page 27: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/27.jpg)
Graph cuts: Pros and Cons
• Pros– Very fast inference– Can incorporate recognition or high‐level priors– Applies to a wide range of problems (stereo, image labeling, recognition)
• Cons– Not always applicable (associative only)– Need unary terms (not used for generic segmentation)
• Use whenever applicable
![Page 28: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/28.jpg)
Further reading and resources
• Normalized cuts and image segmentation (Shi and Malik)http://www.cs.berkeley.edu/~malik/papers/SM‐ncut.pdf
• N‐cut implementation http://www.seas.upenn.edu/~timothee/software/ncut/ncut.html
• Graph cuts– http://www.cs.cornell.edu/~rdz/graphcuts.html
– Classic paper: What Energy Functions can be Minimized via Graph Cuts? (Kolmogorov and Zabih, ECCV '02/PAMI '04)
![Page 29: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/29.jpg)
Recap of Grouping and Fitting
![Page 30: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/30.jpg)
Line detection and Hough transform• Canny edge detector = smooth derivative thin threshold link
• Generalized Hough transform = points vote for shape parameters
• Straight line detector = canny + gradient orientations orientation binning linking check for straightness
![Page 31: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/31.jpg)
Robust fitting and registration
Key algorithm
• RANSAC
![Page 32: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/32.jpg)
Clustering
Key algorithm
• Kmeans
![Page 33: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/33.jpg)
EM and Mixture of Gaussians
Tutorials: http://www.cs.duke.edu/courses/spring04/cps196.1/.../EM/tomasiEM.pdfhttp://www‐clmc.usc.edu/~adsouza/notes/mix_gauss.pdf
![Page 34: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/34.jpg)
Segmentation• Mean‐shift segmentation
– Flexible clustering method, good segmentation
• Watershed segmentation– Hierarchical segmentation from soft boundaries
• Normalized cuts– Produces regular regions
– Slow but good for oversegmentation
• MRFs with Graph Cut– Incorporates foreground/background/object
model and prefers to cut at image boundaries
– Good for interactive segmentation or recognition
![Page 35: Graph based Segmentation - University Of Illinoisdhoiem.cs.illinois.edu/courses/vision_spring10/lectures/Lecture12 - Graph-based...graph: G=(V, E) – Vertex for each pixel – Edge](https://reader033.vdocuments.us/reader033/viewer/2022051909/5ffd3c2d852ad97b2f5320e8/html5/thumbnails/35.jpg)
Next section: Recognition
• How to recognize– Specific object instances
– Faces
– Scenes
– Object categories
– Materials