cutting images: graphs and boundary finding
DESCRIPTION
09/14/10. Cutting Images: Graphs and Boundary Finding. Computational Photography Derek Hoiem, University of Illinois. “The Double Secret”, Magritte. Last two classes: Color and Light. Questions?. LAB Color Space. Luminance = brightness Chrominance = color. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/1.jpg)
Cutting Images: Graphs and Boundary Finding
Computational PhotographyDerek Hoiem, University of Illinois
09/14/10
“The Double Secret”, Magritte
![Page 2: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/2.jpg)
Last two classes: Color and Light• Questions?
![Page 3: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/3.jpg)
LAB Color Space
Luminance = brightnessChrominance = color
![Page 4: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/4.jpg)
If you had to choose, would you rather go without luminance or chrominance?
![Page 5: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/5.jpg)
If you had to choose, would you rather go without luminance or chrominance?
![Page 6: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/6.jpg)
Most information in intensity
Only color shown – constant intensity
![Page 7: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/7.jpg)
Most information in intensity
Only intensity shown – constant color
![Page 8: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/8.jpg)
Most information in intensity
Original image
![Page 9: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/9.jpg)
Project 1 Results• Class Favorite Projects (tie)
– Charles Park– Jia-Bin Huang
• Class Favorite Results (tie): – “Underwear woman(lion???) to the rescue” by Wei (Alex)
Lou – “Who’s sitting there?” by Guenther Charwat– Brimeley and Cat (Set #1) by Steve Melnicki
• About half of the class got at least one vote for favorite project or result
![Page 10: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/10.jpg)
Project 2: Images of the Russian Empire: Colorizing the Prokudin-Gorskii photo collection• Due Monday (Sept 20), 11:59pm
• Simple image alignment• Coarse-to-fine alignment• Bells and whistles
– Automatically crop borders– Automatically adjust colors– Align using FFT
![Page 11: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/11.jpg)
This month: The digital canvas
Camera models, single-view geometry, and 3D reconstruction
Image warping and object morphing
Cutting and pasting objects, filling holes, and blending
![Page 12: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/12.jpg)
This Lecture: Finding Seams and Boundaries
Segmentation
![Page 13: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/13.jpg)
This Lecture: Finding Seams and Boundaries
Retargetting
http://swieskowski.net/carve/
![Page 14: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/14.jpg)
This Lecture: Finding Seams and Boundaries
Stitching
![Page 15: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/15.jpg)
This Lecture: Finding seams and boundaries
Fundamental Concept: The Image as a Graph• Intelligent Scissors: Good boundary = short path• Graph cuts: Good region is has low cutting cost
![Page 16: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/16.jpg)
Semi-automated segmentation
User provides imprecise and incomplete specification of region – your algorithm has to read his/her mind.
Key problems1. What groups of pixels form cohesive regions?2. What pixels are likely to be on the boundary of regions?3. Which region is the user trying to select?
![Page 17: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/17.jpg)
What makes a good region?• Contains small range of color/texture• Looks different than background• Compact
![Page 18: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/18.jpg)
What makes a good boundary?• High gradient along boundary• Gradient in right direction• Smooth
![Page 19: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/19.jpg)
The Image as a Graph
Node: pixel
Edge: cost of path or cut between two pixels
![Page 20: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/20.jpg)
Intelligent ScissorsMortenson and Barrett (SIGGRAPH 2005)
![Page 21: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/21.jpg)
Intelligent Scissors
A good image boundary has a short path through the graph.
Mortenson and Barrett (SIGGRAPH 2005)
1 2 1
4
1
6
9
1
3
1
4
113
2
3
5
Start
End
![Page 22: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/22.jpg)
Intelligent Scissors• Formulation: find good boundary between seed
points• Challenges
– Minimize interaction time– Define what makes a good boundary– Efficiently find it
![Page 23: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/23.jpg)
Intelligent Scissors: method1. Define boundary cost between neighboring pixels2. User specifies a starting point (seed)3. Compute lowest cost from seed to each other pixel4. Get new seed, get path between seeds, repeat
![Page 24: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/24.jpg)
Intelligent Scissors: method1. Define boundary cost between neighboring pixels
a) Lower if edge is present (e.g., with edge(im, ‘canny’))b) Lower if gradient is strongc) Lower if gradient is in direction of boundary
![Page 25: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/25.jpg)
Gradients, Edges, and Path Cost
Gradient Magnitude
Edge Image
Path Cost
![Page 26: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/26.jpg)
Intelligent Scissors: method1. Define boundary cost between neighboring pixels2. User specifies a starting point (seed)
– Snapping
![Page 27: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/27.jpg)
Intelligent Scissors: method1. Define boundary cost between neighboring pixels2. User specifies a starting point (seed)3. Compute lowest cost from seed to each other pixel
– Djikstra’s shortest path algorithm
![Page 28: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/28.jpg)
Djikstra’s shortest path algorithmInitialize, given seed s:• Compute cost2(q, r) % cost for boundary from pixel q to neighboring
pixel r• cost(s) = 0 % total cost from seed to this point • A = {s} % set to be expanded• E = { } % set of expanded pixels• P(q) % pointer to pixel that leads to q
Loop while A is not empty1. q = pixel in A with lowest cost2. for each pixel r in neighborhood of q that is not in E
a) cost_tmp = cost(q) + cost2(q,r)
b) if (r is not in A) OR (cost_tmp < cost(r))i. cost(r) = cost_tmp ii. P(r) = qiii. Add r to A
![Page 29: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/29.jpg)
Intelligent Scissors: method1. Define boundary cost between neighboring pixels2. User specifies a starting point (seed)3. Compute lowest cost from seed to each other pixel4. Get new seed, get path between seeds, repeat
![Page 30: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/30.jpg)
Intelligent Scissors: improving interaction
1. Snap when placing first seed2. Automatically adjust as user drags3. Freeze stable boundary points to make new seeds
![Page 31: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/31.jpg)
Where will intelligent scissors work well, or have problems?
![Page 32: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/32.jpg)
Grab cuts and graph cuts Grab cuts and graph cuts
User Input
Result
Magic Wand (198?)
Intelligent ScissorsMortensen and Barrett (1995)
GrabCut
Regions Boundary Regions & Boundary
Source: Rother
![Page 33: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/33.jpg)
Segmentation with graph cuts
edgesji
jii
i datayydataydataEnergy,
21 ),;,(),;(),;( y
Source (Label 0)
Sink (Label 1)
Cost to assign to 0
Cost to assign to 1
Cost to split nodes
![Page 34: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/34.jpg)
Segmentation with graph cuts
edgesji
jii
i datayydataydataEnergy,
21 ),;,(),;(),;( y
Source (Label 0)
Sink (Label 1)
Cost to assign to 0
Cost to assign to 1
Cost to split nodes
![Page 35: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/35.jpg)
Colour Model Colour Model
Gaussian Mixture Model (typically 5-8 components)
Foreground &Background
Background
Foreground
BackgroundG
R
G
RIterated graph cut
Source: Rother
![Page 36: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/36.jpg)
Graph cuts Boykov and Jolly (2001)
Graph cuts Boykov and Jolly (2001)
Image Min Cut
Cut: separating source and sink; Energy: collection of edges
Min Cut: Global minimal enegry in polynomial time
Foreground (source)
Background(sink)
Source: Rother
![Page 37: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/37.jpg)
Graph cuts segmentation1. Define graph
– usually 4-connected or 8-connected
2. Set weights to foreground/background– Color histogram or mixture of Gaussians for background
and foreground
3. Set weights for edges between pixels
4. Apply min-cut/max-flow algorithm5. Return to 2, using current labels to compute
foreground, background models
2
2
21 2
)()(exp),(_
ycxc
kkyxpotentialedge
));((
));((log)(_
background
foreground
xcP
xcPxpotentialunary
![Page 38: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/38.jpg)
What is easy or hard about these cases for graphcut-based segmentation?
![Page 39: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/39.jpg)
Easier examples
GrabCut – Interactive Foreground Extraction 10
![Page 40: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/40.jpg)
More difficult Examples
Camouflage & Low Contrast Harder CaseFine structure
Initial Rectangle
InitialResult
GrabCut – Interactive Foreground Extraction 11
![Page 41: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/41.jpg)
Lazy Snapping (Li et al. SG 2004)
![Page 42: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/42.jpg)
Limitations of Graph Cuts
• Requires associative graphs– Connected nodes should prefer to have the same
label
• Is optimal only for binary problems
![Page 43: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/43.jpg)
Other applications: Seam Carving
Demo: http://swieskowski.net/carve/
Seam Carving – Avidan and Shamir (2007)
![Page 44: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/44.jpg)
Other applications: Seam Carving
• Find shortest path from top to bottom (or left to right), where cost = gradient magnitude
Demo: http://swieskowski.net/carve/
Seam Carving – Avidan and Shamir (2007)
http://www.youtube.com/watch?v=6NcIJXTlugc
![Page 45: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/45.jpg)
Other applications: stitching
Graphcut Textures – Kwatra et al. SIGGRAPH 2003
![Page 46: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/46.jpg)
Other applications: stitching
+
Graphcut Textures – Kwatra et al. SIGGRAPH 2003
Ideal boundary:1. Similar color in both images2. High gradient in both images
![Page 47: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/47.jpg)
Summary of big ideas
• Treat image as a graph– Pixels are nodes– Between-pixel edge weights based on gradients– Sometimes per-pixel weights for affinity to
foreground/background
• Good boundaries are a short path through the graph (Intelligent Scissors, Seam Carving)
• Good regions are produced by a low-cost cut (GrabCuts, Graph Cut Stitching)
![Page 48: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/48.jpg)
Take-home questions09/14/10
Possible factors: albedo, shadows, texture, specularities, curvature, lighting direction
Slides: http://www.cs.illinois.edu/~dhoiem/courses/vision_spring10/lectures/LightandColor.pdf
1.
![Page 49: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/49.jpg)
Take-home questions
2. What would be the result in “Intelligent Scissors” if all of the edge costs were set to 1?
09/14/10
3. Typically, “GrabCut” will not work well on objects with thin structures. How could you change the boundary costs to better segment such objects?
![Page 50: Cutting Images: Graphs and Boundary Finding](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813d05550346895da6aa04/html5/thumbnails/50.jpg)
Next class
• Discussion of project 1
• Texture synthesis and hole-filling