segmentation from examples
DESCRIPTION
Segmentation from Examples. By: A’laa Kryeem Lecturer: Hagit Hel-Or. What is Segmentation from Examples ?. Segment an image based on one (or more) correctly segmented image(s) assumed to be from the same domain Effective when making a semantic segmentation. Why to use Examples. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/1.jpg)
Segmentation from Examples
By: A’laa KryeemLecturer: Hagit Hel-Or
![Page 2: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/2.jpg)
What is Segmentation from Examples? Segment an image based on
one (or more) correctly segmented image(s) assumed to be from the same domain
Effective when making a semantic segmentation
![Page 3: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/3.jpg)
Why to use Examples The example defines the
granularity of the desired output
Give us the ability to characterize meaningful parts in the image
Using example allow us to use non-parametric model
![Page 4: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/4.jpg)
The example defines the granularity of the desired output:
Training image
Test Image
Desired Segmentation
Induced Segmentation
![Page 5: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/5.jpg)
Why to use Examples The example defines the
granularity of the desired output
Give us the ability to characterize meaningful parts in the image
![Page 6: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/6.jpg)
Give us the ability to characterize meaningful parts in the image
![Page 7: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/7.jpg)
Semantic Segmentation from an example
We want to segment an image into semantically meaningful parts
Required in various applications
![Page 8: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/8.jpg)
Semantic Segmentation from an example
We want to segment an image into semantically meaningful parts
Required in various applications Problems:
Meaningful parts are often too complex
Semantic interpretation is highly subjective, depending on both the application, and the user
![Page 9: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/9.jpg)
Meaningful parts are often too complex
![Page 10: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/10.jpg)
Semantic Segmentation from an example
We want to segment an image into semantically meaningful parts
Required in various applications Problems:
Meaningful parts are often too complex
Semantic interpretation is highly subjective, depending on both the application, and the user
![Page 11: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/11.jpg)
Example of image different segmentation
![Page 12: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/12.jpg)
Semantic Segmentation from an example
So, How to achieve semantic segmentation Getting segmented training
image(s) as input
![Page 13: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/13.jpg)
Training set
![Page 14: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/14.jpg)
Semantic Segmentation from an example
So, How to achieve semantic segmentation Getting segmented training
image(s) as input Using non-parametric
representation
![Page 15: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/15.jpg)
non-parametric model
Each semantic part is represented by a set of square patches
![Page 16: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/16.jpg)
Semantic Segmentation from an example
So, How to achieve semantic segmentation Getting segmented training
image(s) as input Using non-parametric representation Over-segmenting the Test image
into small fragments
![Page 17: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/17.jpg)
Over segmented image
![Page 18: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/18.jpg)
Semantic Segmentation from an example
So, How to achieve semantic segmentation Getting segmented training
image(s) as input Using non-parametric representation Over-segmenting the Test image into
small fragments Compute costs for fragment-
label pairs
![Page 19: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/19.jpg)
(fragment,label )cost example
?
![Page 20: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/20.jpg)
Semantic Segmentation from an example
So, How to achieve semantic segmentation Getting segmented training
image(s) as input Using non-parametric representation Over-segmenting the Test image into
small fragments Compute costs for fragment-label
pairs Graph-cuts multi-label
optimization
![Page 21: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/21.jpg)
Why do we need graph-cuts Graph-cuts optimization is used to label
each fragment in a globally optimal manner
![Page 22: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/22.jpg)
Training set
Test image
Fragmentation
Fragments
Patch sets
Clas
sifica
tion
Classification scores
Graph-Cuts optimization
Result
![Page 23: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/23.jpg)
Over segmenting Fragment: small arbitrarily-shaped
and simply-connected pixel clusters
We assume that small homogeneous regions always belong to the same semantic part
![Page 24: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/24.jpg)
Over segmenting Fragment: small arbitrarily-shaped
and simply connected pixel clusters We assume that small
homogeneous regions always belong to the same semantic part
Advantages: Enforces a locally coherent labeling Reduces the computational complexity
![Page 25: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/25.jpg)
Graph-cuts multi-label optimization
For each fragment we have k cost values, we need to determine which one is the optimal
Using expanded version of the graph-cuts we saw at Jad’s lecture, where we may have more than two labels (background , object)
![Page 26: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/26.jpg)
Algorithm for semantic segmentation1.Pixel labeling costs2.Fragmentation3.Fragment labeling costs4.Graph-cuts optimization
![Page 27: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/27.jpg)
Algorithm for semantic segmentation1.Pixel labeling costs2.Fragmentation3.Fragment labeling costs4.Graph-cuts optimization
![Page 28: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/28.jpg)
Pixel labeling costs Given Itrain and Ltrain
Representing each label(segment) in Ltrain by a set of square patches
We get k sets {Sl} l=1,…,k one for each label
![Page 29: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/29.jpg)
Pixel labeling costs (cont.) Next, we define φ (p, l) for each
(pixel,label) pair The cost of assigning label l to pixel p Itest
= minP’Sl
p:pixel at Itestl : label in Ltrain
ssd(P,P’) is the sum of squared distances between P,P’M:mxmx3P:mXm neighborhood centered at pP’:mxm patch
![Page 30: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/30.jpg)
![Page 31: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/31.jpg)
Algorithm for semantic segmentation1.Pixel labeling costs2.Fragmentation3.Fragment labeling costs4.Graph-cuts optimization
![Page 32: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/32.jpg)
Fragmentation We partition Itest into small,color-
homogeneous regions using mean shift segmentation
![Page 33: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/33.jpg)
Fragmentation We partition Itest into small,color-
homogeneous regions using mean shift segmentation
Fragment size is adjusted according to Itest.(fragments are smaller in more detailed areas of Itest, and larger in more homogeneous regions)
![Page 34: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/34.jpg)
Fragmentation We partition Itest into small,color-
homogeneous regions using mean shift segmentation
Fragment size is adjusted according to Itest.(fragments are smaller in more detailed areas of Itest, and larger in more homogeneous regions)
Fragment boundaries align with edges in the image
![Page 35: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/35.jpg)
Fragmentation (cont.)
Random colorization Detailed close-up
![Page 36: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/36.jpg)
Algorithm for semantic segmentation1.Pixel labeling costs2.Fragmentation3.Fragment labeling costs4.Graph-cuts optimization
![Page 37: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/37.jpg)
Fragment labeling costs
Voting scheme in order to compute labeling costs of each fragment
For each fragment fItest we pick a few representative pixels:
Rep(f)={pi f } i=I,…,Rf
Rf is proportional to |f|
![Page 38: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/38.jpg)
Fragment labeling costs (cont.)
We talked about the cost of assigning a pixel into a label
Rep(f)}
![Page 39: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/39.jpg)
Fragment labeling vs. pixel labeling
Fragment labeling reducing complexity We have n=Assume |Rep(f)|
= ,then we need to compute costs only for pixels
![Page 40: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/40.jpg)
Fragment labeling vs. pixel labeling Enforces a locally coherent labeling
Training image Training seg. Input image
Fragment labelingPixel
labeling
![Page 41: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/41.jpg)
Algorithm for semantic segmentation1.Pixel labeling costs2.Fragmentation3.Fragment labeling costs4.Graph-cuts optimization
![Page 42: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/42.jpg)
Graph-cuts optimizationAfter calculating labeling cost for all image fragments we get k images. Image i describes the cost assigning each pixel at the test image to label i
fragment labeling costs. Costs range in the interval [0,1]
![Page 43: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/43.jpg)
Graph-cuts optimization Now for each pixel pItest we have a labeling
cost We need to find Ltest the globally optimal
labeling Requirements:
Minimizes the total labeling cost Consistent with presence (or absence) of
edges
![Page 44: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/44.jpg)
Graph-cuts optimization (cont.)
For each pair of neighboring pixels we define:
Ψ(p,q,L(p),L(q))=
L(p),L(q) : labels assigned to p,q : difference between pixels p,q (RGB euclidian distance)
![Page 45: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/45.jpg)
Graph-cuts optimization (cont.)
In order to force pixels within each fragment to be labeled the same, and reduce complexity we specify the energy term E(L) (E(L) the value of a labeling scheme) in terms of fragments instead of pixels :
: cost of assigning fragment f to label L(f), weighted by the size of each fragment.: neighboring fragments in Itest.
= :controls trade-off between regions and boundaries
![Page 46: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/46.jpg)
Intuition:
Big value:For pixels p,q even with different colors the graph-cuts step prefers to connect them to the same label to have and reduce the energy. Instead of 1- althout can be big too. This mean we prefer continues regions and not edges.
𝛼∑⟨𝑝 ,𝑞 ⟩
Ψ (𝑝 ,𝑞 ,𝐿 (𝑝 ) ,𝐿 (𝑞))
![Page 47: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/47.jpg)
Intuition:
s value:For pixels p,q even with similar colors the graph-cuts step won’t care about connecting them to different labels because of small value. Even with big value at1-multiplying still give us small E value. Favors boundaries, holding out non-continues regions .
𝛼 ∑⟨ 𝑓 1, 𝑓 2⟩
Ψ( 𝑓 1 , 𝑓 2 ,𝐿 ( 𝑓 1 ) ,𝐿 ( 𝑓 2 ))
![Page 48: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/48.jpg)
Graph-cuts optimization (cont.)
Finally Ltest is determined by solving Ltest=minL E(L)
Fragment labeling Labeling after Graph-Cuts Optimization
![Page 49: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/49.jpg)
Multi-label graph-cut
Colored nodes:labels Squares : fragmentsFor each (fragment,label) pair we have an edge.Edge weigh according to φ.Edges between two squares weighed according to Ψ .
![Page 50: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/50.jpg)
Multi-label graph-cut
Induced graphEach fragment connected to a single label .
![Page 51: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/51.jpg)
Multi-label graph-cut is NP-complete problem! Using Isolation Heuristic we can get an
approximation of E(L)
1. For 1i k construct a minimum weight isolating cut Ei for label Li .
2. Determine h such that Eh has max weight.3. E=.4. Return E.
![Page 52: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/52.jpg)
effect
Training image Training segmentation
Input image
=0.1 =5 =1
![Page 53: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/53.jpg)
Algorithm for semantic segmentation1.Pixel labeling costs2.Fragmentation3.Fragment labeling costs4.Graph-cuts optimization
![Page 54: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/54.jpg)
Algorithm results
Training set
a b c
![Page 55: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/55.jpg)
Bear results
invariant to the number of instances of each semantic part within the image, and insensitive to the shape of each part.
We can’t separate multiple objects belonging to the same label (c).
![Page 56: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/56.jpg)
Algorithm results
Training set
a b c d
![Page 57: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/57.jpg)
Summary We saw that giving segmented example from
the same domain of an image can effectively perform a semantic segmentation
Using example also defines the granularity of the desired output
determining whether an entity belongs to a particular semantic part is more easily done at the fragment level, than on a pixel-by-pixel basis
Using graph-cuts with multi-label support can help making global optimization step for finding optimal labeling
Only one parameter needed , controlling the trade-off between regions and boundaries
![Page 58: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/58.jpg)
Thank You For Listening
![Page 59: Segmentation from Examples](https://reader035.vdocuments.us/reader035/viewer/2022062305/56816100550346895dd0423a/html5/thumbnails/59.jpg)
References
Inducing Semantic Segmentation from an Example,
Yaar Schnitman, Yaron Caspi, Daniel Cohen-Or, and Dani Lischinski. "Segmentation by Example“,
Sameer Agarwal and Serge Belongie. Christoudias, C.M., Georgescu, B.: Edge detection
and image segmentation (edison) system. Boykov, Y., Veksler, O., Zabih, R.: Fast
approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23 (2001) 1222–1239