earth mover's distance and the applicationshuding/selected/emd.pdf · 2018-03-06 · earth...
TRANSCRIPT
![Page 1: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/1.jpg)
Earth Mover’s Distance and The Applications
Hu Ding
Computer Science and Engineering, Michigan State University
![Page 2: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/2.jpg)
The Motivations
• It is easy to compare two single objects: the pairwise“distance”. For example:
1 two points in Euclidean space;2 two colors in color space “Color Science: Concepts and
Methods, Quantitative Data and Formulae” by Wyszecki andStiles.
• How about two distributions of objects?
• Hausdorff distance (many-to-one matching);• one-to-one matching;• Kullback-Leibler (KL) divergence (from information theory),
but the correspondence should be given in advance;• and many others.
• Here we introduce a very general & robust measurement,Earth Mover’s Distance (EMD) “The Earth Mover’sDistance as a Metric for Image Retrieval” by Rubner et al.
![Page 3: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/3.jpg)
The Motivations
• It is easy to compare two single objects: the pairwise“distance”. For example:
1 two points in Euclidean space;2 two colors in color space “Color Science: Concepts and
Methods, Quantitative Data and Formulae” by Wyszecki andStiles.
• How about two distributions of objects?• Hausdorff distance (many-to-one matching);• one-to-one matching;• Kullback-Leibler (KL) divergence (from information theory),
but the correspondence should be given in advance;• and many others.
• Here we introduce a very general & robust measurement,Earth Mover’s Distance (EMD) “The Earth Mover’sDistance as a Metric for Image Retrieval” by Rubner et al.
![Page 4: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/4.jpg)
The Motivations
• It is easy to compare two single objects: the pairwise“distance”. For example:
1 two points in Euclidean space;2 two colors in color space “Color Science: Concepts and
Methods, Quantitative Data and Formulae” by Wyszecki andStiles.
• How about two distributions of objects?• Hausdorff distance (many-to-one matching);• one-to-one matching;• Kullback-Leibler (KL) divergence (from information theory),
but the correspondence should be given in advance;• and many others.
• Here we introduce a very general & robust measurement,Earth Mover’s Distance (EMD) “The Earth Mover’sDistance as a Metric for Image Retrieval” by Rubner et al.
![Page 5: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/5.jpg)
Earth Mover’s Distance
• Given two weighted point-sets A = {a1, · · · , an} andB = {b1, · · · , bm} in some metric space (e.g., Rd) with theweights {αi | 1 ≤ i ≤ n} and {βj | 1 ≤ j ≤ m}, which arecalled the “signatures”:
1 Color image: the histogram bins in color space.2 Grey image: the histogram bins in one dimensional space.3 Or represented by a set of weighted feature points in R2.
• For any pair of ai and bj , define ||ai − bj || as the “grounddistance” (i.e., the distance in the metric space).
![Page 6: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/6.jpg)
Earth Mover’s Distance
• Given two weighted point-sets A = {a1, · · · , an} andB = {b1, · · · , bm} in some metric space (e.g., Rd) with theweights {αi | 1 ≤ i ≤ n} and {βj | 1 ≤ j ≤ m}, which arecalled the “signatures”:
1 Color image: the histogram bins in color space.2 Grey image: the histogram bins in one dimensional space.3 Or represented by a set of weighted feature points in R2.
• For any pair of ai and bj , define ||ai − bj || as the “grounddistance” (i.e., the distance in the metric space).
![Page 7: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/7.jpg)
Earth Mover’s Distance (cont.)
• EMD(A,B) = minF∑n
i=1
∑mj=1 ||ai − bj ||fij where F = {fij}
is a feasible flow from A to B, s.t.,• ∀fij ≥ 0;•
∑ni=1 fij ≤ βj ;
•∑m
j=1 fij ≤ αi ;
•∑n
i=1
∑mj=1 fij = min{
∑ni=1 αi ,
∑mj=1 βj}.
• Intuitively, EMD measures the least amount of work formoving a mass of earth (A) to a collection of holes (B).
![Page 8: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/8.jpg)
Earth Mover’s Distance (cont.)
• EMD(A,B) = minF∑n
i=1
∑mj=1 ||ai − bj ||fij where F = {fij}
is a feasible flow from A to B, s.t.,• ∀fij ≥ 0;•
∑ni=1 fij ≤ βj ;
•∑m
j=1 fij ≤ αi ;
•∑n
i=1
∑mj=1 fij = min{
∑ni=1 αi ,
∑mj=1 βj}.
• Intuitively, EMD measures the least amount of work formoving a mass of earth (A) to a collection of holes (B).
![Page 9: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/9.jpg)
Earth Mover’s Distance (cont.)
• Comparing with other measurements:• Many-to-one matching: all αi s are +∞ and all βjs are 1.• One-to-one matching: all αi s and βjs are 1.• KL-divergence: need to know the correspondence in advance.
• EMD is more general and robust, but suffers from highcomputational complexity (we will see later).
• In addition, the continuous version of EMD is the famous“Monge-Kantorovich problem” in transportation theory.Leonid Kantorovich won the Nobel prize in economics dueto his contribution in this area.
![Page 10: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/10.jpg)
Earth Mover’s Distance (cont.)
• Comparing with other measurements:• Many-to-one matching: all αi s are +∞ and all βjs are 1.• One-to-one matching: all αi s and βjs are 1.• KL-divergence: need to know the correspondence in advance.
• EMD is more general and robust, but suffers from highcomputational complexity (we will see later).
• In addition, the continuous version of EMD is the famous“Monge-Kantorovich problem” in transportation theory.Leonid Kantorovich won the Nobel prize in economics dueto his contribution in this area.
![Page 11: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/11.jpg)
Earth Mover’s Distance (cont.)
• Comparing with other measurements:• Many-to-one matching: all αi s are +∞ and all βjs are 1.• One-to-one matching: all αi s and βjs are 1.• KL-divergence: need to know the correspondence in advance.
• EMD is more general and robust, but suffers from highcomputational complexity (we will see later).
• In addition, the continuous version of EMD is the famous“Monge-Kantorovich problem” in transportation theory.Leonid Kantorovich won the Nobel prize in economics dueto his contribution in this area.
![Page 12: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/12.jpg)
The Algorithms for EMD
• EMD(A,B) = minF∑n
i=1
∑mj=1 ||ai − bj ||fij where F = {fij}
is a feasible flow from A to B, s.t.,• ∀fij ≥ 0;•
∑ni=1 fij ≤ βj ;
•∑m
j=1 fij ≤ αi ;
•∑n
i=1
∑mj=1 fij = min{
∑ni=1 αi ,
∑mj=1 βj}.
• It is easy to observe that EMD is special min cost max flowproblem. So directly, we have
• linear programming by network simplex algorithm;• or several other combinatorial algorithms (see Prof. David
Karger’s online course https://www.youtube.com/channel/
UCtv9PiQVUDzsT4yl7524DCg).
• But the drawback: high running time (usually at least cubicO(VE )), not utilizing the geometric properties.
![Page 13: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/13.jpg)
The Algorithms for EMD
• EMD(A,B) = minF∑n
i=1
∑mj=1 ||ai − bj ||fij where F = {fij}
is a feasible flow from A to B, s.t.,• ∀fij ≥ 0;•
∑ni=1 fij ≤ βj ;
•∑m
j=1 fij ≤ αi ;
•∑n
i=1
∑mj=1 fij = min{
∑ni=1 αi ,
∑mj=1 βj}.
• It is easy to observe that EMD is special min cost max flowproblem. So directly, we have
• linear programming by network simplex algorithm;• or several other combinatorial algorithms (see Prof. David
Karger’s online course https://www.youtube.com/channel/
UCtv9PiQVUDzsT4yl7524DCg).
• But the drawback: high running time (usually at least cubicO(VE )), not utilizing the geometric properties.
![Page 14: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/14.jpg)
The Algorithms for EMD
• EMD(A,B) = minF∑n
i=1
∑mj=1 ||ai − bj ||fij where F = {fij}
is a feasible flow from A to B, s.t.,• ∀fij ≥ 0;•
∑ni=1 fij ≤ βj ;
•∑m
j=1 fij ≤ αi ;
•∑n
i=1
∑mj=1 fij = min{
∑ni=1 αi ,
∑mj=1 βj}.
• It is easy to observe that EMD is special min cost max flowproblem. So directly, we have
• linear programming by network simplex algorithm;• or several other combinatorial algorithms (see Prof. David
Karger’s online course https://www.youtube.com/channel/
UCtv9PiQVUDzsT4yl7524DCg).
• But the drawback: high running time (usually at least cubicO(VE )), not utilizing the geometric properties.
![Page 15: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/15.jpg)
The Geometric Algorithms for EMD
• A heuristic algorithm called thresholded EMD by pruning thelong distances ||ai − bj || so as to reduce the size of E . “Fastand Robust Earth Mover’s Distances” by Pele and Werman.
• Several approximate EMD algorithms with theoretical qualityguarantee, such as “ A near-linear time ε-approximationalgorithm for geometric bipartite matching” by Sharathkumarand Agarwal.
• Random shift + Quadtree to compress the pairwise distances.• The running time is nearly linear on max{
∑ni=1 αi ,
∑mj=1 βj}.
![Page 16: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/16.jpg)
The Geometric Algorithms for EMD
• A heuristic algorithm called thresholded EMD by pruning thelong distances ||ai − bj || so as to reduce the size of E . “Fastand Robust Earth Mover’s Distances” by Pele and Werman.
• Several approximate EMD algorithms with theoretical qualityguarantee, such as “ A near-linear time ε-approximationalgorithm for geometric bipartite matching” by Sharathkumarand Agarwal.
• Random shift + Quadtree to compress the pairwise distances.• The running time is nearly linear on max{
∑ni=1 αi ,
∑mj=1 βj}.
![Page 17: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/17.jpg)
The Applications
• Pattern matching. “The Earth Mover’s Distance underTransformation Sets” by Cohen and Guibas; “SupervisedEarth Mover’s Distance Learning and Its Computer VisionApplications” by Wang and Guibas.
• Image retrieval (embedding + LSH). “Fast Image Retrievalvia Embeddings” by Indyk and Thaper; “Fast ContourMatching Using Approximate Earth Mover’s Distance” byGrauman and Darrell.
![Page 18: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/18.jpg)
The Applications
• Pattern matching. “The Earth Mover’s Distance underTransformation Sets” by Cohen and Guibas; “SupervisedEarth Mover’s Distance Learning and Its Computer VisionApplications” by Wang and Guibas.
• Image retrieval (embedding + LSH). “Fast Image Retrievalvia Embeddings” by Indyk and Thaper; “Fast ContourMatching Using Approximate Earth Mover’s Distance” byGrauman and Darrell.
![Page 19: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/19.jpg)
The Applications (cont.)
• Deep learning. “Wasserstein GAN” by Arjovsky et al., “AGeometric View of Optimal Transportation and GenerativeModel” by Lei et al.
• Word Mover’s Distance for analyzing documents. “FromWord Embeddings To Document Distances” by Kusner et al.
![Page 20: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/20.jpg)
The Applications (cont.)
• Deep learning. “Wasserstein GAN” by Arjovsky et al., “AGeometric View of Optimal Transportation and GenerativeModel” by Lei et al.
• Word Mover’s Distance for analyzing documents. “FromWord Embeddings To Document Distances” by Kusner et al.
![Page 21: Earth Mover's Distance and The Applicationshuding/selected/emd.pdf · 2018-03-06 · Earth Mover’s Distance Given two weighted point-sets A = f a 1; ngand B = fb 1; ;b mgin some](https://reader033.vdocuments.us/reader033/viewer/2022060309/5f0a4d017e708231d42afba8/html5/thumbnails/21.jpg)
Thank You!
Any Question?