simultaneous segmentation and 3d pose estimation of humans philip h.s. torr pawan kumar, pushmeet...
Post on 21-Dec-2015
222 views
TRANSCRIPT
![Page 1: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/1.jpg)
Simultaneous Segmentation and 3D Pose Estimation of Humans
Philip H.S. TorrPawan Kumar, Pushmeet Kohli, Matt Bray
Oxford Brookes University
Arasanathan Thayananthan, Bjorn Stenger, Roberto CipollaCambridge University
![Page 2: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/2.jpg)
Algebra
Unifying Conjecture
Tracking = Detection = Recognition Detection = Segmentation
• therefore Tracking (pose estimation)=Segmentation?
![Page 3: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/3.jpg)
Objective
Image Segmentation Pose Estimate??
Aim to get a clean segmentation of a human…
![Page 4: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/4.jpg)
Developments
ICCV 2003, pose estimation as fast nearest neighbour plus dynamics (inspired by Gavrilla and Toyoma & Blake)
BMVC 2004, parts based chamfer to make space of templates more flexible (a la pictorial structures of Huttenlocher)
CVPR 2005, ObjCut combining segmentation and detection.
ICCV 2005 Dynamic Graph Cuts ECCV 2006, interpolation of poses using the MVRVM
(Agarwal and Triggs) ECCV 2006 combination of pose estimation and
segmentation using graph cuts.
![Page 5: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/5.jpg)
Tracking as Detection (Stenger et al ICCV 2003)
Detection has become very efficient,e.g. real-time face detection, pedestrian detection
Example: Pedestrian detection [Gavrila & Philomin, 1999]: Find match among large number of exemplar templates
Issues: Number of templates needed Efficient search Robust cost function
![Page 6: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/6.jpg)
Cascaded Classifiers
![Page 7: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/7.jpg)
First filter : 19.8 % patches remaining
1280x1024 image, 11 subsampling levels, 80sAverage number of filter per patch : 6.7
![Page 8: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/8.jpg)
Filter 10 : 0.74 % patches remaining
1280x1024 image, 11 subsampling levels, 80sAverage number of filter per patch : 6.7
![Page 9: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/9.jpg)
Filter 20 : 0.06 % patches remaining
1280x1024 image, 11 subsampling levels, 80sAverage number of filter per patch : 6.7
![Page 10: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/10.jpg)
Filter 30 : 0.01 % patches remaining
1280x1024 image, 11 subsampling levels, 80sAverage number of filter per patch : 6.7
![Page 11: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/11.jpg)
Filter 70 : 0.007 % patches remaining
1280x1024 image, 11 subsampling levels, 80sAverage number of filter per patch : 6.7
![Page 12: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/12.jpg)
Hierarchical Detection Efficient template matching (Huttenlocher & Olson,
Gavrila) Idea: When matching similar objects, speed-up by
forming template hierarchy found by clustering Match prototypes first, sub-tree only if cost below
threshold
![Page 13: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/13.jpg)
Trees
These search trees are the same as used for efficient nearest neighbour.
Add dynamic model and • Detection = Tracking = Recognition
![Page 14: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/14.jpg)
Evaluation at Multiple Resolutions
One traversal of tree per time step
![Page 15: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/15.jpg)
Evaluation at Multiple Resolutions
Tree: 9000 templates of hand pointing, rigid
![Page 16: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/16.jpg)
Templates at Level 1
![Page 17: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/17.jpg)
Templates at Level 2
![Page 18: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/18.jpg)
Templates at Level 3
![Page 19: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/19.jpg)
Comparison with Particle Filters
This method is grid based,• No need to render the model on line• Like efficient search• Can always use this as a proposal process for
a particle filter if need be.
![Page 20: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/20.jpg)
Interpolation, MVRVM, ECCV 2006
Code available.
![Page 21: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/21.jpg)
Energy being Optimized, link to graph cuts
Combination of• Edge term (quickly evaluated using chamfer)• Interior term (quickly evaluated using integral
images)
Note that possible templates are a bit like cuts that we put down, one could think of this whole process as a constrained search for the best graph cut.
![Page 22: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/22.jpg)
Likelihood : Edges
Edge Detection Projected Contours
Robust EdgeMatching
Input Image 3D Model
![Page 23: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/23.jpg)
Chamfer MatchingInput image Canny edges
Distance transform Projected Contours
![Page 24: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/24.jpg)
Likelihood : Colour
Skin Colour ModelProjected Silhouette
Input Image 3D Model
Template Matching
![Page 25: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/25.jpg)
Template Matching =
Template Matching = constrained search for a cut/segmentation?
Detection = Segmentation?
![Page 26: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/26.jpg)
Objective
Image Segmentation Pose Estimate??
Aim to get a clean segmentation of a human…
![Page 27: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/27.jpg)
MRF for Interactive Image Segmentation, Boykov and Jolly [ICCV 2001]
EnergyMR
F
Pair-wise Terms MAP SolutionUnary likelihoodData (D)
Unary likelihood Contrast Term Uniform Prior(Potts Model)
Maximum-a-posteriori (MAP) solution x* = arg min E(x)x
=
![Page 28: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/28.jpg)
However…
This energy formulation rarely provides realistic (target-
like) results.
![Page 29: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/29.jpg)
ObjCut (yesterday)
Unary potential
Pairwise potential
Pose parameters
Labels
Pixels
Prior Potts model
Pose-specific MRF
![Page 30: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/30.jpg)
Layer 2
Layer 1
Transformations
Θ1
P(Θ1) = 0.9
Cow Instance
Do we really need accurate models?
![Page 31: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/31.jpg)
Do we really need accurate models?
Segmentation boundary can be extracted from edges
Rough 3D Shape-prior enough for region disambiguation
![Page 32: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/32.jpg)
Energy of the Pose-specific MRFEnergy to be
minimizedUnary term
Shape prior
Pairwise potential
Potts model
But what should be the value of θ?
![Page 33: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/33.jpg)
The different terms of the MRF
Original image
Likelihood of being foreground given a
foreground histogram
Grimson-Stauffer
segmentation
Shape prior model
Shape prior (distance transform)
Likelihood of being foreground
given all the terms
Resulting Graph-Cuts
segmentation
![Page 34: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/34.jpg)
Can segment multiple views simultaneously
![Page 35: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/35.jpg)
Solve via gradient descent
Comparable to level set methods
Could use other approaches (e.g. Objcut)
Need a graph cut per function evaluation
![Page 36: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/36.jpg)
Formulating the Pose Inference Problem
![Page 37: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/37.jpg)
But…But…
… to compute the MAP of E(x) w.r.t the pose, it means that the unary terms will be changed at EACHEACH iteration and the maxflow recomputed!
However…However… Kohli and Torr showed how dynamic graph cuts can
be used to efficiently find MAP solutions for MRFs that change minimally from one time instant to the next: Dynamic Graph Cuts (ICCV05).
![Page 38: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/38.jpg)
Dynamic Graph Cuts
PB SB
cheaperoperation
computationally
expensive operation
Simplerproblem
PB*
differencesbetweenA and B
A and Bsimilar
PA SA
solve
![Page 39: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/39.jpg)
Dynamic Image Segmentation
Image
Flows in n-edges Segmentation Obtained
![Page 40: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/40.jpg)
First segmentation problem MAP solution
Ga
Our Algorithm
Gb
second segmentation problem
Maximum flow
residual graph (Gr)
G`
differencebetween
Ga and Gbupdated residual
graph
![Page 41: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/41.jpg)
Energy Minimization using Graph cuts
EMRF(a1,a2)
Sink (1)
Source (0)
a1 a2
Graph Construction for Binary Random Variables
![Page 42: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/42.jpg)
Energy Minimization using Graph cuts
Sink (1)
Source (0)
a1 a2
EMRF(a1,a2) = 2a1
2t-edges
(unary terms)
![Page 43: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/43.jpg)
Energy Minimization using Graph cuts
EMRF(a1,a2) = 2a1 + 5ā1
Sink (1)
Source (0)
a1 a2
2
5
![Page 44: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/44.jpg)
Energy Minimization using Graph cuts
EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2
Sink (1)
Source (0)
a1 a2
2
5
9
4
![Page 45: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/45.jpg)
Energy Minimization using Graph cuts
EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2
Sink (1)
Source (0)
a1 a2
2
5
9
4
2
n-edges(pair-wise term)
![Page 46: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/46.jpg)
Energy Minimization using Graph cuts
EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
Sink (1)
Source (0)
a1 a2
2
5
9
4
2
1
![Page 47: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/47.jpg)
Energy Minimization using Graph cuts
EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
Sink (1)
Source (0)
a1 a2
2
5
9
4
2
1
![Page 48: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/48.jpg)
Energy Minimization using Graph cuts
EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
Sink (1)
Source (0)
a1 a2
2
5
9
4
2
1
a1 = 1 a2 = 1
EMRF(1,1) = 11
Cost of st-cut = 11
![Page 49: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/49.jpg)
Energy Minimization using Graph cuts
EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
Sink (1)
Source (0)
a1 a2
2
5
9
4
2
1
a1 = 1 a2 = 0
EMRF(1,0) = 8
Cost of st-cut = 8
![Page 50: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/50.jpg)
Energy Minimization using Graph cuts
• Most probable (MAP) configuration ≡ minimum cost st-cut.
• st-mincut is in general a NP-hard problem - negative edge weights
• Solvable in polynomial time- non-negative edge weights- corresponds to sub-modular (regular) energy functions
![Page 51: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/51.jpg)
• The Max-flow Problem- Edge capacity and flow balance constraints
Computing the st-mincut from Max-flow algorithms
• Notation- Residual capacity (edge capacity – current flow)- Augmenting path
• Simple Augmenting Path based Algorithms- Repeatedly find augmenting paths and push flow.- Saturated edges constitute the st-mincut. [Ford-Fulkerson Theorem]
Sink (1)
Source (0)
a1 a2
2
5
9
42
1
![Page 52: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/52.jpg)
9 + α
4 + α
Adding a constant to both thet-edges of a node does not change the edges constituting the st-mincut.
Key Observation
Sink (1)
Source (0)
a1 a2
2
5
2
1
E (a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
E*(a1,a2 ) = E(a1,a2) + α(a2+ā2)
= E(a1,a2) + α [a2+ā2 =1]
Reparametrization
![Page 53: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/53.jpg)
9 + α
4
All reparametrizations of the graph are sums of these two types.
Other type of reparametrization
Sink (1)
Source (0)
a1 a2
2
5 + α
2 + α
1 - α
Reparametrization, second type
Both maintain the solution and add a constant α to the energy.
![Page 54: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/54.jpg)
s
Gt
original graph
0/9
0/7
0/5
0/2 0/4
0/1
xi xj
flow/residual capacity
Graph Re-parameterization
![Page 55: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/55.jpg)
s
Gt
original graph
0/9
0/7
0/5
0/2 0/4
0/1
xi xj
flow/residual capacity
Graph Re-parameterization
t residual graph
xi xj0/12
5/2
3/2
1/0
2/0 4/0st-mincut
ComputeMaxflow
Gr
Edges cut
![Page 56: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/56.jpg)
Update t-edge Capacities
s
Gr
t residual graph
xi xj0/12
5/2
3/2
1/0
2/0 4/0
![Page 57: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/57.jpg)
Update t-edge Capacities
s
Gr
t residual graph
xi xj0/12
5/2
3/2
1/0
2/0 4/0
capacitychanges from
7 to 4
![Page 58: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/58.jpg)
Update t-edge Capacities
s
G`t
updated residual graph
xi xj0/12
5/-1
3/2
1/0
2/0 4/0
capacitychanges from
7 to 4
edge capacityconstraint violated!(flow > capacity)
= 5 – 4 = 1
excess flow (e) = flow – new capacity
![Page 59: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/59.jpg)
add e to both t-edges connected to node i
Update t-edge Capacities
s
G`t
updated residual graph
xi xj0/12
3/2
1/0
2/0 4/0
capacitychanges from
7 to 4
edge capacityconstraint violated!(flow > capacity)
= 5 – 4 = 1
excess flow (e) = flow – new capacity
5/-1
![Page 60: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/60.jpg)
Update t-edge Capacities
s
G`t
updated residual graph
xi xj0/12
3/2
1/0
4/0
capacitychanges from
7 to 4
excess flow (e) = flow – new capacity
add e to both t-edges connected to node i
= 5 – 4 = 1
5/0
2/1
edge capacityconstraint violated!(flow > capacity)
![Page 61: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/61.jpg)
Update n-edge Capacities
s
Gr
t
residual graph
xi xj0/12
5/2
3/2
1/0
2/0 4/0
• Capacity changes from 5 to 2
![Page 62: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/62.jpg)
Update n-edge Capacities
s
t
Updated residual graph
xi xj0/12
5/2
3/-1
1/0
2/0 4/0
G`
• Capacity changes from 5 to 2- edge capacity constraint violated!
![Page 63: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/63.jpg)
Update n-edge Capacities
s
t
Updated residual graph
xi xj0/12
5/2
3/-1
1/0
2/0 4/0
G`
• Capacity changes from 5 to 2- edge capacity constraint violated!
• Reduce flow to satisfy constraint
![Page 64: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/64.jpg)
Update n-edge Capacities
s
t
Updated residual graph
xi xj0/11
5/2
2/0
1/0
2/0 4/0
excess
deficiency
G`
• Capacity changes from 5 to 2- edge capacity constraint violated!
• Reduce flow to satisfy constraint- causes flow imbalance!
![Page 65: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/65.jpg)
Update n-edge Capacities
s
t
Updated residual graph
xi xj0/11
5/2
2/0
1/0
2/0 4/0
deficiency
excess
G`
• Capacity changes from 5 to 2- edge capacity constraint violated!
• Reduce flow to satisfy constraint- causes flow imbalance!
• Push excess flow to/from the terminals
• Create capacity by adding α = excess to both t-edges.
![Page 66: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/66.jpg)
Update n-edge Capacities
Updated residual graph
• Capacity changes from 5 to 2- edge capacity constraint violated!
• Reduce flow to satisfy constraint- causes flow imbalance!
• Push excess flow to the terminals
• Create capacity by adding α = excess to both t-edges.
G`
xi xj0/11
5/3
2/0
2/0
3/0 4/1
s
t
![Page 67: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/67.jpg)
Update n-edge Capacities
Updated residual graph
• Capacity changes from 5 to 2- edge capacity constraint violated!
• Reduce flow to satisfy constraint- causes flow imbalance!
• Push excess flow to the terminals
• Create capacity by adding α = excess to both t-edges.
G`
xi xj0/11
5/3
2/0
2/0
3/0 4/1
s
t
![Page 68: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/68.jpg)
Complexity analysis of MRF Update Operations
MRF Energy Operation
Graph Operation Complexity
modifying a unary term
modifying a pair-wise term
adding a latent variable
delete a latent variable
Updating a t-edge capacity
Updating a n-edge capacity
adding a node
set the capacities of all edges of a node zero
O(1)
O(1)
O(1)
O(k)*
*requires k edge update operations where k is degree of the node
![Page 69: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/69.jpg)
• Finding augmenting paths is time consuming.
• Dual-tree maxflow algorithm [Boykov & Kolmogorov PAMI 2004]- Reuses search trees after each augmentation.- Empirically shown to be substantially faster.
• Our Idea • Reuse search trees from previous graph cut computation• Saves us search tree creation tree time [O(#edges)]• Search trees have to be modified to make them consistent with
new graphs• Constrain the search of augmenting paths
– New paths must contain at least one updated edge
Improving the Algorithm
![Page 70: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/70.jpg)
Reusing Search Trees
c’ = measure of change in the energy
• Running time– Dynamic algorithm (c’ + re-create search tree )– Improved dynamic algorithm (c’)– Video Segmentation Example
- Duplicate image frames (No time is needed)
![Page 71: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/71.jpg)
Dynamic Graph Cut vs Active Cuts
Our method flow recycling
AC cut recycling
Both methods: Tree recycling
![Page 72: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/72.jpg)
• Compared results with the best static algorithm.- Dual-tree algorithm [Boykov & Kolmogorov PAMI 2004]
• Applications- Interactive Image Segmentation- Image Segmentation in Videos
Experimental Analysis
![Page 73: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/73.jpg)
Experimental Analysis
additional segmentation
cues
user segmentation cuesstatic: 175 msecdynamic : 80 msecdynamic (optimized): 15 msec
static : 175 msec
Interactive Image segmentation (update unary terms)
EnergyMRF =
![Page 74: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/74.jpg)
Experimental Analysis
Image resolution: 720x576 static: 220 msec dynamic (optimized): 50 msec
Image segmentation in videos (unary & pairwise terms)
Graph CutsDynamic Graph Cuts
EnergyMRF =
![Page 75: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/75.jpg)
Experimental Analysis
Image resolution: 720x576 static: 177 msec dynamic (optimized): 60 msec
Image segmentation in videos (unary & pairwise terms)
Graph CutsDynamic Graph Cuts
EnergyMRF =
![Page 76: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/76.jpg)
Experimental Analysis
MRF consisting of 2x105 latent variables connected in a 4-neighborhood.
Running time of the dynamic algorithm
![Page 77: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/77.jpg)
Other uses
Can be used to compute uncertainty in graph cuts via max marginals.
Max marginals can be used for parameter learning in MRF’s.
![Page 78: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/78.jpg)
Inference in Graphical Models
Graphical Model
Topology
Tree Graph with cycles
Belief Propagation and variants
Exact solution
True Marginals/ min-marginals
Approximate solution
Approximate Marginals/ min-marginals
Graph Cuts
No Marginals/
Min-Marginals
Class 1: Max-flow Computation, Exact
Class 2: Alpha expansions, Approximate Solution with bounds
Class 3: Local Minima (with no bounds)
![Page 79: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/79.jpg)
Inference in Graphical Models
Min-Marginals Energies(ψ)- Minimize joint energy over all other variables.
- Related to max-marginals as:
- Can be used to compute confidence as:
σj = µj / Σa µa = exp(-ψi) / Σa exp(-ψa)
µj = (1/z)*exp(-ψj)
![Page 80: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/80.jpg)
Energy Projections and Graph Construction
EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 + Kā2
a1
a2
2
5
9
4
2
1
Sink (0)
Source (1)
∞
A high unary term (t-edge) can be used to constrain the solution of the energy to be the solution of the energy projection.
Alternative Construction
K
![Page 81: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/81.jpg)
Segmentation Comparison
Gri
mson
-G
rim
son
-S
tau
ffer
Sta
uff
er
Bath
ia0
Bath
ia0
44O
ur
Ou
r m
eth
od
meth
od
![Page 82: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/82.jpg)
Face Detector and ObjCut
![Page 83: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/83.jpg)
Segmentation
![Page 84: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/84.jpg)
Segmentation
![Page 85: Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d6c5503460f94a4c256/html5/thumbnails/85.jpg)
Conclusion
Combining pose inference and segmentation worth investigating.
Tracking = Detection Detection = Segmentation Tracking = Segmentation. Segmentation = SFM ??