24/01/2014 - enseeihtgurdjos.perso.enseeiht.fr/vision24janvier2014/bartoli-24jan-tlse.pdfalcov-isit,...
TRANSCRIPT
24/01/2014
1
Shape-from-Template
Adrien BartoliALCoV-ISIT, Clermont-Ferrand
Florent Brunet, Toby Collins, Vincent Gay-Bellile, Mathieu Perriollat, Daniel Pizarro, Richard Hartley
Computer Vision Mini Workshop
Toulouse, January 24, 2014
3D Reconstruction: to Recover Shape from Images
Active methods Passive methods
Image from [Crandall et al, CVPR’11]Using Shape-from-Motion
Image by Visnjic, 2010Using Kinect
Escaping Criticism by del Caso, 1874
Model Town by Matt West, 2006
Blur ShadingMotion OcclusionsStereoscopy Texture
[Gibson ~ 1960 ; Marr ~ 1970]
Passive Single-View Reconstruction: Visual Cues
Blur ShadingMotion OcclusionsStereoscopy
Unapplicableto single-view
Very weak hereUnapplicableto single-view
Weak in general
Very weak here Very weak here
Texture
+ database
[Hoeim et al, SIGGRAPH’05]
Passive Single-View Reconstruction: Shape Priors
Low dimensional shape space
3D Morphable Face Model[Blanz and Vetter, PAMI’03]
High dimensional shape space…
… but simple deformation model!
Ob
ject
leve
l
High dimensional shape space
Scen
ele
vel
Faces Newspapers
Detection Recognition
Shape-from-Template
ShapeImage
Shape-from-Template
MaterialShapeAppearance
Template
Scope: object-specific passive single-view reconstruction using simple physics-based deformation from a known reference shape and a matchable appearance
[Gumerov et al, ECCV’04 ; Salzmann et al, BMVC’05 ; Perriollat et al, BMVC’08]
Detection
Shape-from-Template
Template not found
Template not found
Shape-from-Template
Shape-from-Template
Shape-from-Template
Shape-from-Template
Shape-from-Template
Template not found
ShapeAppearance
Template
Material
24/01/2014
2
Shape-from-Template
Template not found
Template not found
Template not found
Shape-from-Template
ShapeAppearance
Template
Material
Template not found
Augmented Reality for Deformable Surfaces
Differential Geometric Setup
3D
2D
𝑢𝑣-map - Ω ⊂ ℝ2 Image
Shape
Cameraprojection Π
𝜑 ∈ 𝐶2 Ω,ℝ3
Π ∈ 𝐶∞ ℝ3, ℝ2
ShapeAppearance
Template
Material
∘ Π ∘ 𝜑=⇒
⇒ 𝜑 =
Image warp 𝜂
𝜂
Two-Step Approach
3D
2D
𝑢𝑣-map - Ω ⊂ ℝ2 Image
Image warp 𝜂
Cameraprojection Π
Shape
1 – Image registration
2 – Shape inference
Differential Problem Statement
ShapeAppearance
Template
Material
Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3
Projection Π ∈ 𝐶∞ ℝ3, ℝ2Find s.t.
𝜑 =
∘ Π ∘ 𝜑=
Shape-from-Template
Problem Relaxation
ShapeAppearance
Template
Material
Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3
Projection Π ∈ 𝐶∞ ℝ3, ℝ2Find s.t.
𝜑 =
∘ Π ∘ 𝜑=
Shape-from-Template
𝜼 ∈ ? ?𝜼
𝑇𝑉2𝐿2 𝜂 → min
24/01/2014
3
Image Registration
ShapeAppearance
Template
Material
Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3
Projection Π ∈ 𝐶∞ ℝ3, ℝ2Find s.t.
𝜑 =
∘ Π ∘ 𝜑=
Shape-from-Template
𝜼 ∈ ? ?𝜼
𝑇𝑉2𝐿2 𝜂 → min
Find s.t. = ∘ 𝜂Warp 𝜂 ∈ 𝐶2 Ω,ℝ2
1 – Image registration
Factoring the Warp into Embedding and Projection
ShapeAppearance
Template
Material
Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3
Projection Π ∈ 𝐶∞ ℝ3, ℝ2Find s.t.
𝜑 =
∘ Π ∘ 𝜑=
Shape-from-Template 𝜼 = 𝚷 ∘ 𝝋
Shape Inference
ShapeAppearance
Template
Material
Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3
Projection Π ∈ 𝐶∞ ℝ3, ℝ2Find s.t.
𝜑 =
∘ Π ∘ 𝜑=
Shape-from-Template 𝜼 = 𝚷 ∘ 𝝋
Find s.t.
2 – Shape inference
Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3
Projection Π ∈ 𝐶∞ ℝ3, ℝ2𝜑 =
𝜂 = Π ∘ 𝜑
Shape-from-Template
Two-Step Approach
Find s.t.
2 – Shape inference
Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3
Projection Π ∈ 𝐶∞ ℝ3, ℝ2𝜑 =
𝜂 = Π ∘ 𝜑
Template
Find s.t. = ∘ 𝜂Warp 𝜂 ∈ 𝐶2 Ω,ℝ2
1 – Image registration
Appearance Shape Material
Step 1: Image Registration, in a Nutshell
Find s.t. = ∘ 𝜂Warp 𝜂 ∈ 𝐶2 Ω,ℝ2
1 – Image registration
Putative keypoint matches [Lowe, IJCV’04]
Densification[Bookstein, PAMI’89]
Template not found
Local consistency [Pizarro et al, IJCV’12]
1
2
[Schmid et al, PAMI’97]
Step 1: Image Registration, in a Nutshell
Local consistency [Pizarro et al, IJCV’12]
Partial self-occlusion detection[Pizarro et al, IJCV’12]
Densification [Bookstein, PAMI’89]
Self-occlusion aware densification[Pizarro et al, IJCV’12]
Color-based refinement[Gay-Bellile et al, PAMI’10]
2
3
4
5
6
24/01/2014
4
Step 1: Image Registration, Results Step 1: Image Registration, Results
Shape-from-Template
Two-Step Approach
Find s.t.
2 – Shape inference
Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3
Projection Π ∈ 𝐶∞ ℝ3, ℝ2𝜑 =
𝜂 = Π ∘ 𝜑
Template
Find s.t. = ∘ 𝜂Warp 𝜂 ∈ 𝐶2 Ω,ℝ2
1 – Image registration
Appearance Shape Material
Step 2: Shape Inference
3D
2D
𝑢𝑣-map - Ω ⊂ ℝ2 Image
Cameraprojection Π
Image warp 𝜂
Find s.t.
2 – Shape inference
Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3
Projection Π ∈ 𝐶∞ ℝ3, ℝ2𝜑 =
𝜂 = Π ∘ 𝜑
Calibratedpinhole
Zeroth Order Methods
3D
2D
𝑢𝑣-map - Ω ⊂ ℝ2 Image
Cameraprojection Π
Image warp 𝜂
Find s.t.
2 – Shape inference
Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3
Projection Π ∈ 𝐶∞ ℝ3, ℝ2𝜑 =
𝜂 = Π ∘ 𝜑
𝛻𝜑⊤𝛻𝜑 = I
Main result: shape-wise solution of a convex relaxation[Perriollat et al, IJCV’11; Salzmann et al, PAMI’11; Brunet et al, ACCV’10; Östlund et al, ECCV’12]
Zeroth orderreprojection
[Perriollat et al, IJCV’11; Salzmann et al, PAMI’11]
First Order Methods
3D
2D
𝑢𝑣-map - Ω ⊂ ℝ2 Image
Cameraprojection Π
Image warp 𝜂
Find s.t.
2 – Shape inference
Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3
Projection Π ∈ 𝐶∞ ℝ3, ℝ2𝜑 =
𝜂 = Π ∘ 𝜑
𝛻𝜑⊤𝛻𝜑 = I
Zeroth orderreprojection
𝛻𝜂 = 𝛻Π ∘ 𝜑 𝛻𝜑
First orderreprojection
[Bartoli et al, CVPR’12]
24/01/2014
5
First Order Methods
Find s.t.
2 – Shape inference
Embedding 𝜑 ∈ 𝐶2 Ω,ℝ3
Projection Π ∈ 𝐶∞ ℝ3, ℝ2𝜑 =
𝜂 = Π ∘ 𝜑
𝛻𝜑⊤𝛻𝜑 = I
Zeroth orderreprojection
𝛻𝜂 = 𝛻Π ∘ 𝜑 𝛻𝜑
First orderreprojection
𝜂 22𝛻𝛾⊤𝛻𝛾 + 𝛾2𝛻𝜂⊤𝛻𝜂 + 𝛾 𝛻𝛾⊤𝜂⊤𝛻𝜂 + 𝛻𝜂⊤𝜂𝛻𝛾 = I
Let 𝛾 ∈ 𝐶1 Ω, ℝ be the depth function
Shape inference is this first-order quadratic PDE in 𝜸
Main result: exact point-wise solution
[Bartoli et al, CVPR’12]
Intuition: neglect the dependency between 𝛾 and 𝛻𝛾
First Order Methods
𝜂 22𝛻𝛾⊤𝛻𝛾 + 𝛾2𝛻𝜂⊤𝛻𝜂 + 𝛾 𝛻𝛾⊤𝜂⊤𝛻𝜂 + 𝛻𝜂⊤𝜂𝛻𝛾 = I
Isometric developable
Conformal
𝜂 22𝛻𝛾⊤𝛻𝛾 + 𝛾2𝛻𝜂⊤𝛻𝜂 + 𝛾 𝛻𝛾⊤𝜂⊤𝛻𝜂 + 𝛻𝜂⊤𝜂𝛻𝛾 = 𝜈𝛻Δ⊤𝛻Δ
𝜂 22𝛻𝛾⊤𝛻𝛾 + 𝛾2𝛻𝜂⊤𝛻𝜂 + 𝛾 𝛻𝛾⊤𝜂⊤𝛻𝜂 + 𝛻𝜂⊤𝜂𝛻𝛾 = 𝛻Δ⊤𝛻Δ
Isometric non-developable object
Isometric (infinitesimal) weak-perspective
𝛻𝛾 22 + 𝛾2 𝛻𝜂𝛻𝜂⊤ + 𝛼2𝛻𝛾⊤𝛻𝛾 = 𝜈𝛻Δ⊤𝛻Δ
Isometric, unknown focal length
𝑓2 𝛻𝛾 22 + 𝛾2 𝛻𝜂𝛻𝜂⊤ + 𝛼2𝛻𝛾⊤𝛻𝛾 = 𝜈𝛻Δ⊤𝛻Δ
‘true’ 𝑓 = 2040 pixels
3.8% relative error
Estimated 𝑓 = 2118 pixels
𝑓 = 5870 pixels
Non-flattenable Objects
3D
2D
Image warp 𝜂
Cameraprojection Π
Deformation Ψ
Conformal flattening Δ−1
𝑢𝑣-map - Ω ⊂ ℝ2 Image
Δ ∈ 𝐶2 Ω,ℝ3
Ψ = 𝜑 ∘ Δ−1
Step 2: Shape Inference, Results
Ground-truth(Rigid Shape-from-Motion)
Shape-from-Template
Step 2: Shape Inference, Results Step 2: Shape Inference, Results
24/01/2014
6
Some Extensions
ShapeAppearance
Template 1
Material ShapeAppearance
Template 2
Material
Multiobject shape-from-template [Alcantarilla et al, BMVC’12]
etc.
Linear elastic deformations [Malti et al, CVPR’13]
template image shape
Isowarp and Conwarp [Pizarro et al, BMVC’13]
Differential constraints on 𝜂 so that 𝜂 = Π ∘ 𝜑
Main Surface Types
Physical flattening Virtual flattening
Iso
met
ric
def
orm
atio
nEl
asti
cd
efo
rmat
ion
Gynecologic Surgery
Procedures• Benign conditions• Cancer• Infertility• Incontinence
Organs of the female reproductive system
Located in the pelvic cavity
Scope
Abdominal (open, laparotomy)
Hysteroscopic(through vagina)
Laparoscopic(small incisions)
Types
Preoperative Imaging
MRI US
• Diagnosis• Procedure• Type of surgery
Uterine Fibroids or Myomas
• Microscopic to extremely large size• Often several of them
Benign tumors from the myometrium
May be invisible in laparoscopy(and hysteroscopy)
Intramural myomas (type b)
Clearly visible in MRI
Laparoscopic Augmented Reality
3 Displaying
Augmenting
Augmentation data
2
Registration
1 Filming
24/01/2014
7
Registration is Challenging
Blur Occlusions
Bleeding Smoke
Augmentation data
Preoperative MRI Preparation
Axial Sagittal Coronal
Uterus surface First fibroid Second fibroid
Augmented Reality Framework
1 Registration
2 Augmentation
Current frame
Current frame
Registration Φ𝑖 :ℝ3 → ℝ2
Requirements:R1 – deformable R2 – multimodalR3 – realtime R4 – automatic
Two-Step Registration
Registration Φ𝑖 :ℝ3 → ℝ2
1 Registration
Current frame
Reference frame
1b
Reference shape1a
Requirements: (R1), R3, R4
Requirements:R1 – deformable R2 – multimodalR3 – realtime R4 – automatic
Relax requirements with Φ𝑖 = Γ𝑖 ∘ Γ0
Preoperative to intraoperative reference
Intraoperativereference to current frame
WBMTR (Wide-Baseline Multi-Texturemap Registration)[Collins et al, MIAR@MICCAI’13]
Registration Φ𝑖 :ℝ3 → ℝ2
1 Registration
Reference frame
Requirements:R1 – deformable R2 – multimodalR3 – realtime R4 – automatic
Relax requirements with Φ𝑖 = Γ𝑖 ∘ Γ0
Current frame
Shape-from-Motion
Pose with keypoints from best reference frame
Reference frames
Reference shape
WBMTR Registration Results
24/01/2014
8
WBMTR Registration Results Generalizing Rigid Pose to Deformations
Registration Φ𝑖 :ℝ3 → ℝ2
1 Registration
Current frame
Reference shape
Requirements:R1 – deformable R2 – multimodalR3 – realtime R4 – automatic
Relax requirements with Φ𝑖 = Γ𝑖 ∘ Γ0
Ψ𝑖 :ℝ3 → ℝ3
Π:ℝ3 → ℝ2
Deformation
Projection
Shape-from-TemplateTo recover Ψ𝑖 (and Π)
Uterine Shape-from-Template Results
First order, isometricZeroth order, isometric
[Salzmann et al, PAMI’09]First order, conformal
Shape-from-Template
Adrien BartoliALCoV-ISIT, Clermont-Ferrand
Florent Brunet, Toby Collins, Vincent Gay-Bellile, Mathieu Perriollat, Daniel Pizarro, Richard Hartley
Computer Vision Mini Workshop
Toulouse, January 24, 2014