csce 641 computer graphics: image-based modeling
DESCRIPTION
CSCE 641 Computer Graphics: Image-based Modeling. Jinxiang Chai. Outline. Stereo matching - Traditional stereo - Multi-baseline stereo - Active stereo Volumetric stereo - Visual hull - Voxel coloring - Space carving. Multi-baseline stereo reconstruction. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/1.jpg)
CSCE 641 Computer Graphics: Image-based Modeling
Jinxiang Chai
![Page 2: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/2.jpg)
Outline
Stereo matching - Traditional stereo
- Multi-baseline stereo
- Active stereo
Volumetric stereo - Visual hull
- Voxel coloring
- Space carving
![Page 3: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/3.jpg)
Multi-baseline stereo reconstruction
Need to choose reference frames!
Does not fully utilize pixel information across all images
![Page 4: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/4.jpg)
Volumetric stereo
Scene VolumeScene Volume
VV
Input ImagesInput Images
(Calibrated)(Calibrated)
Goal: Goal: Determine occupancy, “color” of points in VDetermine occupancy, “color” of points in V
![Page 5: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/5.jpg)
Discrete formulation: Voxel Coloring
Discretized Discretized
Scene VolumeScene Volume
Input ImagesInput Images
(Calibrated)(Calibrated)
Goal: Assign RGBA values to voxels in Vphoto-consistent with images
![Page 6: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/6.jpg)
Complexity and computability
Discretized Discretized
Scene VolumeScene Volume
N voxelsN voxels
C colorsC colors
33
All Scenes (CN3)Photo-Consistent
Scenes
TrueScene
![Page 7: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/7.jpg)
Issues
Theoretical Questions• Identify class of all photo-consistent scenes
Practical Questions• How do we compute photo-consistent models?
![Page 8: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/8.jpg)
1. C=2 (shape from silhouettes)• Volume intersection [Baumgart 1974]
> For more info: Rapid octree construction from image sequences. R. Szeliski, CVGIP: Image Understanding, 58(1):23-32, July 1993. (this paper is apparently not available online) or
> W. Matusik, C. Buehler, R. Raskar, L. McMillan, and S. J. Gortler, Image-Based Visual Hulls, SIGGRAPH 2000 ( pdf 1.6 MB )
2. C unconstrained, viewpoint constraints• Voxel coloring algorithm [Seitz & Dyer 97]
3. General Case• Space carving [Kutulakos & Seitz 98]
Voxel coloring solutions
![Page 9: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/9.jpg)
Why use silhouettes?
Can be computed robustly
Can be computed efficiently
- =
background background + +
foregroundforeground
backgroundbackground foreground foreground
![Page 10: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/10.jpg)
Reconstruction from silhouettes (C = 2)
Binary ImagesBinary Images
How to construct 3D volume?
![Page 11: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/11.jpg)
Reconstruction from silhouettes (C = 2)
Binary ImagesBinary Images
Approach: • Backproject each silhouette
• Intersect backprojected volumes
![Page 12: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/12.jpg)
Volume intersection
Reconstruction Contains the True Scene• But is generally not the same
• In the limit (all views) get visual hull > Complement of all lines that don’t intersect S
![Page 13: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/13.jpg)
Voxel algorithm for volume intersection
Color voxel black if on silhouette in every image• for M images, N3 voxels
• Don’t have to search 2N3 possible scenes!
O( ? ),
![Page 14: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/14.jpg)
Visual Hull Results
Download data and results from
http://www-cvr.ai.uiuc.edu/ponce_grp/data/visual_hull/
![Page 15: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/15.jpg)
Properties of volume intersection
Pros• Easy to implement, fast
• Accelerated via octrees [Szeliski 1993] or interval techniques [Matusik 2000]
Cons• No concavities
• Reconstruction is not photo-consistent
• Requires identification of silhouettes
![Page 16: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/16.jpg)
Voxel coloring solutions
1. C=2 (silhouettes)• Volume intersection [Baumgart 1974]
2. C unconstrained, viewpoint constraints• Voxel coloring algorithm [Seitz & Dyer 97]
> For more info: http://www.cs.washington.edu/homes/seitz/papers/ijcv99.pdf
3. General Case• Space carving [Kutulakos & Seitz 98]
![Page 17: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/17.jpg)
1. Choose voxel1. Choose voxel2. Project and correlate2. Project and correlate
3.3. Color if consistentColor if consistent(standard deviation of pixel
colors below threshold)
Voxel coloring approach
Visibility Problem: Visibility Problem: in which images is each voxel visible?in which images is each voxel visible?
![Page 18: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/18.jpg)
The visibility problem
Inverse Visibility?known images
Unknown SceneUnknown Scene
Which points are visible in which images?
Known SceneKnown Scene
Forward Visibility - Z-buffer
- Painter’s algorithm
known scene
![Page 19: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/19.jpg)
LayersLayers
Depth ordering: visit occluders first!
SceneScene
TraversalTraversal
Condition: Condition: depth order is the depth order is the same for all input viewssame for all input views
![Page 20: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/20.jpg)
Panoramic depth ordering
• Cameras oriented in many different directions
• Planar depth ordering does not apply
![Page 21: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/21.jpg)
Panoramic depth ordering
Layers radiate outwards from camerasLayers radiate outwards from cameras
![Page 22: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/22.jpg)
Panoramic layering
Layers radiate outwards from camerasLayers radiate outwards from cameras
![Page 23: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/23.jpg)
Panoramic layering
Layers radiate outwards from camerasLayers radiate outwards from cameras
![Page 24: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/24.jpg)
Compatible camera configurations
Depth-Order Constraint• Scene outside convex hull of camera centers
Outward-Lookingcameras inside scene
Inward-Lookingcameras above scene
![Page 25: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/25.jpg)
Calibrated image acquisition
Calibrated Turntable360° rotation (21 images)
Selected Dinosaur ImagesSelected Dinosaur Images
Selected Flower ImagesSelected Flower Images
![Page 26: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/26.jpg)
Voxel coloring results
Dinosaur ReconstructionDinosaur Reconstruction72 K voxels colored72 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI
Flower ReconstructionFlower Reconstruction70 K voxels colored70 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI
![Page 27: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/27.jpg)
Limitations of depth ordering
A view-independent depth order may not exist
p q
Need more powerful general-case algorithms• Unconstrained camera positions
• Unconstrained scene geometry/topology
![Page 28: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/28.jpg)
Voxel coloring solutions
1. C=2 (silhouettes)• Volume intersection [Baumgart 1974]
2. C unconstrained, viewpoint constraints• Voxel coloring algorithm [Seitz & Dyer 97]
3. General Case• Space carving [Kutulakos & Seitz 98]
> For more info: http://www.cs.washington.edu/homes/seitz/papers/kutu-ijcv00.pdf
![Page 29: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/29.jpg)
Space carving algorithm
Space Carving Algorithm
Image 1 Image N
…...
• Initialize to a volume V containing the true scene
• Repeat until convergence
• Choose a voxel on the current surface
• Carve if not photo-consistent
• Project to visible input images
![Page 30: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/30.jpg)
Which shape do you get?
The Photo Hull is the UNION of all photo-consistent scenes in V• It is a photo-consistent scene reconstruction
• Tightest possible bound on the true scene
True SceneTrue Scene
VV
Photo HullPhoto Hull
VV
![Page 31: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/31.jpg)
Space carving algorithm
The Basic Algorithm is Unwieldy• Complex update procedure
Alternative: Multi-Pass Plane Sweep• Efficient, can use texture-mapping hardware
• Converges quickly in practice
• Easy to implement
Results Algorithm
![Page 32: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/32.jpg)
Multi-pass plane sweep• Sweep plane in each of 6 principle directions• Consider cameras on only one side of plane• Repeat until convergence
True Scene Reconstruction
![Page 33: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/33.jpg)
Multi-pass plane sweep• Sweep plane in each of 6 principle directions• Consider cameras on only one side of plane• Repeat until convergence
![Page 34: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/34.jpg)
Multi-pass plane sweep• Sweep plane in each of 6 principle directions• Consider cameras on only one side of plane• Repeat until convergence
![Page 35: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/35.jpg)
Multi-pass plane sweep• Sweep plane in each of 6 principle directions• Consider cameras on only one side of plane• Repeat until convergence
![Page 36: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/36.jpg)
Multi-pass plane sweep• Sweep plane in each of 6 principle directions• Consider cameras on only one side of plane• Repeat until convergence
![Page 37: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/37.jpg)
Multi-pass plane sweep• Sweep plane in each of 6 principle directions• Consider cameras on only one side of plane• Repeat until convergence
![Page 38: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/38.jpg)
Space carving results: african violet
Input Image (1 of 45) Reconstruction
ReconstructionReconstruction
![Page 39: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/39.jpg)
Space carving results: hand
Input Image(1 of 100)
Views of Reconstruction
![Page 40: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/40.jpg)
Properties of space carving
Pros• Voxel coloring version is easy to implement, fast
• Photo-consistent results
• No smoothness prior
Cons• Bulging
• No smoothness prior
![Page 41: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/41.jpg)
Next lecture
Images user input range
scans
Model
Images
Image based modeling
Image-based renderingGeometry+ Images
Geometry+ Materials
Images + Depth
Light field
Panoroma
Kinematics
Dynamics
Etc.
Camera + geometry
![Page 42: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/42.jpg)
Stereo reconstruction
Given two or more images of the same scene or object, compute a representation of its shape
knownknowncameracamera
viewpointsviewpoints
![Page 43: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/43.jpg)
Stereo reconstruction
Given two or more images of the same scene or object, compute a representation of its shape
knownknowncameracamera
viewpointsviewpoints
How to estimate camera parameters?
- where is the camera?
- where is it pointing?
- what are internal parameters, e.g. focal length?
![Page 44: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/44.jpg)
Spectrum of IBMR
Images user input range
scans
Model
Images
Image based modeling
Image-based renderingGeometry+ Images
Geometry+ Materials
Images + Depth
Light field
Panoroma
Kinematics
Dynamics
Etc.
Camera + geometry
![Page 45: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/45.jpg)
How can we estimate the camera parameters?
![Page 46: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/46.jpg)
Camera calibration
Augmented pin-hole camera - focal point, orientation
- focal length, aspect ratio, center, lens distortion
Known 3DKnown 3D
Classical calibration - 3D 2D
- correspondence
Camera calibration online resources
![Page 47: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/47.jpg)
Camera and calibration target
![Page 48: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/48.jpg)
Classical camera calibration
Known 3D coordinates and 2D coordinates - known 3D points on calibration targets
- find corresponding 2D points in image using feature detection
algorithm
![Page 49: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/49.jpg)
Camera parameters
u0
v0
100-sy0
sx аuv1
Perspective proj. View trans.Viewport proj.
Known 3D coords and 2D coordsKnown 3D coords and 2D coords
![Page 50: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/50.jpg)
Camera parameters
u0
v0
100-sy0
sx аuv1
Perspective proj. View trans.Viewport proj.
Known 3D coords and 2D coordsKnown 3D coords and 2D coords
Intrinsic camera parameters (5 parameters)
extrinsic camera parameters (6 parameters)
![Page 51: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/51.jpg)
Camera matrix
Fold intrinsic calibration matrix K and extrinsic pose parameters (R,t) together into acamera matrix
M = K [R | t ]
(put 1 in lower r.h. corner for 11 d.o.f.)
![Page 52: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/52.jpg)
Camera matrix calibration
Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)
![Page 53: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/53.jpg)
Camera matrix calibration
Linear regression:• Bring denominator over, solve set of (over-determined) linear
equations. How?
![Page 54: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/54.jpg)
Camera matrix calibration
Linear regression:• Bring denominator over, solve set of (over-determined) linear
equations. How?
• Least squares (pseudo-inverse) - 11 unknowns (up to scale) - 2 equations per point (homogeneous coordinates) - 6 points are sufficient
![Page 55: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/55.jpg)
Nonlinear camera calibration
Perspective projection:
1100
0
1 3
2
1
3
2
1
0
0
i
i
i
T
T
T
y
x
i
i
z
y
x
t
t
t
r
r
r
vf
uf
v
u
![Page 56: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/56.jpg)
Nonlinear camera calibration
Perspective projection:
1100
0
1 3
2
1
3
2
1
0
0
i
i
i
T
T
T
y
x
i
i
z
y
x
t
t
t
r
r
r
vf
uf
v
u
K R T P
![Page 57: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/57.jpg)
Nonlinear camera calibration
Perspective projection:
2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters:
1100
0
1 3
2
1
3
2
1
0
0
i
i
i
T
T
T
y
x
i
i
z
y
x
t
t
t
r
r
r
vf
uf
v
u
K R T P
33
32302
33
30213021
)(
)(
tPr
ttfPrvrfv
tPr
tuttfPrurrfu
T
yTT
yi
Tx
TTTx
i
![Page 58: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/58.jpg)
Nonlinear camera calibration
Perspective projection:
2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters:
1100
0
1 3
2
1
3
2
1
0
0
i
i
i
T
T
T
y
x
i
i
z
y
x
t
t
t
r
r
r
vf
uf
v
u
33
32302
33
30213021
)(
)(
tPr
ttfPrvrfv
tPr
tuttfPrurrfu
T
yTT
yi
Tx
TTTx
i
K
);,,( iPTRKf
);,,( iPTRKg
R T P
![Page 59: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/59.jpg)
Multiple calibration images
Find camera parameters which satisfy the constraints from M images, N points: for j=1,…,M
for i=1,…,N
This can be formulated as a nonlinear optimization problem:
);,,(
);,,(
ijjji
ijjji
PTRKgv
PTRKfu
M
j
N
iijj
jiijj
ji PTRKgvPTRKfu
1 1
22 ));,,(());,,((
![Page 60: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/60.jpg)
Multiple calibration images
Find camera parameters which satisfy the constraints from M images, N points: for j=1,…,M for i=1,…,N
This can be formulated as a nonlinear optimization problem:
);,,(
);,,(
ijjji
ijjji
PTRKgv
PTRKfu
M
j
N
iijj
jiijj
ji PTRKgvPTRKfu
1 1
22 ));,,(());,,((
Solve the optimization using nonlinear optimization techniques:
- Gauss-newton
- Levenberg-Marquardt
![Page 61: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/61.jpg)
Nonlinear approach
Advantages:• can solve for more than one camera pose at a time
• fewer degrees of freedom than linear approach
• Standard technique in photogrammetry, computer vision, computer graphics
- [Tsai 87] also estimates lens distortions (freeware @ CMU)http://www.cs.cmu.edu/afs/cs/project/cil/ftp/html/v-source.html
Disadvantages:• more complex update rules
• need a good initialization (recover K [R | t] from M)
![Page 62: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/62.jpg)
How can we estimate the camera parameters?
![Page 63: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/63.jpg)
Application: camera calibration for sports video
[Farin et. Al]
images Court model
![Page 64: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/64.jpg)
Calibration from 2D motion
Structure from motion (SFM) - track points over a sequence of images
- estimate for 3D positions and camera positions
- calibrate intrinsic camera parameters before hand
Self-calibration: - solve for both intrinsic and extrinsic camera parameters
![Page 65: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/65.jpg)
SFM = Holy Grail of 3D Reconstruction
Take movie of object
Reconstruct 3D model
Would be
commercially
highly viable
![Page 66: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/66.jpg)
How to get feature correspondences
Feature-based approach
- good for images
- feature detection (corners)
- feature matching using RANSAC (epipolar line)
Pixel-based approach
- good for video sequences
- patch based registration with lucas-kanade algorithm
- register features across the entire sequence
![Page 67: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/67.jpg)
Structure from motion
Two Principal Solutions• Bundle adjustment (nonlinear optimization)
• Factorization (SVD, through orthographic approximation, affine geometry)
![Page 68: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/68.jpg)
Nonlinear approach for SFM
What’s the difference between camera calibration and SFM?
![Page 69: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/69.jpg)
Nonlinear approach for SFM
M
j
N
iijj
jiijj
ji
TRK
PTRKgvPTRKfujj
1 1
22
}{},{,
));,,(());,,((minarg
What’s the difference between camera calibration and SFM?
- camera calibration: known 3D and 2D
![Page 70: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/70.jpg)
Nonlinear approach for SFM
M
j
N
iijj
jiijj
ji
TRKP
PTRKgvPTRKfujji
1 1
22
}{},{,},{
)),,,(()),,,((minarg
M
j
N
iijj
jiijj
ji
TRK
PTRKgvPTRKfujj
1 1
22
}{},{,
));,,(());,,((minarg
What’s the difference between camera calibration and SFM?
- camera calibration: known 3D and 2D
- SFM: unknown 3D and known 2D
![Page 71: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/71.jpg)
Nonlinear approach for SFM
M
j
N
iijj
jiijj
ji
TRKP
PTRKgvPTRKfujji
1 1
22
}{},{,},{
)),,,(()),,,((minarg
M
j
N
iijj
jiijj
ji
TRK
PTRKgvPTRKfujj
1 1
22
}{},{,
));,,(());,,((minarg
What’s the difference between camera calibration and SFM?
- camera calibration: known 3D and 2D
- SFM: unknown 3D and known 2D
- what’s 3D-to-2D registration problem?
![Page 72: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/72.jpg)
Nonlinear approach for SFM
M
j
N
iijj
jiijj
ji
TRKP
PTRKgvPTRKfujji
1 1
22
}{},{,},{
)),,,(()),,,((minarg
M
j
N
iijj
jiijj
ji
TRK
PTRKgvPTRKfujj
1 1
22
}{},{,
));,,(());,,((minarg
What’s the difference between camera calibration and SFM?
- camera calibration: known 3D and 2D
- SFM: unknown 3D and known 2D
- what’s 3D-to-2D registration problem?
![Page 73: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/73.jpg)
SFM: bundle adjustment
SFM = Nonlinear Least Squares problem
Minimize through• Gradient Descent
• Conjugate Gradient
• Gauss-Newton
• Levenberg Marquardt common method
Prone to local minima
M
j
N
iijj
jiijj
ji
TRKP
PTRKgvPTRKfujji
1 1
22
}{},{,},{
)),,,(()),,,((minarg
![Page 74: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/74.jpg)
Count # Constraints vs #Unknowns
M camera poses
N points
2MN point constraints
6M+3N unknowns
Suggests: need 2mn 6m + 3n
But: Can we really recover all parameters???
M
j
N
iijj
jiijj
ji
TRKP
PTRKgvPTRKfujji
1 1
22
}{},{,},{
)),,,(()),,,((minarg
![Page 75: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/75.jpg)
Count # Constraints vs #Unknowns
M camera poses
N points
2MN point constraints
6M+3N unknowns (known intrinsic camera parameters)
Suggests: need 2mn 6m + 3n
But: Can we really recover all parameters???• Can’t recover origin, orientation (6 params)
• Can’t recover scale (1 param)
Thus, we need 2mn 6m + 3n - 7
M
j
N
iijj
jiijj
ji
TRKP
PTRKgvPTRKfujji
1 1
22
}{},{,},{
)),,,(()),,,((minarg
![Page 76: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/76.jpg)
Are we done?
No, bundle adjustment has many local minima.
![Page 77: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/77.jpg)
SFM using Factorization
12
1
2
1
i
i
i
T
T
i
i
z
y
x
t
t
r
r
v
u
Assume an othorgraphic camera
Image World
![Page 78: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/78.jpg)
SFM using Factorization
12
1
2
1
i
i
i
T
T
i
i
z
y
x
t
t
r
r
v
u
Assume othorgraphic camera
Image World
i
i
i
T
T
N
ii
i
N
ii
i
z
y
x
r
r
N
vv
N
uu
2
1
1
1
Subtract the mean
![Page 79: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/79.jpg)
SFM using Factorization
N
N
N
T
T
N
N
z
y
x
z
y
x
z
y
x
r
r
v
u
v
u
v
u
...
...
...
~
~
...
...~
~
~
~
2
2
2
1
1
1
2
1
2
2
1
1
Stack all the features from the same frame:
![Page 80: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/80.jpg)
SFM using Factorization
N
N
N
T
T
N
N
z
y
x
z
y
x
z
y
x
r
r
v
u
v
u
v
u
...
...
...
~
~
...
...~
~
~
~
2
2
2
1
1
1
2
1
2
2
1
1
N
N
N
TF
TF
T
T
NF
NF
F
F
F
F
NF
NF
F
F
F
F
z
y
x
z
y
x
z
y
x
r
r
r
r
v
u
v
u
v
u
v
u
v
u
v
u
...
...
...
~
~
...
...~
~
~
~
~
~
...
...~
~
~
~
2
2
2
1
1
1
2,
1,
2,1
1,1
,
,
2,
2,
1,
1,
,
,
2,
2,
1,
1,
Stack all the features from the same frame:
Stack all the features from all the images:
W
![Page 81: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/81.jpg)
SFM using Factorization
N
N
N
T
T
N
N
z
y
x
z
y
x
z
y
x
r
r
v
u
v
u
v
u
...
...
...
~
~
...
...~
~
~
~
2
2
2
1
1
1
2
1
2
2
1
1
N
N
N
TF
TF
T
T
NF
NF
F
F
F
F
NF
NF
F
F
F
F
z
y
x
z
y
x
z
y
x
r
r
r
r
v
u
v
u
v
u
v
u
v
u
v
u
...
...
...
~
~
...
...~
~
~
~
~
~
...
...~
~
~
~
2
2
2
1
1
1
2,
1,
2,1
1,1
,
,
2,
2,
1,
1,
,
,
2,
2,
1,
1,
NFW 2
~
Stack all the features from the same frame:
Stack all the features from all the images:
W
32 FM NS 3
![Page 82: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/82.jpg)
SFM using Factorization
N
N
N
TF
TF
T
T
NF
NF
F
F
F
F
NF
NF
F
F
F
F
z
y
x
z
y
x
z
y
x
r
r
r
r
v
u
v
u
v
u
v
u
v
u
v
u
...
...
...
~
~
...
...~
~
~
~
~
~
...
...~
~
~
~
2
2
2
1
1
1
2,
1,
2,1
1,1
,
,
2,
2,
1,
1,
,
,
2,
2,
1,
1,
NFW 2
~32 FM
Stack all the features from all the images:
W
NS 3
Factorize the matrix into two matrix using SVD:
NFW 2
~
TNF
TNF VSUMVUW 2
1
32
1
322
~~~
![Page 83: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/83.jpg)
SFM using Factorization
N
N
N
TF
TF
T
T
NF
NF
F
F
F
F
NF
NF
F
F
F
F
z
y
x
z
y
x
z
y
x
r
r
r
r
v
u
v
u
v
u
v
u
v
u
v
u
...
...
...
~
~
...
...~
~
~
~
~
~
...
...~
~
~
~
2
2
2
1
1
1
2,
1,
2,1
1,1
,
,
2,
2,
1,
1,
,
,
2,
2,
1,
1,
NFW 2
~32 FM
Stack all the features from all the images:
NS 3
Factorize the matrix into two matrix using SVD:
NFW 2
~
TNF
TNF VSUMVUW 2
1
32
1
322
~~~
NNFF SQSQMM
31
333333232
~~
![Page 84: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/84.jpg)
SFM using Factorization
N
N
N
TF
TF
T
T
NF
NF
F
F
F
F
NF
NF
F
F
F
F
z
y
x
z
y
x
z
y
x
r
r
r
r
v
u
v
u
v
u
v
u
v
u
v
u
...
...
...
~
~
...
...~
~
~
~
~
~
...
...~
~
~
~
2
2
2
1
1
1
2,
1,
2,1
1,1
,
,
2,
2,
1,
1,
,
,
2,
2,
1,
1,
NFW 2
~32 FM
Stack all the features from all the images:
W
NS 3
Factorize the matrix into two matrix using SVD:
NFW 2
~
TNF
TNF VSUMVUW 2
1
32
1
322
~~~
NNFF SQSQMM
31
333333232
~~
How to compute the matrix ? 33Q
![Page 85: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/85.jpg)
SFM using Factorization
2,2,2,11,1
2,
1,
2,1
1,1
3232 FF
TF
TF
T
T
TFF rrrr
r
r
r
r
MM
M is the stack of rotation matrix:
2,2,
2,1,
1,2,
1,1,
2,12,1
2,11,1
1,12,1
1,11,1
FTF
FTF
FTF
FTF
T
T
T
T
rr
rr
rr
rr
rr
rr
rr
rr
![Page 86: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/86.jpg)
SFM using Factorization
2,2,2,11,1
2,
1,
2,1
1,1
3232 FF
TF
TF
T
T
TFF rrrr
r
r
r
r
MM
M is the stack of rotation matrix:
2,2,
2,1,
1,2,
1,1,
2,12,1
2,11,1
1,12,1
1,11,1
FTF
FTF
FTF
FTF
T
T
T
T
rr
rr
rr
rr
rr
rr
rr
rr
1 010
1 010
Orthogonal constraints from rotation matrix
![Page 87: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/87.jpg)
SFM using Factorization
2,2,2,11,1
2,
1,
2,1
1,1
3232 FF
TF
TF
T
T
TFF rrrr
r
r
r
r
MM
2,2,
2,1,
1,2,
1,1,
2,12,1
2,11,1
1,12,1
1,11,1
FTF
FTF
FTF
FTF
T
T
T
T
rr
rr
rr
rr
rr
rr
rr
rr
M is the stack of rotation matrix:
1 010
1 010
Orthogonal constraints from rotation matrix
TF
TF MQQM 32333332
~~
![Page 88: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/88.jpg)
SFM using Factorization
TF
TF MQQM 32333332
~~
2,2,
2,1,
1,2,
1,1,
2,12,1
2,11,1
1,12,1
1,11,1
FTF
FTF
FTF
FTF
T
T
T
T
rr
rr
rr
rr
rr
rr
rr
rr
1 010
1 010
Orthogonal constraints from rotation matrices:
![Page 89: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/89.jpg)
SFM using Factorization
TF
TF MQQM 32333332
~~
2,2,
2,1,
1,2,
1,1,
2,12,1
2,11,1
1,12,1
1,11,1
FTF
FTF
FTF
FTF
T
T
T
T
rr
rr
rr
rr
rr
rr
rr
rr
1 010
1 010
Orthogonal constraints from rotation matrices:
QQ: symmetric 3 by 3 matrix
![Page 90: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/90.jpg)
SFM using Factorization
TF
TF MQQM 32333332
~~
2,2,
2,1,
1,2,
1,1,
2,12,1
2,11,1
1,12,1
1,11,1
FTF
FTF
FTF
FTF
T
T
T
T
rr
rr
rr
rr
rr
rr
rr
rr
1 010
1 010
Orthogonal constraints from rotation matrices:
How to compute QQ?
least square solution
- 4F linear constraints, 9 unknowns (6 independent due to symmetric matrix)
QQ: symmetric 3 by 3 matrix
![Page 91: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/91.jpg)
SFM using Factorization
TF
TF MQQM 32333332
~~
2,2,
2,1,
1,2,
1,1,
2,12,1
2,11,1
1,12,1
1,11,1
FTF
FTF
FTF
FTF
T
T
T
T
rr
rr
rr
rr
rr
rr
rr
rr
1 010
1 010
Orthogonal constraints from rotation matrices:
How to compute QQ?
least square solution
- 4F linear constraints, 9 unknowns (6 independent due to symmetric matrix) How to compute Q from QQ:
SVD again: 2
1
UQVUQQ T
QQ: symmetric 3 by 3 matrix
![Page 92: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/92.jpg)
SFM using Factorization
2,2,2,11,1
2,
1,
2,1
1,1
3232 FF
TF
TF
T
T
TFF rrrr
r
r
r
r
MM
2,2,
2,1,
1,2,
1,1,
2,12,1
2,11,1
1,12,1
1,11,1
FTF
FTF
FTF
FTF
T
T
T
T
rr
rr
rr
rr
rr
rr
rr
rr
M is the stack of rotation matrix:
1 010
1 010
Orthogonal constraints from rotation matrix
TF
TF MQQM 32333332
~~
QQ: symmetric 3 by 3 matrix
Computing QQ is easy:
- 3F linear equations
- 6 independent unknowns
![Page 93: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/93.jpg)
SFM using factorization
1. Form the measurement matrix
2. Decompose the matrix into two matrices and using SVD
3. Compute the matrix Q with least square and SVD
4. Compute the rotation matrix and shape matrix:
and
NFW 2
~
NS 3
~ 32
~FM
QMM F 32
~ 32
1 ~
FSQS
![Page 94: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/94.jpg)
Weak-perspective Projection
Factorization also works for weak-perspective projection (scaled orthographic projection):
d z0
12
1
2
1
i
i
i
T
T
i
i
z
y
x
t
t
r
r
v
u
![Page 95: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/95.jpg)
SFM using factorization
Bundle adjustment (nonlinear optimization) - work with perspective camera model - work with incomplete data - prone to local minima
Factorization: - closed-form solution for weak perspective camera - simple and efficient - usually need complete data - becomes complicated for full-perspective camera model
Phil Torr’s structure from motion toolkit in matlab (click here)
Voodoo camera tracker (click here)
![Page 96: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/96.jpg)
Results from SFM
[Han and Kanade]
![Page 97: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/97.jpg)
All together video
Click here
- feature detection
- feature matching (epipolar geometry)
- structure from motion
- stereo reconstruction
- triangulation
- texture mapping
![Page 98: CSCE 641 Computer Graphics: Image-based Modeling](https://reader036.vdocuments.us/reader036/viewer/2022062322/568143f1550346895db07984/html5/thumbnails/98.jpg)
Challenging objects
Why are they so difficult to model?
How to model them?