multi-hypothesis projection-based shift estimation for sweeping panorama reconstruction
Post on 04-Jul-2015
340 Views
Preview:
DESCRIPTION
TRANSCRIPT
C A N O N C O N F I D E N T I A L
Multi‐hypothesis projection‐based shift estimation for sweeping panorama reconstruction
Tuan Pham and Philip CoxCanon Information Systems Research Australia
Paper 366 Session OT4 10:50PM-11:10AM on Tue 10 July 2012
Overview
2
1. How correlation can estimate shift incorrectly
2. Multi‐hypothesis projection‐based shift estimation
3. Sweep panorama application
Shift estimation
3
Why do it?
Common techniquesFeature‐based: faster but less accurate
Point matching e.g., Harris corners, SIFT
Area‐based: slower but very accurateGradient‐based e.g., Lucas‐KanadeCorrelation can be applied to 1D image projections for speed
1. Correlation‐based shift estimation: good case
4
Correlation works for small shift (e.g. 81/512 pixels = 15% image width)
correlation peak at offset [81 ‐1] aligned images (superimposed on R and G channels)
Frame 1 (512x340) Frame 8
1. Correlation‐based shift estimation: bad case
5
Correlation fails for shift as small as 25% of image width*
correlation peak at offset [‐191 ‐164] misaligned images. Correct shift is [123 ‐6]
Frame 1 (512x340) Frame 11
* Phase correlation can handle shifts up to 50% of image width
1. Correlation‐based shift estimation: key finding
6
True shift may correspond to a local correlation peak instead
correlation peak at offset [‐191 ‐164]
Frame 1 (512x340) Frame 11True shift appears as a local maximum
stitched image using the correct shift of [123 ‐6]
2. Multi‐hypothesis projection‐based shift estimation
7
2. How testing multiple shift hypotheses helps
8
Various non‐ideal factors corrupt the correlation surface, reducing the global peak of a true shift to a local peak. Robust against large motion in x or y (up to 90% image width):
Nonoverlapping area corrupts correlation surface, causing false peaks
Robust against small rotation (up to 9° under no translation):Rotation also corrupts correlation, reducing strength of the true peak
0 50 100 150 200 250 300 350 400 450 500-500
0
5002D correlation
txty
ground-truth shift tx
Estimated shift by 2D correlation
Simulated shift tx
0 50 100 150 200 250 300 350 400 450 500-500
0
500
ground-truth shift tx
multi-hypothesis projection-based alignment (this paper)Estimated shift by this paper*
Simulated shift tx
-10 -8 -6 -4 -2 0 2 4 6 8 10-20
0
202D correlation
txty
-10 -8 -6 -4 -2 0 2 4 6 8 10-20
0
20
Simulated rotation angle (degrees) Simulated rotation angle (degrees)
Estimated shift by 2D correlation Estimated shift by this paper*
* Experiments performed on the 512x340 images shown earlier
2. Our algorithm is also fast: 20 fps for VGA video under Matlab
9
Competitive speed compared to viewfinder alignment* due to:image subsampling to roughly 256x256 pixels prior to alignmentprojections to 2 directions (x‐ and y‐) instead of 4 (0°,45°,90°,135°)don’t need corner matching step to be robust against small rotation
* Adams, et al., Viewfinder alignment, Eurographics 2008.
-2 0 2 4 6 8 10 12 14 160
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
image size (Mega-Pixels)
alig
nmen
t tim
e in
Mat
lab
(sec
onds
)
Shift estimation run-time versus image size
2D correlation: y=0.132*x+0.008 Adams et.al.: y=0.007*x+0.132 this paper: y=0.010*x+0.023
3. Application: panorama capture from camera sweeping motion
10
frame 1 frame 11 frame 21
The alignment algorithm in this paper is robust against perspective change. The irregular seam stitching* further helps hiding these perspective change artefacts.
* Avidan et al., Seam carving for content‐aware image resizing, SIGGRAPH, 2007.
3. Good alignment is crucial in panorama creation
11
Robust against motion blur:
panorama (3456×704) reconstructed from 12 images (1024×680) under camera motion blur
180° panorama (4000×704) of a busy shopping centre from 9 images (972×648)
Robust against moving objects and large perspective change:
Summary
12
Correlation peak due to image translation may be reduced to a local peak
We proposed testing multiple correlation peaks for a solution
Our multi‐hypothesis projection‐based shift estimation is fast and robust against large translation and some rotation
Our shift estimation technique can be used for panorama stitching under variety of real‐life challenging conditions
13
Thank you
tuan.pham@cisra.canon.com.au
2. Multi‐hypothesis projection‐based shift estimation: the details
14
To improve efficiency:1. Input images are subsampled to roughly 256x256 pixels2. Absolute gradient projection, e.g., To improve robustness to large translation, for each dimension:3. Find k shift hypotheses from k strongest correlation peaks (k=5)4. Shift hypothesis with dominant correlation score is the final shift 5. Crop images by the dominant shift to improve overlap6. Test a full set of k2 2D shift hypotheses in case of no dominant shift
top related