change detection of 3d scene with 3d and 2d information for environment checking
Post on 01-Jul-2015
434 Views
Preview:
DESCRIPTION
TRANSCRIPT
1
Change Detection of 3D Scene with 3D and 2D Information for
Environment Checking
PhD Candidate: Baowei Lin August 12th, 2013
1. Introduction Research Motivation Change Detection
2. 3D Keypoints Based 3D-2D Matching Background 3D Keypoints Detection Evaluation
3. 3D-2D Based Change Detection Background Image Based Change Detection Evaluation
4. 3D-3D Based Change Detection Background Scale Estimation of a Single Point Cloud Scale Ratio Estimation of Two Point Clouds Evaluation
5. Conclusions
2
1. Introduction Research Motivation Change Detection
2. 3D Keypoints Based 3D-2D Matching Background 3D Keypoints Detection Evaluation
3. 3D-2D Based Change Detection Background Image Based Change Detection Evaluation
4. 3D-3D Based Change Detection Background Scale Estimation of a Single Point Cloud Scale Ratio Estimation of Two Point Clouds Evaluation
5. Conclusions
3
4
Changes should be alerted at these areas.
5
Original configuration
Damaged configuration
wave washing
if changed
dangerous
6
• Impossible to check manually Wide range Huge number of blocks
• Important to check automatically
7
• Impractical to check by fixed cameras
8
• possible to check by hand-held devices
9
Finding potential change area.
Sub-goal 1:
10
Estimating accurate changes.
Sub-goal 2: offline
Finding potential change area. online
Sub-goal 1:
1. Introduction Research Motivation Change Detection
2. 3D Keypoints Based 3D-2D Matching Background 3D Keypoints Detection Evaluation
3. 3D-2D Based Change Detection Background Image Based Change Detection Evaluation
4. 3D-3D Based Change Detection Background Scale Estimation of a Single Point Cloud Scale Ratio Estimation of Two Point Clouds Evaluation
5. Conclusions
11
A change is a difference of objects in the scene at time A and at time B.
12
Time A Time B
13
3D point cloud
Training images (2D images)
1. 2D-2D Method
14
Need Fixed camera input
output
Original image
Change image
Changed area
2D-2D
input
output
2. 3D-2D Method
15
Detection is fast but not accurate
Original point cloud
Change image
Changed area
3D-2D
input
output
3. 3D-3D Method
16
Detection is accurate but slow
Original point cloud
Change point cloud
Changed area
3D-3D
17
Potential changed areas 3D information
Camera poses
Online system Offline system
3D-2D based change detection
3D-2D based camera pose estimation 3D-3D based
change detection
Chapter 2 Chapter 3 Chapter 4
3D-3D 3D-2D
1. Introduction Research Motivation Change Detection
2. 3D Keypoints Based 3D-2D Matching Background 3D Keypoints Detection Evaluation
3. 3D-2D Based Change Detection Background Image Based Change Detection Evaluation
4. 3D-3D Based Change Detection Background Scale Estimation of a Single Point Cloud Scale Ratio Estimation of Two Point Clouds Evaluation
5. Conclusions
18
19
3D interesting points
2D interesting points
Camera pose=[R,t]
3D point cloud
2D training images
20
2D-2D
3D-3D
SIFT[Lowe 2004], SURF [Bay 2006], etc.
spin image[Johnson 1998], NARF[Steder 2010], etc.
detector descriptor
detector descriptor
21
3D point cloud
2D training images
3D detector and 2D detector can not be corresponded.
22
Image patch
Point distribution
Can not match
2D image
3D point cloud
23
• Detect keypoints correctly
• Describe keypoints appropriately
2D image
3D point cloud
Introduction ◦ Research Motivation ◦ Change Detection
3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation
3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation
3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation
Conclusions
24
Detected 2D interesting points
25
Feature matching
Obviously, the SIFT features could be used in 3D keypoints detection and description.
Point cloud
26
P1
P2
P3 P4
P5
P6 Camera position
3D keypoint
Projected 3D points
2D images number threshold used for 3D keypoints decision.
the points which can appear on multiple training images
Back face points are not used for computation
3D keypoints
th_v
27
th_v = 1 #3D keypoints ≅10,000
27 training images 105,779 3D points
th_v = 7 #3D keypoints ≅ 1,000
Reconstructed 3D points #3D points ≅30,000
Too many for real time calculating
Smaller number and good distribution
ours
orig
inal
28
2D SIFT keypoints and descriptors
3
3D keypoint& descriptor
Projected 3D points should overlapped to 2D SIFT keypoint
-Keep all 2D descriptors Accurate but slow
Description methods: -Average and Median
SIFT features are different when view directions are different.
Introduction ◦ Research Motivation ◦ Change Detection
3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation
3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation
3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation
Conclusions
29
30
P1 P2 P3 P4 P5 P6
3D point cloud
Ground truth Camera
positions
……
Training images
31
P1 P2 P3 P4 P5 P6
P6’
Camera pose estimation
3D keypoints generation
32
P1
P2
P3 P4
P5
P6
P6’
P6’ =[R ’ |t ’]
P6 =[R | t ]
Tra
nsla
tion e
rror
Rota
tion e
rror[ra
d]
33
1. Our method is accurate 2. th_v does not affect the result
th_v is used for 3D keypoints selection
2 degrees Dataset: 27 training images Image resolution:2256x1504 3D points number:105,779 3D scene size:40x25x5cm Bounding box size:10.6x5.7x1.4
0.24cm
34
3D point cloud Query image
project 3D points
35
Introduction ◦ Research Motivation ◦ Change Detection
3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation
3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation
3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation
Conclusions
36
37
Potential changed areas 3D information
Camera poses
Online system Offline system
3D-2D based change detection
3D-2D based camera pose estimation
3D-3D based change detection
38
Our method: 1. Use local feature instead of color 2. Detect any shape of object
Using laser range finder [Goncalves 2010,
Ryle 2011 and Neuman
2011].
Not for wide area targets.
Not applicable for our round shape or natural scenes.
Matching 3D line segments [Eden 2008].
Using color differences [Sato
2006, Pollard 2007 and Taneja 2011]
Not stable for illumination changes.
Introduction ◦ Research Motivation ◦ Change Detection
3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation
3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation
3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation
Conclusions
39
40
36
1. Find the nearest image
Query image
Nearest image
2. Find changed area Nearest image Query image
changed area
3. Visualization
Project 3D points onto changed area
1st Nearest Query image
41
P1 P2 P3 P4 P5
3D keypoints generation
Need fixed camera
Smallest distance
Ground truth
……
Training images 2nd 3rd
P
42
the 1st nearest image
Query image
Points: 2D keypoints
Blue: correspondence
Red: no correspondence
Blue: correspondence
Red: no correspondence
Non-change area
Uncovered area is the changed area
Estimated changed area
3D point cloud
Visualized 3D points 43
change area
projection
Detected result
Projected 3D points
Introduction ◦ Research Motivation ◦ Change Detection
3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation
3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation
3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation
Conclusions
44
3D point cloud Query image
45
Quantitative results visualization
Changed 3D points Changed area
Results for different thresholds 46
Set as:0, 5, 10, 20, 30, 50, 70 and 90 pixels
0 5 10 20
30 50 70 90
Image resolution:2256x1504 The number of Image: 54 The number of 3D points: 190,845
TP rate= True Positive Ground Positive
FP rate= Ground Negative
False Positive
47
Receiver operating characteristic (ROC) plot
threshold = 30
threshold = 30
Ground truth is set manually
We expect the 1st nearest image perform better than others, but the best result is the 2nd nearest image.
Good performance
Bad performance
48 It is the parameter left for users.
1st
2nd
Query image Detection results
Introduction ◦ Research Motivation ◦ Change Detection
3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation
3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation
3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation
Conclusions
49
50
Potential changed areas 3D information
Camera poses
Online system Offline system
3D-2D based change detection
3D-2D based camera pose estimation
3D-3D based change detection
Different size scale because of the character of Structure-from-Motion (SfM)
51
3D-3D registration is actually, the scale registration
3D point cloud 3D point cloud
3D point cloud
Change points
registration
Point clouds of same scene with different size
3D point cloud 3D point cloud
52
Iterative closest point (ICP) based alignment [Besl 1991].
-Need simple scenes -Need initial pose and scale -Not robust to clutters, occlusions and missing part
spin images [Johnson 1998],
NARF [Steder 2010], shape context [Belongie 2002], etc.
Feature based alignment
-Need appropriate neighborhood size
3D SIFT [Scovanner 2007], 3D SURF [Knopp 2010], etc.
-Not robust to clutters, occlusions and missing part
Easy data
Different data
Fixed scale
Adaptive scale
53
1. Scale estimation
2. Scale Ratio estimation
Keyscale1=0.5 Keyscale2=0.1
Scale ratio=5
Introduction ◦ Research Motivation ◦ Change Detection
3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation
3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation
3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation
Conclusions
54
Bunny point cloud
Width=0.001
55
Similar to each other
Different to each other
Width=0.1
Width=1.0
3D keypoints
Spin images
the minimum of similarity between spin images when the width changes.
Similar to each other
Keyscale
Robust to clutters, occlusions and missing part
Calculate similarity of collected spin images
56
Decide which set of spin images are different to each other by using Contribution rate.
PCA Robust to order of extracted spin images.
Robust to detail
57 minimum
1 5 10 15
sim
ilarity
sim
ilarity
d
w
Similar to each other
Different
Similar to each other
minimum is not unique Finding them is not stable
58
minimum minimum
sim
ilarity
w
59
Bunny point cloud
Finding minimum is not stable
sim
ilarity
w
Introduction ◦ Research Motivation ◦ Change Detection
3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation
3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation
3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation
Conclusions
60
Point clouds
61
Register two plots to get scale ratio
Scale ratio ICP
similarity plots
Overlapping parts
Original bunny curves 5 times larger bunny curves
62
Original bunny curves
5 times larger bunny curves
63
Scale ratio t
Displaced Original bunny curves
5 times larger bunny curves
64
65
Similarity estimation
3D registration
Scale
ratio
estim
atio
n
input
Similarity plots
Scale ratio
alignment
Introduction ◦ Research Motivation ◦ Change Detection
3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation
3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation
3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation
Conclusions
66
67
Original point cloud
The number of points:207,583 Scene size: 35m x 5m
overlapping area
68
Created 1st point cloud
Created 2nd point cloud
estimate scale ratio
Original point cloud
69
The method provides perfect result when the overlap rate is larger than 70%.
Ground truth = 1
Small blocks point clouds
70
Changed block
Introduction ◦ Research Motivation ◦ Change Detection
3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation
3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation
3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation
Conclusions
71
We have proposed three methods for a surveillance system to detect change.
2. Online 3D-2D based change detection
3. Offline 3D-3D based change detection
1. 3D-2D matching
In future: Find a more systematic way for choosing parameters. Improve computation time.
In future: Find the nearest image by considering FOVs. Implement the method on mobile devices.
In future: Accelerate the computation in order to handle much larger number of points.
72
Potential changed areas 3D information
Camera poses
Online system Offline system
3D-2D based change detection
3D-2D based camera pose estimation
3D-3D based change detection
73
Future work:
1. Improve computation speed and detection accuracy for online system. -current computation time: 20 seconds per image
2. Optimize algorithm to operate with huge size data for offline system. -current computation time: 10 minutes for 100,000 points
top related