dr.kawewong ph.d thesis
TRANSCRIPT
PIRF-Nav: An Online Incremental Appearance-based Localization and Mapping in
Dynamic Environments
Aram Kawewong
Hasegawa Laboratory Department of Computational Intelligence and Systems Science
Interdisciplinary Graduate School of Science and Engineering
Tokyo Institute of Technology
1
Simultaneous Localization and Mapping, or SLAM, is a navigation system needed for every kind of mobile robots
In the unfamiliar environment, the robot must be able to perform two important tasks simultaneously
Mapping the new place if the place has never been visited previously
Localizing itself to some mapped place if the place has been visited before
2
Introduction to SLAM
3
Appearance-based Localization and Mapping (FAB-MAP)
Why don’t we just use GPS ? GPS is not always reliable in the crowded city centre
GPS can only locate the coordinate/position of the agent but not the corresponded scene; how can the robot answer the question “look at this picture and tell me where it is ?” or “have you ever visited this place before ? Can you describe about the nearby places ?”
No false positive (can have false negative) If the robot is not confident then it should answer “this is the
new place”. If the robot is to answer “this place is the same place as the place ….”, it must be 100% correct.
100% precision (all answers must be correct)
4
Why Visual SLAM ? What are the Challenging ?
Place Recognition (Computer Vision)
Localization and Mapping (Robotics)
Input Images All testing images are known to come from somewhere in the map
Every input image is a testing image; it might come from somewhere in the map or it might be the previously unseen place
Environment Closed Environment Opened Environment
Precision Precision-1 is not the main concern if the recall rate is reasonably high
Precision-1 is the first priority concern; one false positive may lead the serious error in navigation
5
Appearance-based Localization and Mapping VS Place Recognition
100% Precision with very high Recall Rates
Can run incrementally in an online manner
Life-long Low computation time
Consume less memory
Suitable to navigate in large-scale environments
Can solve 2 main problems: Dynamical Changes
Perceptual Aliasing (Different Places but look similar)
Note: Coordinate-based Localization is not required here
6
Appearance-based SLAM’s Common Objectives
1. FAB-MAP (Cummins & Newman, IJRR’08) Considering the efficiency at 100% precision, the obtained
recall rate of FAB-MAP (a State-of-the-art method) is still not so high.
An offline generation process for dictionary generation is necessary.
2. Fast Incremental Bag-of-words (Angeli, et al. T-RO’08) The system can run incrementally; offline dictionary
generation process is not needed. Accuracy is said to be less than or equal to that of FAB-MAP Consume much higher memory than FAB-MAP
7
Visual SLAM’s Related Works
FAB-MAP (IJRR’08)
Inc. BoW (T-RO’08)
PIRF-Nav (prop.)
Ability to incrementally run without needs for offline dictionary generation process
No
Yes
Yes
Memory Consumption Low High Moderate
Ability to run in real-time Yes Yes Yes
Robustness against dynamical changes*
Moderate (~40% on
City Centre)
Low (~20% on City
Centre)
High (~85% on
City Centre)
8
What Do We Want ? :PIRF-Nav’s Advantages
* The recall rate is considered at 100% precision
Making use of PIRF, we can detect the good landmarks of each individual place
The extracted PIRFs should be sufficiently informative to represent the place so that the system does not need the preliminary generated visual vocabulary
The number of PIRFs is sufficiently small to be used in the real-time application
Because the PIRF is robust against dynamical changes of scenes, the PIRF-based visual SLAM (called PIRF-Nav) become an efficient online incremental visual SLAM
9
Basic Idea & Concept of PIRF-Nav
Outdoor Scenes generally include distant objects whose appearances are robust against the changes in camera position
Averaging the “slow-moving” local features which capture such objects give us the less and more robust features
10
Basic Idea of PIRFs (proposed)
10
11
PIRF Extraction Algorithm
Image Sequence
3 0 0 1 4 3
1 3 6 0 5 0
2 4 1 5 3 1
0 2 6 4 1 3
4 1 0 5 0 0
0 1 5 0 4 2
Sliding Window; w = 3
Sequence of Matching Index Vectors
Exp. 1 Scenes From Suzukakedai
Exp. 1 Scenes From O-okayama
12
Briefly on PIRF’s Performance
Training (640x428) Testing (640x428)
580 489
Training (640x428) Testing (640x428)
450 493
13
PIRF’s Performance
0.00%10.00%20.00%30.00%40.00%50.00%60.00%70.00%80.00%90.00%
100.00%
24.5
4%
31.0
8%
22.2
9%
18.2
3%
93.4
6%
27.5
9%
45.7
5%
36.7
1%
30.2
2%
77.4
8%
Recognition Rate of Suzukakedai and O-okayama
Su
zuka
ked
ai
O-O
kaya
ma
14
Even With These Strong Changes, PIRF Still Works Well !!!
Highly Dynamic Changes in Scenes
Illumination Changes in Scenes
15
PIRF (City Centre Dataset)
Original Descriptors (SIFT)
Position-invariant Robust Feature (PIRF) (proposed)
Overall Processing Diagram
Step 1: Perform simple feature matching. The score is calculated based on the popular term frequency-inverted document frequency weighting
Step 2-3: Adapt the score by considering the neighbors and then perform normalization
Step 4: Perform second integration over the score’s space for relocalization
16
PIRF-Nav Processing Diagram (prop.)
At time t, a map of the environment is a collection of nt discrete and disjoint location
Each of these locations , which has been created from past image , has an associated model
The model is a set of PIRFs
17
Notation Definition
𝐋 = {𝐿1,… ,𝐿𝑛𝑡 }
𝐿𝑖
𝐼𝑖
𝑀𝑖
𝑀𝑖
The current model is compared to each of all mapped models using standard feature matching with distant threshold
Each matching outputs the similarity score s
is model of the location which is a virtual location for the event “no loop closure occurred at time t”.
Based on the obtained score s, the system proceed to the next step if
18
STEP 1: Simple Feature Matching
𝐌𝑡 = {𝑀0,… ,𝑀𝑛𝑡} 𝑀𝑡
𝜃2
𝑀0 𝐿0
𝑎𝑟𝑔𝑚𝑎𝑥(𝑠) ≠ 0
The similarity score s is calculated by considering the term frequency – inverted document frequency (tf-idf) weighting (Sivic & Zisserman, ICCV’ 03) :-
is the number of occurrences of visual word w in is the total number of visual words in is the number of models containing word w N is the total number of all existing models
19
STEP 1: Simple Feature Matching (Continued)
tf − idf =𝑛𝑤𝑖𝑛𝑖
log𝑁
𝑛𝑤
𝑛𝑤𝑖 𝑀𝑖 𝑛𝑖 𝑀𝑖 𝑛𝑤
To be used with PIRF, the function is then converted to
is the number of models , containing PIRFs which match the kth PIRF of the input model
is the number of all matched PIRFs between input and the query model
The system proceeds to STEP 2 if and only if the maximum score does not belong to and is greater than
20
STEP 1: Simple Feature Matching (Continued)
𝑠𝑖 = log 𝑛𝑡𝑛𝑤𝑘
m i
k=1
𝑛𝑤𝑘 𝑀𝑗 , 0 ≤ 𝑗 ≤ 𝑛𝑡 , 𝑗 ≠ 𝑖
𝑀𝑡
𝑀0 𝜏1
𝑚𝑖
Accepting of rejecting loop-closure detection based on the score from only single image is sensitive to noise
This can be handled by considering the similarity score of neighboring image models as:-
The term is the transition probability generated from a Gaussian on the distance in time between i and k
stands for the number of neighbors examined
21
STEP 2: Considering Neighbors
𝛽𝑖 = 𝑠𝑘 ∙ 𝑝𝑇 𝑖, 𝑘 𝑖+𝜔
𝑘=𝑖−𝜔
𝑝𝑇 𝑖,𝑘
𝜔
Done by considering the standard deviation and mean value over all scores
indicates the number of neighbours taken into consideration
The beta-scores are converted into normalized score according to the equation
where 22
STEP 3: Normalizing the Score
ln
𝐶𝑖 =
𝛽𝑖 − 𝜎
𝜇, if 𝛽𝑖 ≥ 𝑇
1, Otherwise
𝑇 = 𝜎 + 𝜇
The obtained location would be accepted as loop closure if
Ideally, the neighboring model scores of location Lj should decrease symmetrically from a model score. However, scenes in dynamic environments usually contains moving objects that frequently cause the occlusion. The score of some assigned location may not be symmetrical.
23
STEP 4: Re-localization
𝐿𝑗 𝛽𝑗 − 𝑇 > 𝜏2
24
Step 4: Relocalization (Sample Problems)
Location assigned from Step 3 does not have a symmetrical score
Performing one more summation can shift the location to the right one
Therefore, we perform the second summation over the neighbouring score model to achieve a more accurate localization
The obtained normalized score for all possible models determines the most potential loop-closure location , where
25
STEP 4: Re-Localization
𝐿𝑗 𝑗 = argmax
𝑖𝐶𝑖′
𝐶𝑖′ = 𝐶𝑘 ∙ 𝑝𝑇 𝑖,𝑘
𝑖+𝜔
𝑘=𝑖−𝜔
Three datasets have been used City Centre (2474 images with size 640 x 480)
The dataset was taken to address to problem of dynamical changes of scenes in the city centre.
New College (2146 images with size 640 x 480) The dataset was taken to address the problem of perceptual aliasing. By
this dataset, a robot walked to the same place many times. Many different places look very similar.
Suzukakedai (1079 images with size 1920 x 1080) The dataset was taken by video camera attached with the omnidirectional
lens. The dataset was taken to address the problem of highly dynamical changes where the different event was organized (i.e. open-campus event)
26
Results & Experiments : DATASETS
City Centre
27
Results & Experiments: DATASETS
New College
28
Results & Experiments: DATASETS
Suzukakedai
29
Results & Experiments: DATASETS
Among many visual SLAM methods, FAB-MAP (Cummins & Newman. IJRR’08) and the fast incremental BoW method of Angeli et al. (T-RO’ 08) are considered to be state-of-the-art.
Both of them are based on Bag-of-words scheme Each of them offer different advantages
FAB-MAP High accuracy with offline dictionary generation Angeli et al. Lower than or equal accuracy to FAB-MAP but
with an online incremental dictionary generation
PIRF-Nav must offer higher accuracy than FAB-MAP while being an online incremental method like Angeli et al.
30
Results & Experiments: BASELINE
31
Evaluation on Appearance-based Loop-closure Detection Problem
Input Image
Loop-Closing ?
Add new place to the map
Find the loop-closure place
Output the loop-closure location
Binary Classification: New place / Old Place
Image Retrieval Problem: Retrieve the most likely place for
loop-closure
PrecisionA = Correct Loop-closure
All Loop-closure
RecallA = Correct Loop-closure
All labeled loop-closure
PrecisionB = Correctly retrieved image
All retrieved images
RecallB = Correctly retrieved image
All labeled images
Actually, performance should be evaluated by two graphs:
Precision A – Recall A curve
Precision B – Recall B curve
However, for compact representation, most works in visual SLAM use Precision B – Recall B curve to show the performance because
The binary classification is currently not so much problematic
Important challenge is given to the performance of image retrieval
32
Evaluation on Appearance-based Loop-closure Detection Problem
33
Evaluation on Appearance-based Loop-closure Detection Problem
(City Centre)
Precision A – Recall A: Focusing on only the problem of
saying “YES/NO” loop-closure detected is currently trivial
Precision B – Recall B: Instead, given that the precision of
the “YES/NO” loop-closure detected is 100%, it is much more
interesting to see how accurate the system can correctly retrieve the
corresponding image
34
Result 1: City Centre Vehicle Trajectory Loop Closure Detection
PIRF-Nav (100% Precision) (proposed) FAB-MAP (100% Precision)
35
Result 1 : City Centre (Precision-Recall Curve)
36
Result 1: City Centre (Computation Time)
*It is noteworthy that all programs of PIRF-Nav were written in MATLAB while FAB-MAP was written in C.
37
Result 2: New College Vehicle Trajectory Loop Closure Detection
PIRF-Nav (100% Precision) (proposed) FAB-MAP (100% Precision)
38
Result 2: New College (Precision-Recall Curve)
39
Result 3: Suzukakedai
Vehicle Trajectory Loop Closure Detection
PIRF-Nav (100% Precision)
40
Result 3: Suzukakedai (Precision-Recall Curve)
41
Result 4: Combined Datasets (Precision-Recall Curve)
Note: We did not test FAB-MAP on this experiment because FAB-MAP completely failed in Suzukakedai Dataset. Also the results on City Centre and New College clearly imply that FAB-MAP will not gain better accuracy in this experiment.
42
Sample Matched Images (Dynamical Changes in Major Part of Scene)
43
Sample Matched Images (Different View-Points)
PIRF-Nav outperforms FAB-MAP in term of accuracy with more than 80% recall rate at 100% precision on all datasets provided by the authors
PIRF-Nav offers an online and incremental ability to run in very different environments
Although the computation time of PIRF-Nav at the same image scale is slower than FAB, PIRF-Nav compensates this drawback by processing on smaller image scale since the accuracy is still considerably much higher than FAB-MAP
44
Conclusions
45
Thank you for Your Kind Attention
“DOUBT IS THE FATHER OF INVENTION”
QUOTED BY GALILEO
Journal 1. A. Kawewong and O. Hasegawa, "Classifying 3D Real-World Texture Images by
Combining Maximum Response 8, 4th Order of Auto Correlation and Colortons, " Jour. of Advanced Comp. Intelligence and Intelligent Informatics, vol. 11, no. 5, 2007.
2. A. Kawewong, Y. Honda, M. Tsuboyama, and O. Hasegawa, "Reasoning on the Self-Organizing Incremental Associative Memory for Online Robot Path Planning," IEICE Trans. Inf. & Sys., vol. E93-D, no. 3, 2009. (impact factor 0.369)
3. 本田雄太郎,Aram Kawewong, 坪山学,長谷川修:"半教師ありニューラルネットワークによる場所細胞の獲得とロボットの自律移動制御",信学論D,2009,採録決定
4. A. Kawewong, N. Tongprasit, S. Tangruamsub and O. Hasegawa, “Online and Incremental Appearance-based SLAM in Highly Dynamic Environments, " Int’l Jour. Robotics Research (IJRR). (To Appear in 2010, impact factor 2.882, rank#1 in robotics)
5. A. Kawewong, S. Tangruamsub and O. Hasegawa, “Position-Invariant Robust Features for Long-term Recognition of Dynamic Outdoor Scenes," IEICE Trans. Inf. & Sys. (conditional accepted)
46
Publication
Conferences 1. A. Kawewong and O. Hasegawa, "3D Texture Classification by Using Pre-testing
Stage and Reliability Table, " IEEE Proc. International Conference on Image Processing (ICIP), (2005).
2. A. Kawewong and O. Hasegawa, "Combining Rotationally Variant and Invariant Features Based on Between-Class Error for 3D Texture Classification, " IEEE Int’l Conf. On Computer Vision (ICCV) Workshop, 2005.
3. A. Kawewong, Y. Honda, M. Tsuboyama, O. Hasegawa, "A Common-Neural-Pattern Based Reasoning for Mobile Robot Cognitive Mapping, " In Proc. Int’l Conf. Neural Information Processing (ICONIP), 2008.
4. A. Kawewong, Y. Honda, M. Tsuboyama, O. Hasegawa, "Common-Patterns Based Mapping for Robot Navigation, " in Proc. IEEE Int’l Conf. Robotics and Biomimetics (ROBIO), 2008.
5. S. Tangruamsub, M. Tsuboyama, A. Kawewong and O. Hasegawa, "Mobile Robot Vision-Based Navigation Using Self-Organizing and Incremental Neural Networks," in Proc. Int’l Joint Conf. Neural Networks (IJCNN), 2009.
47
Publication
Conferences 6. A. Kawewong, S. Tangruamsub, and O. Hasegawa, "Wide-baseline Visible
Features for Highly Dynamic Scene Recognition," in Proc. Int'l Conf. Computer Analysis of Images and Patterns (CAIP), 2009.
7. N. Tongprasit, A. Kawewong and O. Hasegawa, "Data Partitioning Technique for Online and Incremental Visual SLAM," in Proc. Int’l Conf. on Neural Information Processing (ICONIP), 2009. (oral & student travel award)
48
Publication