Paper Note

Real-time rgb-d camera relocalization via randomized ferns for keyframe encoding

Organized by: Jacky Liu

jjkka132.slide (at) gmail.com

03 Feb 2017

National Chung Cheng UniversityRVL (Robot Vision Laboratory)

Paper Note, Jacky Liu, 03 Feb 2017

Why to read?

• To improve the robustness of ElasticFusion[1], one needs to study why relocalization module failed from recover after taking “white wall” scene.

• This paper is the relocalization module of ElasticFusion.

03 Feb 2017 2

[1] Whelan, Thomas, et al. "ElasticFusion: Dense SLAM Without A Pose Graph." Robotics: science and systems. Vol. 11. 2015.

Paper information

• MLA referenceGlocker, Ben, et al. "Real-time rgb-d camera relocalization via randomized ferns for keyframeencoding." IEEE transactions on visualization and computer graphics 21.5 (2015): 571-583.

• Citation until 03 Feb 2017: 9 times

• Key words1. Camera relocalization

2. Tracking recovery

3. Dense tracking and mapping

4. Marker-free augmented reality

03 Feb 2017 3

Outline

1. Real-time method

2. Compact representation of randomized fern

3. No spatial sampling heuristics

03 Feb 2017 4

Keyframe-based relocalization via randomized ferns for frame encoding

1. Frame encoding

2. Frame dissimilarity via Hamming Distance

3. Precision/Recall of BlockHD

4. Harvesting Keyframes

5. Tracking Recovery via pose retrieval

6. Fern Construction

03 Feb 2017 5

Thinking behind topics

A. Frame encoding→ Efficient way to represent frames, instead of using whole

image.

B. Frame dissimilarity via Hamming Distance→ How to compare two frames?

C. Precision/Recall of BlockHD→ BlockHD encoding property

D. Harvesting Keyframes→ How to determine “how densely we record the keyframe?”

E. Tracking Recovery via pose retrieval→ How to use encoded datastructure?

F. Fern Construction→ Detail of randomize fern method

03 Feb 2017 6

A - Frame encoding

03 Feb 2017 7

…m blocks (fern)

n node

Frame code: 𝑏𝑙𝑜𝑐𝑘1 𝑏𝑙𝑜𝑐𝑘2 …𝑏𝑙𝑜𝑐𝑘𝑚 = 0010 1101 … 0100

Frame id in order: 1 2 3 … k Code table for frame id lookup

B - Frame dissimilarity via Hamming Distance

03 Feb 2017 8

𝐵𝑙𝑜𝑐𝑘𝐻𝐷 𝑏𝐶𝐼 , 𝑏𝐶𝐽 =1

𝑚

𝑘=1

𝑚

𝑏𝐹𝑘𝐼 ≡ 𝑏𝐹𝑘

𝐽

Frame code: 𝑓𝑟𝑎𝑚𝑒𝐼 = 0010 1101 … 0100Frame code: 𝑓𝑟𝑎𝑚𝑒𝐽 = 0110 1001 … 0100

𝑏1 𝑏2 𝑏𝐶

Ex.Frame_I has 3 blocksHamming distance between frame I and J is 2

𝐵𝑙𝑜𝑐𝑘𝐻𝐷 𝑓𝑟𝑎𝑚𝑒𝐼, 𝑓𝑟𝑎𝑚𝑒𝐽 =1

𝐶

𝑘=1

𝐶

𝑓𝑟𝑎𝑚𝑒𝐼 ≡ 𝑓𝑟𝑎𝑚𝑒𝐽 =1

3∗ 2 = 0.66

C - Precision/Recall of BlockHD

code length ↗

change that two frame with low BlockHD ↘

precision ↗

recall ↘

vise versa

03 Feb 2017 9

Frame code: 𝑓𝑟𝑎𝑚𝑒𝐼 = 0010 1101 … 0100Frame code: 𝑓𝑟𝑎𝑚𝑒𝐽 = 0110 1001 … 0100

D - Harvesting Keyframes

• How to determine “how densely we record the keyframe?”

• Avoid to use spatial sampling heuristics=> non optimal scene coverage

• Minimum BlockHD K_I

𝐾𝐼 = min∀𝐽𝐵𝑙𝑜𝑐𝑘𝐻𝐷 𝑏𝐶

𝐼 , 𝑏𝐶𝐽= min∀𝐽(𝑚 − 𝑞𝐼𝐽𝑚)

03 Feb 2017 10

If 𝐾𝐼 > 𝑡, which t is predefined thresholdadd current frame as new keyframe

elsediscard current frame

E - Tracking Recovery via pose retrieval

• When tracking fails for an incoming frame, use most similar keyframe to retrieve pose proposals for tracking recovery.

• If reinitialization is unsuccessful for all proposed poses => discard current frame, repeat recovery until tracking is recovered

• Possible enhancementActively hint user to move the camera into areas of good keyframe coverage.

03 Feb 2017 11

F - Fern Construction

• Randomized parameters• Sampling pixel position x (uniform distribution)

• Number of ferns m

• Keyframe acceptance threshold t

• Number of neighbors k

03 Feb 2017 12

Experiment overview

• System based on KinectFusion (model based ICP)

• Two distinct evaluation set• Tracked frame for simulating harvesting of keyframes

• Performance evaluation

• Challenges of dataset1. Ambiguties (repeated steps in ‘Stairs’)

2. Specularities (reflections in ‘RedKitchen’)

3. Motion blur

4. Lighting conditions

5. Flat surfaces

6. Sensor noise

03 Feb 2017 13

Experiment – determine loss

• Plausibility check on the magnitude of camera motion and on the ICP residual error.

• If the relative motion or residual is too large, ICP reports tracking loss and relocalization is initiated

03 Feb 2017 14

Performance

• Percentage of frames for which the pose was successfully recovered.

• Definition of “successful recovery”Compare estimated pose with ground truth

• Translation error < 2cm

• Rotation error < 2 degrees

• Averaging three run to count “randomize” part of the algorithm

03 Feb 2017 15

Pose proposal strategies

• Using which keyframe pose to do ICP?• NN (nearest neighbor) – keyframe with smallest BlockHD

• kNN (k nearest neighbor) – Exam each k keyframe with smallest BlockHD, and pick the one with smallest ICP residual

• WAP (weighted average pose)

03 Feb 2017 16

Result

• Percentages correspond to the number of successfully recovered frames

03 Feb 2017 17

Offline v.s. Real-time

• Offline training methods outperform proposed real-time method. However, offline method is limited to revisited scene.

03 Feb 2017 18

Scalability to larger scenes

• Sufficient for representing large scenes of more than 30m^3 and thousands of frames.

03 Feb 2017 21

Marker-free augmented reality

• AR become more applicable when robustness of KinectFusion been improved by relocalizationalgorithm.

03 Feb 2017 22

Conclusion

• Solve the main cause of tracking failure in KinectFusion

• Compact frame encoding

• Lower performance compare to offline method

Future

• frame-to-frame tracking could give additional information about the relative pose trajectory.

• Loop closure

03 Feb 2017 23

Recall the goal of reading this paper

• To improve the robustness of ElasticFusion[1], one needs to study why relocalization module failed from recover after taking “white wall” scene.

• ElasticFusion loose track (https://youtu.be/mOmqd82EfDY)

• After meet the door, ICP report tracking loss.

• Relocalization module perform recovery.

• No loop closure to fix the tracking drift.

• To improve the robustness of ElasticFusion=> Apply loop closure. Define “uncertainty” of frame sequence, and use it to relocate path.

03 Feb 2017 24