InerVis

Mobile Robotics LaboratoryInstitute of Systems and Robotics

ISR – Coimbra

Contact Person:Jorge LoboEmail: jlobo@isr.uc.pt

Human inertial sensor: The vestibular system

Within the inner ear, the vestibular system measures tilt and angular acceleration

Integration of

Vision and Inertial SensingPhD 2003 – 2007

selected publications: Jorge Lobo and Jorge Dias, "Relative Pose Calibration Between Visual and Inertial

Sensors", International Journal of Robotics Research, Special InerVis Issue, in press. Peter Corke, Jorge Lobo and Jorge Dias, "An introduction to inertial and visual sensing",

International Journal of Robotics Research, Special InerVis Issue, in press. Luiz G. B. Mirisola, Jorge Lobo, and Jorge Dias, "Stereo Vision 3D Map Registration for

Airships using Vision-Inertial Sensing", In The 12th IASTED Int. Conf. on Robotics and Applications, Honolulu, USA, August 2006.

Jorge Lobo, João Filipe Ferreira and Jorge Dias, "Bioinspired Visuo-vestibular Artificial Perception System for Independent Motion Segmentation", In Second Inernational Cognitive Vision Workshop, ECCV 9th European Conference on Computer Vision, Graz, Austria, May 2006.

Jorge Lobo and Jorge Dias, "Relative Pose Calibration Between Visual and Inertial Sensors", Proceedings of the ICRA 2005 Workshop on Integration of Vision and Inertial Sensors - 2nd InerVis, Barcelona, Spain, April 18, 2005.

• Jorge Lobo, Jorge Dias , “Inertial Sensed Ego-motion for 3D Vision”, in Journal of Robotic Systems Volume 21, Issue 1, pp. 3-12, January 2004.

• Jorge Lobo and Jorge Dias, “Vision and Inertial Sensor Cooperation, Using Gravity as a Vertical Reference”, in IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI, 25(12), pp. 1597-1608, December 2003.

• Jorge Lobo, Carlos Queiroz, Jorge Dias, “World Feature Detection and Mapping using Stereovision and Inertial Sensors”, in Robotics and Autonomous Systems, Elsevier Science, vol. 44, Issue 1, pp. 69–81, July 2003.

selected publications: Jorge Lobo and Jorge Dias, "Relative Pose Calibration Between Visual and Inertial

Sensors", International Journal of Robotics Research, Special InerVis Issue, in press. Peter Corke, Jorge Lobo and Jorge Dias, "An introduction to inertial and visual sensing",

International Journal of Robotics Research, Special InerVis Issue, in press. Luiz G. B. Mirisola, Jorge Lobo, and Jorge Dias, "Stereo Vision 3D Map Registration for

Airships using Vision-Inertial Sensing", In The 12th IASTED Int. Conf. on Robotics and Applications, Honolulu, USA, August 2006.

Jorge Lobo, João Filipe Ferreira and Jorge Dias, "Bioinspired Visuo-vestibular Artificial Perception System for Independent Motion Segmentation", In Second Inernational Cognitive Vision Workshop, ECCV 9th European Conference on Computer Vision, Graz, Austria, May 2006.

Jorge Lobo and Jorge Dias, "Relative Pose Calibration Between Visual and Inertial Sensors", Proceedings of the ICRA 2005 Workshop on Integration of Vision and Inertial Sensors - 2nd InerVis, Barcelona, Spain, April 18, 2005.

• Jorge Lobo, Jorge Dias , “Inertial Sensed Ego-motion for 3D Vision”, in Journal of Robotic Systems Volume 21, Issue 1, pp. 3-12, January 2004.

• Jorge Lobo and Jorge Dias, “Vision and Inertial Sensor Cooperation, Using Gravity as a Vertical Reference”, in IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI, 25(12), pp. 1597-1608, December 2003.

• Jorge Lobo, Carlos Queiroz, Jorge Dias, “World Feature Detection and Mapping using Stereovision and Inertial Sensors”, in Robotics and Autonomous Systems, Elsevier Science, vol. 44, Issue 1, pp. 69–81, July 2003.

Integrating Inertial Sensors with Artificial Vision key contributions: a common framework for inertial-vision sensor integration; calibration methods for integrated inertial and vision systems; vertical feature segmentation and 3D mapping; ground plane segmentation; 3D depth map registration; independent motion segmentation.

MEMs Inertial sensors

Analog devicesADXL202 dual axis ±2g accelerometer.

Analog devices ADXRS150 angular rate sensor (gyroscope). (October 2002)

Xsens MTi IMU

Data From Inertial Sensors

Independent Motion Segmentation

Registering Stereo Depth Maps

bodyt

t

ω

a

Inertialsensors

Camera

Images

INS calcs

Imageprocessing

Match featuresacross images Dynamic 3D(t)

reconstructuredworld model

real world

worldt

t

t

t

g

θ

x

v

time

features

lines

flow optical

points

time

sens

or d

ata

fusi

on

• N static poses observing vertical target– Full camera

calibration

– IMU↔CAM Rotation estimated

Sensor Calibration

• Both sensors used to measure the vertical direction•N observations at different camera positions•Unknown rotation determined

• 2N static poses with N rotations about IMU– IMU↔CAM Translation estimated

Swinging pendulum sequence

Seg. motion voxelsRaw voxels background voxels

- =

1)Image optical flow (LK)

2)Estimate optical flow from3D data and reconstructed camera motion assuming static scene

3)Subtract and threshold to segment independentmotion

• Optical flow consistency segmentation

Images

Cameramotion

DepthMap

• Background subtraction1)Quantise registered point cloud to voxel space and accumulate occupancy votes for all frames

2)Threshold to obtain background voxels (apply thinning and growing transformation for noise filtering)

3)Intersect current frame voxels with complement of background voxels to have voxels from moving objects