multimodal sensing-based camera applications

MACHINE VISION GROUP

Multimodal sensing-based camera applications

Miguel Bordallo1, Jari Hannuksela1, Olli Silvén1 and Markku Vehviläinen2

1 University of Oulu, Finland2 Nokia Research Center, Tampere, Finland

Jari Hannuksela, Olli SilvénMachine Vision Group, Infotech Oulu

Department of Electrical and Information EngineeeringUniversity of Oulu, Finland


Outline

Introduction• Modern movile device with multiple

sensorsVision-based User InterfacesSensor data fusion systemApplication case implementations

• Motion-based image browser• Motion sensor assisted panorama

imagingConclusions/Summary


Introduction

• More and more applications and features are being crammed into handhelds

• Causes usability complications given the constraints of current mobile UIs

• Increased computing power not harnessed for UIs

• Keypad and pointer based UIs and/or touchscreens in current devices– User’s hand obstructs the view– Require two handed operation


Modern mobile device with multiple sensors• The phone includes touch screen, GPS,

accelerometers, light sensor, proximity sensor • Two cameras: low resolution for video calls and high

resolution for photography and video capture• Newer phones will include magnetometers,

gyroscopes


Motivation for vision based user interfaces

Allow recognition of the context- Detect user’s actions- Recognize environment

Allow 3D informationProvide interactivity

- Real-time feedback- Single hand use


Limitations of vision based UIs

Fast movements

Low light

Difficult conditions

+


The solution: sensor data fusion

Fusing the data obtained by several sensors

• Ambience light sensor determines illumination conditions

• Video analysis detects ego-movements and analyzes the context

• Accelerometers provide complementary motion data


Video analysis

- Every frame divided into regions- Selection of feature blocks- Estimation of block displacements- Analysis of uncertainty

- Results: 4-paramenter model- X, Y, Z, r


Sensor data fusion: Accelerometers


Sensor data fusion

Model the device movement with the folowing

Define a state vector: position, speed, acceleration

Define a measurement model

Apply Kalman filtering adding accelerometer values: State prediction + state correction


Application cases

• Sensor data fusion method applied in two applications– Implemented on a Nokia N900 mobile phone

• Motion based image browser– Allows browsing large images and maps with one hand operation– Works under different light conditions

• Sensor assisted panorama imaging– Stitches panorama images in real time from video frames– Increased robustness against fast movements and no-texture

frames


Motion based image browser

Uses fusion model from accelerometers + video analysis to generate commands

• Scroll up/down/left/right• Zoom in/out

Light sensor decides:• if camera should be turned on • weighting factors and uncertainties• 3 modes defined:

• Good image quality (video analysis + accelerometer correction)• Bad image (accelerometers have increased contribution)• No image (only accelerometers are used)


Motion based image browser II


Sensor assisted panorama Imaging

•Based on video analysis

•Guides the user with instructions

•>360 degrees panoramas •Real-time registration•Real-time frame evaluation and selection•Real-time frame correction

•Increased robustness via sensor-data integration


Panorama imaging: Sensor uses

•Uses sensor fusion model to compute camera motion•Increased robustness against fast movements and frames with low/smooth texture

Registration


Panorama: Sensor uses II

•Uses accelerometer data to detect blur•Detects unwanted shake/tilt•Integrated in scoring system

Selection


Summary

• Vision based interfaces offer high interactivity with one hand operation

• They present several limitations• Sensor fusion improves motion estimation

adding robusness against fast movements and dark conditions

• The framework can be included in several applications (e.g. as a part of Motion Estimation API)


Conclusions

• We have presented a sensor fusion framework that fuses vide analysis with motion sensors (acelerometers+magnetometers+gyroscopes)

• We have presented two applications cases that make use of sensor data fusion and integration

• The applications presented are by no means the only ways to apply vision or multiple sensors, and one may find new interesting possibilities in further research


Thank you!

Any question???

multimodal sensing-based camera applications

Documents