indoor navigation infraestructure based on augmented reality techniques

[Slide 2]

□ Introduction□ Objectives□ Related Works□ Distributed Architecture

□ Multimodal Positioning System□ Vision Based Methods

□ Results□ Open Challenges

Carlos Gonzalez Morcillo ([email protected]) – University of Castilla-La Mancha

Contents26/10/2012 Intro Objectives Related Architecture Results Future

[Slide 3]






[Slide 4]






[Slide 5]






[Slide 6]


□ Multimodal Positioning System□ Vision-Based Methods




[Slide 7]






[Slide 8]






Intr o

duct

i on

Intro Objectives Related Architecture Results Future 26/10/2012

[Slide 9]Carlos Gonzalez Morcillo ([email protected]) – University of Castilla-La Mancha

Input Bin

Introduction26/10/2012


Intro Objectives Related Architecture Results Future

Input Bin

26/10/2012



What is Augmented Reality? Variation of Virtual Environments where virtual objects are superimposed over the real world.

Characteristics [Azuma97]: Real + Virtual Interactive Registered in 3D

Input Bin

26/10/2012





26/10/2012



More than just «Location»...

26/10/2012




How to measure Camera Pose (6DOF): Tracking

26/10/2012





Identify well-known points

26/10/2012





Table Surface


26/10/2012





Table Surface

Table Surface


26/10/2012



AR Market (Juniper Research)

Total Mobile AR Revenues on Mobile Devices Split by Category

Mill

ions

of D

olla

rs

26/10/2012



More than a «Trending Topic»

Augmented Reality Virtual Reality

Google Trends

26/10/2012



More than a «Trending Topic»

Augmented Reality Virtual Reality

Google Trends

Search Volume Index (by Country)

Spain: 18

Portugal: 31

France: 12

UK: 61

Germany: 54

Netherlands: 80

USA: 46India: 63

Contact Lenses,Contact Lenses,The Future?The Future?

AR Contact Lenses Prototype

26/10/2012



B. Parviz et al. (2009) University of Washington

26/10/2012



Goo g

le «P

roje c

t Glas

s»

Google+: g.co/projectglass

4th April 2012

26/10/2012



A world of Location-Based Services

[Video]

Obj ec

ti ves



Input Bin

Objectives26/10/2012



Not use AR«wish list»

Large Databases

SoftcomputingTechniques

ScalableApproach

Input Bin




To not use AR«wish list»

Large Databases


ScalableApproach

Input Bin




Large Databases


ScalableApproach


Input Bin

Sub-Objectives26/10/2012



Multi-Layer Architecture Task Delegation for Multiple Services Efficient Platform Transparent Location Based on Intelligent Agents (Wooldridge):

Autonomy Social ability Reactivity Proactivity

Input Bin






Rela t

ed W

ork s



Visual Tracking26/10/2012



Compute camera pose using local frame of reference.

Compute Relative Pose

Not only «Visual»... 6DOF26/10/2012



[Poner Vídeo de I. Shutterland]

Tracking or Localization26/10/2012



Relative or Absolute...

+ Known Points+ Precision

«Easy» definition of relations

26/10/2012





«Easy» definition of relations

Tracking or Localization

26/10/2012



Global PositionComposite Matrix



«Simple» definition of relations

Tracking or Localization

26/10/2012



External CamerasObtain camera pose with respect to fixed video sources.

26/10/2012



Visual Tracking Methods

Fiducial MarkersMarkerless Based Methods

Natural Feature Tracking (KeyPoint) Model Based (IPB, PS, ...) On-the-fly reconstruction (SLAM) Hybrid Methods (+IMU...)

26/10/2012



Fiducial Markers

Binarization Components Extraction

Vertexes and Edge Detection

Mar

kers

Sea

rch

2D P

osit i

on

Per

spec

tive

Pat

tern

I D

Get

Tra

n sM

at

3D R

end e

ring

Input Bin

Fiducial Markers: Example of Use26/10/2012



Virtual Museum of Computing

2007 (ESI/UCLM)

http://www.esi.uclm.es/museo/

Fiducial MarkersLimitations

26/10/2012



Marker Size (cm)

Det

ectio

n (D

ist)

(cm

)

Error / Marker Size (cm)

Rot

ati o

n


26/10/2012



Origin of GRS placed at the center of the Marker.

Result of propagating the same error

Be careful when using them as a localization method... Propagation error!


26/10/2012



Origin of GRS placed at the center of the Marker.

Result of propagating the same error

Be careful when using them as localization method... Propagate error!

26/10/2012




Fiducial MarkersMarkerless-Based Methods


26/10/2012






Natural Feature Tracking26/10/2012



Keypoint Extraction (Offline) Keypoint Matching Pose Calculation

26/10/2012






Model Based Tracking26/10/2012



Control Points Sampling Visible Edges Detection Control Points Matching Pose Calculation

26/10/2012






On-the-fly calculation Tracking26/10/2012



PTAMM[Castle 2010]

26/10/2012






26/10/2012



Previous Work: Hesperia Project

Our P

ropo

s al



Input Bin

26/10/2012


GlobalLocation

Architecture Distributed Middleware: ZeroC Ice

Location Providers

Our A

g ent

Pla t

form

26/10/2012



VideoStream

LightweightAgent

LightweightAgent

RFIDAgent

AudioStream

SensorLayer

StaticAgents

MobileAgents

Input Bin

AR Location & Rendering Service26/10/2012



TrackingLayer

Inertial System

EstimatorAgent

Compass

Acelerom.

Gyros.

GPSAgent

Absolute Methods Relative Methods

FiducialAgent

LedsAgent

Point T.Agent

Op. FlowAgent

VideoStream

LightweightAgent

LightweightAgent

RFIDAgent

AudioStream

SensorLayer

StaticAgents

MobileAgents




TrackingLayer

Inertial System

EstimatorAgent

Compass

Acelerom.

Gyros.

GPSAgent


FiducialAgent

LedsAgent

Point T.Agent

Op. FlowAgent

VideoStream

LightweightAgent

LightweightAgent

RFIDAgent

AudioStream

SensorLayer

FusionLayer Historical

PerceptionAgent

Perception FusionAgent

StaticAgents

MobileAgents




TrackingLayer

Inertial System

EstimatorAgent

Compass

Acelerom.

Gyros.

GPSAgent


FiducialAgent

LedsAgent

Point T.Agent

Op. FlowAgent

VideoStream

LightweightAgent

LightweightAgent

RFIDAgent

AudioStream

SensorLayer


PerceptionAgent


3D Repr.ManagerAgent

PhoneAgent

HMDAgent

TabletPCAgent

RepresentationLayer

MultiAgentRender Grid

StaticAgents

MobileAgents




TrackingLayer

Inertial System

EstimatorAgent

Compass

Acelerom.

Gyros.

GPSAgent


FiducialAgent

LedsAgent

Point T.Agent

Op. FlowAgent

VideoStream

LightweightAgent

LightweightAgent

RFIDAgent

AudioStream

SensorLayer


PerceptionAgent



PhoneAgent

HMDAgent

TabletPCAgent

RepresentationLayer

ServiceLayer


3D Model RepositoryAgentFace

IdentificationAgent

InteractionAgent

Authentication Agent

StaticAgents

MobileAgents




Tracking Layer26/10/2012



Fiducial Markers (ARToolKit).

Tool for tuning marker position.




Fiducial Markers (ARToolKit).

On-the-fly Recons-truction (PTAMM).

Tool to obtain pose and scale of PTAM Maps.

Fiducial Markers (ARToolKit). On-the-fly Reconstruction (PTAMM). Visual Descriptors. QBIC.

Lucene Indexes: ColourLayout and Edge Histogram.

Query: Voronoi Tree.




Tracking Layer: Visual Descriptors26/10/2012



Position: 2.5, 5.5Rotation: 135º




Fiducial Markers (ARToolKit). On-the-fly Reconstruction (PTAMM). Visual Descriptors. Stereo Location.

1. Features to track: Shi-Tomasi. 2. Optical-Flow: Lucas-Kanade. 3. Fundamental Matrix: LmedS. 4. Fundamental + Intrinsic = Essential Rotation Estimation (Rodrígues).

TrackingLayer

Inertial System

EstimatorAgent

Compass

Acelerom.

Gyros.

GPSAgent


FiducialAgent

LedsAgent

Point T.Agent

Op. FlowAgent

VideoStream

LightweightAgent

LightweightAgent

RFIDAgent

AudioStream

SensorLayer


PerceptionAgent



PhoneAgent

HMDAgent

TabletPCAgent

RepresentationLayer

ServiceLayer


3D Model RepositoryAgentFace

IdentificationAgent

InteractionAgent

Authentication Agent

StaticAgents

MobileAgents




Fusion Layer26/10/2012



With HistoricalWithout Hist.

Objective: Tracking Stabilization. Solution: Use of a historical vector. Issues: Precision loss.

Fuzzy Controller26/10/2012



OpticalFlow PTAM Weight

NG NP Z PP PG NG NP Z PP PG LOW MED HIGH

Weight

26/10/2012



R1/ if(OF == NG & PT == NG) -> P = HIGH;R2/ if(OF == NP & PT == NP) -> P = HIGH;R3/ if(OF == Z & PT == Z) -> P = HIGH;R4/ if(OF == PP & PT == PP) -> P = HIGH;R5/ if(OF == PG & PT == PG) -> P = HIGH;R6/ if((OF == PP | OF == PG) & (PT == NP | PT == NG)) -> P = LOW;R7/ if((OF == NP | OF == NG) & (PT == PP | PT == PG)) -> P = LOW;R8/ if((OF == NP | OF == Z) & PT == NG) -> P = MED;R9/ if((OF == PP | OF == Z) & PT == PG) -> P = MED;R10/ if(OF == NG & (PT == NP | PT == Z)) -> P = MED;R11/ if(OF == PG & (PT == PP | PT == Z)) -> P = MED;

Fuzzy Controller (11 Rules):

Fuzzy Controller

26/10/2012



Fuzzy Controller

Res u

lts



Input Bin

Results26/10/2012



Futu

re W

o rks



26/10/2012



Open Challenges

indoor navigation infraestructure based on augmented reality techniques

Engineering