Mobile AR and Electronic Secretary

Wouter Pasman

Overview

1. Who am I

2. About UbiCommobile AR

3. About Cactuselectronic secretary

Wouter Pasman

1987-1991: Student Computer Science, Masters completed with honours (Complexity theory, correctness proving, ..)

1991-1993: Student Musicology, completed first 2 years

1993-1997: Ph.D. Industrial Design Engineering, Delft University.

1997-2001: Post-doc work: UbiCom

2001-2002: Post-doc work: Cactus

One of my hobbies…

UbiCom project

Ubiquitous Communications

Mobile Augmented Reality (AR)

VIDEO: UbiCom short intro

Overview

UbiCom structureOutdoor AR TrackingLow latency mobile AR renderingDynamic simplification Mathematical model for rendering ARC framework QoS, Accuracy curves VRML integration

UbiCom structure

Project leader: R.L. Lagendijk (E. Deprettere)

Personal supervisorF.W. JansenComputer Graphics &CAD

P1'wireless' P2 'visual info'

me

P3 'protocols'

P1. Adaptive wireless comm transceivers.

Subproject-leader Wouter Serdijn

5 Ph.Ds

OFDM codec hardwareOFDM on infrared5 and 17GHz radio linkSignal propagation measurements

P2: Audio-visual information processing.

Subproject leader: Richard Heusdens

2 postdocs, 4 PhDs

TrackingLow-latency renderingVideo CodingVisual Presentation FilterRetinal Scanning DisplayGIS database handling

QuickTime™ and aPhoto - JPEG decompressor

are needed to see this picture.

P3: System architecture and protocols.

Subproject leader: Henk Sips

3 postdocs, 2 PhDs

ARC QoS systemLART mobile sysVoltage ScalingP123 interfaceJava compiler

VIDEO: Outdoor AR Tracking

VIDEO: Low latency mobile AR

rendering

Dynamic Simplification

Dynamic LoD generation in backbone

Maximize perf/cost ratio in headset.

Mathematical model per object

• Estimate link and CPU load, memory usage, lifetime of objects, etc • Est screenspace error and geometric distortions

D=0.001R=1m

QoS: Scheduling of resources

Funkhouser,Séquin (1992), Mason,Blake (1997)

Goal: maximize benefit within a cost budget.

Benefit: pixel size on screen,accuracy heuristic

Cost: # triangles and pixel size on screen

Mason showed this is NP complete..

- Iterative approx giving in worst case half

the maximum possible quality

- Quality only known after iteration

- Only feedback loop with application possible

QoS Management

Phase I

At top level, acceptable options for operation of the server are specified or changed.

UbiCom-wide QoS mechanism: ARC

Modeserveroperation spaceModeclientclientoptions

Phase IIa

Server searches for possibilities, considering • internal state• ARC services

A number of options within the requested range are returned.

clientoptionsserver optionsModeserveroperation spaceoperation space 3operation space 2

Phase IIb

An optimizer filters the best options and returns them to the client

Modeclientclientoptionsserver optionsModeoptimizerfeasibleoptionsserveroperation space

Phase III

At top level the operation point is chosen and a contract is established. The server probably needs to set its own contracts accordingly.

Operation pointModeserveroperation spaceModeclient

Accuracy Curves

To fit graphics rendering to ARC, each node in scene graph is seen as a 'client' in ARC. Each is assigned an accuracy curve

RA• required resources as function of accuracy target

• monotonically increasing.

R->#polygons

Measurement of geometric distortion d as function of number of polygons n

d~ C/n.

Accuracy a =1/d

Resource usage r = K a + R0

-> piecewise linear function

cowhousebunnyknotbuddha

100 1k 10k 100k

0.1m

1m

0.01

Number of polygons

Propagating accuracy curves

Leaf nodes: accuracy curve from (1) mathematical model or (2) measurements

Other nodes: propagate curve upwards through scenegraphLODChild 1Child 2ARARAR

Res

ourc

e U

sage

LoD node behaviour

AccuracyLOD curveCurve ofchild 2Curve of child 1

LODChild 1Child 2ARARAR

Different end points

curve 1

curve 2

Accuracy

Reso

urc

e u

sag

e

incorrectextension

incorrect minimum

20 50

Complexity

Root client/node in scene graph: Target Accuracy -> Resources requiredIdem R -> A (so not NP-complete)

RAmax.Rmax.Atarget A

Optimizing curve updates

Upto now: screenspace error=visual accuracyRefresh required if user moves

Optimizations:

(1) Determine range where a curve is ‘accurate enough’ as long as viewer is within the range.

(2) Visual accuracy is derived from geometric distortion - which is viewpoint independent

Relative acc = geometric accuracy object radius

Using relative accuracy curves

Slight changes in algorithms:

• grouping -> 'object' diameter changes.

• Conversion to visual accuracy needed. Group's bbox is much closer to viewer than the individual bboxes -> convert to visual accuracy at reasonable distance.

VRML integration

Accuracy curves & simplified objects: valid only in part of space. Efficient checking with ConicRange.

d

Imposter nodes

Replace children with image. Automatic refresh & calc of accuracy curve

SimpleImposter{ MFNode children [ ] SFVec2f size 2 2 SFBool autosize true SFVec3f center 0 0 0 SFBool autocenter true}

LOD

LOD picks valid level requiring least resources

LODSimpleImposterrange 10 1000MeshedImposterCylinder {}

Statue on the campus

-Prototype implementation of all previous

-Very complex, implementation was simplified at several places (caching, prediction, etc)

Statue application

Tracker was not yet working -> tracker stubvideo position & orientation

renderingfrontendpositionlookupVGAmixer

finalvideo file

Statueapplication

Statue VRML filebackbonemobile systemrendering

compilerSimplification

machinestracker ‘stub’Offlineanalysis

DEMO VIDEO

Cactus project

Context Aware Communication, Terminal and User

Overview

Scenario sketchCactus structure, manpowerFirst ideas for architecture

Cactus Scenario: cultural outing

Current electronic secretary (PCA)

Existing PCA's:• enhancing comm between customer and business representative• 'Unified Messaging': voice,email,fax• often also agenda and conference booking• Menu-like voice interface

KPN Eileen• Additional: news, weather, teletekst, tv programs, travel info, etc• Human operator

Cactus electronic secretary

Adding context sensitivityTracking the user and estimating user plansPro-active secretary

Project structure

1 of 13 projects of Freeband Kennisimpuls project (www . freeband.nl)(EZ and OCW)

Cactus consists of 2 phases

2002-2004: Cactus Impulse

2004-2006: Cactus

Cactus Impulse structure

Project leader: R.L. Lagendijk

UseT TermiNet

me

Personal supervisorF.W. JansenComputer Graphics &CAD

UseT (User & Terminal)

Three subprojects:

Trust, Consistency and Interaction (1 Ph.D.)

User-context Analysis, Modeling and Sensing

(1 Ph.D.)

i-DEA Proof-of-Concept (me + 1 Ph.D.)

Questions UseT.1 "Trust"

Will the user see the system as consistent and trust it?

Which cognitive and user interface factors are most important wrt trust and consistency?

eg, when should system ask and not ask? how should information and questions be presented?

Wizard of Oz studies to test our systemand ideas

Questions UseT.2 "Sensing"

1. Which context- and personalization parameters are relevant for Eileen?

2. How can these parameters be sensed?

3. How can this info best be stored?

Questions T.3 "Proof of Concept"

1. What is proper architecture for UseT System?

2. What reasoning system is appropriate?

3. What filtering of information based on user preferences and context is possible?

Knowledge required

• Knowledge of applications

• Knowledge of user

• Knowledge on inferring user plans, goals etc

• Action planning system

• Knowledge about relatives of user

Etc..

Architecture sketchuser

others9292 info

shopsvoice mail

etc

inputselectparsersemanticssecretary

applic.historylogplans, themesapplicknowl.knowl. about user

i-DeaOCRemailapplic.

useremaili-Deaemail

agendaapplic.

useragendaText to Speechstandard PCOS messages

Primitives-hooks

TextgrabberPhoneinterfaceuser planinferenceactionplanner

voicerecorder

voiceplaybackrecorded audio

(Minsky)

Easy

Few anyNUMBERS OF CAUSES

Using magnitudes helpes makingcomparisons by hiding

Find betterrepresentation!

Intractable

ConnectionistNeural Netw orkFuzzy Lo gic

Analogy-basedReasoning

Case-basedReasoning

Societyof Mind

ClassicalAl

TraditionalComputing

Or dinar yQualitativ eReasoning

SymbolicLo gicReasoning

Linear ,Statistical

Scaleof

Effect

Large

Small

Figur e 1. Causal-diversity matrix

Appropriate Reasoning system

In our case, subtile differences have large effects, but not so many causes -> classical AI

Proposal: logical goal- and plan-inference

In style of Wilensky's Plan Application Mechanism

Plan/goal graph required for this (figure)Task analysis is good idea anywayLearning opportunities for system

Multimodality

Use speech semantic structures to represent other modalities as well (pen input, pointing, scrolling, etc).

Advantages: uniform representation, easy modality crossing, late modality binding, language independent application, …

Problem: stills & video images

Open questions when I left Cactus..

• Which subset of requirements do we start with?

• How to get there without doing an impossible amount of programming?

• Video modality not even theoretically clear how to handle in plan-based approach