intelligent sensor and learning challenges for context aware appliances

Séminaire PSIFRE CNRS 254616 Janvier 2003

[email protected]

Intelligent sensorIntelligent sensor

and learning challengesand learning challengesfor context aware appliancesfor context aware appliances

Intelligent sensorIntelligent sensor

and learning challengesand learning challengesfor context aware appliancesfor context aware appliances

>> Stéphane Canu [email protected]

asi.insa-rouen.fr/~scanu

INSA Rouen, France - EU

Laboratoire PSI


[email protected]: La souris et leMacintoch

200X : la nouvelle rupture "break through"


[email protected] technologie d'aujourd'hui

• Loi de Moore

• Communication "sans fil"

• L'ère des données

Quelles applications ?


[email protected]


[email protected]

Olympus Optical Co., Ltd. is pleased to announce its new wearable user interface technologies. Employing gestures and other hand movements for input, the system is an ideal match for new wearable PCs.


[email protected]

http://www.redwoodhouse.com/wearable/index.htmlhttp://wearables.cs.bris.ac.uk/public/wearables/esleeve.htmhttp://www.ices.cmu.edu/design/streetware/


[email protected] on wearable


[email protected]

Wearable


[email protected] aware appliances

Phone by night

http://mediacup.teco.edu/overview/engl/m_what.html

The mediacup(calm version of the active badge)


[email protected] Motors and CMUThe car- drives together- informs you- in a parking…

GM/CMU Companion driver interface systemGM/CMU Companion driver interface system


[email protected]! Where is my car?• Old fashion software design: process

1.Match the sentence 2.Send the query to the satellite3.Satellite send query to the car on its own frequency4.Car answers…

- Tell the computer what to do (where is the switch)

• Distributed software design: interaction- Software agents talk together

• Future way: Programming by Example- Show the computer what to do

• Today's solution: Louis my 3 years old son

Disappearing computer >> Your Wish is My Command: Programming by Example Henry Lieberman, editor, Published by Morgan Kaufmann, 2001.


[email protected] technology• Ubiquitous computing

- One people - many computer

• Technology at our service- Reactive to what user do- Proactive - Prepare what to do next- Situated – sharing context

(Hans Gellersen, Sensing in Ubiquitous Computing)

• Adapted to our needs- New functionalities and new behaviors- New way of communicating- Learn to adapt

Machines have to know their context

>> M. Weiser "The Computer for the 21st Century." Scientific American, September 1991


[email protected]

• user- activity (available/meeting)- location, - identity, profile

• environment monitoring- time, day/night, temperature, weather, - resources (networks, services…)

• appliance - proprioception- usage - functionalities- maintenance- resources (energy…)

What is the context?

Abstract representation of the situationKnowledge?

How to find it from data?

+ history…

ExplicitOutput

actuators

Context input

Context output

ExplicitInput

sensors

Context-awareapplication

Adapted From Henry Lieberman and Ted Selker, Out of Context: Computer Systems That Adapt To, and Learn From, Context,IBM Systems Journal 39, 2000.


[email protected] context from the environmentpresentation roadmap

>> Kristof Van Laerhoven, Kofi Aidoo: Teaching Context to Applications In Personal and Ubiquitous Computing, Volume 5 Issue 1 (2001) pp 46-49

1. Data

2. Representation

3. Information retrieval

4. Context evolution

5. User interaction


[email protected] from data

• Unbelievable capacity- Moore’s law

• New sensors- Artificial nose- Bio sensor

• “Personal” data- humor: affective computing

Data Era!

http://www-stat.stanford.edu/~donoho/lectures.html

DataRepresentationInformation retrievalContext evolution

>>


[email protected] sensors

http://www.teco.edu/tea/sensors.html

How are you?


>>


[email protected]

http://markov.ucsd.edu/~movellan/mplab/index.html

Machine Perception LabFace Detection and Expression Recognition

Expression recognitionDataRepresentationInformation retrievalContext evolution

>>


[email protected] much informationkills information

• Critic of the "Data Era"

• Data smog

• Non measurable things

• Ethical consequences- the Orwellian future

Filter data!


>>

"We are drowning in information and starving for knowledge." - Rutherford D. Roger


[email protected] sensors• Requirements:

- Data- Accuracy and confidence- Self diagnostic- Self calibration

• How to do it?- Uncertainty management- Learning ability

• Network + database- Adaptation ability- Fault detection mechanism

Associated software sensors


>>

>>S. Canu et al., "Black-box Software Sensor Design for Environmental Monitoring" , in International Conference on Artificial Neural Networks , Skovde, Sweden. Sep 2-4, 1998 (and related work on data validation within the EM2S project)


[email protected] validation• Mono sensor validation

- Static validation• Mean, variance

- Dynamic validation• Cusum (control charts)• Trend analysis

• Multisensor validation- Residual analysis- Fusion: Joint probability estimation- Prior knowledge: Balanced relations

• Hierarchical validation- Multisensor perception

Interactive matrix of smart sensors

>> http://www.accenture.com/xd/xd.asp?it=enWeb&xd=services\technology\research\tech_sensor_matrix.xml>> K. Van Laerhoven, A. Schmidt and H.-W. Gellersen. "Multi-Sensor Context-Aware Clothing". In Proceedings of the 6th International Symposium on Wearable Computers, 2002


>>


[email protected] sensor

• Value + confidence interval + validity domain• How to build it ?

- From a model: tracking = Kalman filter- When no model is available: learn it!

Raw data v(t)

Raw data x(t)

Raw data y(t)

Raw data z(t)

learning = Black box modeling


>>

environment


[email protected] proprioceptors

• Learn

• How to learn?- Gaussian mixture + EM- Include prior: Bayesian networks- Deal with uncertainty: Evidence framework

• Use to:- Detect non nominal situations- Replace missing data

vxxx di ,,...,...,Pr 1

d = Curse of dimensionality (Belman)


>>

>> E. Petriu et al., "Sensor based information appliances",


[email protected] is data?

• Individuals or measurements• Associated variables• Data set (matrix)

- line = measurements

- column = variable

• Data: point clouds- Data exploration: recognize patterns

too many data: SUMARIZE


>>


[email protected] data

• Non linear components analysis- Feature space: kernel (PCA or ICA)- Local linear- Quantisation (SOM)- Relevant distance

• Select features- Local adapted representation- Feature selection

• Select relevant situations- Sparse learning- Kernel learning


>>

>> J. Mäntyjärvi, J. Himberg, P. Korpipää, H. .Mannila, "Extracting the Context of a Mobile Device User", 8th Symposium on Human-Machine Systems-HMS,Kassel, Germany, 2001.

Kernel representation


[email protected]

Example in 2 dimension of the influence map of the "black circle". Red color denotes a high influence while the low influence zones are in blue.

Data

Dat

a'

Influence map

Kernel representationDistance maps

Analyze data proximity through the kernel map

),(exp),( jidistjiI

i

j

>> B. Scholkopf and A. Smola, "Leaning with Kernels", MIT Press, 2001


[email protected] of kernel map

Data clouds in two dimensions Associated kernel map

Class 2

Class 1

Class 2

Even in d dimensions you can visualize

>> S. Canu and al., "Functionnal learning through kernels", invited lecture at the NATO institute in Leuven, 2002


[email protected]

>> Balázs Kégl http://www.iro.umontreal.ca/~kegl/research/pcurves/

Looking for hiden shapes

• Data point = information + noise

• Principal curve- Non linear PCA

• Independent curve - Non linear ICA


>>

Kernel representation + linear analysis


[email protected]

>> J. B. Tenenbaum, V. de Silva and J. C. Langford http://isomap.stanford.edu/handfig.html

Navigatein high dimensional space Data

RepresentationInformation retrievalContext evolution

>>


[email protected] retrieval

• What for- User profiling- User identification- Battery discharge rate- Sequence induction…

• Classification problem- Decision theory- Example based programming- Learning machine

Select relevant cases


>>


[email protected] brief historical perspectiveof machine learning

• Before machines- Statistics: PCA, DA, regression, CART, kNN

• 70's - Learning is logic- Grammatical inference in expert systems

• 80's - Learning is human- Neural networks: backprop

• 90's - Learning is a problem: COLT- Kernel machines: SVM- Mixture of experts: adaboost

What is the learning problem?


>>

>> T. Hastie, R. Tibshirani and J. Friedman, "The elements of statistical learning", Springer, 2001


[email protected]

• Data- Training set

- Test point looking for such that

• Learning is balancing

1. Hypothesis set (Neural networks, Kernels)

2. Fitting criterion (least square, absolute value)

3. Compression criterion (penalization, Margin)

4. Balancing mechanism (cross validation, generalization)

What is learning?

Learning is summarizing

)(ˆ

,,...,,,...,,

111

11

nnn

nnii

xfyfx

yxyxyx

Fit Summarize data data


[email protected] discriminationseparable case

+

+

+ +

+

++

+

+

+

+

+

+wx+ b=0

(w,b) ???

Use hyperplane


>>

How to correctlyclassify

all points?

Occam Razor's


[email protected] discriminationseparable case

+

+

+ +

+

++

+

+

+

+

+

+wx+ b=0

Be sparse


>>


all points?


[email protected]

+

+

+ +

+

++

+

+

+

+

+

+

The classifier Margin

wx+ b=0Margin

Margin

Be sparse


>>


all points?


[email protected]

+

+

+ +

+

++

+

+

+

+

+

+

Maximize the marginBe sparse

wx+ b=0

wx+ b=-1

wx+ b=1

Margin

Margin

Support Vector Machines: SVM


>>


all points?

>> V. N. Vapnik, "The nature of statistical learning theory", Springer-Verlag, 1995


[email protected] is learning?

Learning is summarizing

SVM

Fit Summarize data data

• Data- Training set

- Test point looking for such that

• Learning is balancing

1. Hypothesis set (Neural networks, Kernels)

2. Fitting criterion (least square, absolute value)

3. Compression criterion (penalization, Margin)

4. Balancing mechanism (cross validation, generalization)

)(ˆ

,,...,,,...,,

111

11

nnn

nnii

xfyfx

yxyxyx

>> S. Canu, A. Rakotomamonjy, Ozone peak and pollution forecasting using Support Vectors, IFAC workshop, Yokohama, 2001.


[email protected] Inputadaptive scaling

• Enumerate all combination …and score

• Preprocessing- Information theory - Statistical test

• Wrapper- Use a relevance index - Learn and select together

Global formulation

Example of relevance index for a toy problemwith 2 relevant features and 50 irrelevants


>>

>> Y. Gandvalet and S. Canu, "Adaptive Scaling for Feature Selection in SVMs", accepted for publication at NIPS 2002


[email protected]

Dimension reduction by >> multi-resolution analysis

(just like in your eyes…)

Learn at the relevant scale >> multi scale representation

Efficient implementation - ridgelets, curvelets - wavelets’ kernel


>>

Summarize patterns

"Kernelize" wavelets>> A. Rakotomamonjy and S. Canu, "Frame, Reproducing Kernel, Regularization and Learning", accepted in JMLR 2002


[email protected] machines challenges1. Hypothesis set

Multi scale data representation: wavelets Use context: mixture of experts

2. Fitting criterion Sparse distance criterion Select relevant input (adaptive scaling) Relevant distance: adapt the kernel

3. Compression criterion Information issues Global optimization

4. Balancing mechanism Efficient direct algorithm (one shot learning)

Towards Context based learning


>>


[email protected] assessment• Deal with uncertainty

- plausibility / credibility- unknown states / ability to evolve- data fusion: evidence theory

• Take into account prior knowledge: transitions- temporal representation- uncertain transitions- learn probabilities or possibilities

• Learn the model- don't start from scratch- create and delete contexts

• Adapt context determination to user- from a global imprecise context to specific context

How to implement context?

DataRepresentationInformation retrievalContext evolution>>


[email protected] implementation• Context = state

- List of variables- Petri's nets

• State = stochastic- Markov model- Bayesian networks

• Identify = decision theory (data fusion)- Information retrieval

• Learn context- Knowledge discovery- Create / delete- Context hierarchy (time granularity)

Context is a languageHow to retrieve the context?

Henry Lieberman: http://web.media.mit.edu/~lieber/


[email protected] idea to deal with context• Current context: working memory

- Prior knowledge: transition law

• Available information: evidence- Data fusion

• Learn context- Transition law- Context retrieval from data

• Context is a language

• Speech recognition- Markovian model- Evidence- Language + previous state- Locator's adaptation

Adapt speech recognition ideas to contexthttp://htk.eng.cam.ac.uk/



[email protected]: Research chalenges• Inputs

- Deal with uncertainty (and missing data)- Representation- Data fusion (multimedia fusion)

• Context- Define a language- Represent previous state- Learn transition

• Feed Back to inputs

• Adapt transition to the user- Loop the user: reinforcement- Control mechanism (stability/plasticity dilemma)

Challenging research issues

http://cslu.cse.ogi.edu/tutordemos/nnet_training/tutorial.html



[email protected] through

• What is information?- Computer science- Coding- Signal

• Mathematics- Statistics & computer science- Pattern recognition- Functional analysis- ??????

>> L. Devroye, L. Györfi and G. Lugosi, "A Probabilistic Theory of Pattern Recognition", Springer-Verlag 1996.

…remember Albert and relativity

Theoretical models are essentials (Mark Weiser, Computer Science Challenges for the Next Ten Years)


[email protected] long bet

Before 2050We will faced a scientific revolutionregarding information definitionComparable with the one induced in physics by the relativity theory

$ 500To greenpeace

Long bet fundation at San Francisco http://www.longbets.org/


[email protected] challenges• create context

- how to define prior contexts: user’s needs- how to represent contexts: stochastic automaton- learn from data: modify, create and destroy context

• decide context - validate data software sensors- select relevant inputs representation + distance- select relevant patterns wavelets- select relevant situations SVM and kernel- make decision using data fusion Dempster-Shafer + EM

• loop with the user- reinforcement learning- user’s needs

Bayesian networks

Integrate: create relevant learning architecture


[email protected]?• Asia

- Scurry™, Wearable & Virtual Keyboard - Samsung, - K. Doya for reinforcement

• America- Context Aware Computing group - Media lab MIT- CMU, Stanford- Georgia tech: Future Computing Environments - Smart Matter Integrated Systems (Xerox PARC)- Montreal – learning lab

• Australia- ANU for learning- University of South Australia - wearable computer lab

• Europe- Telecooperation Office (TecO) at the University of Karlsruhe- The disappearing computer, a EU-funded proactive initiative- The Smart-Its project- Equator project focuses on the integration of physical and digital interaction - Perceptual Computing in general and Computer Vision in ETH Zurich- IDIAP for machine learning and speech recognition- PSI, France for learning

Some context aware references


[email protected]


[email protected] macroscopic…DataRepresentationInformation retrievalContext evolution

>>


[email protected]

http://www.mcell.cnl.salk.edu/

MCell Simulation of miniature endplatecurrent generation at the neuromuscular junction.

Image rendered with Pixar Photorealistic RenderMan.

…to Microscopic dataDataRepresentationInformation retrievalContext evolution

>>


[email protected] detection >> E-Motions Data

RepresentationInformation retrievalContext evolution

>>

>> R. Picard, Affective Computing, MIT Press, 1997 http://graphics.usc.edu/~dfidaleo/Emotion/ http://www.mis.atr.co.jp/~mlyons/facial_expression.html

Towards affective computing


[email protected] accuracy

• Find (a,b) such that

• Model the model- The sandwich estimator, (Tibshirani, 1996)

• Likelihood Based on the Hessian matrix

- Confidence machine (Gammerman RHC, 1999)• Confidence: 73.11% - Credibility: 51.37%

• Sample the models- Bootstrap (Heskes 1997)

• Learn the error- Train using absolute error

How to compute error bars?

>> R. Tibshirani, "A comparison of some error estimates for neural network models," Neural Computation, 8, 152-163, 1996.>> Tom Heskes, "Practical confidence and prediction intervals", Advances in Neural Information Processing 9, eds. Mozer, M., Jordan, M. and Petsche. T., pp. 176-182, 1997.

http://nostradamus.cs.rhul.ac.uk/~leo/pCoMa/


>>

1ˆˆPr bvvavAccuracy confidence


[email protected]

Locally linear representations

Looking for hiden shapes

>> Sam T. Roweis & Lawrence K. Saul http://www.cs.toronto.edu/~roweis/lle/


>>


[email protected] synthesis from text

Turing proof

>> Tony Ezzat and Tomaso Poggio http://cuneus.ai.mit.edu:8000/research/mary101/mary101.html

…from text to movie


>>


[email protected] one expression to anotherDataRepresentationInformation retrievalContext evolution

>>


[email protected] euclidian metrics

http://cs.unm.edu/~joel/NonEuclid/


>>


[email protected] Self-Organizing MapDataRepresentationInformation retrievalContext evolution

>>

What is the "distance" between two objects


[email protected]

http://www.techfak.uni-bielefeld.de/ags/ni/projects/hsom/hsom.html

- 650 documents - from 16000 reviews - Internet Movie Database

Disney's animationMovies are closed


>>

Example on movies - HSOM


[email protected] on moviesDataRepresentationInformation retrievalContext evolution

>>


[email protected]


>>

Curvlets

Deal with high dimensional spacehttp://www-stat.stanford.edu/~jstarck/


[email protected]


>>

Curvlets

Deal with high dimensional space

The original image 64,536 coefficients.

http://www-stat.stanford.edu/~jstarck/


[email protected]

-2 0 2-3

-2

-1

0

1

2

3

-1

0

0 1

1

1

Select relevant situations

• Relevant representation- "Invent" features- Select features

• Relevant "distance"- map- Use kernel

• Summarize the examples- Define a relevant global criterion to be minimized- Support vector machines (SVM)

Be sparse


>>


[email protected] architecture

• Agent - Data base - Communication• Metadata• Context language• Adaptability: control mechanism• Pre programming: anticipation• Open – modular – distributed

- The Ektara Architecture (MIT for wearable)- Nexus - A Platform for Context-Aware Systems- The Context-Toolkit (Geargia Tech)

How to debug such software?

http://web.media.mit.edu/~rich/



[email protected] appliances?

• Deal with the context- Recognize- Adapt- Create

• Inference, Learning, discovery,- Represent- Decide- Deal with time

• From user interface to user interaction- Reinforcement learning- Human factors

• How to know what we need?

Human factors: cool technology is at our service

intelligent sensor and learning challenges for context aware appliances

Documents