Mobile Phone based Inference Models using People-centric FeaturesNicholas D. Lane

Results

Approach: Use features that leverage the ability of people to synthesize complex multivariate data

Recognition ProcessExploratory Experiment

Problem: Inferences about society and where we live are challenging with mobile phones.

Example: Spoken Words as features for Activity Recognition.

• 10^9 mobile phones are in daily use but with limited sensing capabilities (e.g., localization accelerometer, microphone).

•Important inferences are difficult based on these sensors (e.g., What are people doing? Are they sick? Are they safe?).

Hypothesis: Even when recognizing only a fraction of ambient spoken words it is possible to perform complex forms of activity recognition using only a simple bag-of-words model.

Future Work

Methodology: Build proof-of-concept iPhone-based prototype. Capture 19 hours of audio while doing different activities over 2 weeks.

• Evaluate other examples of People-centric features particularly those found in other modalities and across other time scales

• Develop models that combine these examples with more conventional features.

Audio Signals

Collection of Words

Activities

MFCC feature vectors from audio frames

LBG-based vector quantization

Isolated word based discrete HMMs

Stemming & Stop Word Removal

Activity class based bayesian “bag-of-word” modelsfast food

coffee• With 17% of words recognized and using word only features mean activity recognition accuracy was 71%.

bank gym fast food coffee100% 0% 100% 81%

•Recognizes different instances of classes (e.g., fast food) and does not confuse these with similar classes (e.g., restaurants).

• Differentiates activity uses (e.g., coffee or book purchase) in the same physical space (e.g., bookstore).

unknown class

Can I have a coffee?

Here is a coffee.

Thanks for my coffee.

words selection

non-verbal sounds (sneeze)

mobility patterns

behaviour

people and the environment