overview of life logging september 4, 2008 sung-bae cho
TRANSCRIPT
![Page 1: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/1.jpg)
Overview of Life Logging
September 4, 2008
Sung-Bae Cho
![Page 2: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/2.jpg)
• Life logging
• Context-aware computing
• Sensory data for activity recognition
• Life logging with mobile devices
• Summary
Agenda
![Page 3: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/3.jpg)
0
0.2
0.4
0.6
0.8
1
2002 2003 2004 2005 2006 2007
Dis
k ca
pac
ity
(TB
)
Source: Microsoft 2003
Advances of Digital Devices
Right now, it is affordable to buy 40 GB (in 2003)
In 3 years 1TB/year is affordable!
It is hard to fill a terabyte/year, but you can:Look at 9,800 pictures a day (300 KB JPEGs)
Read 2,900 documents a day (1MB files)
Listening to audio or view compressed video 24 hours/day (it takes more than 256 kb/s to fill a TB in a year)
Watch 1.5 Mb/s video 4 hours each day.
3/48
![Page 4: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/4.jpg)
4/48
Digital Convergence
Properties of Mobile Devices
Multi-function Mobility
Large Memory Personal Device
Collect Various Log Data of
User’s Everyday Life
Advances of Mobile Devices
![Page 5: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/5.jpg)
Everyday Life with Mobile Devices
I’m always I’m always with you!with you!
5/48
![Page 6: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/6.jpg)
Available Information from Mobile Devices
Photo
Where is the place?
Who is in the media?
Video
How many people in the media?
User Created Contents
What is the activity?
Sensor Log
Bluetooth
Who is nearby?
Bio-sensor
Sleep? Activity Level?
GPS
Location
Vision Sensor
Contexts of environments
Audio
Noisy level?
User Data
Scheduler Personal Profile
Details on daily events Preference
Human Relationships
Demographical information
Address Book
Usage Log
Call Log
DurationReceiver/Caller
Time
SMS Log
ContentsReceiver/Sender
Time
Application Usage
Mp3 Player
6/48
![Page 7: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/7.jpg)
Life(Experience)
DigitalInformation
Life Record
Context DataAlways Bringing Contents Creation
Human Memorize Experience
Life Logging with Personal Digital Devices
Personal Digital Device
7/48
![Page 8: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/8.jpg)
Necessity of Context-aware Computing
Integration of Various Typeof Information
User Modeling fromPersonal Information
Data Mining fromPersonal Information
Context-aware Computing
Personal Information Retrievalbased on Semantic Structure
Episodic Management ofPersonal Information
Summary ofEveryday Life
8/48
![Page 9: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/9.jpg)
• Life logging
• Context-aware computing
• Sensory data for activity recognition
• Life logging with mobile devices
• Summary
Agenda
![Page 10: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/10.jpg)
Definition of Context-aware Computing
• Context : Dey & Abowd (1999)
– Any information that can be used to characterize the situation of an entity, where an entity can be a person, place, physical or computational object
• Context-aware computing
– The use of context to provide task-relevant information and/or services to a user, wherever they may be
• Three important context-aware behaviors
– The presentation of information and services to a user
– The automatic execution of a service
– The tagging of context to information for later retrieval
• Importance of context
– The context of user is changed frequently and drastically in ubiquitous and mobile environment (Pascoe, et al., 1998)
10/48
![Page 11: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/11.jpg)
Trends on Context Awareness
• TEA: EU project for enabling context awareness
– 1998 ~ 2000, with TeCo, Starlab, Omega and Nokia
– scenarios, technologies, market research and demonstrators
• DrWhatsOn concept project
– concept and user scenarios for context aware office PDA device
– main focus in usability and user interface design
• Earlier the focus was in sensor-based context recognition
– recognize and utilize information from user's activity and environment properties
• Now more directions and possibilities exist
– Context-aware computing
– Context-aware communication
– Context-aware services, connectivity, information retrieval, affective computing, etc.
TEA
TEA
DrWhatsOn
11/48
![Page 12: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/12.jpg)
Why Context Awareness is Difficult?
• Context recognition is never 100% reliable
– Contexts are vague, dynamic, overlapping, and ill-defined
• Adaptive user interfaces are scary!
– May need an adaptive, learning SW
• Sensors & algorithms may need constant recalibration
– They may also be too CPU intensive
• Application development frameworks & support are missing
– Security, privacy concerns a lot
• Connectivity to environment
– Limited I/O through wireless links
– What short range connectivity technology should be used?
• Productization
– Where’s the business?
– What everyday problems it really solves?
– Where do we get sensors for sensor-based context recognition?
12/48
![Page 13: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/13.jpg)
Some Technical Challenges
• Understanding structure and behavior of context information
– context are fuzzy, overlapping, and changing in time
• Lack of sensing methods for context information from various sources
• Lack of methods for fusing context information
• Lack of format of context information
• Sensor-based context recognition
– hard to obtain reliable data
– signal processing and recognition consume memory, energy and processing time
– results may be ambiguous
• Connectivity to environment and other devices
– Are profiles available; how about location information?
– Any privacy & security risks?
13/48
![Page 14: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/14.jpg)
Why do we Need Context Awareness?
Maybe avoid disturbing people
at wrong times
Right information, right time, right place
Receding to background
(Calm computing)
New services
(Location-based services are the obvious ones)
Increased user
satisfaction
Better social acceptance
of technology
14/48
![Page 15: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/15.jpg)
• Life logging
• Context-aware computing
• Sensory data for activity recognition
• Life logging with mobile devices
• Summary
Agenda
![Page 16: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/16.jpg)
MIT Media LAB (1)
• Area: Visual Contextual Awareness in Wearable Computing (1998)
• Sensor: Vision
• Probabilistic object recognition
– Probabilistic dependence analysisbased on neighbor vector & object recognition
• O: object, M: measurement
– Task recognition with HMM
16/48
![Page 17: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/17.jpg)
MIT Media LAB (2)
• Activity recognition based on accelerometer (2004)
– 20 activities
• Sensor: Accelerometer on diverse points of body
• Geometric analysis
– Mean, energy, frequency-domain entropy, and correlation
• Classification method
– Decision Tree C4.5, IBL, Naive Bayes
17/48
![Page 18: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/18.jpg)
MIT Media LAB (3)the number of subjects collected under laboratory (L) or naturalistic (N) settings
18/48
![Page 19: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/19.jpg)
MIT Media Lab (4)
• Aggregate confusion matrix for C4.5 classifier
– leave-one-subject-out validation for 20 subjects
19/48
![Page 20: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/20.jpg)
eWatch Sensor Platform
• CMU Computer Science Lab in 2005– Defense Advanced Research Projects Agency (DARPA)
• Activity recognition, improving power consumption, location recognition
• Hardware
– LCD, LED, vibration motor, speaker,Bluetooth for wireless communication
– Li-Ion battery with a capacity of 700mAh
• Sensors
– a two-axis accelerometer (ADXL202; +/- 2g)
– Microphone, light & temperature sensors
• Method
– multi-class SVMs
– HMM based selective sampling
20/48
![Page 21: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/21.jpg)
Classifier
• multi-class SVMs with Gaussian Radial Basis Function kernels
– frequency spectrum-based classification
– time-domain-based classification with SVM
• means, variances, square root of the uncentered second moment, the median absolute differences
21/48
![Page 22: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/22.jpg)
University of Alberta
• National ICT Australia Project: University of Alberta, Canada
• Human activity & gesture recognition
• Sensor
– active, magnetic field, acoustic, laser, camera sensor
• Method
– Coupled hidden Markov model (CHMM)
• Extended HMM model for combining and utilizing concurrent information stream more effectively (By M. Brand et al., 1997)
– M. Brand, N. Oliver, and A. Pentland, “Coupled hidden Markov models for complex action recognition,” in IEEE Intl. Conf. Comp. Vis. Pat. Rec., pp. 994-999, 1997.
22/48
![Page 23: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/23.jpg)
University of Bologna
• Micrel Lab: University of Bologna, Italy (2004)
• Research
– Construction of Ubiquitous environments
– Sensor-based gesture recognition
• Sensor
– Wireless MOCA (Motion capture with integrated accelerometers) sensor
• Accelerometer, gyroscope
• Small size, small power cosume
• Wireless operation
• Sticking on diverse points of human body
• Method
– Hidden Markov Model
23/48
![Page 24: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/24.jpg)
Activity Recognition Summary
• Mostly focused on static classification for pose recognition (relatively easy)
USAMIT
FinlandNokia
JapanATR
Aus. Curtin Univ.
SwissETH
Zurich
BelgiumStarlab.
ChinaSJT Univ.
CanadaCMU
Static classification ○ ○ ○ ○ ○ ○ ○
Temporal classification
○ △
Accelerometer ○ ○ ○ ○ ○ ○ ○ ○
Vision sensor ○ ○
Instant behavior ○ △
Repeated behavior (Pose)
○ ○ ○ ○ ○ ○ ○
Geometric calculation
○ △ ○ △ △ △ ○ △
HMM △ ○ △
SOM ○
24/48
![Page 25: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/25.jpg)
• Life logging
• Context-aware computing
• Sensory data for activity recognition
• Life logging with mobile devices
• Summary
Agenda
![Page 26: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/26.jpg)
Context-aware Mobile Device Trends
• Increasing demands for the multi-functional and high-performance phone
• Prospects
– Increasing demands for Smart Phone
– Decreasing basic functional phone
• Enlarging role as a computing equipment
– Internet surfing
– External storage
• Increasing investment for adding value of phone
• Digital convergence with other functionality
– Mp3 player
– Personal Media Player
– Camera & Camcoder
26/48
![Page 27: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/27.jpg)
Context-aware Phone on Media
The official newsletter of the institute for Complex Engineered Systems (CMU) Sep/Oct 2003
New Scientist, Nov 2004 MIT Sloan Management Review Fall 2004
27/48
![Page 28: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/28.jpg)
SenSay (Sensing and Say)
• Carnegie Mellon University (2003)
• A context-aware mobile phone
– Adapting to dynamically changing environmental and physiological states
– Manipulating ringer volume, vibration, and phone alerts
– Providing remote callers with the ability to communicate the urgency of their calls
– making call suggestions to users when they are idle
– Providing the caller with feedback on the current status of the Sensay user
• Sensors
– accelerometers, light, and microphones
– mounted at various points on the body
28/48
![Page 29: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/29.jpg)
SenSay (2)
• Context recognition by thresholding
29/48
![Page 30: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/30.jpg)
SenSay (3)
• Context classification based on self-organizing map
30/48
![Page 31: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/31.jpg)
MIT Reality Mining Group
• Utilizing Context application from the University of Helsinki (Raento et al., 2005)
• Capturing mobile phone usage patterns
– from one hundred people (MIT students)
– for an extended period of time
• Providing
– insight into both the users
– the ease of use of the device itself
• Method: Bayesian inference
31/48
![Page 32: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/32.jpg)
Nokia
• Nokia 7650 (2002)
– Backlight time adjustment based on proximity & light sensor
– Speaker-phone mode change by proximity sensor
• Cancellation of speaker-phone mode near ear
• Nokia 6230, 6820, 7200
– Presence-enhanced chat service
• Presenting and sharing the user’s status
– SMS messaging based on other’s status
Light SensorProximity Sensor
32/48
![Page 33: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/33.jpg)
Microsoft Research (1)
33/48
![Page 34: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/34.jpg)
Microsoft Research (2)
• Alarm method for incoming call
– Quiet ringing
• Volume down with hand touching
– Acknowledging and ignoring calls
• Calling response: Tilting to body of phone on the pocket
• Notification stop: Holding phone with hand without movement on pocket
– Target device for notifications
• Device selection for alarm: Selecting the recent device if there are several devices of user
– Vibration notification
• Vibration mode change: Holding phone for a long time
34/48
![Page 35: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/35.jpg)
Tilt Sensor & User Behavior
• Upper: forward/back tilt
• Below: left-right tilt
35/48
![Page 36: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/36.jpg)
VTT Electronics (Finland)
• Supported by Nokia
• VTT Electronics – Advanced Interaction Systems – Context Awareness
• Fuzzy/Bayesian Approach
Summary of Context Recognition Procedure(From VTT Publications 511)
Backlight level & font size adjustment
36/48
![Page 37: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/37.jpg)
VTT Technical Research Center of Finland
• Context representation Fuzzy logic
• Context reasoning Naive Bayes, Markov Chain
• Service Fuzzy Control
Context OntologyFuzzification
37/48
![Page 38: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/38.jpg)
VTT Technical Research Center of Finland (2)
• Sensor based contexts
– Bottom: high-level contexts
38/48
![Page 39: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/39.jpg)
VTT Technical Research Center of Finland (3)
• Audio based context
– Bottom: high-level context
39/48
![Page 40: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/40.jpg)
VTT Technical Research Center of Finland (4)
• Naive Bayes based classification
40/48
![Page 41: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/41.jpg)
TEA Project
• High-level context recognition
– Method: Rule & SOM http://www.teco.edu/tea
41/48
![Page 42: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/42.jpg)
• Life logging
• Context-aware computing
• Sensory data for activity recognition
• Life logging with mobile devices
• Summary
Agenda
![Page 43: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/43.jpg)
Summary
• Key components for context-aware applications (life logging)
– Sensor technology to acquire various contextual information
– Intelligence technology to model complex contexts
– Agent technology to provide seamless services
• Future requirements
– Context modeling and cognitive technologies to provide users with more advanced services
43/48
![Page 44: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/44.jpg)
DrWhatsOn (Laerhoven, 2003)
• Concept and user scenarios for context aware office PDA device
• Main focus in usability and user interface design
– Information fusion: Sensor data, device status, personal preference, schedule
– Peripheral attention: supporting appropriate information at appropriate time under prepared conditions
• Scenario for a context sensitive phone of Nokia
– A day of a Finland student Dude
44/48
![Page 45: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/45.jpg)
DrWhatsOn Scenario (1)
• [Nokia’s DrWhatsOn Concept Video, Urpo Tuomela, Nokia Research Laboratories]
– Red mark: Abstracted information from sensor
45/48
![Page 46: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/46.jpg)
DrWhatsOn Scenario (2)
46/48
![Page 47: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/47.jpg)
DrWhatsOn Scenario (3)
47/48
![Page 48: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/48.jpg)
DrWhatsOn Scenario (4)
48/48
![Page 49: Overview of Life Logging September 4, 2008 Sung-Bae Cho](https://reader036.vdocuments.us/reader036/viewer/2022081802/56649e535503460f94b48bd0/html5/thumbnails/49.jpg)
Homework #1: Due 9/9
• Survey the state-of-the-art research on the life logging at KIST, Nokia, Microsoft and MIT (10 min presentation per each institute)
– KIST
– Nokia
– Microsoft
– MIT
49/48