From Sensor Networks From Sensor Networks to Smart Environments to Smart Environments
and Social Networksand Social Networks
From Sensor Networks From Sensor Networks to Smart Environments to Smart Environments
and Social Networksand Social NetworksHamid AghajanHamid AghajanAmbient Intelligence Research LabAmbient Intelligence Research LabStanford University, USAStanford University, USA
and Social Networksand Social Networksand Social Networksand Social Networks
Ambient Intelligence Hamid Aghajan
IntroductionOutlineOutline
Introductiono Trends in technology and researcho From Smart Environments to Ambient Intelligenceo New potentials in user-centric, context-aware systems
Our labo User-centric ambient intelligence applications
Human activity analysisHuman activity analysiso Source of context in smart environments
Adaptive smart homes Well-being applicationsAdaptive smart homeso Learning from user feedback
Well being applicationso Avatars and social interactions
Meetings of the future Environment discoveryMeetings of the futureo User-centric performance evaluation
Environment discoveryo User interactions as contextual clues
Vision: Application LandscapeVision: Application LandscapeVision: Application LandscapeVision: Application Landscape
Ambient Intelligence Hamid Aghajan
Application LandscapeApplication LandscapeVision offers rich Vision offers rich contextualcontextual data:data:
3D modeling
Applications established in monitoring and surveillanceApplications established in monitoring and surveillanceUserUser--centric, homecentric, home--based application market ? based application market ? Need value propositionNeed value proposition
modeling
Urban sensing
GamingAvatars
Intelligent vehicles
Face profile: tele-presence, remote gaming
Ambient Intelligence Hamid Aghajan
Ambient IntelligenceAmbient IntelligenceRetail: interactive ads Ambience controlSeminar rooms
Intelligent light control 3D tele-presence Assisted living
Ambient Intelligence Hamid Aghajan
Ambient IntelligenceAmbient IntelligenceA li ti i i t d li iApplications in assisted living
Event and scene descriptionEvent and scene description
Monitoring patient events
Key to adoption:Key to adoption:Reliability, ease of use, pReliability, ease of use, privacy management, data security, user control (opt in/out)rivacy management, data security, user control (opt in/out)
Research & Development TrendsResearch & Development TrendsResearch & Development Trends Research & Development Trends Personalization of ServicesPersonalization of Services
Ambient Intelligence Hamid Aghajan
UserUser--centric application spacecentric application spaceTrendsTrends
pp ppp p
Smart Environments
Ambient Intelligence
user adaptation
behavior models gAdaptive, unobtrusive, Intuitive,context-aware
Sense, Interact,
perceive, interpret,
behavior models
UserContext
Social
services tailored to learned user preferences
pproject,
react, anticipate
InteractionsMulti-modal, pervasive
connectivity / tele-presencefor sharing an experience g pwith others in a flexible,
layered visual representation
Ambient Intelligence Hamid Aghajan
TrendsTrends
Smart Environments
Ambient Intelligence
user adaptation
behavior models gbehavior models
UserContext
Social
Interdisciplinary field of research:Interdisciplinary field of research:• Engineering
Interactions
Engineering• Multimodal sensing, distributed processing, networking, reasoning, interface design• Ambient communication, media convergence• User profile, behavior model, preferences, context
• Psychology:• Psychology:• Human factors, user activity, emotions, history, skills, limitations, sensitivities, privacy options
• Sociology:• Social networks, multi-user interactions, large-scale databases, behavior patterns
Ambient Intelligence Hamid Aghajan
Pervasive sensing processing and communicationTrendsTrends
Pervasive sensing, processing, and communication (widespread technology support)
Personal devices: smart phones, laptops, smart cars, personal multimedia and gaming consoles (constant interface with the user)
ConvergenceConvergence of sensor nets media and virtual data domains:of sensor nets media and virtual data domains:ConvergenceConvergence of sensor nets, media, and virtual data domains:of sensor nets, media, and virtual data domains:Mobile device as interface of user’s physical world with digital worldAlso as carrier of user profile
o Physical world: sensors, user context, activities, interactions, events, home appliances
o Digital world: Media, TV, internet, search, home automation, g , , , , ,outdoor services, virtual
oo User profileUser profile: Store and update user behavior model, habits, : Store and update user behavior model, habits, preferences in different contexts for service adaptationpreferences in different contexts for service adaptationp pp p
Ambient Intelligence Hamid Aghajan
TrendsTrendsUser • Behavior model, profile, preferences
Context-Aware
Modeling
• Inference of tasks and intentions
, p , p• Ambience control and smart home services• Social networks, pervasive communication
FusionFusionddrrrr
High-Level
Co e a eProcessing
• Human activity recognition (semantics)
Inference of tasks and intentions• User context for service provision, interrupts• Semantic labeling of things and effects
andandSemanticsSemantics
ehav
ioeh
avio
ehav
ioeh
avio
• Human pose analysis (graphical models, ti l filt h t f t )
High Level Reasoning
• Human activity recognition (semantics)• Object recognition with user interactionsSignalSignal
and and FusionFusionl l t
o B
eto
Be
l l to
Be
to B
e
Multi-Camera Vision
particle filters, heterogeneous features)• Face angle estimation, face profiles• Decision fusion for event detection
S tS tSign
alSi
gnal
Sign
alSi
gnal
Smart Camera Networks
• Occupancy counting• 3D reconstruction of events• Distributed embedded processing for real-
time human pose analysis
SystemSystemand and
SignalSignal
From
SFr
om S
From
SFr
om S
Computer Vision
Wireless Sensor
NetworksSample Methods Sample Methods and Applicationsand ApplicationsSample Methods Sample Methods and Applicationsand Applications
FFFF
Ambient Intelligence Hamid Aghajan
TrendsTrendsSmartSmart S i t t i i t t j t t ti i t
Occupancy sensing and services
Smart Smart EnvironmentsEnvironments
Sense, interact, perceive, interpret, project, react, anticipate
– Provide services based on location, event type, number of occupants
• Smart lighting for energy efficiency / Ambient lighting• Conference room analytics, smart meetings and presentations
– Smart buildings / occupancy maps:• Real-time maps for emergency management and rescue
guidance
• History data for resource utilization: space, lighting, walkways, energy, activity patterns
Ambient Intelligence Hamid Aghajan
TrendsTrendsSmartSmart S i t t i i t t j t t ti i t
Human activity
Smart Smart EnvironmentsEnvironments
Sense, interact, perceive, interpret, project, react, anticipate
– Analyze and react to pose-basedevents:
• Fall detection, gaming, HCI, gesture control, home automation, pervasive communication, smart presentations
– Reconstruct actions, expressions, 3D models, avatars
– Semantic labeling based on user interaction• User activity as context to discover the environment
Time
Ambient Intelligence Hamid Aghajan
TrendsTrends
Ambient communication
A Vision for Future: Novel Application DomainsA Vision for Future: Novel Application Domains
Ambient communicationTele-presence, pervasive communication, avatars, social networks
Well-being applicationsActivity monitoring, exercise, daily routines, patient and elderly care
Energy efficiencyOccupancy based context aware adaptive to personal modelOccupancy-based, context-aware, adaptive to personal model
MultimediaContext-aware ambience control and games, sensor-enabled social networks g ,
Smart meetingsConference room analytics, adaptive and flexible tele-conferences, speaker assistance systemsassistance systems
Ambient Intelligence Hamid Aghajan
UserUser--centric Designcentric Design
S t & Al ithSystem & Algorithm
Application
Algorithm design often ignores:• Ease of installation / use • Interruptibility of the user• Unobtrusiveness• User preferences and sensitivities
p y• User engagement level• User skills / limitations
• Privacy issues • User perception of control
Ambient Intelligence Hamid Aghajan
S t & Al ith • Real time vision and visualization
UserUser--centric Designcentric DesignSystem & Algorithm • Real-time vision and visualization
• 3D reconstruction of action• Appearance-based or avatar• Avatar repertoire database• Action mapping, expression mapping
B h i d l M d f t ti
User acceptance
Social aspectsUser preferences• Behavior models• Context• Option to override
• Modes of representation• Privacy issues
• User-friendly installation and use • Context-aware operation and responseAttributes of user-centric design:
Application
User friendly installation and use
• Self configuration / automated environment discovery
• Learning behavior models for the user
Context aware operation and response
• Privacy options in multi-user communication applications
• Adaptation to user defined preferences• Learning behavior models for the user and the environment
• Adaptation to user-defined preferences– Balance of automation and user query
Affective interfaces: Regard for user’s skills, limitations, personality characteristics, emotional state in services or interrupts, infer user’s intention
Ambient Intelligence Hamid Aghajan
Privacy ManagementPrivacy ManagementMulti-layered privacy handling approach neededMulti layered privacy handling approach needed
Need policy, convincing trust metricOpt in/out, user-centricOwners of privacy options: the elderly, family, nursing facility, insurance, legislation
Multi-modal sensingU th t t i t
Owners of privacy options: the elderly, family, nursing facility, insurance, legislation
– Use other sensors to trigger cameras upon event– Activate voice communication first to check status– Image query only possible by authorized personnel
Smart cameras
– Raw video saved locally for post-event analysis
Turn video into text in normal user stateTransform person’s gesture into:
silhouette, avatar
Designing a Practical Vision SystemDesigning a Practical Vision SystemDesigning a Practical Vision SystemDesigning a Practical Vision System
Ambient Intelligence Hamid Aghajan
Interfacing VisionInterfacing Vision
Observe → interpret → build up behavior models → react
Quantitative knowledge + Qualitative assessmentSensing Context
Processing Behavior Model
R i t tResponsiveness to events– Adapt services– Employ additional sensors– Send alerts
InteractivityB d t l ti i f i t t f
Real-time visiona necessity
– Based on gesture, location, region of interest of user
Ambient Intelligence Hamid Aghajan
Interfacing VisionInterfacing Vision
Vi i• Pose estimation, activity recognition• Face and gaze analysisVision
algorithms• Face and gaze analysis• Event reconstruction• Tracking, identification
Ambient Intelligence Hamid Aghajan
Interfacing VisionInterfacing Vision
Vi iVision algorithms
Camera Node
Task:T ki ?
Energy consumption?
Data aggregation?Distributed
Observations
OperationVision System:
Mono or stereo?Resolution?
Field-of-View?
Camera orientation?
Tracking?Counting?
Data Exchange:Type of data?Traffic load?
Multi-camera hardware & network Network topology?
Camera orientation?Placement?
Vision algorithm:Local vs. central processing
Application Requirements:Accuracy? Coverage?
Network Lifetime?
Which cameras sense?
Ambient Intelligence Hamid Aghajan
Interfacing VisionInterfacing Vision
Vi i• Pose estimation, activity recognition• Face and gaze analysisVision
algorithms• Face and gaze analysis• Event reconstruction• Tracking, identification
Operation • Local processing and centralized processing• Communication bandwidth• Latency of real-time results• Resolution in image view and time
Multi-camera hardware & network
Resolution in image view and time• Temporal alignment (synchronization)• Camera view overlaps, data redundancies• Data exchange methods
Ambient Intelligence Hamid Aghajan
Interfacing VisionInterfacing Visionbehavior models
High-level
behavior context events adaptation
high-levelreasoning
observation knowledgei t falgorithms
vision
user interface 3D model
validationg
accumulationi n t e r f a c e
pose / activity face / gaze
Vi i
Data fusion
• Pose estimation, activity recognition• Face and gaze analysisVision
algorithms• Face and gaze analysis• Event reconstruction• Tracking, identification
Operation • Local processing and centralized processing• Communication bandwidth• Latency of real-time results• Resolution in image view and time
Multi-camera hardware & network
Resolution in image view and time• Temporal alignment (synchronization)• Camera view overlaps, data redundancies• Data exchange methods
Ambient Intelligence Hamid Aghajan
Interfacing VisionInterfacing Vision
High-level
• Behavior models and user preferences• Contextual data• Knowledge-base from historic dataalgorithms • Knowledge-base from historic data• Communication mode and user availability
• Relative confidence levels in space and time• Fusion based on hybrid features
Vi i
Data fusion
• Pose estimation, activity recognition• Face and gaze analysis
• Assignment of priorities and tasks
Vision algorithms
• Face and gaze analysis• Event reconstruction• Tracking, identification
Operation • Local processing and centralized processing• Communication bandwidth• Latency of real-time results• Resolution in image view and time
Multi-camera hardware & network
Resolution in image view and time• Temporal alignment (synchronization)• Camera view overlaps, data redundancies• Data exchange methods
Ambient Intelligence Hamid Aghajan
Traditional VisionTraditional VisionGeneric features: Can become unavailable or un-interestingGeneric features: Can become unavailable or un interesting
Frame scope: Frames can have zero or misleading information value
Calibration: User-dependent function
Point-to-point accuracy metric: May not be relevant to end application
E i t d fi itiEnvironment definition: Manual, need to repeat upon movement
Fixed vision function: To avoid overloading the processor in real-time app.g p pp
Uniform value for data: Misleading data can bias results
Ambient Intelligence Hamid Aghajan
VisionVision for Ambient Intelligencefor Ambient IntelligenceGeneric features: Can become unavailable or un-interestingGeneric features: Can become unavailable or un interesting
Opportunistic features, context-driven
Frame scope: Frames can have zero or misleading information valueFlexible fusion window, reliability models
Calibration: User and environment-dependent functionObservation-based methods, application needspp
Point-to-point accuracy metric: May not be relevant to end applicationApplication-driven metric
E i t d fi itiEnvironment definition: Manual, need to repeat upon movementObservations of user interactions
Fixed vision function: To avoid overloading the processor in real-time app.g p ppAlg. switching, Task assignment (active vision)
Uniform value for data: Misleading data can bias resultsConfidence level based on content and historyConfidence level based on content and history
Enabled by two-way interfacing of vision with higher processing layers
Ambient Intelligence Hamid Aghajan
Vision and HighVision and High--Level ReasoningLevel Reasoning
Maintain knowledge base by accumulating historic dataMaintain knowledge base by accumulating historic dataGuide vision processing based on history
Compensate for imperfect vision processing output, enhance robustness
Assess relative value of information:Value of current estimates based on past interpretationsValue of low-level features for addressing a high-level taskValue of low-level features for addressing a high-level taskValue of observations made by different cameras
Task assignment to different cameras based on recent
Case study (later): Interfacing vision with inference engine for object recognition based on user interactions
results, context, and current observations
Ambient Intelligence Hamid Aghajan
Models, Context, FeedbackModels, Context, FeedbackHierarchical processing structure:Hierarchical processing structure:
Vision processing has a higher dependency on the environment and placement of cameras, semantic reasoning is more generalEach module can be designed and modified separatelyg p yThe same high-level model can be used in different environments
behavior environment events adaptationhigh-level
ireasoningobservation validation
knowledgeaccumulation
c o n t e x t i n t e r f a c e
vision
motion pattern 3D modelpose / activity face / gaze
vision hardware & network
Ambient Intelligence Hamid Aghajan
Context in Vision ProcessingContext in Vision ProcessingSpatiotemporal constraintsGeographic informationOther modalitiesMulti-camera networkCamera priors
EnvironmentalContext
StaticContext
Domain knowledgeDomain knowledgeHigh-level reasoning
Context Context
User-centric Context
Dynamic Context
User interfaceUser interfaceUser location and activityUser behavior modelUser preferencesSocial settinggRepresentation mode
K. Henricksen, J. Indulska, A. Rakotonirainy, “Modeling context information in pervasive computing systems”, in Proc. of the First Int. Conference on Pervasive Computing, 2002.
Ambient Intelligence Hamid Aghajan
Models, Context, FeedbackModels, Context, Feedback
camera network model(geometry, topology, roles, priors, confidence levels)
environment discovery(objects, temporal/affine
usage relations)
user model(routines, preferences,
emotional state)
abnormal event detection(activities, routines, accident,
control zones, crowds)
behavior modelscontext
semantic-level context
behavior environment events adaptationhigh-level
i
context
reasoningobservation validation
knowledgeaccumulation
c o n t e x t i n t e r f a c e signal-level
context
vision
motion pattern 3D modelpose / activity face / gaze
context
system levelvision
hardware & network
system-levelcontext
Ambient IntelligenceAmbient IntelligenceAmbient IntelligenceAmbient IntelligenceCase Studies in UserCase Studies in User--centric Application Designcentric Application Design
Ambient Intelligence Hamid Aghajan
UserUser--centric Contextcentric ContextUser behavior model– Inference of intention– Detection of abnormal events– Emotional state (affective computing)– Demographical information
f– User profiles in personal devices: • Preferences, skills, special needs, knowledge, expertise, limitations(mobile phones, laptops, cars, smart rooms, multimedia / game consoles)
How can this help?– Analyze / anticipate events using knowledge base
Ch ll– Challenges:– Method to summarize / abstract sensed input into behavior models– How to query and deduct from the model in real-time responsesq y p– Modeling / tracking change in user behavior– Dealing with ambiguity or uncertainty in captured user information
Ambient Intelligence Hamid Aghajan
UserUser--centric Contextcentric ContextUser preferencesp– Feedback
• Explicit (system-directed confirmation)• Implicit (user-directed confirmation)
– Interrupts: Provide the right service / data at the right time (proactive )g ( )• System-directed initiative (novice user)• Mixed initiative (experienced user)
How can this help?– Turn generic settings into adaptive services
User’s perception of being in control of the system– User s perception of being in control of the system
– Challenges:– What is proper level & type of user query?p p yp q y– Being proactive while keeping user interruption at minimum
Ambient Intelligence Hamid Aghajan
UserUser--centric Contextcentric ContextRepresentation modep– Communication channel
• Visual / non-visual options• Raw video, voxel, shadow, avatar
How can this help?– Offer multiple options to preserve user privacy in visual communication
Employ suitable vision processing functions– Employ suitable vision processing functions
Ambient Intelligence Hamid Aghajan
Our LabOur LabWSNLWSNL:: WWirelessireless SSensorensor NNetworks Labetworks Lab Multi-camera algorithms:WSNLWSNL: : WWireless ireless SSensor ensor NNetworks Labetworks Lab
• Occupancy sensing• Face detection• Best view selection• Vision-based data and
decision fusion• Distributed processing
WSNL.Stanford.edu
Ambient Intelligence Hamid Aghajan
A I RA I R Lab:Lab: AAmbientmbient IIntelligencentelligence RResearch Labesearch LabOur LabOur Lab
AIRlab.Stanford.eduA I RA I R Lab: Lab: AAmbient mbient IIntelligence ntelligence RResearch Labesearch Lab
AdaptiveAdaptive, , contextcontext--awareawareapplications:applications:
• Activity classification• Ambience control• User behavior model• User feedback, HCI,• Energy efficiency• Social networks• Well-being, connectedness
Ambient Intelligence Hamid Aghajan
User centric application design:Ambient IntelligenceAmbient Intelligence
User-centric application design:Ambient communication
Tele-presence and modes of representationp p
Adaptive smart homesLearning from user feedback
E i itExercise monitorAvatars and real-time social interactions
Meetings of the future / Speaker assistanceg pUser-centric performance evaluation
Environment and object discoveryUser interactions as source of contextUser interactions as source of context
TeleTele--Presence, Human Pose AnalysisPresence, Human Pose AnalysisTeleTele Presence, Human Pose AnalysisPresence, Human Pose AnalysisAvatars, Modes of RepresentationAvatars, Modes of Representation
Ambient Intelligence Hamid Aghajan
M lti 3D t ti V l
Ambient Communication / TeleAmbient Communication / Tele--PresencePresenceMulti-camera 3D reconstruction - Voxels
Tommi Maataa
http://wsnl.stanford.edu/videos/examples/visualhull_cmp.avi
http://wsnl.stanford.edu/videos/examples/voxel_skin.mpg
WSNL - Stanford40
Ambient Intelligence Hamid Aghajan
Ambient Communication / TeleAmbient Communication / Tele--PresencePresenceCustomized visualization:– User availability mode + avatar preferences:
B t i ( id )• Best camera view (video)• 3D shadow• Avatar replicating user’s gesture• Avatar with pre-defined gesture
– Change the 3D view based on viewer’s position
Applications in home-to-home communication and remote meetings
T. Määttä, A. Härmä, and H. Aghajan, “Home-to-Tome Communication Using 3D Shadows“, Immerscom 2009.
Ambient Intelligence Hamid Aghajan
Multi-camera pose analysis - Avatars
Human Pose AnalysisHuman Pose AnalysisMulti-camera pose analysis - Avatars
http://wsnl.stanford.edu/videos/gesture/combine1.avi
http://wsnl.stanford.edu/videos/gesture/rotate2.avi http://wsnl.stanford.edu/videos/gesture/jogging1.avi
http://wsnl.stanford.edu/videos/gesture/combine1.avi
WSNL - Stanfordhttp://wsnl.stanford.edu/videos/gesture/pang.avi
Ambient Intelligence Hamid Aghajan
Multi-camera pose analysis - Avatars
Human Pose AnalysisHuman Pose AnalysisMulti-camera pose analysis - Avatars
http://wsnl.stanford.edu/videos/gesture/realtime1.avi
43
p g
C. Wu and H. Aghajan, “Real-Time Human Pose Estimation: A Case Study in Algorithm Design for Smart Camera Networks“, Proceedings of the IEEE, Nov. 2008.
Ambient Intelligence Hamid Aghajan
A user-centric system needs to support
Ambient Communication / TeleAmbient Communication / Tele--PresencePresenceA user centric system needs to support
a variety of visualization modescommunication
best view selection
3D reconstruction
avatarmapping textual report
visualization
c o n t e x t i n t e r f a c e
observation validation
user preferences and availability
3D modelmotion patternface / gazepose / activity
vision
vision hardware & networkSwitching between different vision
algorithms based on user’s choice
WellWell--Being ApplicationsBeing ApplicationsWellWell Being ApplicationsBeing ApplicationsAvatars, Social Networks, Health ReportsAvatars, Social Networks, Health Reports
Ambient Intelligence Hamid Aghajan
WellWell--Being ApplicationsBeing ApplicationsSmart Homes: Health and wellSmart Homes: Health and well--beingbeing
Input from heterogeneous sensorsEvaluation: query the user, confirm with an expert, demographic comparisons
Smart Homes: Health and wellSmart Homes: Health and well--beingbeing
comparisonsMonitoring patient events
Ambient Intelligence Hamid Aghajan
Exercise at Home?Exercise at Home?WellWell--Being ApplicationsBeing Applications
Exercise at Home?Exercise at Home?Benefits:– Comfort and privacy superior to gym– Comfort and privacy superior to gym– Flexibility in time User preferences in the
avatar domain
Disadvantages:– Absence of a personal coach or trainer– Lack of social motivation factors
(social atmosphere, competition)
Staying with exercise plan more likely with:likely with:– Measurements and record keeping– Social connectedness
Ambient Intelligence Hamid Aghajan
Home Exercise MonitorHome Exercise MonitorWhat technology can offer– Instant measurements, instructions, and feedback
(Vision and HCI)– Interactive links with trainer
(C i ti d I t t)(Communication and Internet)– Different visualization options
(Computer Graphics)(Computer Graphics)– Virtual games with peers
(Social Networks)( )– User history and knowledge base
(Databases and Artificial Intelligence)
Also use of Psychology to assess user acceptance
Ambient Intelligence Hamid Aghajan
Home Exercise MonitorHome Exercise MonitorWhat can this technology do?What can this technology do?– An exercise system at home in front of a TV equipped with a camera– The camera will project user’s image on the TV screen while you exercise
The image can be presented to the user in three ways: – Mirror image– “Body shape” avatar– Skeletal “stick figure” avatar
I b h d ith thImage can be shared with others– Fitness coach– Exercise buddies and friends– Other users in social network
The fitness coach could provide instructions, guidance, and corrective f db kfeedback– User may also be able to see the exercising movements of a coach, friends, and
other users through this system
Ambient Intelligence Hamid Aghajan
Home Exercise MonitorHome Exercise Monitor
Requirements for acceptance:Requirements for acceptance:
– Real-time gesture recognition• Adaptation to user silhouette set
– Social connectedness• Live links with coach, peers
– User preferencesp• Instructions and feedback• Options on avatars
Ambient Intelligence Hamid Aghajan
Application concepts:Exercise MonitorExercise Monitor
Application concepts:• Monitor, measure home exercise
– Instant feedback, instructions, progress report• Connection with a coach• Connection with a coach• Social networking
– Shared experience with peers via avatars
User preferences • Instructions and feedback
Instructions
Measurements
Progress report
Coach
FeedbackAvatar Social
Network
i t ti i tiinteractive communicative• Social connectedness• Group games
Ambient Intelligence Hamid Aghajan
Home Exercise MonitorHome Exercise MonitorExercise MonitorExercise Monitor
Home Exercise MonitorHome Exercise Monitor
Use of context in silhouette search:• Prior knowledge of the exercise routine
Observed user silhouettes
• Prior knowledge of the exercise routine• Pose in previous frames
Reduce size of search regionsilhouettes
L b l d ( d b d j i t )
• Update silhouette bank with acquired user’s silhouettes
Adapt to user’s appearance over timeLabeled (pose and body joints) generic silhouette set
J. Cui, Y. Aghajan, J. Lacroix, A. van Halteren, H. Aghajan, “Exercising at home: Real-time interaction and experience sharing using avatars”, Journal of Entertainment Computing, Dec. 2009.
Ambient Intelligence Hamid Aghajan
Exercise MonitorExercise Monitor
http://wsnl.stanford.edu/videos/exercise/Record5.avi http://wsnl.stanford.edu/videos/exercise/Record2av.avi
Human Activity LabelingHuman Activity LabelingHuman Activity LabelingHuman Activity LabelingSource of Context for Smart Home ServicesSource of Context for Smart Home Services
Ambient Intelligence Hamid Aghajan
Human ActivityHuman Activity
Home environment is complex for activity analysis with visionanalysis with vision– Previous work usually have clean background or
well-positioned cameras
Fine-level activity classification is needed for ambient intelligence– Activities in the living room, kitchen, and study
roomroom
Ambient Intelligence Hamid Aghajan
Human ActivityHuman ActivityCan serve as context for providing services in smart homes
0m
Philips Research
Home Lab
p g4.
30
9.42m
5.12
m
http://wsnl.stanford.edu/videos/envdisc/views_DiningTable1_T.avi
http://wsnl.stanford.edu/videos/envdisc/event_DiningTable1_T.avi
User Location and User Location and ActivityActivity
Ambient Intelligence Hamid Aghajan
Human ActivityHuman ActivityHierarchical classification of activitiesHierarchical classification of activities
Location
Pose-based activities
Motion-based activities
Ambient Intelligence Hamid Aghajan
Human ActivityHuman Activity1ta − ta 1ta + 2ta +
1ty − ty 1ty + 2ty +
Conditional random field (CRF) Classes: standing, sitting, lyingFeatures: height, aspect ratio
• Secondary activities:
• Standing pose + Global motion “Walking”g p g
• Sitting pose + Dining table + Local motion “Eating”
• Sitting pose + Living room + Gazing “Watching TV”
Ambient Intelligence Hamid Aghajan
Human ActivityHuman ActivityMotion-based activities
Location Activity
Kitchen Cutting, scrambling, vacuuming, other
Motion-based activities
Dining table Eating, vacuuming, other
Living room Reading, vacuuming, other
S d T i di i hStudy room Typing, reading, vacuuming, other
Ambient Intelligence Hamid Aghajan
Human ActivityHuman ActivityBag-of-features approachBag-of-features approach
Extract space-time interest points
descriptorscodebook, size = N
K-means clustering
episode,t seconds SVM
feature vector
activity classes
histogram feature vector for the episode
Ambient Intelligence Hamid Aghajan
Human ActivityHuman ActivityMulti-camera fusion
Independent classification
Combined-view fusion (concatenate feature vectors)
Mixed-view fusion (mix feature vectors)Mixed-view fusion (mix feature vectors)
C. Wu, A. Khalili, and H. Aghajan, “Multiview Activity Recognition in Smart Homes with Spatiotemporal Features“, ICDSC 2010.
Use of space-time bag-of-features
Ambient Intelligence Hamid Aghajan
Human ActivityHuman Activity
http://airlab.stanford.edu/videos/activity/airlab_activities_xvid.avi
Amir Khalili, Chen Wu, and H. Aghajan, “Hierarchical Preference Learning for Light Control from User Feedback“, CVPR 2010 Workshop on Human Communicative Behavior Analysis.
Ambient Intelligence Hamid Aghajan
Behavior ModelingBehavior Modeling
Modeling transitions between location contexts
Dining
Kitchen
StudyRoom
gTable
LivingRoom
µ(Activiy{state}, Location),
δ(Living Room, Study Room)
Ambient Intelligence Hamid Aghajan
Behavior ModelingBehavior Modeling
Storyline of the experimentsDining TableLiving Room Study RoomKitchen
Sample daily activity sequences
Dining TableLiving Room Study RoomKitchen
Watching Tv(9)
Studying(14)
Cutting(19)
Vacuuming(8)
Scrambling(14)
Cutting(15)
Scrambling(12)
Eating(32)
Studying(14)
Watching Tv
Scrambling(25)
Eating(9)
Watching Tv(8)
Studying(22)
Watching Tv(24)
Vacuuming(2)
Typing(6)
Studying(4)
Watching Tv(19)
Ambient Intelligence Hamid Aghajan
Smart Homes: Prompting ServiceSmart Homes: Prompting Service
WellWell--Being ApplicationsBeing ApplicationsPRIME: Prompting Interactive Mobile Engagement systemTask and activity inference and user behavior modelingMobile device:
Smart Homes: Prompting ServiceSmart Homes: Prompting Service
Mobile device:Interactive prompting and instructions / tracking of responseCarrier of user behavior modelMaintain long term report on lifestyle and cognitive healthMaintain long-term report on lifestyle and cognitive health
Adaptive Smart HomesAdaptive Smart HomesAdaptive Smart HomesAdaptive Smart HomesLearning from User FeedbackLearning from User Feedback
Ambient Intelligence Hamid Aghajan
Smart HomesSmart HomesUser comfort / ambient lighting and multimedia
Goal:• Minimize need for explicit user interaction / query• Minimize cost of service (energy)• Minimize cost of service (energy)
User-centric design:Accumulate knowledge about user lifestyle, preferences in service parameters
Ambient Intelligence Hamid Aghajan
Occupancy-based context-awareSmart HomesSmart Homes
Occupancy-based, context-awareContextual data: Real-time activity, history / behavior model
Adaptation to user settings and preferences
Monitoring patient events Measuring exercise routinesMonitoring patient events Measuring exercise routines
Energy-efficient light control based on user location and activity
Ambient Intelligence Hamid Aghajan
Occupancy reasoning for dynamic lighting control:
Smart HomesSmart HomesOccupancy reasoning for dynamic lighting control:
– Adjust light level based on location and activity of the user– Minimize energy use while keeping the intensity (utility function) at comfort level
Smart cameras for user sensing:– Simple in-camera bounding box extraction– Low-bandwidth communication Light intensity Low bandwidth communication g y
utility functions for different poses
http://wsnl.stanford.edu/videos/occupancy/ video_huang_intensity.avi
Ambient Intelligence Hamid Aghajan
Smart Homes Smart Homes –– Ambience ControlAmbience Control
Making the system user-centric:Learn and adapt to user preferences over timeLearn and adapt to user preferences over time
User-centric design choices:F ll t t d t OR th h– Fully automated system OR query the user on every change
– Explicit input / feedback from the user OR implicit feedback
Learn user’s preference (user modeling)Learn user’s preference (user modeling)– Learn from explicit / implicit feedback– Explicit: Reward / penalty direct scoring optionp p y g p– Implicit: changing the offered service, leaving the area
Add other services: background music, ambient lighting
Ambient Intelligence Hamid Aghajan
Adaptive Smart HomesAdaptive Smart HomesG lGoal:• Adaptation to user preferences• Activity-level granularity
User Location, Activity
Context Module
Activity level granularity• Minimize energy usage
ActivityOBSERVER
Method (user-centric design):U P fil
knowledge base
Adaptation Module Environment
User Feedback( g )
• Measure real-time user context• Learn user preferences via feedback• Develop behavior model for user
User ProfileMODEL
FeedbackINTERFACE
• Develop behavior model for user• Contextual data:
Real-time activity, location, time, user profileAmbience
ControlACTOR
Reinforcement Learning
LEARNER
Energy Cost
Ambient Intelligence Hamid Aghajan
Adaptive Smart HomesAdaptive Smart Homes
LOCATION
User Location, Activity
Context Module
ACTIVITY
ActivityOBSERVER
Light intensity utility
User-adapted Utility Functions
U P fil
knowledge base
Adaptation Module Environment
User Feedbackg y y
functions for different contexts
User ProfileMODEL
FeedbackINTERFACE
AmbienceControl
ACTOR
Reinforcement Learning
LEARNER
Amir Khalili, Chen Wu, and H. Aghajan, “Hierarchical Preference Learning for Light Control from User Feedback“, CVPR 2010 Workshop on Human Communicative Behavior Analysis.
Energy Cost
Ambient Intelligence Hamid Aghajan
Convergence of different services / heterogeneous sources (internet,
Smart Homes and Media ConvergenceSmart Homes and Media ConvergenceConvergence of different services / heterogeneous sources (internet, TV, gaming, social networks
Mobile device: content search delivery point for media access homeMobile device: content search, delivery point for media, access home sensor network (from inside home and outside), carry user profile
Ambience setting based on context and user preferencesAmbience setting based on context and user preferences
Ambient Intelligence Hamid Aghajan
Smart Homes Smart Homes –– Ambience ControlAmbience ControlService Features ValuesService Features Values
AmbientGenre
Blues, Christian, Classical, Country, Dance, Electronic, Folk, Hip_Hop, Holiday, Jazz, Latin, New_Age, Oldies, Pop, R_n_B, Reggae, Rock
Music Mood Calm, Exciting, Happy, Neutral, Sad
Volume Silence, Low, Medium, High
P ttDigital Window
Pattern Cloudy, Starry, Fireworks, Solid
Color Purple, Blue, Gray, Green, Orange, Red, Yellow
Brightness Bright, Medium, Dark, Off
AmbientMusic (‘Genre value’, ’Mood value’, ’Volume value’)
DigitalWindow(‘Pattern value’, ’Color value’, ’Brightness value’)
Ambient Intelligence Hamid Aghajan
Smart Homes Smart Homes –– Ambience ControlAmbience ControlProvide services to user based on context
Action according to preferenceMutual discovery (element of randomness)
– State = (time, location, activity)– Service: ambient music, digital window
Environment(Home and User)
Sensory
(element of randomness)
Sensory Motor ActorUser preference
Attribute = ([classical, nostalgic],[dim, starry])
Reinforcement Reinforcement LearningLearning
Ambient music Digital window
Q LearningState (time, loc, activity)
User feedback User Modelaction a(t)
s(t)
Ambient music Digital window
User feedbackLike? / dislike?
QoS – Reward/Penalty R(t+1) 1s
2s
1l 2l 3l nlKsf
actio
n
ates
M satis
(light setting)l
Sta
Ambient Intelligence Hamid Aghajan
Adapt to preference changeSmart Homes Smart Homes –– Ambience ControlAmbience Control
Adapt to preference change– Online and continuous learning (preferences may change)
– Role of randomness:• Allow mutual discovery (exploration vs. exploitation)• Preserve novelty of service • Mimic a human-like behavior (reliable but not 100% predictable)Mimic a human like behavior (reliable but not 100% predictable)
A. Khalili, C. Wu, and H. Aghajan, “Autonomous Learning of User’s Preference of Music and Light Services in Smart Home Applications“, Behavior Monitoring and Interpretation Workshop at German AI Conf, Sept. 2009.
Meetings of The FutureMeetings of The FutureMeetings of The FutureMeetings of The FutureUser Experience, Analytics,User Experience, Analytics,P li d P f R tP li d P f R tPersonalized Performance ReportsPersonalized Performance Reports
Ambient Intelligence Hamid Aghajan
Meetings of the FutureMeetings of the Future
User Experience Analytics
Speaker • Non-laser pointer• Highlighting and actuation• Gesture control of content
Learning-based score• Gaze distribution
B d l h d
Real-Time Gesture Speaker Report
Assistance • Gesture control of content• Voice keywords commands• Smart documents
• Body language, hands• Walking pattern• Repetitive moves, habits
Meeting• Best camera view selection• Faces of the sides of Q/A
• Faces and interactions• Social signals, body language
View Selection Meeting Report
Meeting Manager • Gesture-based view control
• Geometry, places, gazes• Shared content control
• Participation balance, flow• Models and roles• Personal report card
Ambient Intelligence Hamid Aghajan
Speaker Assistance SystemSpeaker Assistance System
User Experience Analytics
Speaker • Non-laser pointer• Highlighting and actuation• Gesture control of content
Learning-based score• Gaze distribution
B d l h d
Real-Time Gesture Speaker Report
Assistance • Gesture control of content• Voice keywords commands• Smart documents
• Body language, hands• Walking pattern• Repetitive moves, habits
Ambient Intelligence Hamid Aghajan
Speaker Assistance SystemSpeaker Assistance System
User feedback
User behavior adaptation
User feedback
Real-time gesture
User behavior model User
preferencesSpeaker report
Smart presentations
UserUser--centric centric behaviorbehavior
Ambient Intelligence Hamid Aghajan
(Face angle + pointing gesture) Intention interpretation Algorithm switchingSpeaker Assistance SystemSpeaker Assistance System
Adaptation:User System
Context( g p g g ) p g g
Calibration-free system: User System
http://wsnl.stanford.edu/videos/gesture/pdemo.avi
“Non-Laser Pointer”
C. Wu and H. Aghajan, “Context-aware Gesture Analysis for Speaker HCI“, Workshop on Artificial Intelligence Techniques for Ambient Intelligence (AITAmI), co-located with ECAI, July 2008.
Ambient Intelligence Hamid Aghajan
Speaker Assistance SystemSpeaker Assistance System
User feedback
User behavior adaptation
User feedback
Real-time gesture
User behavior model User
preferencesSpeaker report
Smart presentations
UserUser--centric centric behaviorbehavior
Ambient Intelligence Hamid Aghajan
Speaker Assistance SystemSpeaker Assistance System
Measurement pool
ObservationWalking Screen gaze
Face orientation distribution
Use of hands
Standing pose
Walking pattern
Hand gestures Sky gaze
Voice level / tone
Other repetitive gestures / habits
Speaker Report• Gestures and body language• Desired / undesired habits• Comparison with historic data
Ambient Intelligence Hamid Aghajan
Speaker Assistance SystemSpeaker Assistance System
Measurement pool
History Observation Smart Document
Face orientation distribution
Use of handsUser behavior model
• Emphasis points
Document context
Standing pose
Walking pattern
• History of scored (un)desired habits
p p• Transitions• Interactions with
audience
Voice level / tone
Other repetitive gestures / habits
Speaker Report• Gestures and body language• Desired / undesired habits• Comparison with historic data
Ambient Intelligence Hamid Aghajan
A t E l
Speaker Assistance SystemSpeaker Assistance SystemAspect Examples
ContentTopic; Logical organization;
Presentation slides (text picture animation)Presentation slides (text, picture, animation)
SpeechEmphasis; Repetition; Confidence;
Emotion; Pitch variation;
VisualGesture (use of hands); Posture; Movement;
Eye contact / gaze; Facial expression
Focusing on the visual cues:Focusing on the visual cues:• Head trajectory encodes global movement
• Face/head orientation encodes eye contactFace/head orientation encodes eye contact
• Near-body movement encodes use of hand
Ambient Intelligence Hamid Aghajan
Use of face orientation to evaluate:
Speaker Assistance SystemSpeaker Assistance SystemUse of face orientation to evaluate:– Distribution of gaze towards the audience– Frequency and duration of looking at display– Sky gazing occurrences– Looking at the floor
WSNL - StanfordSemantic labeling of clusters
Ambient Intelligence Hamid Aghajan
Speaker Assistance SystemSpeaker Assistance System
• User-centric design:• User as definer of a value metric:• User as definer of a value metric:
– Intuitive, high-level input by user
– Self evaluation system: • Allow subjective metric (user definitions of desired / undesired)
• Personal system (privacy)
• Training of a scoring system based on user’s own metric
Ambient Intelligence Hamid Aghajan
Speaker Assistance SystemSpeaker Assistance SystemUser-centric learning of performance scores:– Training set is created from user’s lectures– User scores own performance in segmented clips (episodes)
• Intuitive, high-level scoring scheme
– The scoring function is trained based on user’s scores (e.g. 1 – 10)
Interactive Machine Learning
Scored Episodes
User-specific scoring Processing Unit
Vision Processing
Sensing / recording
EpisodesLearning Scoring
Function
Ambient Intelligence Hamid Aghajan
Vision processing:Speaker Assistance SystemSpeaker Assistance System
p g– Face orientation classification
• A head detector localizes face• Speaker’s gaze angle is classified into 6 angles
0.2
0.4- Head orientation histogram
– Global movement:• Based on KF head tracker
150
200
250
300
x100
110
120
130
140
150
y
01 2 3 4 5 6
ased o ead t ac e– Amount of movement– Span of movement
– Hand movement:0 100 200 300 400 500 600 700 800 900
0
50
100
150
time
x
0 50 100 150 200 250 30050
60
70
80
90
100
x
y
• Local motion inside the ‘body box’– Frequency of motion near the body– Span of motion near the body– Frequency of motion far from the body
An 11-D feature vector is produced and used in score trainingT. Gao, C. Wu, H. Aghajan, “User-centric Speaker Report: Ranking-based Effectiveness Evaluation and Feedback”, ICCV 2009 THEMIS Workshop.
Ambient Intelligence Hamid Aghajan
User Experience Analytics
Meetings of the FutureMeetings of the FutureUser Experience Analytics
• Best camera view selectionF f th id f Q/A
• Faces and interactionsSocial signals body language
View Selection Meeting Report
Meeting Manager
• Faces of the sides of Q/A • Gesture-based view control• Geometry, places, gazes• Shared content control
• Social signals, body language• Participation balance, flow• Models and roles• Personal report card
Ambient Intelligence Hamid Aghajan
Meetings of the FutureMeetings of the Future
Improve experience of remote participantsremote participants
Ambient Intelligence Hamid Aghajan
Which view to stream to remote participants?
Meetings of the FutureMeetings of the FutureWhich view to stream to remote participants?
Speaker’s gesture, body languageInteraction with the audienceAd t t f f th t ti i tAdapt to preferences of the remote participant
Ambient Intelligence Hamid Aghajan
Signals Models Roles
Meetings of the FutureMeetings of the FutureSignals
• Gaze direction• Eye contact
Models
• Interaction• Eye contact
Roles
• Driver / follower• Info seeker / giver
Personal Features Interpersonal Features Inferred Features
• Eye contact• Nodding• Posture• Use of hands• Body language
Eye contact• Dialogue• Mimicry• Influence• Consistency
D i
Info seeker / giver• Supporter / neutral /
challenger
• Walking • Dominance
• Statistical, anonymized
Team Report• Confidential
Personal Report
• Only pre-selected data types• User-selected features
Meeting Report• Synergetic / polarized / boring / balanced in interaction• Uniform participation / apathy• Adequate discussion / group-thinking
g p
Inference via User InteractionsInference via User InteractionsInference via User InteractionsInference via User InteractionsUser Activity as Source of ContextUser Activity as Source of Context
Ambient Intelligence Hamid Aghajan
Case Study: Object RecognitionCase Study: Object Recognition
Environment and object discovery in smart homes based on user interactionsbased on user interactions
Interfacing computer vision with high-level inferenceContext-driven data and decision fusion
Camera priors multi camera fusionCamera priors, multi-camera fusionUser location and activityCommonsense-based logic
Framework for probabilistic reasoning data fusion feedback toFramework for probabilistic reasoning, data fusion, feedback to vision, user-centric learning methods
Ambient Intelligence Hamid Aghajan
Object recognition remains a challenging vision problemEnvironment DiscoveryEnvironment Discovery
Object recognition remains a challenging vision problem– Manual:
• Labeling of objects in view at camera deploymentAppearance based / model based:– Appearance-based / model-based:
• Large training set, may fail with variations, update with new object designs• Many objects may not be relevant to user’s lifestyle and routines
Th l bj t d fi th bj t’ ti l• The way a user employs an object defines the object’s semantic role
A user-centric approach:• User activity as context to discover the environment• User activity as context to discover the environment• User gives semantic definition to each object the way s/he uses it• User behavior model as a by-product
Ambient Intelligence Hamid Aghajan
Simple cases:
Direct (Simple) InferencesDirect (Simple) InferencesSimple cases: – Based on user interactivity over time, discover:
Where the floor isWhere the doors areWhere the cameras are w.r.t. each other
These can be established using short-term observationsHow to develop a structured reasoning system?How to develop a structured reasoning system?• Which interacts with, queries from, and guides the vision module• And which considers the challenges in vision processing
Ambient Intelligence Hamid Aghajan
Structuring A Knowledge BaseStructuring A Knowledge BaseSemantic labeling of objects based on user interaction:
– Three logic types:
1. Direct logic Appear/disappear
SofaSitting + lyingUser
DoorUser
2 Sequential-actions logic Take object Put object EatingUser
Appear/disappear DoorUser
2. Sequential actions logicout in + wait Eating
Fridge Microwave
User
Kitchen context
3. Concurrent-actions logic Sitting & WatchingSofa
User
Kitchen context
TVLiving room context
Ambient Intelligence Hamid Aghajan
Structuring A Knowledge BaseStructuring A Knowledge BaseUser
Watching gestureDropping action
posepose
Sitting action
Sequential Object behaviorConcurrent
Object behavior
pose
Sofa
TV
Object behavior
Chair
Concurrent
Reclined pose
Coffee tableRules embed functional models for objects
Concurrent
Rules embed functional models for objectswatching: :TVlying: :sofasitting: chair, sofa
User activity used to give weight to rulesuser never reclined on sofa before
seen reclined is perceived as a new event
99 WSNL - Stanford
Spatial / temporal relationshipsobjects used concurrently - sofa: :TV: :coffee tableobjects used sequentially - fridge: :microwave
Behavior models of the userdrops on sofa to watch evening news on TV
Ambient Intelligence Hamid Aghajan
Fusion and Inference FlowFusion and Inference FlowUser activities
Walking(t), Sitting(t), Gazing(t), HandMotion(t), …InKitchen(t), InLivingRoom(t), …
0.99 Walking(t) ^ Floor(t-1) -> Floor(t) Domain knowledge
( ), g ( ),
g( ) ( ) ( )0.5 Walking(t) ^ Chair(t-1) -> Floor(t)0.9 Sitting(t) ^ NA(t-1) -> Chair(t) v Sofa(t)0.9 Lying(t) ^ Chair(t-1) -> Sofa(t)0.1 Lying(t) ^ Floor(t-1) -> Sofa(t)
Domain knowledge in MLN
y g( ) ( ) ( )………
Grounding weight
Markov Random Field
Feature function
Pr(query)Number of true groundings of a formula
Ambient Intelligence Hamid Aghajan
Fusion and Inference FlowFusion and Inference FlowMarkov Logic Network: Make the constraints soft by assigning weights to themg y g g g
0.99 Walking(t) ^ Floor(t-1) -> Floor(t) 0.5 Walking(t) ^ Chair(t-1) -> Floor(t)0 9 Sitting(t) ^ NA(t-1) -> Chair(t) v Sofa(t)0.9 Sitting(t) NA(t-1) -> Chair(t) v Sofa(t)0.9 Lying(t) ^ Chair(t-1) -> Sofa(t)0.1 Lying(t) ^ Floor(t-1) -> Sofa(t)
………
MLN a model to handle:• Imperfect visual output• Rules with weights
C. Wu and H. Aghajan, “Using Context with Statistical Relational Models – Object Recognition from Observing User Activity in Home Environment”, ICMI-MLMI Workshop on Use of Context in Vision Processing, Nov. 2009.
Ambient Intelligence Hamid Aghajan
Markov Logic NetworkMarkov Logic NetworkRules (constraints): encode domain knowledge in first-order logic syntaxWeights Weights
(soft constraints)(soft constraints)
Walk FloorSit Chair or SofaLie Sofa
1.00.50 8
Examples of Direct logic
Example of Concurrent-actions logicLie Sofa
Stand still in kitchen (preparing food)
Gaze TV0.80.6
0.6
Example of Sequential-actions logic
then Enter living room then Sit with hand motion (eating) Dining Table
UncertaintyDomain knowledge
Observations (vision output)Grounding (probabilities dj t d ith b ti Observations (vision output)
Walk (t) with Probability 0.7adjusted with observations over time)
Ambient Intelligence Hamid Aghajan
Fusion and Inference FlowFusion and Inference Flow
Semantic processing
Event processingp g
Individual
Central unit Vision
processing
CAM 1 CAM 2 CAM n
cameras
Ambient Intelligence Hamid Aghajan
Local ProcessingLocal Processing• Simple vision processing employed in each view
• Possible flaws in detectionPossible flaws in detection
• Frame based probability of pose between views• Frame-based probability of pose between views
• Different cameras may see the user concurrently• Different cameras may see the user concurrently• Decision fusion based on declared pose
Ambient Intelligence Hamid Aghajan
MappingMapping• Map observed poses to spatial grid (e.g. walking)
• Evolve probabilities for each grid point over time with observations
WSNL - Stanford
Ambient Intelligence Hamid Aghajan
Build up probability map based on pose at location
MappingMappingu d up p obab ty ap based o pose at ocat o
Walk => FloorSit => Chair or Sofa
Direct relations:
Recline => Sofa
Gazing => TV
Spatial relations:
• Can exploit spatial relationship between grid points
Ambient Intelligence Hamid Aghajan
Context in ProcessingContext in Processing• Features used in pose analysis:
• Bounding box (size, aspect ratio), calibrated height
Camera priors as context:• Range of acceptable foreground bounding boxes
Two SVM classifiers to learn range of too large and too small bounding boxesU dj t f f i t t d l b li f t i i t
• Performance of past pose classification reports- Only cameras 2 & 5 can see lying pose- Camera 2 never reports lying correctly
C 5 h 34% ll 63% i i
Use adjacent frames for semi-automated labeling of training set
- Camera 5 has 34% recall, 63% precision
Rows: ground truthColumns: estimates
Ambient Intelligence Hamid Aghajan
For each grid point:
Living Room ExampleLiving Room ExampleFor each grid point:
Evolve probabilities over time with observations
Time
Ambient Intelligence Hamid Aghajan
(1) SequentialSequential--actionsactions logiclogicKitchen ExampleKitchen Example
Stand still in kitchen (preparing food) then Enter living room then Sit with hand motion (eating) Dining Table(2)Sit ith h d ti i li i ( ti )
qq gg
1
Sit with hand motion in living room (eating)then Enter kitchenthen Stand still in kitchen (putting away plates) Sink
1
2Living Room
Dining table
Fridge
0.39
0.36
0.38
6
5.40m
3
5
6
Workspace
Sink0.28
0.30 0.38 0.44
4
Kitchen0.380.38
4
C. Wu and H. Aghajan, “User-centric Environment Discovery with Camera Networks in Smart Homes”, IEEE Trans. on Systems, Man, and Cybernetics Part A, 2010.
Ambient Intelligence Hamid Aghajan
E i t DiE i t DiFloor Chair S fDirect logic example
Living Room ExampleLiving Room ExampleEnvironment DiscoveryEnvironment DiscoveryFloor Chair SofaDirect logic example
More chair area discovered
Possible TV locations Some possible TV locations are ruled out as more activities are observed
Concurrent-actions logicexample
Ambient Intelligence Hamid Aghajan
C t k
Next StepsNext StepsCurrent work:– Incorporate explicit user query and feedback
Walk FloorSit Chair or SofaLie SofaGaze TV
1.00.50.80 6
Stand still in kitchen (preparing food) then Enter living room th Sit ith h d ti ( ti ) Di i T bl
Gaze TV0.6
0.6
then Sit with hand motion (eating) Dining Table
• When to query the user to confirm or disambiguate a result?When to query the user to confirm or disambiguate a result?• How to score user input vs. accumulated knowledge?• How to use user input to adjust the weight of the rules?j g
Interactive Machine Learning
Ambient Intelligence Hamid Aghajan
Open QuestionsOpen Questions
How to model semantics, ontology, knowledge representation– History of observations– User input– Use of the internet, search-based, community-basedUse of the internet, search based, community based
definitions
Ambient Intelligence Hamid Aghajan
Iterative DiscoveryIterative Discovery
Object ObjectS
knowledge base
Activity Object
knowledge base
Spatial relationPose relation
Primary AuxiliaryPrimary Object
inference
AuxiliaryObject
inference
Object ActivityMotion relation
knowledge base
Activity inference module Object inference module
Ambient Intelligence Hamid Aghajan
User Modeling and AdaptationUser Modeling and Adaptation
Learn the user’s habit
tablecoffeeeating tablediningeating
2
1
⇒⇒
ww
deskeating 3 ⇒w
Summary and OutlookSummary and OutlookSummary and OutlookSummary and OutlookFuture Research DirectionsFuture Research Directions
Ambient Intelligence Hamid Aghajan
Vision: Challenges & OpportunitiesVision: Challenges & OpportunitiesGeneric features: Can become unavailable or un-interestingGeneric features: Can become unavailable or un interesting
Opportunistic features
Frame scope: Frames can have zero or misleading information valueFlexible fusion window
Calibration: User-dependent functionObservation-based self calibration methods
Point-to-point accuracy metric: May not be relevant to end applicationApplication-driven metric
E i t d fi itiEnvironment definition: Manual, need to repeat upon movementObservations of user interaction
Fixed vision function: To avoid overloading the processor in real-time app.g p ppAlg. switching, Task assignment (active vision)
Uniform value for data: Misleading data can bias resultsConfidence level based on content and historyConfidence level based on content and history
Enabled by two-way interfacing of vision with higher processing layers
Ambient Intelligence Hamid Aghajan
In vision processing:
Vision: Challenges & OpportunitiesVision: Challenges & OpportunitiesIn vision processing:– Use context (incl. multimodal sensing) to guide vision– Algorithm switching based on contextg g– Associate confidence with data
Multi-camera vision:Multi-camera vision:– Handling redundancy, methods of data fusion, role selection– Privacy management (smart cameras)y g ( )
Address network deployment challenges:Online calibration / Calibration free methods– Online calibration / Calibration-free methods
– Automated setup and environment discovery
I i i f f i i i h i d lInteractive interface of vision with reasoning modules – Develop behavior models for user, environment, etc.
Ambient Intelligence Hamid Aghajan
Models and InterfacesModels and Interfaces
communicationmodes
behaviormodels
high-levelreasoning
graphic visualization
user modeling environment discovery
camera network model
abnormality detection
best view selection
3D reconstruction
avatarmapping textual report
knowledgeaccumulation
user preferences and availability
observation validation
context interface
vision
pose / activity face / gaze motion pattern 3D model
WSNL - Stanford
vision
multi-camera hardware & network
Ambient Intelligence Hamid Aghajan
Models, Context, FeedbackModels, Context, Feedback
camera network model(geometry, topology, roles, priors, confidence levels)
environment discovery(objects, temporal/affine
usage relations)
user model(routines, preferences,
emotional state)
abnormal event detection(activities, routines, accident,
control zones, crowds)
behavior modelscontext
semantic-level context
behavior environment events adaptationhigh-level
i
context
reasoningobservation validation
knowledgeaccumulation
c o n t e x t i n t e r f a c e signal-level
context
vision
motion pattern 3D modelpose / activity face / gaze
context
system levelvision
hardware & network
system-levelcontext
Ambient Intelligence Hamid Aghajan
Well-beingApplication DomainsApplication Domains
Well beingMonitor all relevant activities to user’s health and well-beingInput from heterogeneous sensorsSocial interaction: query the user confirm with an expert demographic comparisonsSocial interaction: query the user, confirm with an expert, demographic comparisons
Energy efficient smart environmentsContextual data: Real-time activity, history / behavior model, user preferences
Smart buildingsReal-time occupancy maps for emergency evacuation and rescue guidanceResource utilization: space lighting walkways energy activity patternsResource utilization: space, lighting, walkways, energy, activity patterns
Ambient multimediaConvergence of different content, interrupt prioritiesAmbience and atmosphere based on context and user preferencesSensor-enabled applications in social networks
Smart conference roomsGestures and interactions, smart documents, internet search, broadcast TV, streaming video, switching to best views, history and behavior modelsAnalytics: speaker performance card, roles and interactions in meetings
Ambient Intelligence Hamid Aghajan
Design methodology for multiple applications on the same systemNew FrontiersNew Frontiers
g gy p pp y– Context-aware multi-purpose smart spacesFixed + mobile camera networks
Calibration free methods– Calibration-free methodsHow to model semantics (use of observations, history, user input, internet)– Ontology, knowledge representation, generalizationHuman-in-the-loop methodologies– Level & type of user interaction and query– “Logical” user vs. discovery based on “experimenter” userSystems with adaptive behavior– Exploit “system user” adaptation in designEvent modeling:Event modeling: – Adaptive granularity in space, time (when to declare an event, levels of description)User Behavior change:
A lt f ki ith b ti ( i i ) t– As a result of working with an observation (vision) system– As a result of working with an adaptive system– How to influence and measure it?
Ambient Intelligence Hamid Aghajan
St d t & C ll b t
CreditsCreditsStudents & Collaborators:
Chen WuAmir KhaliliHuang LeeJingyu CuiTianshi GaoStephan HengstlerItai KatzItai KatzNan HuAli Maleki-TabarArezou Keshavarz
Marleen Morbee, Linda Tessens (Ghent U., Belgium), Tommi Maataa (Philips, TU Eindhoven, Netherlands)
Ronald Poppe (Univ. of Twente, Netherlands), Jacopo Staiano (Unit. Of Trento, Italy)Ralph Braspenning, Aki Harma, Joyca Lacroix, Aart van Halteren (Philips Research)