guest lecture @stanford aug 4th 2015
TRANSCRIPT
![Page 1: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/1.jpg)
CAN SUPERVISED AND UNSUPERVISED LEARNING MARRY HAPPILY?
USE CASES ON HUMAN ACTIVITY RECOGNITION
Natalia Díaz RodríguezVisiting scholar
University of California, Santa [email protected]
4th August 2015, Mobilize Center, Stanford University
![Page 2: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/2.jpg)
OUTLINE
•ABOUT ME & MY RESEARCH•RECENT PROJECTS•SUPERVISED AND UNSUPERVISED MACHINE LEARNING•Use Cases on Activity Recognition
![Page 3: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/3.jpg)
ABOUT ME: What excites me in AI?
• Semantic computing• Interpretable, intuitive, human-readable knowledge representation
• Unsupervised and Deep Learning (e.g. Computer Vision) • Automatic and powerful statistical learning
• Can semantics and statistics blend together?• Cognitive neuroscience
• Can the brain’s learning mechanism inspire machine learning?• Quantified Self
• Actionable, personalized well-being
![Page 4: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/4.jpg)
RECENT ACTIVITIES
•Google Anita Borg Scholar •/SYS/TUR Global•Women in Math and CS
•India collab.NIT Meghalaya:Wearables activity recognition
![Page 5: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/5.jpg)
RECENT ACTIVITIES: India collab: IIT KharagpurVirtual tutor: India folklore dance:
Kinect touch free interaction
![Page 6: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/6.jpg)
RECENT ACTIVITIES: SICN 2015 Fellow Summer Institute of Cognitive Neurosciences
http://sicn.cmb.ucdavis.edu/ Program http://sicn.cmb.ucdavis.edu/SI_2015_Week_1_Schedule_1_1_15.pdf http://sicn.cmb.ucdavis.edu/SI_2015_Week_2_Schedule_1_2_15.pdf
![Page 7: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/7.jpg)
RECENT ACTIVITIES:Smart Dosing (with Nursing Science Dept., Finland)
Medication tray filling and dispensing in hospital wards
![Page 8: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/8.jpg)
RECENT ACTIVITIES: First Person Vision for Activity RecognitionUnsupervised annotation (collab. with A. Betancourt, Netherlands)• Recording public Autographer dataset
Betancourt, A., Morerio, P., Barakova, E. I., Marcenaro, L., Rauterberg, M., & Regazzoni, C. S. (2015). A Dynamic Approach and a New Dataset for Hand-Detection in First Person Vision. In International Conference on Computer Analysis of Images and Patterns. Malta.
![Page 9: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/9.jpg)
![Page 10: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/10.jpg)
![Page 11: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/11.jpg)
Currently: Visiting Scholar
HYBRID POSSIBILISTIC AND PROBABILISTIC SEMANTIC MODELLING OF UNCERTAINTY FOR SCALABLE HUMAN ACTIVITY RECOGNITION
•Prof. Lise Getoor•Probabilistic Soft Logic (PSL)
![Page 12: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/12.jpg)
RESEARCH AGENDA
• Supervised and unsupervised learning USE CASES:1. Remote rehabilitation with Kinect2. Human activity recognition in Ambient Intelligence3. Semantic lifestyle profiling with wearables4. Conciliating probabilistic and possibilistic Activity
Recognition with PSL• Mentally stimulating future challenges
![Page 13: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/13.jpg)
Classical data-driven Machine Learning
![Page 14: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/14.jpg)
Knowledge-based Machine Learning
•Event calculus
•Situation calculus
•Rule-based systems
•Fuzzy logic
![Page 15: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/15.jpg)
Knowledge-based Machine Learning
WHY Semantic Technologies & Ontologies?
•Semantic Web: well-defined meaning
•Ontology:
• In Philosophy: study of entities and their relations
• In Artificial Intelligence: “Explicit specification of a
conceptualization” [Gruber, 93]
• Web Ontology Language (OWL)
![Page 16: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/16.jpg)
Background: Ubiquitous computing and Ambient Intelligence
•Smart Space (SS)
•Context-awareness: •Infrastructure and architectures•End-user programming frameworks for AmI
16
![Page 17: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/17.jpg)
Background: Ambient Assisted Living
•Usage of technology to provide assistance to peoplewho needs it in their daily activities, in the less obstrusive way
•Aim: support older/disadvantaged people, independent living, safety
•Includes: methods, systems, products and services
17
![Page 18: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/18.jpg)
Case study 1: A Kinect ontology for physicalexercise annotation and recognition
•Active Healthy Ageing project (EIT Digital)
with Philips Personal Health Labs (PHL)
•Sensor data aggregation platform
18
![Page 19: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/19.jpg)
19
![Page 20: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/20.jpg)
Kinect for remote rehabilitation DEMOS
•Sit-to-stand testhttps://www.youtube.com/watch?v=g8HOtFTk80c• Remote monitoring of post-surgery rehabilitation exercises on shoulder, knee, hip
https://www.youtube.com/watch?v=XL4JexDNs-Q
![Page 21: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/21.jpg)
Ontology features
Skeleton tracking (bone joint rotations + bone orientations)
21
![Page 22: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/22.jpg)
Exercises & Workouts Ontology
22
![Page 23: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/23.jpg)
Kinect ontology: examples of use
23
• Example 1: Defining basic movement (Stand, BendDown, TwistRight, MoveObject, etc.)
• Example 2: Provide workout feedback (# series in time, quality comparison with medical guidelines)
![Page 24: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/24.jpg)
Kinect ontology: examples of use
24
• Example 3: Historic analysis can monitor posture quality in time. E.g. having back less straight than 1 year ago-> notify to correct/prevent on time.
• Example 4: Notifications for office workers sitting too long/ not properly
![Page 25: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/25.jpg)
Case Study 2: PhD (Cum laude, Finland-Spain): Semantic and fuzzy modelling for human behaviour recognition in Smart Spaces, a case study on Ambient Assisted Living
Ros et. Al. 2011
![Page 26: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/26.jpg)
26
SPAIN: 15m of elders in 2049 (1/3 of the population) (INE)FINLAND population 65+ years: 18.14% [1]
• [1] http://www.finnbay.com/media/news/government-prepares-to-set-out-new-requirements-for-senior-caretakers/
![Page 27: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/27.jpg)
PHD OBJECTIVES
27
•Understand Smart Spaces•Human Activity Modelling and Recognition
•Program Smart Spaces
![Page 28: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/28.jpg)
PHD OBJECTIVES
•Understand Smart Spaces•Human Activity Modelling and Recognition
•Program Smart Spaces
28
![Page 29: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/29.jpg)
Activity Recognition in Smart Spaces 29
[Image: http://www.businesskorea.co.kr/sites/default/files/field/image/smart%20home.jpg + The noun project]
![Page 30: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/30.jpg)
Human Activity Recognition
30
![Page 31: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/31.jpg)
Handling uncertainty, vagueness and imprecision
• Broken/ missing sensors
• Incomplete data, vagueness
• Different ways of performing activities• Different object usage, duration, etc.
• Behaviour change
31
![Page 32: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/32.jpg)
Tools
32
![Page 33: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/33.jpg)
33
[CONON Context Ontology]
Methods: Ontologies
![Page 34: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/34.jpg)
34
JULIOANA MARIA
NATALIA
Has Brother
Has Mother
Methods: Ontologies
![Page 35: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/35.jpg)
35
JULIOANA MARIA
NATALIA
Has Brother
Has UncleHas Mother
Methods: Ontologies
![Page 36: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/36.jpg)
Methods: Fuzzy LogicWHY fuzzy (description) logics and fuzzy ontologies?
• Real life is not black & white• Classical (Crisp) Logic: True/False
• Fuzzy Logic: [0, 1]
• e.g. blond, tall
• For automatic reasoning about uncertain, vague or imprecise knowledge
• For natural language expressions
[Bobillo 2008 fuzzyDL: An Expressive Fuzzy Description Logic Reasoner: http://gaia.isti.cnr.it/straccia/software/fuzzyDL/intro.html] 36
![Page 37: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/37.jpg)
37
[Image: http://www.harmonizedsystems.co.uk/]
Example: Take medication
![Page 38: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/38.jpg)
A fuzzy ontology for activity modelling and recognition
38
Classes, Individuals, Data Properties and Object Properties
SUBJECT PREDICATE OBJECT
User performs activity Taking medicine =
(0.3 User performs sub-activity reach Cup or Medicine Box)
(0.3 User performs sub-activity move Cup or Medicine Box)
(0.1 User performs sub-activity place Cup or Medicine Box)
(0.1 User performs sub-activity open Medicine Box)
(0.1 User performs sub-activity eat Medicine Box)
(0.1 User performs sub-activity drink Cup)
![Page 39: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/39.jpg)
39
SurveyingActivityRecognitiontechniques
![Page 40: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/40.jpg)
40
SurveyingActivityRecognitiontechniques
![Page 41: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/41.jpg)
Surveyingontologies for activitymodelling
41
![Page 42: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/42.jpg)
AR Ontologies ranking: domain coverage evaluation
42
![Page 43: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/43.jpg)
OOPS! (OntOlogy Pitfall Scanner!) evaluation
43
![Page 44: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/44.jpg)
Case study 1: A fuzzy ontology for AR in the office/workenvironment
44
![Page 45: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/45.jpg)
45
![Page 46: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/46.jpg)
2-phased algorithm:
1. Sub-activities (data-driven phase)
2. High-level activities (knowledge-based phase)
Validation: CAD-120 dataset:
•10 sub-activities, 10 activities, 10 objects, 4 users46
Hybrid activity recognition with fuzzy ontologies
![Page 47: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/47.jpg)
Cornell Activity Dataset [Koppula et al. 2013]
47
![Page 48: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/48.jpg)
Hybrid data-driven andknowledge-basedactivity recognition
48
![Page 49: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/49.jpg)
a) Data-driven sub-activity recognition phase
49
![Page 50: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/50.jpg)
b) Knowledge-driven sub-activity recognition phase
50
Ontological
definitions
![Page 51: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/51.jpg)
Ontological definitions: object interaction
51
![Page 52: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/52.jpg)
Ontological definitions: object affordances
52
![Page 53: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/53.jpg)
10 semanticrules
53
![Page 54: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/54.jpg)
SUB-ACTIVITY prediction accuracy
54
![Page 55: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/55.jpg)
55
ACTIVITY prediction accuracy
![Page 56: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/56.jpg)
Activity recognition - comparison with state-of-the-art
56
![Page 57: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/57.jpg)
Activity recognitiontimes (ms)
57
![Page 58: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/58.jpg)
PHD OBJECTIVES
•Understand Smart Spaces•Human Activity Modelling and Recognition
•Program Smart Spaces
58
![Page 59: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/59.jpg)
Deployment: Programming Smart Spaces
59
Ros et. Al. 2011
![Page 60: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/60.jpg)
A visual language to configure the Smart Space behaviour
•TARGET USER: a) Developerb) Non-technical background
•AIM: •Rapid & easy programming of applications•Improve interoperability and usability
60
![Page 61: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/61.jpg)
Programming environments for novice programmers
61
[Scratch] [IFTTT]
![Page 62: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/62.jpg)
PROPOSAL: SS visual language mapping to OWL 2
62
![Page 63: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/63.jpg)
PROPOSAL: Smart Space visual programming
63
![Page 64: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/64.jpg)
PhD main contributions
1. A set of ontologies to model human behaviour and tackleuncertainty and vagueness inherent to real life
2. An architecture that integrates Semantic Web and Fuzzy Logicfor interpretable activity recognition
3. A hybrid knowledge-based and data-driven algorithm for real-time, robust activity recognition (84.1% prec.)
4. Design & development of a toolbox for non-expert users and rapid programming of Smart Spaces
[4 Journals -3 on Q1-, 9 conf. Papers. Google Anita Borg, Nokia and HLF scholar. University entrepreneurship award]
64
![Page 65: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/65.jpg)
Startup technology transfer:
65
![Page 66: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/66.jpg)
USE CASE 3: Semantic lifestyle profiling with wearables
Can we recognize lifestyle patterns automatically?1. Provide meaning to large heterogeneous data
-Interpretable, actionable insights2. Knowledge-based methods & uncertainty handling
-Behavior vs Profile recognition
Day routines and lifestyles:• Work/shop-aholics, Gym addicts• Pet/ Partner/ Kids • Retired/ Worker/ On holiday
![Page 67: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/67.jpg)
USE CASE 4: Activity recognition with Probabilistic Soft Logic
•Can manual work be automated?•Can we improve…
•Model & rule learning •Accuracy•Scalability •Genericity
![Page 68: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/68.jpg)
Probabilistic Soft Logic (PSL) :
Expresses collective inference problems mapping logical rules to convex functions(defining a hinge-loss Markov Random field).• FOL Predicate: relationship, property or role• Atom: (continuous) random variables• Rule: dependencies or constraints• Set: aggregatesPSL Program = Rules + Input DB
[PSL (open-source): psl.umiacs.umd.edu]
![Page 69: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/69.jpg)
PSL advantages against fuzzy OWL
•Statistical relational learning •Adds probabilistic component to possibilistic one•Captures cyclic dependencies•Rule weight & latent variable learning
•Scalable (convex optimization) learning•Most probable explanation (MPE) inference
![Page 70: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/70.jpg)
Current work @LINQS Lab
•Can we seamlessly blend…• Knowledge-based and data-driven mechanisms• Supervised and unsupervised learning
for a general activity recognition framework?
•Can TIMED streams be handled naturally?•Cost-sensitive, Online model learning and evolution
•Can models balance•Flexibility and reproducibility?•Accuracy VS deviation/anomaly detection?
![Page 71: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/71.jpg)
General activity recognition framework
AIM: Automate heuristics while maintaining rich semantic granularity:
• Context-awareness • Object interaction, cardinality, recursion, rule subsumption
• Unordered/ordered (sub)sequences of sub-activity-object pairs • Min/ max pattern sequential repetition in Δt
![Page 72: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/72.jpg)
Activity rule examples (PSL)
performsSubActivity(Object, ObjPosition, Time)
m.add rule:
performsMove(MedicineBox, P, T1) &
PerformsDrink(WaterGlass, P, Tn)) >>
PerformsTakingMedicine(Tn),
weight : 10;
![Page 73: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/73.jpg)
Future Challenges in Activity Recognition
•Multiple human sensing•Parallel/interleaved activities
•Automatic ontology learning and evolution•Reduce manual work
•(FOL/DL) Logics support for temporal constraints73
![Page 74: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/74.jpg)
• Unsupervised activity modelling• First camera vision AR
• Automatic dataset annotation
• Wearables sparsity and uncertainty• Scaling and real-timeness
74
Future Challenges in Activity Recognition
![Page 75: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/75.jpg)
TAKE HOME MESSAGE
• Supervised ML: Statistical and probabilistic approaches
• Unsupervised ML: Interpretable knowledge-based techniques
Both are needed!
Domain expert knowledge and common sense knowledge: Classical ML & Deep Learning: unable to exploit it!
![Page 76: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/76.jpg)
THANK YOU!
Brainstorming/collaboration ideas, tips, pointers are welcome!
Natalia Díaz Rodríguez [email protected]
https://research.it.abo.fi/personnel/ndiaz
COST Action on Architectures, Algorithms and Platforms for Enhanced Living
Environments: aapele.eu and Finnish Foundation for Technology Promotion
![Page 77: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/77.jpg)
Kinect for remote rehabilitation DEMOS
•Sit-to-stand testhttps://www.youtube.com/watch?v=g8HOtFTk80c• Remote monitoring of post-surgery rehabilitation exercises on shoulder, knee, hip
https://www.youtube.com/watch?v=XL4JexDNs-Q
![Page 78: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/78.jpg)
Comparison with existing state-of-the-art (sub-activity and activity recognition modules)
78
![Page 79: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/79.jpg)
Human Activity Recognition
A crucial but challenging task in Ambient Intelligence and AAL. Requires:
Context-awareness and heterogeneous data sources
Training data: examples
Common-sense knowledge /domain experts
Adaptation of behavioursAlzheimer, Parkinson
79
![Page 80: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/80.jpg)
ACTIVITY prediction accuracy (ideal situation)
(100% accurate sub-activity prediction) 80
![Page 81: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/81.jpg)
SUB-ACTIVITY prediction: accuracy results
81
![Page 82: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/82.jpg)
82
Fuzzy KB and rules in fuzzyDL
![Page 83: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/83.jpg)
Smart Space Architecture: Smart-M3
83
![Page 84: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/84.jpg)
84
![Page 85: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/85.jpg)
Equivalent SPARQL Query
85
Each rule is converted into a SPARQL query, which can be transformed into a Smart-M3 subscription.
![Page 86: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/86.jpg)
Crisp to fuzzy OWL query mapping to improvesemantics and usability
86
![Page 87: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/87.jpg)
Handling uncertainty reasoning whenprogramming Smart Spaces
• Fuzzy reasoners: expressivity VS
computational requirements and
platform versatility:
• Best compromise: fuzzyDL
87
![Page 88: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/88.jpg)
Activity recognition Algorithm
![Page 89: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/89.jpg)
89
![Page 90: Guest lecture @Stanford Aug 4th 2015](https://reader031.vdocuments.us/reader031/viewer/2022022413/58a172db1a28ab04278b5341/html5/thumbnails/90.jpg)
Ontology classes, data & object properties
90