mit | media lab | affective computing exploratory …(digital pen, voice input and video) kientz,...

Exploratory Tool for Autism Spectrum Conditions

Rana el Kaliouby

Alea Teeters

Rosalind Picard

http://www.media.mit.edu/affect

BSN 2006 Workshop

MIT | Media Lab | Affective Computing

challengesdemomind-reading machinesautism

{kaliouby, teeters, picard}@media.mit.eduMIT | Media Lab | Affective Computing

Autism Spectrum Conditions

Center for Disease Control and Prevention (2005)

– 1 child in 166 has ASC



New Initiative: Autism Wearables

Communication Social interaction

Repetitive, obsessivebehavior

Related work• Monitoring• Assessment• Natural environment



Examples of Autism Wearables

Recognition of stimming behavior (e.g. flapping, rocking)Bluetooth accelerator and HMMs

Westeyn, Vadas, Bian, Starner, and Abowd (ISWC 2005)

Automated capture to support therapists during intervention sessions(Digital Pen, Voice Input and Video)

Kientz, Broing, Abowd, Hayes(Ubicomp 2005)



Autism Spectrum Conditions

Communication Social interaction

Repetitive, obsessivebehavior

Our research• Intervention• Assistive • Natural environment



Mind-Read > Act > Persuade

hmm … Roz looks busy. Its probably not a good time to bring this up

Analysis of nonverbal cues Inference and reasoning about mental states

Modify one’s actionsPersuade others



Real time Mental State Inference

Feature point tracking*

Head pose estimation

Facial feature extraction

Head & facial action unit recognition

Head & facial display recognition

Mental state inference

hmm … Let me thinkabout this

El Kaliouby and Robinson (2005)

* Nevenvision face-tracker



Complex Mental States(subset)

Agreeing

Concentrating

Disagreeing

Interested

Thinking

Unsure

Assertive

Committed

Persuaded

Sure

AbsorbedConcentratingVigilant

DisapprovingDiscouragingDisinclined

AskingCuriousImpressedInterested Brooding

ChoosingThinkingThoughtful

BaffledConfusedUndecidedUnsure

Affective-Cognitive Mental States

Agreeing

AssertiveCommittedPersuadedSure

Baron-Cohen et al.AUTISM RESEARCH CENTRE, CAMBRIDGE



Accuracy > Posed > Actors

Accuracy of system when trained and tested with posed actrors



Posed > Non-actors

AgreeingDisagreeingConfusedConcentratingThinkingInterested

IEEE CVPR Conference, 2004



Posed > Non-actors



Accuracy > Posed > Non-actors

Accuracy of panel of 18 peopleclassifying the videos

Accuracy of system(as good as the top 6%)



Real-time Performance

34.1%Total

24.7%6 Hz41.10 ms6 Mental State DBNs

0.1%6 Hz0.14 ms9 Display HMMs

0.3%30 Hz0.09 msAction Units

9.0%30 Hz3.00 msFaceTracker

LoadFrequencyTime on 3.4GHz p4

Level



Sense > Explore > Assist

Comfortable sensing innatural Environment

Explore socio-emotionalcues in self and others

Assist in communication (how to respond to disinterest)

Partner with behavioral programs already in place

(Groden Center)



Self-Cam

Monitoring self-expressions (along with other body sensors)Opportunity to learn about emotion expression in selfNetworked > exchange of social-emotional cues

Videos recorded by myDejaView camera



2-person interaction > Monologue

challenge to BSNdemomind-reading machinesautism


Challanges

Form-factorWearableHigh-resWide-angleHigh frame ratePower consumption

On-body processingAnalysisInference PredictionPrivacy

Data sensor fusion

Infrastructure for exchanging this information

Novel apps



Social Sensor Networks

PAN > Alea

PAN > Roz PAN > Seth

Sensor sampling

Sensor data analysis

Mental state inference

Share state

Peer-to-Peer PANs



Social Sensor Networks

• Networks to exchange social-emotional cues– Real-time– Not limited to facial expressions

• Examples of cues:– Facial expressions– Affect in speech – Physiology – Affective-Cognitive States– Activity – E.g.: is it a good time to interrupt

• On-Body Processing– Alleviates privacy concerns– You choose what and who to share your affective-cognitive states with

sense > explore > assist

wearable social sensor networks

autism and mind-reading

Acknowledgements:www.myDejaview.com for after-the-fact cameraswww.nevenvision.com for face tracking technologyMatthew Goodwin, Groden CenterNSF and TTT consortium for funding this research

{kaliouby, alea, picard}@media.mit.edu



Expression Capture

Opportunity to replay/reflect on expressions of people you interact withFun using a cameraImproved ability to look at, recognize, and respond to expressions

Camera by myDejaView

mit | media lab | affective computing exploratory …(digital pen, voice input and video) kientz,...

Documents