The Assisted Cognition Project

Henry Kautz, Dieter Fox, Gaetano BorielloLin Liao, Brian Ferris, Evan Welborne

(UW CSE)

Don Patterson(UW / UC Irvine)

Kurt Johnson, Pat Brown, Mark Harniss(UW Rehabilitation Medicine)

Matthai Philipose(Intel Research Seattle)

Trend 1: Sensing Infrastructure

Robust direct-sensing technologyo GPS-enabled phoneso RFID tagged productso Wearable multi-modal sensors

Rapid commercial deployment

Trend 2: Healthcare Crisis

Demand for community integration of the cognitively disabledo 100,000 @ year disabled by traumatic brain injuryo 7.5 million in US with mental retardationo 4 million in US with Alzheimer’s

Family burnout Nationwide shortage of professionals

Assisted Cognition Technology to support independent living

by people with cognitive disabilitieso at homeo at worko throughout the community

byo Understanding human behavior from sensor

datao Actively prompting and advisingo Alerting human caregivers when necessary

Building Partnerships

UW Assisted Cognition seminaro CSE, medicine, nursing, Intel

ACCESSo UW CSE & Rehabilitation Medicineo Grant from NIDDR (Dept. of Education)o Help cognitively disabled use public

transportationo Prototype: Opportunity Knocks

Intel Proactive Health efforto Computing for wellness & caregivingo Promote partnerships with government,

universities, healthcare organizationso Intel Seattle: sensors for activity tracking

Example

Way-finding Assistanto Help user travel throughout community

On foot Using public transportation

o Detect user errors Proactively help user recover “You missed your stop, so get off at the next

stop and then wait for the #16 bus...”

o Potential users TBI, MR, mild memory impairment

Example ADL Assistant

o Activities of daily living Eating, bathing, dressing, ... Cooking, cleaning, emailing, ...

o Monitoring Changes in ADLs signal changes in health

o Reminding / prompting “Time to take your blue meds”

o Step-by-step guidance “Turn on the tap ... now pick up the brush

...”

o Potential users Disabled, ordinary aging

General Model

usermodel

common-sense

KB

geospatialDB

wearablessensors

environmentalsensors

interventiondecisionmaking

userinterface

caregiveralerts

General Model

common-sense

KB

geospatialDB

wearablessensors

environmentalsensors

interventiondecisionmaking

userinterface

caregiveralerts

physical motion& position

cognitive state

goals

activity

Deciding to Intervene

A = system intervenes

G = user actually needs help

ACCESSWay-finding Assistant

supported by

National Institute on Disability & Rehabilitation Research

DARPA IPTO

The Need: Community Access for the Cognitively

Disabled

Problems in Using Public Transportation

•Learning bus routes and numbers

Problems in Using Public Transportation

•Learning bus routes and numbers

•Transfers, complex plans

Problems in Using Public Transportation

•Learning bus routes and numbers

•Transfers, complex plans

•Recovering from mistakes

Result

•Need for extensive life-coaching

•Need for point-to-bus service

Result

•Need for extensive life-coaching

•Need point-to-bus service

•Isolation

Current GPS Navigation Devices

Designed for drivers, not bus riders!o Should I get on this bus?o Is my stop next?o What do I do if I miss my stop?

Requires extensive user inputo Keying in street addresses no fun!

Device decides which route is “best”o Familiar route better than shorter one

“Catastrophic failure” when signal is lost

New Approach

User carries GPS cell phone System infers transportation mode

o Position, velocity, geographic information

Over time, system learns about usero Important places o Common transportation plans

Breaks from routine = possible user errorso Ask user if help is needed

GPS readingzk-1 zk

Edge, velocity, positionxk-1 xk

k-1 k Data (edge) association

Time k-1 Time k

mk-1 mk Transportation mode

tk-1 tk Trip segment

gk-1 gk Goal

ck-1 ck Cognitive mode { routine, novel, error }

User Model

Error Detectio

n: Missed

Bus Stop

GPS camera-phone “Knocks” when there is

an opportunity to help

o Can I guide you to a likely destination?

o I think you made a mistake!

o This place seems important – would you photograph it?

Prototype: Opportunity Knocks

Status

User needs study Algorithms for learning and

predicting transportation behavioro Best paper award at AAAI-2004

Proof of concept prototype Now: user interface studies

o Modality: Audio, Graphics, Tactile, ...o Guidance strategies: Landmarks,

User frame of reference, Maps, ...

ADL Monitoring from RFID Tag

Data

UW CSE

Intel Research Seattle

demo at Intel this afternoon

Object-Based Activity Recognition

Activities of daily living involve the manipulation of many physical objectso Kitchen: stove, pans, dishes, …o Bathroom: toothbrush, shampoo, towel,

…o Bedroom: linen, dresser, clock,

clothing, … We can recognize activities from a

time-sequence of object touches

Sensing Object Manipulation

RFID: Radio-frequency identification tagso Smallo Long-lived – no

batterieso Durable

Easy to deploy Bracelet touch sensor Wall-mount

movement sensor

Example Data Stream

Example Activity Model

Creating Models of ADLs

Hand-built Learn from sensor data Mine from natural-language texts All of the above...

Experiment: Morning Activities

10 days of data from the morning routine in an experimenter’s homeo 61 tagged objects

11 activities o Often interleaved and interruptedo Many shared objects

Use bathroom

Make coffee Set table

Make oatmeal

Make tea Eat breakfast

Make eggs Use telephone Clear table

Prepare OJ Take out trash

DBN with Aggregate Features

88% accuracy6.5 errors per episode

Improving Robustness

Tracking fails if novel objects are used

Solution: smooth parameters over abstraction hierarchy of object types

Status

Accurate tracking of wide variety ADLs Active collaboration with Intel Current work

o Detecting user errors in ADL performanceo Learning more complex ADLs

Preconditions/effects Multi-tasking Temporal constraints

o Reminding & prompting

Concluding Remarks

Research on Assisted Cognition going great guns at UW and (a few) other universitieso CMU / Pitt / U Michigan (Nursebot,

Autominder – M. Pollack)o Georgia Tech (Aware Home, G.

Abowd)o MIT (House N, Stephen Intille)

Some Thoughts on Funding

Getting funding for work in this area is currently challengingo We were fortunate once with NIDRR,

but less than 1% of their budget is for research

o NIH & NIA spend relatively little on caregiving research New NIH “Roadmap” for interdisciplinary

exploratory research completely leaves out caregiving!

o NIN has good people, but no real money

Some Thoughts on Funding

Getting funding for work in this area is currently challengingo NSF supports some of the underlying,

multi-use technology, but not medically-oriented applications Exception: helping disabled use computers

o Industry support is vital, but more for collaboration than actual dollars Good industry grant = 1 grad student

o There’s a gap waiting to be filled...