presenter: jennifer c. hou ‡

Personal Assistant System: Use of Computer Technologies to Facilitate Elderly People with Assisted Living

Presenter: Jennifer C. Hou‡ Collaboration with Linda Ball*, Stanley Birge*, Marco Caccamo‡, Carl Gunter‡, Karrie G. Karahalios‡, Narasimhan Nitya #, Lui Sha‡,

Yang Yu#

‡ Dept of Computer Science University of Illinois at Urbana Champaign*College of Medicine, Washington University in Saint Louis#Pervasive Platforms and Architecture Lab, Motorola [email protected], 217-265-6329

Population Aging

Table compiled by the U.S. Administration on Aging based on data from the U.S. Census Bureau.

Aging of the baby boomer has become a social and economical issue.

In the United States alone, the number of people over age 65 is expected to hit 70 million by 2030, almost doubling from 35 million in 2000.

Percentage of People of 65+ and 85+

People over age 65 are expected to constitute 20% of the population in 2030.

Similar increases are expected worldwide.

Table compiled by the U.S. Administration on Aging based on data from the U.S. Census Bureau.

Similar Expenses Worldwide

SOURCE: United Nations ▪ “Population Aging ▪ 2002”

2002

Similar Expenses Worldwide

SOURCE: United Nations ▪ “Population Aging ▪ 2002”

2030

Consequences Are …..

Along with the increase of elderly people population, the expenditures of the United States for health care will project to rise to 15.9% of the GDP ($2.6 trillion) by 2010.

-- Health care industry study, Digital Foresight

Many elderly people will stay at home, rather than being consigned to expensive retirement homes.

Even today, only 10% of elderly people of age 65-85 and 25% of those of age >= 85 are institutionalized.

Many elderly people choose to stay at home also for privacy/dignity issues.

Overview of the Talk

Why Do We Care About Senior Care? Overview of Personal Assistant System (PAS) Underlying Research that Makes Things Work

Systems Architecture Wireless communications Tracking and localization Security and privacy Software safety, reliability, and availability Human computer interface

Do Elderly People Accept This Technology? Pilot studies

Where Do We Go From Here?

Why Do We Care About Senior Care

Because your parents are the next in line to be qualified as seniors, and you are next to the next in line.

Can advances in sensing, object localization, wireless communications technologies

enable elderly people to regain their capability of independent living?

make possible unobtrusive supervision of basic needs of frail elderly and thereby replicate services of on-site health care providers?

We believe the answers are Yes!

We Believe Technology Can Help

Time Driven Reminders of Daily Activities:

Home PC Serves as the

intelligence. Sends reminder

messages to wireless-enabled appliances.

Closes the loop with HCI/ localization techniques.

Takes action in the lack of response: A reminder can be sent more times, after which a designated on-site personnel or a healthcare provider is notified.

Jennifer, It is 8:30am.Time to take your Insulin injection before breakfast.

Overview of PAS

ALSP

Clinician Service

Gateway

Assisted Living Hub (ALH)

MedicalDevice

A drop-box architecture in which A security-enhanced, assisted living device called Authentication

Manager for You (AMY) co-exists with a home PC, called the Assisted-Living Hub (ALH).

The ALH is equipped with multiple wireless interfaces and serves as the local intelligence. It also communicates with the Assisted Living Service Provider (ALSP).

Through web interfaces, ALSP allows healthcare providers and clinicians to retrieve/analyze data and give instructions.

A More Technical View of PAS

Home Environment

. AMY Monitoring

ServiceInternet

ALH

IEEE 802.11WLAN

Clinicians

ALSP

• Easy to deploy.• Few entities that are subject to security attacks.• Numerous applications can be built on top of it.

What Applications Are Most Critical

% of Residents % of Residents asNeed With Need Primary Cause________________________________________________________________________Needs prompting to take medications 95 42Risk of injury due to falls 42 17Unable to get up after a fall 20 17Monitoring of vital signs too labor intensive 12 20Needs physical assistance with Activities of Daily Living 90 67Needs prompting to toilet on a schedule 67 17Needs prompting to go to meals 33 10Needs prompting to bathe 75 0Gets lost in apartment 17 0May wander out of facility 12 10Needs monitoring of blood sugar frequently 20 8Needs monitoring of weight daily/weekly 25 0

Factors Contributing to Loss of Independence and Institutionalization*

* Respondents include 8 geriatricians, 10 nurse administrators of assisted living facilities, and 6 home health clinicians at Washington University in Saint Louis

Application I: Time-based Reminder Services

PAS can help reminding residents of daily activities. ALH obtains from the ALSP updated prescription and

appointment records of a resident. When it is time for the resident to carry out their time

driven routines, the ALH locates active wireless-enable devices and sends reminder messages to one or more devices that are in the proximity of the resident.

Whether or not these routines are followed as advised is detected in a non-intrusive manner by exploiting sensor localization techniques.

ALSP

Clinician Service

Gateway

Assisted Living Hub (ALH)

MedicalDevice

Application II: Monitoring of Physiological Functions A number of physiological functions critical to maintaining homeostasis for different

medical conditions can be measured by Bluetooth-enabled medical devices, transmitted to the ALH and then to the ALSP to be evalauted by healthcare providers.

Measures will have a prescribed desired range and deviations from that range will generate an alert from the ALSP to the health care provider.

This enables prompt intervention before the situation deterioates to a point requiring hospitalization and may simply be in the form of additional instructions to the resident.

Home Environment

. . AMY Monitoring ServiceInternet

IEEE 802.11WLAN

ALSP

Application III: Non-intrusive Monitoring of Daily

Activities and locations Detection of early warning signs for depression and/or other chronic diseases:

The location of a resident and the duration in which he/she stays at that location are profiled by Motes-based (or Ubisense-based) sensing and tracking in a privacy-preserving manner.

Movement profiles are transmitted to the ALSP and analyzed (if necessary) for Early warning signs for severe depression (e.g., not taking medicine, not eating, staying

in bed for very long time) Preventive measures (behavior changes) for chronic diseases commonly seen in elderly

people (such as Parkinson’s disease and/or Alzheimer’s disease).

Application IV: Fall Detection

Residents wear sensors equipped with accelerometers (with fall detection algorithms that detect falls with the combination of speed and orientation changes).

In the case of fall detection, the sensor device beeps and an alert message is sent to the ALSP and to a designated healthcare provider.

In the case of false alarm, the resident can press a button and disable the message sending.



Systems Architecture Wireless communications Tracking and localization Security and privacy Software safety, reliability, and availability Human computer interface



Software Infrastructure Needed

A software infrastructure that integrates sensing, communication, and event/information management.

Understand, analyze, structure and control the complex interactions across the layers of computing, communication and sensing along the dimensions of robustness, reliability, QoS, security and privacy.

Interface mgmt

Events/data management

Communication

Sensing

Dependability Real-timeSecurity

& Privacy

Evolvability

Reminders/social interaction

Monitoring Emergency event processingTelemedicine

J2ME API for AccessingBT Nodes and Internet

Under Normal Situation

When the Gateway is Available

Active Application A

BT StackFixedALHOS

TCP/IPComm.

API

ALSP Server

TCP/IP Stack

ALSPServer

OS

Cellphone Network

+ Internet

Bluetooth

TCP/IP Stack

802.11Wireline

MAC

IP Routing Service

802.11MAC

Gateway

ServerSocket ss = new ServerSocket(90);

ss.accept();

Socket s = new Socket(myaslp.com:90)

Vital Sign Meters

Networked Software Architecture

System Architecture of the ALH

OS Layer in the ALH Architecture

The OS layer Is equipped with

various communication stacks and corresponding platform-dependent APIs

Depending on the family of peripherals to support, other stacks and APIs may include Bluetooth, Zigbee, and Infrared.

Middleware Layer in the ALH Architecture

Middleware Device monitoring

daemons: monitor the join/leave of peripheral devices in the environment and registers/de-registers the devices in the Device Registry Services.

The register-de-register process includes creating/destroying proxy or singular proxy stub objects for the device.

The proxy/proxy stub allows applications to be built upon well-known device APIs instead of vendor specific APIs/semantics. It also allows off-the-shelf devices to be integrated into the PAS system, as long as the vendor provides the semantics specifications on how to communicate with the device.


Device registry service: maintains a database of peripherals available in the environment. Each entry of the database is a proxy (or proxy stub) object created for a specific peripheral device.

Applications will query this service for appropriate proxies. When an appropriate proxy is found, a clone of the proxy migrates to the application.


The Unified Peripheral Communication APIs

Abstracts different network stack programming interfaces into a consistent paradigm, which basically follows java.net.* APIs.

Allows users to specify (in XML) their QoS demands when establishing communication links, and the middleware returns approved QoS guarantees.

This allows proxies to be developed independently of specific network stack programming interfaces.


Internet Heartbeat Daemon:

periodically checks the availability of Internet access through the gateway router.

When the gateway router fails/recovers, this daemon activates/deactivates the Bluetooth cell phone to access to the ALSP.

ALH Main Daemon: is in charge of managing (start, suspend, stop, restart etc.) all the application daemons and middleware daemons on the ALH.



Systems Architecture Wireless communications and networking Tracking and localization Security and privacy Software safety, reliability, and availability Human computer interface



ALSP

Gateway

ALH

TV

earplug Internet

WLAN(e.g. IEEE 802.11)

Networked Software Architecture Peripheral Bluetooth Network

Internet-capableMedicalMeters





BT StackFixedALHOS

TCP/IPComm.

API

ALSP Server

TCP/IP Stack

ALSPServer

OS

Internet

Bluetooth

TCP/IP Stack

802.11Wireline

MAC

IP Routing Service

802.11MAC

Gateway


ss.accept();


Vital Sign Meters



Passive Application A

BT StackFixedALHOS

TCP/IPComm.

API

ALSP Server

TCP/IP Stack

ALSPServer

OS

Cellphone Network

+ Internet

Bluetooth

TCP/IP Stack

802.11Wireline

MAC

IP Routing Service

802.11MAC

Gateway

Socket s = new Socket(128.174.11.11:90)


ss.accept()

Dumb Dev

PC IP: 128.174.11.11




What If the Internet Gateway Is Down?

One major deficiency of PAS is that all the traffic is transported through the gateway AMY, which becomes unavailable when it fails or when the resident is away from home.

To enhance robustness and ubiquity of PAS, we have used cell phones (transparent to users) as both a backup AMY and the local intelligence for data aggregation and acquisition.

we have leveraged the programming capability of Motorola EZX platforms (mainly the A780, E680 and E680i phones) to incorporate the following novel features that are not currently present in any cell phones:

Enabling cell phones to serve as local data storage/fusion intelligence.

Enabling cell phones as a delivery endpoint for reminder messages

ALSP

ALH

TV

earplug Internet



Peripheral Bluetooth Network

When the Gateway is Not Available


Base Station

cellphone

GPRS network

or WiFi

mobile medicaldevices

APIs (TAPI/NAPI, File system, etc.) J2ME + Bluetooth Dial-Up Networking Profile

(BT-DUN) for Internet AccessService Layer (Network, Connectivity, Telephony, etc)

GPRSBT StackCellphone

OS


BT StackFixedALHOS

TCP/IPComm.

API

ALSP Server

TCP/IP Stack

ALSPServer

OS

Cellphone Network

+ Internet

Bluetooth

802.11Wireline

MAC

IP Routing Service

802.11MAC

A. L. Device

Bluetooth MedicalMeter


ss.accept();


Cellphone DUN Modem(service provided by most BT enabled cellphones)

The cellphone modem is activated

J2ME + Bluetooth Dial-Up Networking Profile

(BT-DUN) for Internet Access


BT StackFixedButler

OS

TCP/IPComm.

API

ALSP Server

TCP/IP Stack

ALSPServer

OS

Cellphone Network

+ Internet

Bluetooth

802.11Wireline

MAC

IP Routing Service

802.11MAC

Gateway


The cellphone modem is activated Socket s = new Socket(128.174.11.12:90)


ss.accept()

APIs (TAPI/NAPI, File system, etc.) Service Layer (Network, Connectivity, Telephony, etc)

GPRSBT StackCellphone

OS

Cellphone DUN Modem(service provided by most BT enabled cellphones)

ALSP

mobile medicaldevices

earplug Internet



Peripheral Bluetooth Network

When cell phone takes the roll of ALH


Base Station

GPRSor WiFi

Cellphone

APIs (TAPI/NAPI, File system, etc.) J2ME

Service Layer (Network, Connectivity, Telephony, etc)

GPRSOr WiFi

BT StackCellphone

OS


BT StackFixedButler

OS

TCP/IPComm.

API

ALSP Server

TCP/IP Stack

ALSPServer

OS

Cellphone Network

+ Internet

Bluetooth

802.11Wireline

MAC

IP Routing Service

802.11MAC

A. L. Device



ss.accept();


Application 1(Listening to PC)

The cellphone is used as a smart device

Application 2(Sending data to ALSP

Server)

QoSbased


When cell phone takes the roll of ALH


BT StackFixedALHOS

TCP/IPComm.

API

ALSP Server

TCP/IP Stack

ALSPServer

OS

Internet

Bluetooth

GPRSOr WiFi

GPRS network

or 802.11

IP Routing Service

Base Station


ss.accept();


Vital Sign Meters


Real-Time Tracking and Localization

We have built a system that uses a combination of ultrasonic and RFID technologies as the underlying sensing mechanism for real-time tracking of both human and objects.

Human Tracking: Ultrasonic transmitters,

called beacons and denoted by Bi, occasionally sends out an ultrasonic pulse and a short RF signal at the same time.

Due to the speed difference between an RF signal and an ultrasonic signal, the listener denoted by Lj on the resident can infer its distance from the beacon using Time Difference Of Arrivals of the two signals.

This distance measurement can be reported to the ALH for real-time tracking of the use (marked as (2) and (3)).

Real-Time Tracking and Localization

Object tracking The RFID (radio

frequency identification) reader attached on the wristband can read RFID tags of objects touched by the user.

The objects can be tracked, whenever the user touches and/or carries them.

The whereabouts of the objects are logged.

How Medical Data Can Be Securely Transmitted ?

SOAP Envelope

Header

MedicalInformation

SOAP Envelope

Header

MedicalInformation

Sign. on Msg.

Sign. on Key

PubC KF

PubC/PrivC

KFPWAP

PWAP

PubAS/PrivAS

SOAP Envelope

Header

MedicalInformation

Sign. on Msg.

PubAS

PWAP

[Michael et al. 06]

PubC/PrivC

Sign. on Msg.

KFPWAP

PubAS/PrivAS

SOAP Envelope

Header

MedicalInformation

Sign. on Msg.

PubAS

Sign. on KeyPWAP

SOAP Envelope

Header

MedicalInformation

PubAS/PrivAS

PubC KF

PWAP

PWAPPWAP


PubC/PrivC

KFPWAP

PubAS/PrivAS

SOAP Envelope

Header

MedicalInformation

PubAS/PrivAS

PubC KF

PWAPPWAP

Sign. on Msg.PrivAS

SOAP Envelope

Header

MedicalInformation

Sign. on Msg.

PubC

Sign. on KeyPrivAS


PubC/PrivC

KFPWAP

PubAS/PrivAS

SOAP Envelope

Header

MedicalInformation

PubAS/PrivAS

PubC KF

PWAPPWAP

PubAS

SOAP Envelope

Header

MedicalInformation

Sign. on Msg.

PubC

Sign. on KeyPrivAS

PubC/PrivC

SOAP Envelope

Header

MedicalInformation

PubC

Sign. on Msg.PrivAS

SOAP Envelope

Header

MedicalInformation

PubC KF

PubAS PubC/PrivC

MedicalInformation

PubAS


Implementation

Apache Axis(WSDD / SOAP, XML-RPC / Java RPC, EJB, HTTP / WSDL Tools)

AssistedPerson

Clinicians / Family or Friends / Assisted Person

ALSPCompany

MedicalDevice

AL Hub

MessageGenerator

EncryptData

ALSPServer

SAAJ

Computer

AL SecurityEngines

Add Signatures/

EncryptEnvelope

DB Access

XMLSec

DecryptEnvelope/

VerifySignatures

MessageGenerator

Add Signatures/

EncryptEnvelope

Add Signatures/

EncryptEnvelope

MessageGenerator

DecryptEnvelope/

VerifySignatures

DecryptData

DecryptEnvelope/

VerifySignatures

UserInterface

JCE

Drop-box Prototype

AL Messages

Implementation

H/W Device: Digital Pulse Oximeter AL Hub: A labtop, Windows ALSP: A desktop, Linux Clinician’s computer: A labtop, Windows

S/W Language: Java Database: MySQL Web service: Apache web server, tomcat, axis 1.4 Libraries: XML-SEC, SAAJ, avetanaBluetooth, etc.

Security tokens

The following security tokens are set up Doctor’s public key certificate The ALSP’s public key certificate A secret key shared among AP and AP’s family members or friends AP’s and family/friends’ password to ALSP Access control matrix (ALSP), DataID table (AL Hub)

• Doc: Doctors office’s public key certificate.

• AS: The ALSP's public key certificate. • U*; V*: URL pointers to the ALSP's public key certificate. U*; V* are the ones which AP and clinician have, respectively.• KAP: A secret key shared the AP and her family members.

• PPA; PFA: AP's and Family's secret passwords (respectively) shared with the ALSP.• ACLAP: An access control list for the AP's records at the ALSP.• ΔAP: A mapping table from a serial number of a medical device to DataID

Doc, AS, KAP, U*; V*, PPA , PFA , ACLAP , ΔAP

doc, AS, KAP, U*, V*, PPA, PFA, ACLAP, ΔAP

Bootstrapping

1) Clinician sets up relationship with ALSP / Exchange URL pointers2) Assisted person (AP) subscribes to an ISP / Set up a gateway for WPA network3) AP visits clinician. Clinician recommends a list of devices, hubs, and ALSPs. AP

buys devices and hub and adds them to home network4) AP subscribes to ALSP. ALSP sends a URL pointer and a password token. AP loads

them into the hub.5) AP creates a shared “family key” for family members’ and friends’ access to data

on the ALSP. Family members register passwords to ALSP.6) Hub generates ΔAP , a set of DataID. A dataID is mapped to a device serial number

and recipients’ URL pointers. Each entry is a tuple (serial number, dataID, URL pointer)

7) With the recipient information (clinicians and family members), ALSP creates access control list ACLAP.

AP

Clinician

FamilyALSP

Pilot Study at Nazareth Living Center*

With the help of geriatricians at Washington University in Saint Louis, we have carried out a 3-week pilot study at the Nazareth Living Center for Assisted Living in June-July 2006.

This facility houses 110 well-educated, predominantly female residents, whose average age was 88. Of 30 residents who attended a presentation, 14 agreed to participate.

After administering a standard cognitive assessment, two residents were consented to test the PAS prototypes (application 1 and applicatoin2) and ten residents to carry/wear a placeholder device for a period of two weeks.

* Nazareth Living Center is a Catholic, nonprofit skilled nursing and assisted living community, sponsored by the Sisters of St. Joseph of Carondelet.

Pilot Study at Nazareth Living Center

Residents lack in confidence in PAS when it did not work properly:

Wireless connectivity is made possible by a low-end Linksys WRT54G wireless router. This, coupled with concrete walls between residents' rooms and the nurse station, led to intermittent connectivity.

Residents, as a result, were not confident in relying solely on PAS for medical monitoring.

To address this problem for the time being, we will add a wireless repeater (i.e., an additional Linksys WRT54G router configured in the client-mode and forwarding all packets to another, master router) between residents' rooms and the nurse station.

Residents found PAS useful and were willing to wear the sensing apparatus:

The two residents using the PAS prototype found it to be quite useful.

Residents not chosen to use the working prototype expressed their desire to use the working version as well.

Pilot Study at Nazareth Living Center

Nurses/caretakers desire interfaces that provide security/privacy.

With a high resident-to-nurse ratio, nurses were usually very busy and could not ensure that the information being displayed would not be viewed by unauthorized personnel. The need for privacy should be addressed by designing adequate access control to PAS.

Nurses inquired whether or not medical data could be securely transmitted via wireless technology. This implies they also had concerns about PAS security.


Technology enhancement with respect to software robustness/reliability, security/privacy, and HCI.

Extensive, hypothesis-driven clinical trials with Washington University in Saint Louis and Buehler Center on Aging, Health & Society, Northwest University

The average cost of nursing home stay is about $8,300/month*. Can PAS delay transfers of elderly people to skilled nursing facilities (SNFs) and improve the quality of their lives (by preserving independence)?

Delaying institutionalization by simply one month would cover the cost of deploying PAS.

*Phibbs CS, Holtz J-EC, Goldstein MK et al. The effect of geriatrics evaluation and management on nursing home use and health care costs: Results from a randomized trial. Med Care 2006;44:91-5.

Concluding Remark

With most of the PIs being baby boomers, it is our sincere hope that we will be the first generation to live independently and with dignity with the help of PAS when we age, and the project will benefit the healthcare system and the society as a whole.

presenter: jennifer c. hou ‡

Documents

elderly people of age

number of people

percentage of people

united nations population

health care providers

united states

population agingtable

health care industry