INCA: A Software INCA: A Software Infrastructure to Facilitate Infrastructure to Facilitate

the Construction and the Construction and Evolution of Ubiquitous Evolution of Ubiquitous

Capture and Access Capture and Access ApplicationsApplications

Truong and Abowd, Georgia Institute of Technology, 2004

GoalGoal

To build a toolkit to facilitate creation To build a toolkit to facilitate creation of capture and access applications -> of capture and access applications -> rapid prototypingrapid prototyping

MethodMethod

Identify a well defined class of Identify a well defined class of applications to supportapplications to support

Identify relevant design abstractions Identify relevant design abstractions for these applicationsfor these applications

Develop an infrastructure according Develop an infrastructure according to these abstractionsto these abstractions

Implement this infrastructureImplement this infrastructure Develop ‘interesting and complex’ Develop ‘interesting and complex’

applications to validate infrastructureapplications to validate infrastructure

MotivationMotivation

People have bad memoriesPeople have bad memories Manual capture and access are Manual capture and access are

incomplete and inconvenientincomplete and inconvenient Current research cannot leverage Current research cannot leverage

past work because lack of past work because lack of generalized platformgeneralized platform– Also hinders evaluation and iteration Also hinders evaluation and iteration

processprocess

Related Work in Several Related Work in Several DomainsDomains

ClassroomClassroom– eClass, Cornell eClass, Cornell Lecture BrowserLecture Browser, MANIC, , MANIC,

AutoAuditoriumAutoAuditorium, STREAMS, Authoring on the , STREAMS, Authoring on the Fly, Audio Notebook, StuPad, DEBBIEFly, Audio Notebook, StuPad, DEBBIE

MeetingsMeetings– DOLPHIN, TeamSpace, Tivoli, Dynomite, DOLPHIN, TeamSpace, Tivoli, Dynomite,

FiloChatFiloChat OtherOther

– Forget-Me-Not, Conference Assistant, Forget-Me-Not, Conference Assistant, Comic DiaryComic Diary

Lessions Learned from Lessions Learned from eClasseClass

Need for flexibility (esp. in Need for flexibility (esp. in information storage and access information storage and access methods)methods)

System must be able to evolveSystem must be able to evolve Temporal vs. Functional division of Temporal vs. Functional division of

componentscomponents

INCA (finally)INCA (finally)

Supports Key FunctionalitiesSupports Key Functionalities– Capture dataCapture data– StorageStorage– Transduce (data format/type conversion)Transduce (data format/type conversion)– Access to multiple, related, or integrated Access to multiple, related, or integrated

streams of info via context-based queriesstreams of info via context-based queries Supports multiple instances of each Supports multiple instances of each

functionalityfunctionality

Capture and TaggingCapture and Tagging

Data represented as raw bytes with Data represented as raw bytes with tagged attributes (metadata)tagged attributes (metadata)

Capture device advertises availability Capture device advertises availability of data (?)of data (?)

Tagger object adds metadata to data Tagger object adds metadata to data from capture devicefrom capture device

StorageStorage

Each StorageModule stores one type Each StorageModule stores one type of dataof data

Specifies list of attributes of data it’s Specifies list of attributes of data it’s interested ininterested in

Listens for data availability (capture Listens for data availability (capture function calls store callback)function calls store callback)

Announces to other components Announces to other components what type of data it houseswhat type of data it houses

Storage (cont)Storage (cont)

Repository is the generalized Repository is the generalized StorageModuleStorageModule

Is extendableIs extendable

AccessAccess

Components can subscribe to Components can subscribe to captured datacaptured data

Capture function does ‘handle’ Capture function does ‘handle’ callback to actually get datacallback to actually get data

Request creates context-based queryRequest creates context-based query

Transducing (File Transducing (File Conversion)Conversion)

Module advertises what kinds of file Module advertises what kinds of file formats it accepts and what it formats it accepts and what it converts toconverts to

Automatically run by the runtime Automatically run by the runtime systemsystem

Typical conversions: text to speech, Typical conversions: text to speech, video to image frames (and vice video to image frames (and vice versa)versa)

Other GoodiesOther Goodies

Attribute-triggered automated garbage Attribute-triggered automated garbage collectioncollection

ObserveModule and ControlModuleObserveModule and ControlModule Library of reusable components to Library of reusable components to

support capture of audio, video, web support capture of audio, video, web visits and inkvisits and ink

Network layer that handles Network layer that handles synchronous and asynchronous eventssynchronous and asynchronous events

Registry maintains list of componentsRegistry maintains list of components

Diagram of layersDiagram of layers

INCA in Action!INCA in Action! Used to create application for classroom systemUsed to create application for classroom system CaptureModulesCaptureModules: BoardSurface, : BoardSurface,

WaveCapturer, WebMemex, eScannerWaveCapturer, WebMemex, eScanner StorageModuleStorageModule: Repository: Repository Uses Uses ControlModuleControlModule and and ObserverModuleObserverModule to to

determine available capture servicesdetermine available capture services AccessModuleAccessModule: (instantiated by JSP) : (instantiated by JSP)

AudioPlayer, NotesPlayer, ExtendedSurfaceAudioPlayer, NotesPlayer, ExtendedSurface Also created a system for collection and Also created a system for collection and

analysis of behavioral data of children with analysis of behavioral data of children with autism autism

Evaluation (?)Evaluation (?)

Many other applications built using Many other applications built using INCA, at Georgia Tech and INCA, at Georgia Tech and Universidade de Sao PauloUniversidade de Sao Paulo

QuestionsQuestions

In terms of evaluation, is it good In terms of evaluation, is it good enough to know that INCA was used enough to know that INCA was used by others to build applications? by others to build applications? What about ease of use? What about ease of use? Performance issues?Performance issues?

Can a prototyping tool that involves Can a prototyping tool that involves end-users (like a CAPpella) or non-CS end-users (like a CAPpella) or non-CS people (like Topiary) be created to people (like Topiary) be created to achieve the same purpose?achieve the same purpose?

Sidenote on AutismSidenote on Autism ““Although autism is defined by a certain set Although autism is defined by a certain set

of behaviors, children and adults can exhibit of behaviors, children and adults can exhibit any combinationany combination of the behaviors in of the behaviors in any any degree of severitydegree of severity. Two children, both with . Two children, both with the same diagnosis, can act very differently the same diagnosis, can act very differently from one another and have varying skills.” from one another and have varying skills.”

“…“…whatever the diagnosis, children with whatever the diagnosis, children with autism can learn and function productively autism can learn and function productively and show gains with appropriate education and show gains with appropriate education and treatment.” and treatment.”

http://www.autism-society.org/http://www.autism-society.org/

Designing Capture Designing Capture Applications to Support Applications to Support

the Education of the Education of Children with AutismChildren with Autism

Hayes, Kientz, Truong, White, Abowd, Pering, Hayes, Kientz, Truong, White, Abowd, Pering, Georgia Institute of Technology, 2004Georgia Institute of Technology, 2004

OverviewOverview

Caregivers and educators of Autistic Caregivers and educators of Autistic children record data to evaluate children record data to evaluate effectiveness of therapyeffectiveness of therapy

Introduced 3 prototypes for capture Introduced 3 prototypes for capture and accessand access

Found that end-user iteration was key Found that end-user iteration was key to capture and access applications for to capture and access applications for this particular domainthis particular domain

Three PrototypesThree Prototypes

Walden Monitor: caregiver wears a Walden Monitor: caregiver wears a camera and records notes on a TabletPCcamera and records notes on a TabletPC

Abaris: sensors built into the Abaris: sensors built into the environment, notes taken in a form on a environment, notes taken in a form on a TabletPC, data is synchronized (as best TabletPC, data is synchronized (as best as it can)as it can)

CareLog: child wears a personal server, CareLog: child wears a personal server, and any caregiver present can record and any caregiver present can record notes (stored on PS) using any wireless notes (stored on PS) using any wireless enabled deviceenabled device

Constraints (Social Concerns)Constraints (Social Concerns)

Support the ‘Care Cycle’?Support the ‘Care Cycle’? Allow users to control amount of data Allow users to control amount of data

and type of data recorded while and type of data recorded while allowing them to capture rich data? allowing them to capture rich data? (not get bogged down with data)(not get bogged down with data)

Effort required to use system Effort required to use system (inconvenience)(inconvenience)

Privacy and control of dataPrivacy and control of data CostCost

Evaluation Metrics (Technical Evaluation Metrics (Technical Concerns)Concerns)

Integration of manually and Integration of manually and automatically captured dataautomatically captured data

Level of distributionLevel of distribution Data analysis and visualizationData analysis and visualization

Care CycleCare Cycle

1.1. Diagnosis based Diagnosis based on observationon observation

2.2. Goal settingGoal setting3.3. Learning and Learning and

behavior behavior modificationmodification

4.4. Evaluation of Evaluation of progress based on progress based on observationobservation

5.5. (start over again)(start over again)

PrototypPrototypee

Level of Level of SupportSupport

WMWM Medium: Single and Medium: Single and multiple session multiple session analysis possibleanalysis possible

AbarisAbaris Low: analyze only Low: analyze only single instances of single instances of step 4, can’t see step 4, can’t see trends over time or trends over time or across therapistsacross therapists

CareLogCareLog High: analysis portion High: analysis portion allows user to view allows user to view data over any period data over any period of time (over many of time (over many cycles). Capture cycles). Capture interface can be interface can be iteratively modified.iteratively modified.

Balancing Need for Rich Data Balancing Need for Rich Data vs. Capturing Too Much Datavs. Capturing Too Much Data

Need to pair Need to pair narrative with narrative with discrete datadiscrete data

Difficulty parsing Difficulty parsing narrativesnarratives

Difficulty Difficulty retrieving and retrieving and analyzing rich analyzing rich data (which data (which supplies the supplies the narrative)narrative)

PrototypPrototypee

Achieve Balance?Achieve Balance?

WMWM YES: Video, only YES: Video, only relevant datarelevant data

AbarisAbaris Somewhat: Videos, Somewhat: Videos, only relevant data, if only relevant data, if assume that CWA and assume that CWA and user only go into user only go into environment during environment during therapy session and therapy session and that entire therapy that entire therapy session should be session should be capturedcaptured

CareLogCareLog MAYBE: No rich data MAYBE: No rich data (yet), doesn’t capture (yet), doesn’t capture too muchtoo much

Reducing InconvenienceReducing Inconvenience

Technology should Technology should not distract not distract caregivers from caregivers from their primary jobs – their primary jobs – caring for and caring for and supporting a child supporting a child with autismwith autism

PrototypPrototypee

Level of Level of InconvenienInconveniencece

WMWM Medium – Medium – awkward and awkward and heavyheavy

AbarisAbaris Low – automation Low – automation of activitiesof activities

CareLogCareLog High – all data High – all data captured requires captured requires user attentionuser attention

Privacy and Control of DataPrivacy and Control of Data

Schools don’t want Schools don’t want to be liableto be liable

Video requires Video requires parental consentparental consent

Integrated Integrated classrooms – CWA classrooms – CWA students with students with regular studentsregular students

PrototypPrototypee

Protects Protects privacy?privacy?

WMWM Camera can be Camera can be focused on CWA, focused on CWA, user has control of user has control of what’s recordedwhat’s recorded

AbarisAbaris Users can control Users can control what’s recorded, what’s recorded, deployed only in deployed only in special purpose special purpose locationslocations

CareLogCareLog Child’s data stored Child’s data stored in his PS; single in his PS; single point of failure/data point of failure/data loss; can have data loss; can have data on unrelated on unrelated individualsindividuals

CostCost

Taking care Taking care of a CWA is of a CWA is costly costly alreadyalready

PrototypPrototypee

CostCost

WMWM High: dedicated High: dedicated caregiver, wearable caregiver, wearable camera, tabletPCcamera, tabletPC

AbarisAbaris Single environment: Single environment: Low (pays for itself*)Low (pays for itself*)

Multiple: High Multiple: High (network of cameras, (network of cameras, multiple tablePCs)multiple tablePCs)

CareLogCareLog Low: personal server Low: personal server + whatever PC + whatever PC schools/families schools/families already have + already have + wireless connectivitywireless connectivity

Integration of dataIntegration of data

Integration and Integration and synchronization of synchronization of data streams is data streams is importantimportant

PrototypPrototypee

Achieves Achieves integration?integration?

WMWM YES: Video and YES: Video and notes are taken notes are taken simultaneouslysimultaneously

AbarisAbaris NO: may look into NO: may look into activity recognitionactivity recognition

CareLogCareLog NO: open challengeNO: open challenge

DistributionDistribution Storage important Storage important

for privacy for privacy concernsconcerns

Also has Also has repercussions on repercussions on costcost

Distribution Distribution means flexibility in means flexibility in interfaces interfaces (capture devices (capture devices and analysis tools)and analysis tools)

PrototypPrototypee

Level of Level of distributiondistribution

WMWM Not: standalone Not: standalone systemsystem

AbarisAbaris Not Not

CareLogCareLog High: data capture High: data capture can come from can come from anyone, anyone, anywhere*anywhere*

Data AnalysisData Analysis

Use the data to Use the data to inform decisions inform decisions on therapyon therapy

Want to find Want to find trendstrends

PrototypPrototypee

Analysis Analysis TechniqueTechnique

WMWM Single session Single session analysis and analysis and aggregate data aggregate data across sessionsacross sessions

AbarisAbaris Analysis of single Analysis of single therapist or therapist or program, but no program, but no comparison, can comparison, can export dataexport data

CareLogCareLog Can choose any Can choose any interval of time for interval of time for data analysisdata analysis

ConclusionConclusion

End-user Iteration is keyEnd-user Iteration is key– Users should be able to iteratively Users should be able to iteratively

configure their systems to balance these configure their systems to balance these constraintsconstraints

– Distributed modular systems are best Distributed modular systems are best suited for iterationsuited for iteration

QuestionsQuestions

Did they adequately discuss the Did they adequately discuss the constraints and how each prototype constraints and how each prototype fulfilled/violated them? (ex. Privacy issues fulfilled/violated them? (ex. Privacy issues – who’s privacy should the application – who’s privacy should the application protect? How will applications deal with protect? How will applications deal with control of data?)control of data?)

Are there other constraints that they didn’t Are there other constraints that they didn’t consider?consider?

Are there other flaws with the prototypes Are there other flaws with the prototypes that the writers did not consider?that the writers did not consider?