Representations of ambient audio for the Deaf

Catherine PalmerSchool of Health and Rehabilitation Sciences

University of Pittsburgh

April 27, 2005

1

1 Introduction

People use sound in many subtle ways to gain awareness of the state of the world aroundthem. Ambient sounds give people an understanding of serendipitous events (a crowd gath-ering down the street, neighbors arriving home upstairs, children playing in the next room),the state of everyday devices (doorbells, tea kettles, telephones), problems (faucet dripping,fire alarm low-battery indicator, cell phone ringing at inappropriate times), and criticalinformation (fire alarm, knocking on door) relevant to their current situation or location.However, maintaining this awareness is difficult and sometimes impossible for the deaf.

In 1997 there were 3.4 million Americans with difficulty hearing, of whom 227,000 weredeaf [3]. Though there are some tools and techniques used by people who are deaf to monitorambient sounds, there is no collection of tools that provides continuous awareness of all thesounds in home, office, and mobile environments. A large gap remains between the experi-ences of deaf and hearing individuals. Previous work on assistive technology for the deaf hasconcentrated mostly on verbal communication, including supporting the automatic transla-tion of American Sign Language (ASL) [7, 35], but little research has addressed awarenessof ambient sounds. Our proposal is to develop a suite of tools that can help the deaf tomonitor the sounds going on around them, in home, office, and mobile environments.

A key property of ambient audio is its peripheral nature. Ambient audio is rarely at thefocus of a person’s attention. Rather, it supports background awareness of what is going onaround that person. Additionally, ambient audio may take the form of notifications, bothurgent (such as alarms) and informative (such as the beep of a dryer indicating that it isfinished). The design and evaluation of peripheral displays is a challenging research area,and the work proposed will contribute to our understanding of that area. We are proposingto create peripheral displays for both fixed (home, office) and mobile environments that cantranslate ambient audio into a medium that is perceptible to people who are deaf. As anexample, Figure 1 shows an early prototype of a design for showing recognized sounds (left,a door was heard opening or closing), and historical information about past sounds (right).

Thus, our contributions lie in two domains, assistive technology, and peripheral displays.Assistive technology as a field often focuses on desktop access, or support for basic activitiesof daily living. In the case of the deaf, this has focused on communication (sign languagetranslation and recognition). In contrast, we are focusing instead on ambient sounds. Thedevelopment of working technological solutions for displaying arbitrary ambient sounds inboth static and mobile environments, and the study of their uptake and use, represent animportant contribution of this grant. Additionally, our work will contribute a better under-standing of a particularly neglected domain, the experience of deafness in mobile contexts,and the impact of technology on that experience.

Peripheral displays is a burgeoning, important sub-domain of human computer interac-tion. The advent of ubiquitous computing, in particular, has pushed researchers to considerwhat happens when computers are active in a user’s environment, but not contributing toa user’s primary task. For example, a peripheral display might support awareness aboutthe arrival of new email, by playing a sound, showing a message on a computer screen, or

2

Figure 1: Prototype including (left) sound recognition, showing that a door was openedor closed with high probability, and mockups of (middle) spectrograph with recognitioninformation and (right) historical information (height indicates volume).

more abstractly causing something in the user’s environment to glow, move, or otherwiseindicate the email’s arrival. Peripheral displays have been built for both static contexts(rooms, desktop computers, etc.), and mobile settings. However mobile solutions are few innumber and have typically relied on audio to convey information [27, ?]. Thus, by buildingand deploying working mobile peripheral displays, we will contribute to an understanding ofhow to convey information on the periphery when the user is mobile instead of engaged in atask in a fixed location. Second, an important open question is how best to study peripheraldisplays in field settings, particularly with regard to their positive or negative impact onthe user’s primary task. We will explore this question in depth in the course of our workdeploying and testing working ambient audio displays with the help of deaf participants.

In summary, we will create peripheral displays for mobile, desktop, and home settingsthat show ambient audio and notify users about important events. Our displays will provideusers with information such as sound location, pitch, volume, and in some cases informationsuch as what a sound was about or overheard words. Additionally, displays will give usersan option to review historical information about what happened. We will explore multipledesigns, that will be vetted with the help of deaf participants, and the best of these designswill be tested in the field for many months, with a number of deaf users. As stated above,our research will include contributions in Assistive Technology and in the area of PeripheralDisplays. Additionally, our software solutions will be made available for download by anyonewith a need for the software.

1.1 Proposal roadmap

Next, we discuss in more depth the needs of people who are deaf, basing our discussion ona survey of related work, and our own interviews with deaf participants. We go on to detailthe set of solutions we are proposing to develop, and then discuss our validation plans beforelaying out our work plan and conclusions.

3

2 Needs Analysis & Challenges

We have interviewed eighteen deaf participants and six hearing participants over the pastthree years, to find out what is important to them about ambient sounds. Our participantsincluded people who are profoundly deaf and do not use hearing aids, and people who areable to hear some sounds either with or without the help of a hearing aid or cochlear implant.

Interview questions investigated both what sounds participants currently monitor andwhat sounds they are interested in. Additionally, we explored what differences existed be-tween home, work, and mobile environments, and asked about current tools and techniquesin use, and where they succeeded or failed. We also showed participants sketches and pro-totypes, to gain additional insight into their needs and preferences.

2.1 Needs and preferences

Hard-of-hearing participants felt most limited in loud environments, because their hearingaids or implants did not enable them to distinguish multiple, simultaneous sounds verywell. Profoundly deaf participants felt most limited in environments that involved spokencommunication (e.g., bars) or announcements (e.g., an airport).

Aside from communication, participants listed sounds at home, work, in the car, andwhile walking that would be useful to monitor. In an office, people were most interestedin knowing about the presence and activities of coworkers, as well as the absence of others(i.e. when they were alone), emergency alarms, phone ringing, coworkers trying to get theirattention, and faxes.

At home people wanted to know about emergency alarms, wake-up alarms, doorbell andknocking, phone ringing, people shouting, intruders, children knocking things over, faucetsdripping, water boiling, food cooking, timers, the garbage disposal, the washer and dryerrunning, problems with appliances, and gas hissing. One participant told a story about hiswife cooking something that burned, causing the fire alarm to go off. Both of them beingdeaf, they did not know the fire alarm was going off until a hearing friend visited and toldthem. Another participant told a story about leaving water on to boil all night, since shecould not hear the kettle whistle. A third participant told us, “Once I left the vacuumcleaner on all night; I must have bumped into it and turned it on accidentally.” The sameparticipant also told us that when she would like a wake-up alarm because, “Before an earlyflight, I will stay up all night.”

While walking or running outside, people wanted to know about dogs barking, vehicles,honking, bikes or people coming up behind them, and if they were blocking another person(e.g., “excuse me,” “watch out”). One participant told about problems while running,“When I first moved to LA I was surprised at how some drivers are aggressive on the roadsand at the intersection. I had some close calls.” While driving, people were interested inknowing about other cars honking, sirens, and sounds indicating problems with the car. Oneparticipant said, “When there is something wrong with the car, it tends to go unnoticeduntil it is very expensive to fix.” Another participant said she used to own a sports car and

4

Technique Application Characteristics Cons

Vibration sensing Awareness of sounds that

create vibration, e.g. sensing

footsteps, feeling that a

computer is on

Does not require focus of

attention Supports some

ambient and some notification

sounds

Depends on infrastructure

(e.g., having hardwood floors)

Flashing lights Awareness of telephones,

doorbells (Notification sounds)

Supports notification sounds Fixed visual attention

Must hook up each device

Hearing Dogs Awareness of all sounds Supports notification sounds

only

Requires ongoing maintenance

Requires a priori training per sound

Visual Inspection Multiple applications, e.g.,

Steam for a kettle, Looking out

the window for the arrival of a

guest

Sometimes is the only

alternative Some ambient,

some notification

Different for each sound

Polling rather than interrupt based

Hearing Aids; Cochlear

Implants

Enhancing existing hearing, but

not at the fidelity of “normal”

hearing

Enhances awareness of all

sounds

Requires training for interpretation of

sounds Results vary by case

Table 1: Currently available tools for monitoring ambient audio

would drive with the top down to be more visually aware.In addition to general awareness, one participant was particularly interested in learning

about sounds. When asked what sounds she would like to know about, she exclaimed, “Theocean!” She also expressed a love for music: she loved watching live musicians and feelingthe vibrations through the floor or speakers.

2.2 Current commercial solutions

We also asked participants about current tools and techniques they used for maintainingsound awareness. All participants emphasized visual awareness as key: “I tend to lookforward ahead of me much further than typical people. . .My eye sight is so important I’vecome to depend on it.” One participant with a deaf husband said they got each other’sattention by sending IMs to the other’s SidekickTM[30]. Another participant stayed aware ofher surroundings with vibrations, “I use vibration to feel grounded.” The same participantalso said, “I read other people, though this doesn’t always work well.”

For phone ringing, doorbells, and emergency alarms, some participants have flashers orstrobe lights. Table 1 contains an overview of the most popular solutions currently on themarket. Typically, each sound being monitored requires its own specialized system. Mostparticipants did not have these tools because of the expense and difficulty of buying andinstalling them. Other participants had one or two but not all.

2.3 Research solutions

Most research on assistive technology for the deaf has focused on support for communicationbetween deaf and hearing people, including automatic recognition of sign language [7], andtranslation of spoken language into text signed by an avatar [35]. One question for our work

5

is whether recognized non-speech sounds (such as a ringing doorbell) are best displayedabstractly, with written English, or with the use of an avatar signing the relevant word.

Tools for visualizing unrecognized audio graphically have reached the mainstream in thedomain of speech therapy [6]. These tools traditionally employ waveforms and spectrographsas visualizations, and are targeted at expert users. Additionally, they are not meant todisplay non-speech sounds, or to support monitoring and notification.

In addition to the commercial solutions for recognizing specific, pre-defined non-speechsounds summarized in Table 1, researchers have begun to create recognizers for dynamicallyidentifying the cause of an ambient audio event. For example, Malkin and Waibel developed anon-speech audio recognition system that can classify ambient audio events based on trainingdata [19]. However, this research has not yet been applied to the problem of supportingawareness of ambient audio for the deaf.

Displays that support monitoring of peripheral information without grabbing the usersattention are often referred to as peripheral or ambient displays [18, 20, 32, 33, 27, 28].For example, the “dangling string,” an art installation, was a string that spun at a speedproportional to the current network load [32]. These displays lie on the boundary betweenbackground and foreground awareness. They must notify the user of interesting informationwithout impeding the performance of a primary task, a goal that requires careful design [29].To be successful, our system must support monitoring by displaying information continuouslywithout distracting the user, unless a sound is of particular interest. In this case, it shouldcapture the user’s attention, and we refer to this as notification. Successful peripheraldisplays exist in both of these categories. For example, TimeAura supports continuousmonitoring of the passage of time by a lecturer [20], while the Scope system notifies usersabout communication events of interest [18]. Additionally, our display must function in bothmobile and static settings. Many varieties of static peripheral displays exist [?, ?, 32, 33].Existing examples of mobile peripheral displays have used audio as their medium [27, 28].However, aside from our own preliminary work in prototyping solutions for ambient audiodisplays [9], peripheral display research has not focused on the needs of the deaf.

2.4 Summary of needs

In summary, there is still a gap between the sound experience of a hearing person andthe experience of a deaf person. For example, although there are several methods used bythe deaf to provide awareness of certain notification sounds, there is little effective supportfor monitoring (a strength of peripheral displays). Also, notifications are not tuned to theimportance of sounds – whether an alarm is going off for a trip to the airport, or a salesmanis ringing the doorbell, the signal received by the user is identical. Another issue raised byour needs analysis is the dependence of many techniques on sounds that are known a priori.

Additionally, although we have completed an extensive analysis already, the particularneeds of the deaf in mobile settings have not been a focus of past research Catherine, isthis true? Do you have any refs on this?. One piece of our proposed work is to extendour needs analysis to that domain. We will interview and observe the deaf in mobile settings

6

to try to better understand where breakdowns occur, and we will ask their feedback onpotential solutions for those problem areas.

3 Development of ambient audio displays

Our plan is to explore a variety of solutions to the problem of displaying ambient audio inmobile, home, and work environments. In particular, we plan to explore implementationsthat show sound location, sound identity, volume, pitch, and a history of past sounds. Theseissues represent the two main axes of the design space identified by our interviews: place(static, controlled environment; static uncontrolled environment; and mobile, uncontrolledenvironment); and information (sound location, sound recognition, volume, pitch, history ofsound events, and so on). Our goal is to develop technological solutions that explore a rangeof the most useful displays within this design space.

3.1 Design of peripheral displays of ambient audio

We will investigate questions regarding how to optimally design peripheral displays of am-bient audio at all stages of development, from the sketches shown in Figure 2 prototypesthat participants can try out in lab or field settings before giving us feedback. This iterativeprocess will help us to hone in on the best solution for the population we are supporting.Below we discuss some key design considerations for these displays.

3.1.1 Static displays (for home and office)

Although the implementation of static, visually based peripheral displays has been exploredby others in the past (see our paper on toolkit support for peripheral display design for areview of some of the issues involved [26]), much work still remains to be done in under-standing how best to design a display to convey information of interest. Implementation ofdisplays that convey information but consistently remain peripheral is a fundamentally hardproblem, because of the need to balance attention demand very carefully so as not to disturbthe user unnecessarily. For example, how does one design a visual signal to be noticeable inthe same ways as the corresponding audio signal it is based on? Many other design issuesmust be addressed for a display to be adopted. For example, should a display be part orall of a computer screen, or should it be based on a physical map? Although our past workhas begun to address how one designs peripheral displays [22], few carefully designed andevaluated examples exist in the literature as guidance for developers.

There are two particularly interesting problems that will have to be solved in the designof a display of ambient audio – how best to display sound location, and how to displaydense historical information. In the case of sound location, we plan to develop a series ofautomated visualization techniques for showing spatialized audio. On the abstract end, ourplan is to explore aesthetically pleasing visualizations reminiscent of those shown when amedia player such as Windows Media Player plays an audio file. We will investigate how to

7

Recognition

Location

History

Mobile

Figure 2: Example design sketches

8

show easily readable information in this format without sacrificing aesthetics. On the moreconcrete side, we plan to explore map-based visualization that clearly locate sounds for theparticipant. An open question is how to easily facilitate the creation of such maps, and howto calibrate the system to accurately display sounds on the map.

History of information displayed in the past has been notably absent from most peripheraldisplays described in the literature. Yet it seems crucial for displays that are often ignored, orglanced at only occasionally. We will explore different visualization techniques for conveyinghighly readable information about past sound events. Our goal is to create a visualizationthat can show a high volume of events that area easily identifiable visually. We will alsoexplore different options for displaying history, ranging from incorporating it into the displayon a continuous basis, to showing it only on demand.

3.2 Displays for mobile settings

A key contribution of this proposal will be the design of a mobile ambient display of ambientaudio. Unlike past displays [27, 28], ours will not be able to depend on using audio as amedium for conveying information, since it is intended for use by the deaf. Some of thequestions we will need to address include the optimal modality for displaying informationin mobile settings, the choice to display continuous information versus notifying the useronly about important sounds, and how to support access to historical information, whichwe expect will be especially crucial in mobile settings. Another important question is whattechnology to use (a PDA, a cellphone or pager, or an in-glasses display are all possibleoptions).

To begin our investigation, we plan to look at specific settings, such as driving, publictransportation, and so on. We believe that the mobile solutions in these settings will besignificantly different. However, these will help us to learn about the design space for mobileperipheral display design.

3.2.1 Place

As stated above, place was a key axis derived from our needs analysis. We have alreadydiscussed some of the challenges involved in creating displays for mobile and static settings.Some additional issues we intend to explore include how different locations change the re-quirements for display size, function, and other characteristics. For example, we expect thatparticipants may prefer tactile vibrations as notifications of sound events in mobile settings,while they may prefer visual indications of sound events in static settings. Similarly, weexpect that small, lightweight displays will be preferred in mobile settings, while our inter-views indicated that larger, aesthetically pleasing displays may be appropriate in the home.At work, displays may need to be small and integrated into the work setting (for examplearound the edges of a computer monitor).

9

3.2.2 Information

Information was the second key axis for display design derived from our needs analysis. Wewill investigate the importance of displaying different properties of audio, such as pitch,volume and location. Additionally, we will investigate the utility of displaying recognizednon-speech and speech sounds, even when that recognition is error-prone. The question ofhow to help a user deal with error while minimizing impact on the end user was the topic ofthe P.I.’s Ph.D. thesis work [25, 24]. Some possible solutions we will explore include differentapproaches do displaying certainty, displaying multiple possible interpretations, and allowingthe user to correct misinterpretations.

Note that our work is not focused on creating or improving audio recognition systems,rather we plan to use off-the-shelf solutions for this including a non-speech audio recognitionsystem created by researchers in our department [19] and one of several commercially avail-able speech recognition technologies, such as DragonDictateTMor IBM ViaVoiceTM. However,there are technical problems associated with training, which currently requires intensive ef-fort on the part of a hearing person who can capture and segment sounds of interest, beforerunning the training algorithm on them. Among other issues, we will explore approaches forallowing a person who cannot hear to be involved in training the system. Our system willbe designed with the assumption that a hearing person is able to help with initial trainingand installation as this up-front training is lengthy and involved with current state-of-the-art technology. However, as sound identification needs change, the system may need to bere-trained. When the user notes that a sound of interest was not recognized by the system,he or she will be able to select that sound from the history for training. If the sound hada clear start and end, this will be fairly straightforward. In cases where the start and endof the sound is not easy to identify visually, we will provide lightweight support for askinga “goto person” (a hearing helper) to provide this information. The end user will still beexpected to do other training-related tasks such as labeling the sound.

We will also explore the value of history in use contexts (as opposed to for training).History can be of use both for a deaf person to review what just happened, and in emergencysituations to ask a hearing person to translate if the system is unable to. History can taketwo forms – an actual audio recording of recent sound events (or a continuous recording of thelast n minutes of sound), or visual representations of sound showing any of the dimensionsmentioned above (volume, pitch, location, sound identity, and so on). We will implementan audio buffer, including the capability to capture only “important” sound events (eventsidentified by the recognizer). We will also provide support for keeping a history of soundsincluding all known characteristics about them.

We have begun initial investigations into many of these issues. For example, in past work,we compared two display designs in a wizard-of-oz experiment, to see whether location andvolume, or pitch and volume, were more useful for correct sound identification [9]. Figure 2shows some of the design sketches we are currently showing to interviewees for feedback, andFigure 1 shows our first pass at a prototype that includes both sound recognition and histor-ical information. Much work remains to be done in understanding the relative importanceof all these issues, and creating both visual and tactile designs for use in different settings.

10

3.3 Summary of displays

We are proposing to build a variety of displays covering the design space derived from ourneeds analysis, including places such as static and mobile settings, and information such ashistory, location, and sound identity, pitch and volume.

In order to explore this design space, we will have to address several challenging problemsin the design space of peripheral displays. A key contribution of this proposal, will be theexploration of issues in the design of a mobile ambient display of ambient audio. Anotherimportant contribution will be our exploration of approaches to showing historical informa-tion. Both of these issues have been relatively unexplored in existing work in the domain ofperipheral displays. Finally, the validation component of our work (described next), whichincludes evaluating our displays in use in the field over the course of an entire year, willcontribute to an understanding of successful peripheral display design that go beyond mostpast work.

4 Validation

Our validation will be focused on showing that we have made progress in both key areas ofthis proposal: Assistive Technology and Peripheral Displays.

4.1 Validation of Assistive Technology contributions

As stated above, a major focus of this grant is the development of working solutions fordisplaying arbitrary ambient sounds in both static and mobile environments. A thoroughvalidation of assistive technology must address three factors: Does the technology solve theproblem, will people adopt it, and will they continue using it over time need citation forwhat determines assistive tech success?. For this reason, our validation will need tocombine studies of effectiveness in the lab with studies of long-term field use of the technology.

In controlled, short-term studies, we plan to focus on measuring the effectiveness of thetechnology. These studies will take place both in the lab and outdoors in mobile contexts.We will pay particular attention to the accuracy of recognition components of the system,and the system’s ability to evoke similar responses to those experienced by hearing peoplein the same environment. For example, loud sounds, or important announcements shouldboth be more noticeable than quieter sounds, ongoing background hums and noises, andunimportant sounds such as nearby conversations between strangers. We plan to run studiesof this sort during each year of the grant, and to use the data we get from them to iterativelyimprove the design of our system.

In addition to our short-term, controlled studies, we will conduct a long-term field study.This study will focus on issues of adoption and retention, and try to identify the valueof the system over time to deaf users. For this study we will recruit between 10 and 20participants, and ask them to use the system for a full year. During the course of the yearwe may upgrade pieces of the system as necessary, and we will provide ongoing support. Due

11

to the experimental nature of our technology, we believe that this is necessary, despite thefact that it may effect retention. We will gather log data about system use, and regularlygather qualitative data through interviews about the user experience.

Because one major focus of this grant is mobile settings, both our formative (pre-implementation) and summative (post-implementation) work will pay particular attentionto users who are mobile. As a side effect, we hope to develop a better understanding of wherebreakdowns occur for people who are deaf in mobile, and particularly in public settings. Ourhope is that this will allow us to make recommendations not only for how technology canmediate for deaf users, but also about how society can make public settings more accessiblefor the deaf.

4.2 Validation of Peripheral Display contributions

We will apply known techniques to explore the success of our display at remaining on theuser’s periphery while still supporting awareness. These include using a modified form ofheuristic evaluation (developed by the P.I.) during early design [22], employing dual taskstudies (in which the user’s performance on a primary task, such as typing, is measuredwhile the user is also monitoring our display peripherally, e.g. [1], and field studies.

Most peripheral displays show “synthesized” information such as stock data, in/out statusof people in the office, network load, bus arrivals and departures, email arrival, and so on.This means that there is no existing standard against which their success can be compared.Our work stands in strong contrast to this: We are displaying information that is alreadyavailable, peripherally to the hearing population. That means that we can compare theexperience of people who are deaf, with our displays, to the experience of people who arehearing the same sounds.

Because of this unique property of our work, we will be able to directly compare thesuccess of our display against a “gold standard” that works exceptionally well (albeit onlyfor hearing people) – ambient audio. Our controlled studies will look at several metrics:How aware are people of sounds, how distracting are those sounds, how do sounds impactthe user’s primary task. As with past studies of peripheral displays in the lab, we willemploy a dual-task methodology, comparing our displays to existing solutions such as thosediscussed in Table 1. Additionally, we will compare our display’s performance to hearingpeople in the same environment.

Evaluation of peripheral displays in the field is a hard problem because, while it is crucialto know how distracting peripheral displays are, it is very difficult to directly monitor theireffect on the user’s primary task. This is because, in an uncontrolled setting, it is impossibleto know exactly what the user’s primary task is, and the user’s primary task may be changing.Thus measuring distraction, in particular, is especially hard. We have begun to exploreapproaches for field evaluation of peripheral displays [11], and the systems built for thisproposal represent an excellent testbed for further exploring those issues.

Our field studies will build on our past work [11], using a combination of redundantmethods including self-reporting, logging of activity, and regular experience sampling to

12

F 2006 S 2007 F 2007 S 2008 F 2008 S 2009

Expand and finish needs analysis

Expand analysis of issues in mobile settings

Develop office/home displays

Develop mobile display

Heuristic evaluation of displays

Lab studies & controlled tests

Ongoing iteration of field study technique

Field study of office/home displays

Field study of mobile displays

Table 2: Research plan for the next three years. Darker grey boxes indicate where we expectthe majority of the effort for each task to take place.

build a picture of how the displays are used in the field. As stated above, we plan to deployour displays with ten to twenty users. This will require us to develop tools that will make itfeasible to scale our field study process. Our plan is to build these tools into the peripheraldisplay toolkit we will be using to develop our displays [26].

5 Work Plan

Our plan of approach for this work, as illustrated in Table 2, is as follows: In the first year,we will focus our efforts on finishing our needs analysis, and developing multiple iterations ofall three displays. We will use a combination of heuristic evaluation, feedback from potentialusers of our system, and controlled studies to explore and iterate on each prototype. Wehope to conclude the majority of the development of the home and office by the middle ofthe second year (although we will continue to iterate on and improve them as necessary afterthat). The development of the mobile displays will take more time because of the difficultiesinherent in dealing with mobility and the open questions about designing effective peripheraldisplays for mobile settings. This will continue throughout year two and into year three.

Our controlled lab studies will span most of our work on the grant, first providing forma-tive feedback for our iterative design, and next providing detailed quantitative informationabout how they perform in comparison to existing solutions and the hearing experience ofambient audio. Our field study will ramp up during year two, and continue throughout yearthree. Our efforts in designing studies and recruiting participants will be aided by our ac-cess to the resources at the University of Pittsburgh’s School of Rehabilitation Technology.While we conduct the field study, we will slowly iterate on the technique we use, exploringalternatives that provide the researchers with valuable data without overburdening the users,or requiring them to give up too much privacy.

In order to complete this work, we will need to fund one graduate student for the entire

13

three years to work on the major research questions raised in the grant, including needsanalysis, building the home, office, and mobile displays, and evaluating them. We are alsoasking for funds to pay an undergraduate research assistant starting in year two to work onthe mobile display, and to help with running the field evaluations of the displays.

6 Results From Prior Funding

Our current and prior funded projects provide a solid base for carrying out this work. Inthis section, we consider results from three of these projects.

6.1 NSF IIS-0209213 “Web Accessibility for Low-Bandwidth In-put” 2002-2005, Jennifer Mankoff

This project focused on automatically adjusting web browsers and web pages to make themmore accessible to people with severe motor impairments. For example, a web page maybe modified to show preview information about a selected link to the user to avoid the costof following a wrong link and then backing out again. As part of this work, we developeda list of seven requirements and related web page modifications [21, 12]. For any solutionto be proved effective, it must be tested. As part of our work on this proposal, we havebeen developing methodologies for testing the accessibility of web pages and applicationsby users with a variety of disabilities [8, 23]. Finally, because the main focus of this grantwas on users with motor impairments, we have been working to develop improved word-prediction tools specifically for this need, and developed tools to better support users withmotor impairments and developers testing interfaces for accessibility problems relating tomotor impairments [5, 4, 31, 23].

The guidelines and tools that we developed for making web browsing accessible directlyapplicable to the problem of adapting desktop interfaces to meet the needs of people withmotor impairments. They represent a solution focused on a broad class of online activity,web browsing. Our investigations considered the needs of a variety of motor impairments,including people with severe motor impairments using single switch input and people withmild tremors.

6.2 NSF ITR-0205644 “Human-centered Design of Context-awareComputing: Scalability, Usability and Privacy” 2002-2007, 5P.I.s including Jennifer Mankoff

NSF ITR-0205644 focuses on a subset of Ubiquitous Computing, context-aware computing.The main focus of the ITR grant is on infrastructure issues relating to scalability, usabilityand privacy. In the ITR grant, we proposed to build two applications: an augmentedwheelchair that uses context to support word prediction and an emergency response system.To date, we have built an infrastructure in support of Context Aware Computing and begun

14

initial work on both applications [5, 17, 10, 34, 13], as well as application-level support forend-user privacy [14, 16, 15, 5, 4]. Peripheral displays of ambient audio are related to contextawareness in that they help users to maintain awareness of their own contextual environment.As part of this grant, we have developed the Peripheral Displays Toolkit [26], and exploredissues in designing and evaluating peripheral displays [2, ?, 22, 11].

6.3 Something from Catherine Palmer

7 Conclusions & Broader Impact

In conclusion, we are proposing to develop a solution for displaying information about am-bient audio to the deaf. Although ambient audio provides key information to hearing peopleabout peripheral events in their environment, this information is typically not available topeople who are deaf.

Together, the P.I., with her extensive experience in Ubiquitous Computing, PeripheralDisplays, and Assistive Technology, and the Co-P.I. with her deep knowledge of issues facingthe deaf, are a team uniquely capable of addressing these problems.

Our contributions will include the development and testing of effective, peripheral dis-plays of ambient audio for home, office, and mobile environments. Some key problems wewill have to solve in this work include: (1) Understanding the mobile deaf experience; (2)Building usable mobile peripheral displays; (3) Incorporating history into peripheral displays;(4) Visual representation of ambient audio; (5) Techniques for field evaluation of peripheraldisplays.

With our tool in place, a deaf person will not miss out on serendipitous meetings aroundthe water cooler; will know when a transit emergency requires a change of trains; will be ableto sleep through the night before an early flight; will not have to worry if the tea kettle orthe vacuum cleaner was left running. Our hope is to develop solutions that will eventuallybe widely available to the deaf, should they wish to use them.

7.1 Broader Impact

In terms of dissemination, evaluation techniques can be shared simply by writing about howto use them, and we plan to publish this work in premier conferences and journals in humancomputer interaction and ubiquitous computing. Additionally, we will make the displays wedevelop publicly available through the Three Rivers Center for Independent Living (TRCL),a clearinghouse for services and information for the disability community in and aroundPittsburgh,q and through the Internet.

Our work in this arena will be leverage and support the educational goals of an IGERTin assistive technology currently shared by CMU and the University of Pittsburgh, includinggraduate student education in the area of assistive technology. In addition to involvinggraduate and undergraduate students directly in our research, we will regularly teach coursesin the area of assistive technology and accessibility. These courses will take place at the

15

University of Pittsburgh, and in the School of Computer Science, and one of our goals is tobroad range of students to the study of assistive technology and its connections to computerscience.

Finally, both P.I.’s have a strong record of working with women and minorities. Cur-rently, over half of the P.I.’s graduate students are female. Similarly, the The Co-P.I.’sresearch group includes both ethnic minorities and women. The P.I. also mentors womenin programs aimed specifically at them, including the CRA undergraduate summer internprogram (sponsored by CRAW), and a CRA program to support women-only undergraduateresearch groups (CREW).

16

References

[1] Lynn Bartram and Mary Czerwinsky. Design and evaluation of notification interfacesfor ubiquitous computing. Ubicomp 2002 Workshop 9, September 29-October 1 2002.

[2] S. Carter, J. Mankoff, and P. Goddi. Building connections among loosely coupledgroups: Hebb’s rule at work. Journal of CSCW, 13(3-4):305–327, August 2004.

[3] Employment, earnings, and disability – tables: From the survey of income and programparticipation. U.S. Census website, 1997. Available at http://www.census.gov/hhes/www/disability/publications.html.

[4] Anind Dey and Jennifer Mankoff. Designing mediation for context-aware applications.ACM Transactions on Computer-Human Interaction, 12(1):53–80, March 2005. Specialissue on Sensing-Based Interactions.

[5] Anind Dey, Jennifer Mankoff, Gregory Abowd, and Scott Carter. Dis-tributed mediation of ambiguous context in aware environments. In Pro-ceedings of the 15th annual ACM symposium on User interface software andtechnology (UIST-02), pages 121–130, New York, October 27–30 2002. ACM,Available at: http://www.cs.berkeley.edu/ dey/ambiguous-context-mediation.pdfACM Press.

[6] Ellis D.P.W. The weft: A representation for periodic sounds. In Int. Conference onAcoustic, Speech & Sig. Proc. ICASSP-97., pages 1307–1310, 1997.

[7] Alistair D. N. Edwards. Progress in sign languages recognition. In International GestureWorkshop on Gesture and Sign Language in Human Computer Interaction, volume 1371,pages 13–21. Springer-Verlag, 1997.

[8] H. Fait. Simulation of user interaction experiences to improve evaluation for accessibility.Master’s thesis, Uc Berkeley, 2003. Jennifer Mankoff (first reader); John Canny (secondreader).

[9] Wai-ling Ho-Ching, Jennifer Mankoff, and James Landay. Can you see what i hear?the design and evaluation of a peripheral sound display for the deaf. In ACM CHI’03Conference on Human Factors in Computing Systems, pages 161–168, Ft. Lauderdale,FL, USA, 2003. ACM Press.

[10] Jason I. Hong and James A. Landay. An architecture for privacy-sensitive ubiquitouscomputing. In Proceedings of the Mobisys’04 Conference on mobile Systems, Applica-tions, and Services, pages 177–189, Boston, MA, 2004.

[11] Gary Hsieh and Jennifer Mankoff. A field study evaluation of peripheral displays. InSubmitted to Ubicomp 2005, 2005.

17

[12] Melody Y. Ivory, Jennifer Mankoff, and Audrey Le. Using automated tools to improveweb site usage by users withe diverse abilities. Information Technology and Society,3(1):195–236, 2003.

[13] Xiaodong Jiang, Nick Y. Chen, Jason I. Hong, K. Wang, Leila A. Takayama, andJames A. Landay. Siren: Context-aware computing for firefighting. In Proceedings ofthe Pervasive’04 Conference on Pervasive Computing, pages 87–105, Vienna, Austria,2004.

[14] S. Lederer. Designing disclosure: Interactive personal privacy at the dawn of ubiquitouscomputing. Master’s thesis, UC Berkeley, 2003. Jennifer Mankoff & Anind Dey (firstreaders); John Canny (second reader).

[15] Scott Lederer, Anind K. Dey, and Jennifer Mankoff. Managing personal informationdisclosure in ubiquitous computing environments. Technical Report UCB-CSD-03-1257and IRB-TR-03-015, Computer Science Division, University of California, Berkeley andIntel Research Berkeley, July 2003.

[16] Scott Lederer, Jennifer Mankoff, and Anind K. Dey. Who wants to know what when?privacy preference determinants in ubiquitous computing. In Extended Abstracts of CHI2003, ACM Conference on Human Factors in Computing Systems, pages 724–725. ACMPress, April 5-10 2003.

[17] Yang Li, Jason I. Hong, and James A. Landay. Topiary: A tool for prototyping location-enhanced applications. In Proceedings of ACM UIST’04 Symposium on User InterfaceSoftware and Technology, page ??, 2004.

[18] E. Horvitz M. van Dantzich, D. Robbins and M. Czerwinski. Scope: Providing awarenessof multiple notifications at a glance. In Proc. of the 6th Intl Working Conf. on AdvancedVisual Interfaces (AVI ’02), page To Appear. ACM Press, May 2002.

[19] Robert Malkin and Alex Waibel. Classifying user environment for mobile applicationsusing linear autoencoding of ambient audio. In Proceedings of ICASSP 2005, page ToAppear, 2005.

[20] Lena Mamykina, Elizabeth Mynatt, and Michael A. Terry. Timeaura: Interfaces for pacing. In Proceedings of ACM CHI 2001 Con-ference on Human Factors in Computing Systems, pages 144–151,Available at: http://www.acm.org/pubs/articles/proceedings/chi/365024/p144-mamykina/p144-mamykina.pdf2001.

[21] Jennifer Mankoff, Anind Dey, Udit Batra, and Melody Moore. Web accessibility for lowbandwidth input. In Proceedings of ACM ASSETS 2002, pages 17–24, Edinburgh, UK,2002.

18

[22] Jennifer Mankoff, Anind Dey, Gary Hsieh, Julie Kientz, Scott Lederer, and MorganAmes. Heuristic evaluation of ambient displays. In Proceedings of CHI2003, page ToAppear. ACM Press, 2003.

[23] Jennifer Mankoff, Holly Fait, and Ray Juang. Evaluating accessibility thorugh simulat-ing the experiences of users with vision or motor impairments. IBM Systems Journal,page ??, To Appear.

[24] Jennifer Mankoff, Scott E. Hudson, and Gregory D. Abowd. Interaction techniques forambiguity resolution in recognition-based interfaces. In Proceedings of the ACM UIST2000 Symposium on User Interface Software and Technology, pages 11–20. ACM Press,November 2000.

[25] Jennifer Mankoff, Scott E. Hudson, and Gregory D. Abowd. Providing integratedtoolkit-level support for ambiguity in recognition-based interfaces. In Proceedings ofACM CHI 2000 Conference on Human Factors in Computing Systems, pages 368–375.ACM Press, 2000.

[26] T. Matthews, A. Dey, J. Mankoff, S. Carter, and T. Rattenbury. A toolkit for managinguser attention in peripheral displays. In Proceedings of the ACM UIST’04 Symposiumon User Interface Software and Technology, pages 247–256. ACM Press, October 2004.

[27] Elizabeth Mynatt, Maribeth Back, Roy Want, Michael Baer, and Jason B. Ellis. De-signing Audio Aura. In Proceedings of the Conference on Human Factors in ComputingSystems (CHI ’98). ACM Press, 1998.

[28] Nitin Sawhney and Chris Schmandt. Nomadic radio: Scaleable and contextual notifica-tion for wearable audio messaging. In Proceedings of ACM CHI 99 Conference on HumanFactors in Computing Systems, volume 1 of Wearable and Tangible Audio, pages 96–103, Available at: http://www.acm.org/pubs/articles/proceedings/chi/302979/p96-sawhney/p96-sawhney.pdf1999.

[29] J. Somervell, D. S. McCrickard, C. North, and M. Shukla. An evaluation of in-formation visualization in attention-limited environments. In Proceedings of thesymposium on Data visualisation 2002, pages 211–216. Eurographics Association,Available at: http://www.eg.org/EG/DL/WS/VisSym02/p211-216-somervell.pdf2002.

[30] T-Mobile. Available at: http://www.t-mobile.com/products/overview.asp?phoneid=229040Sidekick ii.

[31] Jingtao Wang and Jennifer Mankoff. Theoretical and architectural support for input de-vice adaptation. In Proceedings of the ACM CUU’03 Conference on Universal Usability,pages pp. 85–92, 2003.

19

[32] Mark Weiser and John Seeley Brown. Designing calm technology. Internet: http:

//www.ubiq.com/weiser/calmtech/calmtech.htm, 1995.

[33] Craig Wisneski, Hiroshi Ishii, Andrew Dahley, Matt Gorbet, Scott Brave, Brygg Ullmer,and Paul Yarin. Ambient displays: Turning architectual space into an interface betweenpeople and digital information. In Proceedings of International Workshop on CooperativeBuildings (CoBuild ’98), pages 22–32, Darmstadt, Germany, February 1998. SpringerPress.

[34] Leila A. Takayama Xiadong Jiang, Jason I. Hong and James A. Landay. Ubiquitouscomputing for firefighters: Field studies and prototypes of large displays for incidentcommand. In Proceedings of CHI 2004, CHI Letters, pages 679–686. ACM, 2004.

[35] Liwei Zhao, Karin Kipper, William Schuler, Christian Vogler, Norman I. Badler, andMartha Palmer. A machine translation system from english to american sign language.In The Fourth Conference of the Association for Machine Translation in the Americason Envisioning Machine Translation in the Information Future, volume 1934, pages54–67. Springer-Verlag, 2000.

20