Sensors Know When to Interrupt You in the Car: Detecting Driver Interruptibility Through Monitoring

of Peripheral Interactions

SeungJun Kim1, Jaemin Chun2, and Anind K. Dey3 Human-Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, PA 15213 USA

{sjunikim, anind}@cs.cmu.edu1,3, and jaemchun@andrew.cmu.edu2

ABSTRACT Interruptions while driving can be quite dangerous, whether these are self-interruptions or external interruptions. They increase driver workload and reduce performance on the primary driving task. Being able to identify when a driver is interruptible is critical for building systems that can mediate these interruptions. In this paper, we collect sensor and human-annotated data from 15 drivers, including vehicle motion, traffic states, physiological responses and driver motion. We demonstrate that this data can be used to build a machine learning classifier that can determine interruptibility every second with a 94% accuracy. We present both population and individual models and discuss the features that contribute to the high performance of this system. Such a classifier can be used to build systems that mediate when drivers use technology to self-interrupt and when drivers are interrupted by technology.

Author Keywords Interruptions; Naturalistic Driving; Sensor Data Mining

ACM Classification Keywords H.5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous

INTRODUCTION Human attention is a finite resource [25]. When interrupted while performing a task, this resource is split between two interactive tasks. People have to decide whether the benefits from the interruptive interaction will be enough to offset the loss of attention from the original task. They may choose to ignore or delay dealing with the interruption to a more convenient time. They may alternately choose to immediately address the interruption but this comes with a risk of reduced performance on the primary task or a delay in resuming their primary task [19, 22, 27].

The issue of dealing with peripheral tasks (both self-interruptions and external interruptions) is particularly critical in driving situations. Under normal driving conditions, drivers should have an appropriate following distance

(ideally 2-3 seconds behind the car in front of you [e.g., 12]) for safe driving. This following distance provides the driver with enough reaction time to decide whether to stop, slow down, or otherwise react to changing driving conditions.

However, driving guidelines like this assume that the driver is fully attending to the driving task. If a driver is performing peripheral tasks, those not directly related to the driving task (e.g., turning off a smartphone alarm or changing the radio station), she needs to manage when to divide her attention. Depending on the frequency and duration of these additional tasks, drivers need to adapt how they apply safe driving guidelines.

Fortunately, drivers naturally adapt to changing driving conditions when dealing with peripheral tasks, e.g., waiting for a red light to attend to these tasks, and not performing them in heavy traffic. They determine appropriate timings for changing the positions of their hands on the steering wheel, controlling the foot pedals, all while monitoring adjacent traffic. However, if a demand for peripheral interaction arrives at an inappropriate or unexpected time (e.g., phone rings while changing lanes to exit the highway), it can lead to dangerous driving situations which may result in a driving violation, accident or even loss of life. In fact, 25% of car accidents in the U.S. are related to phone use [8].

By identifying situations or patterns when drivers are not able to attend to peripheral tasks due to their current state and driving situation, a workload manager (e.g., Green [9]), may regulate the flow of information to drivers that could otherwise interfere with driving. Intelligent systems will be able to mediate the delivery of external interruptions, or even disable phone/infotainment systems to mediate self-interruptions [e.g., 24], to support safer driving.

In this context, we investigate when and how drivers perform peripheral activities while driving. We identify driver and driving states and road configurations in which drivers choose to deal with interruptions.

Specifically, we explore three overarching questions:

What are the driver and driving states in which drivers perform peripheral activities?

What sensors and sensor features best explain the states in which drivers perform peripheral interactions?

How quickly and accurately can systems assess opportunities for peripheral interaction?

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the fullcitation on the first page. Copyrights for components of this work owned by othersthan ACM must be honored. Abstracting with credit is permitted. To copy otherwise,or republish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee. Request permissions from Permissions@acm.org. CHI 2015, April 18 - 23 2015, Seoul, Republic of Korea Copyright 2015 ACM 978-1-4503-3145-6/15/04…$15.00 http://dx.doi.org/10.1145/2702123.2702409

Through an in-car data collection with 15 drivers, we collected a large variety of sensor data to understand driver state and driving situations including vehicle motion states, road conditions, driver motion changes, and physiological responses, while capturing driver interactions and traffic states during naturalistic driving. We demonstrate that this data can be used to build a population model that is 94% accurate at determining the interruptibility of a driver each second. We discuss the most informative features that contribute to this performance, and how such a model can be used in practice for mediating interruptions while driving. We start with an overview of background work motivating the need for sensor-based assessment of driving experiences during naturalistic driving, especially in the situations of divided attention due to in-car interruptions.

BACKGROUND WORK

Driver Experience Sampling Traditionally, user experience has been sampled by asking people to stop mid-task and note their experience [17]. The point is for users to record in situ aspects of experience like mental effort or emotion, based on their own judgment. When users are engaged in naturalistic and uncontrolled real-world tasks, that approach provides low-resolution data since sampling rates need to be low to avoid disrupting the user too much, but are often too low to track dynamically varying user states [10]. This problem is especially disadvantageous in mobile contexts such as driving.

In automotive contexts, when directed to self-report experience, drivers must divert their attention from the driving task. This can diminish cognitive capabilities for the primary task by drawing attention to interruptive demands for peripheral interaction; while driving, interruptions can negatively impact primary task performance. As well, when sampling user state at the end of a driving session, in situ variations of that state can be blurred in relation to the overall user experience. Due to these potential issues, driver experience sampling based on self-reports has typically been explored for evaluating user interfaces (e.g., dashboard designs) or within driving simulations (e.g., [13],[16]) where interruptions are less dangerous.

In our work, we seek to better understand the relationship between a driver’s performance of peripheral tasks and in-situ driver states and driving contexts. This necessitates the continuous sampling of user state while driving; therefore, we mainly explore sensor-based assessments to sample driver states. This approach provides quantifiable, fine-grained data in real-time. The feasibility of this approach has been validated in simulated driving experimentation (e.g., [12], [21]), and we apply this to naturalistic driving.

Interruptions As the physical world becomes increasingly connected with our information spaces, so too is the likelihood that information will be pushed to people during the performance of real-world tasks. At best, those

interruptions may alert users to important warnings or messages, inquire about people’s status (e.g., affective states or health conditions for health care), or deliver information that can benefit task performance. Despite this potential value, dealing with these interruptions through peripheral interaction (interaction not directly related to the primary task) demands cognitive attention that can negatively and variably impact user experience. An improved understanding of user availability or interruptibility is necessary for mediating this impact.

Task interruptions result in a time lag before users can resume their primary task, and thus decrease primary task performance [19, 27]. Appropriate timings of interruptions can reduce the impact on users. For example, in the context of desktop computing, interruptions delivered at points of lower mental workload reduced resumption lags and minimized disruption in primary task performance compared to interruptions at points of higher mental workload [1, 6]. In an experiment in which participants were interrupted with emails about consumer products and prices [22], users who experienced deferrable interruptions during high cognitive workload tasks frequently disregarded the notifications until they reached low workload periods. The results of another experiment showed that when peripheral tasks involving reasoning, reading or arithmetic interrupt the execution of primary tasks, users require more time to complete the primary tasks, commit more errors, and become more annoyed and increasingly anxious than when those same peripheral tasks were presented at the boundary between primary tasks [2].

These studies offer important insights for designing human-centered interruptions; however, they have mostly explored static, on-screen tasks mediated with conventional computers or mobile devices (e.g., the impact of call notifications for smartphone users [3]). Little research has been conducted to replicate these findings or approaches for delivering interruptions in situations in which users cannot fully divert their attention from the primary task (e.g., driving cars) and in which interruption timings can critically impact the user experience. We attempt to address this in our work.

Naturalistic driving To ensure driver safety, driving studies about dual task paradigms have mostly been conducted in simulated environments [e.g., 21]. When using sensors to track drivers’ eye gazes or physiological responses in these simulations, the experimental design requires special attention to achieve valid and realistic data. Due to this issue of driver safety, existing naturalistic driving datasets mostly include audiovisual records, traffic and lane information from vision-based systems [e.g., 18], and researcher-estimated driver states using image processing techniques [e.g., 15] on videos from multiple cameras installed in cars.

Recent advances in wearable technologies have resulted in sensors that are less intrusive and more comfortable to wear;

nthdm(inaavthssncw

EHns

PSPctw

Pathr(tirar

TdthfoWwsa

nevertheless lithat tracks driv

during naturalmostly evoked(e.g., auditory sntervals [26])

accelerometers assess driver vehicle states (hese sensing te

streams and festate and the novelty lies continuously mwithout artificia

EXPERIMENTAHere we descrnaturalistic drisituations when

Participants: SD=14.3, age raParticipants wecompensated wwo hours to co

Participants haaverage (SD=1hat they drove

range: 2 - 35 t(SD=4.7, rangeimes per week

routes 1.2 timeaverage drivingrange: 10 - 100

Test-bed: As sdiagnostics (OBhey were aske

four acceleromone chest belt We installed twwindshield andseat. The phonand drivers’ a

Figure 1: Expbody-worn se

ttle research (ever body motilistic driving. d by imposingstimuli at pre-s. On-board don the steeriaggressivenes

e.g., [7, 11]). Iechnologies toeatures are mo

performance in detecting

monitoring natually manipulati

AL SETUP ribe our experiiving data forn peripheral int

We recruitedange: 19 - 69, gere asked to d

with $20. The emplete, includi

ad 14.5 year3.2, range: 1 -

e on city streettimes) and on e: 1 - 20 times)k (SD=9.4, ranges per week (Sg trip duration

0 minutes).

shown in FiguBD) device ined to wear fiv

meter sensors fosensor for trac

wo smartphoned the other on ne cameras recactivities in th

erimental setupensors, and one

e.g., [23]) has on or physiolo

Driver workg artificial duastructured interdiagnostic (OBing wheel havss, driving enIn our work, wo help us identiost useful for of peripheral

g driver inteuralistic drivering cognitive d

imental methor the purposeteractions occu

d 25 drivers gender: 14 femadrive their ownexperiment tooing about 1.25

rs of driving - 52 years). Ths 8.7 times perhighways 4.5 . They used fa

ge: 2 - 50 timesSD=1.0, range: n was 30.4 mi

ure 1, we pluggn study particive body-worn for capturing bcking physiolos in each car - the head-rest oorded traffic ihe cars. They

p – two smartphon-board diagn

been performogical responskload has beal-task demanraction times anBD) systems ve been used nvironment an

we leverage all ify which sens

tracking drivl activities; oerruptibility br / driving statdemands.

od for collectine of identifyinur.

(age M=32.ale and 11 malen cars and we

ok approximatehours of drivin

experience ohey self-reportr week (SD=7.times per we

amiliar routes 8s) and unfamili

0-3 times). Thnutes (SD=19.

ged an on-boaipants’ cars, an

sensor devicebody motion anogical response

one on the froof the passengn adjacent lany also receiv

hone cameras, fnostics in the ca

ed ses en

nds nd or to nd of

sor ver our by tes

ng ng

.0, e). ere ely ng.

on ed .1, ek

8.8 iar he .2,

ard nd es: nd es. ont ger nes ed

informvia Blu

Main naturalwere sThe rodrivingany pr(accordaway).drivingdowntodrivingdriversto reac

Some same dBy fixidriversterms othe lenhill grain the on fam

Sessionsessionthem devicecomplecomple

For sywearabdevice tasks. timesta

For coto grippositiobreath.worn s

In thecollectand infperform

In the pwere completo driperform

In the pthat thwhen Specifi

five ars.

mation from theuetooth and log

task: Study plistic field driselected to comoute in the 1st

g to a shoppinreferred or fading to Google. The route ing on city streown area, alog compared tos were asked tch the destinatio

participants peday while otheing the destinas were exposeof the number ngth of highwadient. By leav1st session, we

miliar and unfam

ns: There wasn: we installed wear the phys and collectete a pre-questete a post-task

yncing the deble sensors for

setup both prThose time-w

amps across da

ollecting baselip the steering ons, while clo. We collectedsensors for 60 s

pre-questionnted demographformation abouming periphera

post-questionnasked to de

eted session, (iving routes, mance) and com

post-interviewhe driver and d

drivers canfically, we pres

e OBD devicegged sensor da

participants peiving. Driving mbine both cit

session consing mall. Drivefamiliar route e maps, the den the 2nd sessieets to a sportong with shoro the previous to drive using on (Google ma

erformed bothers performed ations, and in ted to similar c

of signal lightways and street

ving flexibilitye were able to miliar routes.

s a set of activthe devices in

ysiological seted baseline tionnaire, perfoquestionnaire

evices, experir a period of twrior to and aftewindows were ata streams from

ine measuremwheel at the

osing their eyd sensor data seconds of this

naire prior to hic informatiout driving freqal interactions i

naire after eachescribe situati(e.g., number o

traffics, semplete a NASA

w, we sought todriving contextn perform psented drivers w

e and body-woata streams in r

rformed two sroutes in bot

ty streets and isted mainly oers were allow

to get the estination was ion consisted ts stadium in rter lengths osession. In tha provided Ga

aps: 4.8 miles)

h driving sessiothem on diffe

the 2nd sessionconfiguration ots and signs to ts, road curvey about the roucollect data fro

vities performn drivers’ cars ensors, time-s

data, and haorm the drivingand interview.

imenters shoowenty seconds,er completing t

used for synm the worn sen

ents, drivers wthree and nin

yes after takinstreams from

s “at rest” beha

1st driving son of study pquency and pattin cars.

h driving sessiions during tof passengers, ensor comforA-TLX assessm

o validate our ts can help usperipheral inwith 17 images

orn sensors eal-time.

sessions of th sessions highways.

of highway wed to take

destination 14.7 miles mainly of our city’s

f highway his session, armin GPS .

ons on the erent days. , the route, of roads in encounter,

edness, and ute to take om driving

med in each and helped

synced the ad drivers g task, and .

ok all the , once after the driving

nchronizing nsors.

were asked ne o’clock ng a deep

the body-avior.

ession, we participants terns when

ion, drivers their just-familiarity

rt, driving ment.

hypothesis determine

nteractions. s of drivers

performing secondary tasks while driving (e.g., eating, car radio interaction), and then asked them to comment on all the activities that they conducted during their driving session, explain what factors led them to perform those peripheral interactions, including traffic, in-car and on-road situations, and then comment on whether the moments when they chose to initiate peripheral interactions while driving would also be good for being interrupted.

Peripheral Interaction Moments Correspond to Driver Interruptibility In regular driving situations, there are many instances of drivers engaging in peripheral interactions: actions that take place in the car, but are not related to driving the car. We believe that these instances can be used as ground truth of moments when the driver believes he is able to enter a situation of divided attention [5], and, therefore can better deal with interruptions. This belief needs to be validated.

For example, if a driver eats or drinks while at a red light and she usually interacts with the car radio interface searching for a particular radio station while driving at a constant speed on a street with no noticeable grade or curve in the road, these situations represent conditions in which the driver feels she is able to divide her attention. Drivers may consciously or subconsciously know about the moments in which they can perform multiple tasks at the same time. If they are not actually engaged in any peripheral interaction during such a situation, this may imply that this moment can be used to interrupt the driver with external information (e.g., upcoming traffic information, text message or even advertisements), more safely than in other conditions where no peripheral interactions are ever performed (e.g., during acceleration or driving on a sharp curve). It is also important to separate out interactions with activities that are performed to directly support the driving task – for example, operating blinkers for changing lanes and wipers for better visibility when raining. Moments involving these types of tasks cannot necessarily be viewed as opportunities for performing peripheral interactions or interrupting the user.

Validation: In the post interviews, all drivers described moments in which they initiated peripheral interactions during the driving session they just completed. They stated that these moments were appropriate for being interrupted.

One participant said he would perform peripheral interactions “usually if I was at a red light or if I was stuck in traffic, also depends on the situation - if I don't have to make a turn and it's a straight road. I didn't do anything while I was making a turn or going through an intersection. I waited until it was a straight stretch of road with no places where cars would pull out and low traffic.”

Another said “In most cases, I chose when to have the interaction - when to turn on the wipers, when to change the radio station or adjust the volume... except when I needed more attention on driving. Wipers I turned on

because there was too much stuff on the view. Radio, I changed channels when I thought it was a good time too. If I were stopped in traffic, then yes, I would think that that's also a good time to [receive interruptions].”

TIME-SERIES DATA Based on these interview results validating that moments in which peripheral interactions are performed are times when drivers could be interrupted, we proceeded with the analysis of the captured sensor data to automatically identify these moments. We started this process with identifying ground truth about what drivers were doing during driving sessions.

Videos and Annotation features Camera 1: One of the two smartphone cameras was used for examining drivers’ activities in cars. We manually labeled the captured videos for moments when peripheral interactions were performed and not performed. We used five labels. DRIVING_I includes activities that are quite central to the primary driving task (e.g., changing grip positions, operating levers for blinkers or wipers, switches for opening side windows), whereas PI includes activities that are not directly central to the primary task (e.g., eating food, manipulating the air conditioner or car radio, talking on the phone). ONE_HAND_DRIVE_WITH_NO_PI was used to label moments when one of the driver’s hands was off the steering wheel but that hand was not performing any specific activity. NO_HAND_DRIVE was used for moments when both hands were off the steering wheel and not performing any peripheral activity. STEERING_ONLY was used for moments when both hands were on the wheel. From our interviews, moments that are labeled with PI or NO_HAND_DRIVE indicate that drivers have higher interruptibility, compared with moments of DRIVING_I or STEERING_ONLY.

Camera 2: The other smartphone camera was used for assessing the traffic around the driver. We labeled the videos for the amount of traffic visible in front of the car, to the left and right of the car, and in the oncoming lanes. The traffic labels we used were NO_TRAFFIC, IGNORABLE, SOME, A_LOT based on the approximate distance of adjacent cars and the number of cars. If the traffic in the oncoming lane was occluded by cars in the left lane or a center guardrail on a highway, we labeled it as OCCLUDED. We also had a label indicating the car’s movement, whether it was STOPPED or MOVING.

Three experimenters, each having more than 4-years experience in psychology or human computer interaction, reviewed videos from the smartphones and labeled the kind, start time, and end time of drivers’ peripheral activities, and traffic. Note that while we manually labeled the videos from both cameras, these are labels that could be automatically provided using computer vision (e.g., [4]).

Six Sensors and Applied features An OBD device provided information about the status of the vehicle (sampled at 1Hz) including longitude, latitude,

athTBr

Fpofinuaatrsacinra

Aawfathdacin1mmFbFrhYtos

DOdds3rdedath

Wd

altitude, car hrottle positio

The data wBluetooth toresponsible for

Four YEI 3-Spplaced on eachon the front of foot. The nformation abo

using a tri-axiaaccelerometer, all at 4-5 Hz.ransmitted via

smartphone. Ina BioHarness collected drivencluding ele

rate, respirationand activity at 2

After time syaggregated, andwere calculatedfeatures (referraddition, we dehe basic featu

derived road cuand latitude centrifugal forcnformation, ty

1st derivative ofmotion informamoved from thFrom the BH breathing-in, bFigure 2 – toreferred to as ‘had 100 basic YEI: 40, BH: 4o traffic from

states around th

DATA COLLECOf our 25 drivdriving sessiondrove for an session (SD = 33m 18.3s in trange: 25m 36sdrivers, most deither when OBdrivers’ kneesapplications crheir mounts or

When examinidrivers’ data. T

speed, enginon, and fuel fwere transmito the smrecording traff

ace devices wh of the driverthe head, and t

devices out the driversal gyroscope, a

and compass The sensor

a Bluetooth to n addition, driv

(BH) chest ers’ physiologectrocardiogramn rate, body or20Hz sampling

yncing the sed their means (d for every 1red to as ‘baserived a series ures. For examurvedness by tcoordinates (ces by combin

ypes and gradief altitude data. ation on how mhe baseline pos

data, we quanreathing-out, aop-left). These‘derived’ featufeatures and 5

40) and 5 manuvideos (one ca

he vehicle – fro

CTION STATUvers, 17 driverns without anyaverage of 437m 23.5s, ran

the second drivs - 51m 32s) onata loss came BD devices ws or when rashed. Occasir turned enough

ing the sensorTheir heart rate

ne RPM, flow rate. tted via

martphone ffic video.

were used, rs’ wrists, top of the collected ’ motions a tri-axial s sensors, data was the other

vers wore belt that

gical data m, heart rientation g rate.

ensors, all sen(µ) and standar-second segmeic’ features inof additional f

mple, from theracking variatiSee Figure 2

ning car speed ent of road sloFrom the YEI

much each monsition (See Figuntified levels and holding-bre supplementaures in this pap52 derived featually annotatedar driving stateont, right, left, a

US s successfully

y loss in data.3m 59.2s in tnge: 28m 42s ving session (n average. Forfrom Bluetooth

were accidentalthe Android

ionally smartph to capture irr

r data, we fure data containe

F

nsor data werd deviations (ent as statisticn this paper). features based oe OBD data wions of longitu2 – top-righand curvedne

ope by using thdata, we deriv

nitored body paure 2 – bottomand duration

reath states (Sary features aper. In total, wtures (OBD: 7

d features relate and four traffand oncoming)

completed bo The 17 drivethe first drivin- 54m 14s) anSD = 7m 42.4

r the remainingh disconnectiolly dislodged b

d data logginphones fell frorelevant data.

rther excluded ed errors (high

Figure 2: Exam(top-l

ere (σ) cal In on we de

ht), ess he ed art

m). of ee

are we 72, ed fic .

oth ers ng nd 4s, g 8 ns by ng

om

2 her

than 2sessionworn tsessionthe senanalysi

STATISWith thhow drstateWITH_N

For anaconducmodel as posvariancas effedata orTest fowas exrating post-ho

RESUL

Driver In totadriversactivitisummeactivitiradio, and drchangithe stee

mples of appliedleft), road curve

200 beats pern). This happetoo loose arounn. We did not nsor data analyis of the questi

STICAL ANALhe data from oriver and drivi

classes: SNO_PI, DRIVIN

alyzing the concted a univariaand then used

st-hoc tests ces (i.e., Leveect size. For orr fixed level dollowed by Maxamined as effdata, Friedma

oc test were co

LTS

Activities: Freal, 53 differents. In the remaiies that had a ed across multies include inteoperating blinkrinking, drivining hand grip, ering wheel, an

d features from edness (top-righ

r minutes duened when thend the driver’sinclude data

ysis although wionnaires and in

LYSIS METHOour 15 remainiing states diffeSTEERING_ONLNG_I, PI, and N

ntinuous measate ANOVA bd either Tukeyafter checkin

ene Statistic), wrdinal measuredata), we conduann-Whitney Ufect size. In thean tests and a onducted.

requency and t activities weining analysis, a total durationtiple instanceseracting with t

nkers or windshng with one h

resting by remnd so forth.

sensor data streht), hand devia

uring the entie BioHarness ss torso during tfrom these 10

we did include tnterviews.

OD ing drivers, weer across our fiLY, ONE_HANNO_HAND_DRIVE

sures (e.g., car by using a geny HSD or Gamng the homogwhere ηp

2 wases (e.g., humanucted the Krus

U test as post-hoe analysis of L

Wilcoxon Sig

Duration ere detected fr

we focus on 2n of at least 9s. As shown inthe air conditiohield wipers, ehand, smokingmoving both h

eams – breathintion (bottom)

ire driving sensor was the driving

0 drivers in them in the

e examined five driving ND_DRIVE_ VE.

speed), we neral linear mes-Howell geneity of s examined n-annotated skal Wallis oc, where r Likert-scale gned Rank

rom the 15 22 of these 90 seconds n Table 1, oner or car eating food g, dancing, hands from

ng states

Opscosd(s

H

H

p

F(wO

TwpawtiaththwpOF

One-handed drper subject, wisegment; See considered, sumover a 10-minuseconds, respecduration per ins(21.7 seconds)smoking, and o

Driver activit

ONE_HAND_DGEAR_SHI

TOUCHING_SELF REST_OFF_W

L_BLINKER_BLINKECAR_RAD

R_GRIP_CHAHEAD_ TURNING_W

CELLPHONL_GRIP_CHA

HEAD_TURNING_WGPS_PORTA

AC_CONTROL_SWBOTH_HANDS_OF

FOOD_OR_DOTHERS_MISCELL

(e.g., shouting oOTHERS_MU

SMOKINGWIPER_WASAC_AIR_VEN

OTHERS_DA

Table 1. Freqactivities sorte

per unit, in whi

For most of the(48.0%) or onwithout any perONE_HAND_DRIV

The time spentworkload (e.g.peripheral activare consciouslyworkload for time spent on

availability forhat the averaghe five interact

wise differencepost-hoc test wONE_HAND_DRIVFigure 3 – two

riving most freith 6.9 instanc

Table 1). mmed up for ute segment, it ctively). Howestance, using o), followed b

one-handed driv

Occurre

ties / subject

DRIVE 57.1 FT 37.1 (e.g., face) 24.5

WHEEL 20.6 R 15.7

ER 14 IO 9.2

ANGE 8 WAY_LEFT 7.3 NE 6.7

ANGE 6.7 WAY_RIGHT 5.7 ABLE 4.5 WITCHES 3.1 FF_DRIVE 2.3 RINK 2.6 LANEOUS

outside) 2.7

USIC 2.7 G 1.9 HER 1.7 NTS 1.1

ANCE 0.1

Total

quency and dured by a descendich top-five mea

e driving time,e hand (22.8%ripheral interacVE_WITH_NO_P

t in performin, [10]). Simil

vity during drivy or subconscthat activity. Inn a peripheralr being interrue duration wastion states (F4,4

e between eacwith the excepVE_WITH_NO_P

o rightmost bar

equently occures for each 10When total each participawas also first

ver, when rankone’s cellphoneby eating fooving.

ence Duration (sec

/ 10-min

drive / subject

/ 10mindrive

6.9 15.4 117.4.5 2.7 19.03.0 7.6 28.52.5 14.7 37.21.9 3.9 8.61.7 4.4 9.41.1 5.9 9.51.0 2.9 6.30.9 4.2 4.80.8 13.5 17.60.8 1.8 2.00.7 2.6 1.90.5 14.3 6.50.4 1.8 1.00.3 2.2 3.00.3 11.6 5.9

0.3 6.5 3.3

0.3 1.4 5.20.2 1.2 4.30.2 2.9 1.60.1 3.5 1.50.0 0.7 0.2

23

ration of 22 top-ding order basedasures and activ

, drivers used %) for controlction (See STEEPI in Figure 3).

ng a task is inarly, duration ving may indicciously estiman other wordsl interaction,

upted. In this ss significantly 4546=116.2, p =

ch state was siption of the duPI and NO_HANrs). This impli

rred (57.1 tim0-minute drivindurations we

ant and summ(15.4 and 117

king these by the was the longeod or drinkin

c) Duration (sec / occurrenc

0-n e

M (SD)

.0 17.0 (26) 0 4.2 (6.7) 5 9.7 (21.5) 2 15.0 (14.1)

6 4.6 (16.8) 4 5.6 (22) 5 8.6 (16.7) 3 6.6 (12.5) 8 5.5 (12.1) 6 21.7 (26.9)

0 2.5 (3.1) 9 2.8 (2.7) 5 12.1 (13.5) 0 2.5 (1.5) 0 11.0 (11) 9 18.8 (16.6)

3 9.9 (10.7)

2 16.2 (15.3) 3 18.3 (41.8) 6 7.5 (12.5) 5 11.7 (19.2) 2 11.0 (8.5)

35.3 28.3 5.7 13.410

(19

-ranked driver d on occurrencevities are in bol

both their hanlling the vehicERING_ONLY an.

ndicative of usof a particul

cate how driveating interactios, the longer ththe greater th

study, we foundifferent amon

= .000). The paiignificant in th

urations betweND_ DRIVE (Sies that althoug

mes ng ere ed

7.0 he est ng,

e)

0.4 9.7)

e d.

nds cle nd

ser lar ers on he he nd ng ir-he en ee gh

drivershandedsteerinduratiowill dis

Driver turningaveragsecond

DrivingThe Oregulator takinthe leSTEERIperiphe4), andwhile d

When enginetime significIn partof Figuwhile (statiststates).consid

Figstat

FigurSTE

s did not actuad driving is a dng. Similarly, tons between onscuss this findi

r activities thatg on blinkers)ge, while perds (DRIVING_I

g States: Car OBD data strea

ted car speed ing both handseft-most bar ING_ONLY stateeral interactiond almost reducdriving withou

drivers were e RPM was sigspent on flacantly higher, ticular, as indicure 4, drivers driving with

tically significa. Interestingly,erably high sp

gure 3: Averagetes. *p<.001 com

re 4: Driving staEERING_ONLY

ally execute anydistinct state frthere was no dne handed anding later in the

t corresponded) were complripheral interaand PI in Figu

Speed and Roams revealed twhile perform

s off the steerinin each grou

e, car speed wns were perforced completely

ut either hand o

in either of tgnificantly lowatter roads (compared to tcated by for

were on flattboth hands

ant when comp, one-handed d

peeds, without

e time durationmpared with ST

ates and road costates; p<.001

y peripheral acrom using bothdiscernible disd no-handed de paper.

d to the drivingleted in 2.4 sactions lasted ure 3).

oad Conditionthat drivers si

ming peripherang wheel (See up). Compareas reduced to rmed (PI statey (slower than

on the steering

these two slowwer and the pro(i.e., low grathe STEERING_r the right-moster and less cuoff the steer

pared to each odriving was penecessarily be

n of drivers' inteTEERING ONL

onditions. *p<.01 versus all othe

ctivity, one h hands for stinction in riving. We

g task (e.g., seconds on

for 10.6

ns ignificantly al activities

Figure 4 – ed to the 60% while e in Figure n 3.0km/h) wheel.

wer states, oportion of ades) was _ONLY state. st two bars urvy roads ring wheel of the other erformed at eing on flat

eraction LY states.

001 versus er states.

o

TNrindw

IddND

DTbfppfwwb

DvpIsbbOnh(eNb

ONWp

F

or straight road

These findingsNO_HAND_DRIVErepresent the mnterruptible. H

driving in this were different f

In addition, wdecelerate acrodrive at moreNO_HAND_DRIVEDRIVE_WITH_NO

Driver State: BThe deviation obody pose werefive states acparticular, the performing drivfrom the top). while taking bowithout performbars in Figure 6

Drivers’ breathvaried across p=.000 and F5,3

Interestingly, fsimilarly low baseline sessionbars from FONE_HAND_DRIVnot to NO_HANDholding their (F5,4467=1.99, pexhaling wereNO_HAND_DRIVEbaseline session

Overall, driveNO_HAND_DRIVEWe therefore physiological

Figure 5: Time con

ds.

s imply that VE states are mmoments when However, we group. The drifor ONE_HAND_

when examininoss the five state consistent spVE (73.7%) staNO_PI state (24.

Body Motion aof gravity mease significantly

cross every bright hand m

ving or periphHead movem

oth hands or onming any peri6).

hing states andthe five inte

36565 = 215.0, pfor NO_HAND_Das the resting

n where no driigure 7). AlVE_WITH_NO_PD_DRIVE or bas

breath was p=.075). Rathee significantlVE states, and sn.

er physiologicVE states are mo

confirm thstates, driving

proportion of astant speed acr

driving statusmost similar. In

the driver is lido not inclu

iving state and_DRIVE_WITH_N

ng how drivertes, we see thapeeds during ates than in .6%) (Figure 5

and Physiologsures from the different for e

ody element movement waeral activities

ment was largene hand off thipheral activiti

d heart rate aeraction state

p=.000, respectDRIVE, drivers’g heart rate iving was perfolso, the heaPI state was simseline. The du

not statisticer, durations fly longer fosimilar to whe

cal states forost similar to thhat from oug states and r

acceleration, deross whole drivi

s for the PIn addition, thikely to be moude one-hand

d road conditioNO_PI.

rs accelerate at drivers tend PI (47.8%) the ONE_HAND).

ical States STEERING_ONL

every pair of th(Figure 6).

s distinct whe(2nd and 3rd baer when drivin

he steering wheies (bottom tw

also significants (F5,34693=6.8tively; Figure 7’ heart rate wcaptured in thormed (bottomart rate durinmilar to PI, b

uration of drivecally significafor inhaling anor the PIen resting in th

r the PI anhe resting stateur analysis road condition

eceleration, andng.

or ey

ore ed ns

or to or D_

LY he In en ars ng eel wo

tly 80, 7).

was he

m 2 ng

but ers ant nd or he

nd es. of

ns,

driversHowevany pephysiostates conditi

IdentifGiven states, less inidentifySpecifidetects

To buNO_HANinstanc(INTERfrom thclass; nwe can

Figure

Figure*

s are more intever, interruptiberipheral inter

ological states and the basel

ions differed si

fying Opportuthe statisticaland particular

nterruptible stfy opportune fically, we wans or predicts int

uild this claND_DRIVE statces when RRUPTIBILE). he remaining note that we dnnot know with

e 7: Drivers’ phstreams. *p<.0

e 6: Driver body* p<.01 in every

erruptible thanbility while oneractions is st

are similar tline, but the ignificantly.

nities for intedifferences ac

rly between thtates, we now

moments fnt to build a terruptibility e

assifier, we ctes into a si

drivers cWe correspoclasses into th

do not call thish absolute cert

hysiological stat1 versus STEER

y motion revealy pair of the fiv

n in the remaine-handed driviill unclear sinto the two indriving states

rruptions cross the five he more interruw seek to autfor driver inclassifier that very second.

combined theingle class recan be

ondingly combhe LESS_INTERs “Uninterrupttainty that driv

es revealed in sRING_ONLY sta

led in sensor dave interaction st

ning states. ng without nce driver

nterruptible s and road

interaction uptible and tomatically nterruption.

accurately

e PI and epresenting interrupted bined data RRUPTIBLE tible” since vers are not

sensor data ates.

ata streams. tates.

inth

Ffscfsasp

OcapinRorhrbwsTSds

Nmina

Pin(thw

Hinms1ththince

Wthfthdfa

nterruptible duhey are likely t

For this binaryforest classifiersuch as our sclasses using stfold contains samples from eapplied differesamples that bprocedure for e

Our populaticlassification aacross the datprecision and rnitial list of 14

RPM, GPS beaof the right hanrank-ordered ashuman-annotatranked feature)believe that thwith today’s isecond classifiThe classificatiSD=0.2). This driver interruptsecond using se

Next, we examodels for eandividual diffe

and whether su

Promisingly, tndividual mod

(SD=2.6%, ranhe population

was greater tha

However, the nstances that a

moments corresome individua10, 11, 21, andhe precision rhat are relevannterruptible th

consistent alonevery driver.

We then focusehat contributed

features were che 143 feature

driver’s list of tfrom the bodyannotated data

uring these into be less so.

y classificationr, which runs sensor data. Wtratified 10-fol

approximateleach target clasent sample webelong to eac

each driver and

ion-based claccuracy of 9ta from all 15recall: 0.94 and43 features, ca

aring, centrifugnd deviating frs the most impions about th) and the frontese human animage processer that did noion accuracy d

means that ttibility can be ensor data only

mined the peach of our fierence in whic

uch models nee

the average cdels of our dr

nge: 90.2% frommodel (94.3%

an 90% (See Fi

recall rate (are retrieved = ectly classifiedal differences. d 23) had recalates (i.e., the nt = the percenhat were actuang with the c

ed on the top fd to classificaticommonly usedes, 46 featurestop fifteen featy-worn sensor). Similar to

nteraction state

n problem, weefficiently on

We also handld cross validaly the same ss as the compleights based ch class. We d for each fold.

lassifier had 94.3% classifi5 drivers (SDd 0.89, respectiar speed (OBDgal features, anrom the baselinortant features

he car movemt traffic (7th-rannnotations coulsing technologot use any humdropped by onlthe binary stadiscriminated

y.

erformance offifteen drivers,ch features wed to be persona

classification arivers was 94m 98.2%), alm%) Accuracy fgure 8).

i.e., the fractthe percentage

d as being inFive of our drll rates lower tfraction of ret

ntage of momeally interruptibclassification

fifteen featuresion accuracy, td in the indivis were chosentures (23 from rs, and 2 frothe population

es, but just th

e used a randolarge databas

dled unbalancation where ea

percentage lete set, and thon the ratio automated th

an averagication accura

D=0.3%, averagively). From o

D, km/h), enginnd the movemene position wes, along with tw

ment states (tonked). While wld be automategies, we built man annotationly 2.2% (92.1%

ates representinaccurately eve

f the individu, to investigaere most helpfalized.

accuracy of th.9% on averag

most the same for every driv

tion of relevae of interruptibnterruptible) hrivers (driver 0than 80%, whitrieved instancents identified ble) were fairaccuracy acro

s for each drivto identify whicidual models. On in at least on

OBD device, 2om the human model, engin

hat

om ses ed ch of en of

his

ge cy ge

our ne

ent ere wo op-we ed

a ns. %, ng

ery

ual ate ful

he ge as

ver

ant ble ad

05, ile

ces as

rly oss

ver ch Of ne 21

an-ne

RPM,annotafeaturein our motionforce ffor onl

DICUSIn this featuredetect mopportucontrib

Driver Our seactivitidrivinginitiateon faphysiorestingdriversexperiereporte

An intbut dohand. this staperformstates than weither roads, steer. Tmay hinteracactuallmomenfuture w

Figure

car speed ation features es for more tha

group model,n were also imfor seven drively six drivers.

SSION section, we di

es that contribmoments of in

tunities for fubutions of this p

and Driving sensor data revies when expeg without holde peripheral inairly flat andological signalg states in thes may defer prencing lower ed in prior inter

teresting situatio not perform In general, drate implied lowming peripherfor one-handewhen performaccelerated orand more ofteThis provides ave extra cogn

ctions. It is unly in use for tnt without inwork, we will

e 8: The classificfo

(OBD, km/hwere identifi

an ten of our d, heart rate, fomportant for ers and the fro

iscuss our findbuted to our mnterruptibility. Wuture research paper.

states vealed that drieriencing low

ding the steerinnteractions at ad straight rols were simile baseline sesrocessing peripmental worklorruption studie

ion is when drany periphera

rivers’ physiolwer driver woral interactioned driving we

ming peripherar decelerated wen even than w

an interestingnitive capacity

nclear whetherthe driving ta

nitiating peripinvestigate thi

cation performor our fifteen dr

h), and car fied again as drivers. Howeoot and body five drivers, nt traffic anno

ding on drivingmodel, and ourWe use these t

that can bui

ivers perform er workload. ng wheel, driva near constantoads. Furtherar to those dssion. This im

pheral tasks unoad, confirmines (e.g., [22]).

rivers use onlyal activity witlogical responorkload, closern. However, there much moral interactionwithout avoidi

when using botg hypothesis thy to deal with r this “extra” cask, for enjoyiheral activitie

is issue.

mance of individurivers.

movement important

ver, unlike trunk/torso centrifugal

otation data

g states, the r ability to to point out ild on the

peripheral Similar to

vers tend to t car speed r, drivers’ during the

mplies that til they are

ng findings

y one hand th the idle

nses during r to that of he driving

re dynamic s. Drivers ing curved th hands to hat drivers peripheral

capacity is ing an idle es. In our

ual models

Indicative sensor data features The derived features used to create our classifiers had more discriminative power than the simple statistical (basic) features. For example, the centrifugal feature created with a combination of car speed and road curvedness (more precisely, the square of car velocity divided by the road curvedness feature not considering driver weight) was ranked as the 5th most important in the population model and was in the top-15 features for 7 of the drivers’ individual models. If we incorporated the gravity variation coming from the road slope or the driver’s seating pose, this new feature could be even more powerful.

Similarly, the derived physiological responses (e.g., breathing rates) and road attributes (e.g., categorized slope levels) had statistical differences between driver’s interaction states. However, they did not explicitly contribute to building high performance classification models. Only two basic features, heart rate (in top-15 list for five drivers) and electrocardiography amplitude (ranked 12th in the group model) highly contributed to the classification accuracy.

To explore how to limit the obtrusiveness of our sensing system, we examined the relative values of the different sensors. For building either the group or individual models for predicting driving interruptibility, the most important features come from the cameras and vehicle movement sensors, except for the right hand motion (which could be detected via camera rather than worn sensor). For individual models, though, data from the physiological sensors was also valuable. It is possible that this body part motion could contribute to the group model as well, particularly if combined across body parts or with the car motion. We leave these refinements to future work.

Detecting Driving Interruptibility In this study, we presented promising evidence that driving interruptibility can be accurately assessed in real-time using sensor data streams, independent of human-annotated information. Further, our results also show that accurate personalized models can be constructed from only 2 driving sessions.

However, individual models for a few drivers (i.e., driver 05, 10, 11, 21, and 23) provide insufficient performance in correctly detecting driving interruptibility. Their recall was 75.3% (SD=6%) on average and the precision was 89.0% (SD=1.9%, indicating 11% false discovery rate). This (roughly) implies that one in four known opportunities for interruptions was missed. It also implies that one in ten times the driver was predicted to be interruptible, she was not. This could lead to risky driving situations, although arguably no worse than our current driving situation in which interruptions arrive unmediated to drivers. Given our finding that in a 10-minute driving window, approximately 7 and 3 minutes belong to the LESS_INTERRUPTIBLE and INTERRUPTIBLE states, respectively, this equates to drivers being interrupted when not interruptible for 42 seconds and

missing opportunities for interrupting the driver for 45, over each 10 minute segment of driving. The performance of these individual models should be improved to be used in practice, but our results are already very promising. We may need longer-term data collections from these individuals or more personalized features (e.g., route familiarity or habitual behaviors in the car) to improve the individual model classification accuracy.

Overall, our models performed quite well in identifying moments when the driver is in the NO_HAND_DRIVE or the PI states. While we equated this to the driver being INTERRUPTIBLE, it could be the case that drivers in one of the other states (particularly one-handed driving) could have additional opportunities for being interrupted. In the future, we would like to more thoroughly investigate driver’s interruptibility in situations when they are not performing peripheral activities and not driving no-handed.

CONCLUSION Nowadays, technology is giving us the ability to interact with information anywhere and anytime, even in the car while driving. Whether it is appropriate to interact with information depends a lot on the current driving situation and the driver state. In this paper, we sought to understand these situations and states particularly when drivers engage in peripheral interactions, actions not related to the primary task of driving the car, as they indicate opportunities to interact with information. In particular, we collected sensor and human-annotated driving data from 15 drivers and demonstrated that we could build a machine learning classifier that determined when they were interruptible with 94% accuracy. We discussed the features that contributed to this high accuracy and suggested directions for identifying features to improve on our classifier.

In continuing this work, we have a number of goals. First, we will collect additional data to confirm that our results generalize across a wider driving population and work to build models and apply features that more consistently perform across drivers and for the nuances of specific drivers. Second, with this accurate classifier, and relatively easy-to-deploy system, we can now investigate how to mediate interruptions to drivers. We plan to conduct another field study where we push interruptions at different timings to assess the real-world impact of being able to detect interruptible moments while driving. These interruptions will differ in terms of their temporal urgency, relevance to the driving tasks, and overall importance. Third, we plan to develop generalized guidelines for designing intelligent interruption systems for drivers. We will use these improved models and understanding to identify the attributes of opportune moments detected (e.g., expected duration, expected level of driver engagement). For example, a relevant local advertisement should impose a short interruption during an opportune moment; if a driver has extended interruptibility, interruptions such as an alternate route/highway exit to take can be delivered.

Fourth, we plan to apply our technology to identify breakpoint moments for prompting drivers to safely participate in experience sampling while driving, which will support others doing driving-related research in naturalistic driving situations.

REFERENCES 1. Bailey, B.P., Iqbal, S.T. (2008). Understanding changes

in mental workload during execution of goal-directed tasks and its application for interruption management, ACM TOCHI, 14 (4): 1-28.

2. Bailey, B.P., Konstan, J.A. (2006). On the need for attention-aware systems: Measuring effects of interruption on task performance, error rate, and affective state. Computers in Human Behavior, 22 (4): 685-708

3. Böhmer, M., Lander, C., Gehring, S., Brumby, D.P., and Krüger, A. (2014). Interrupted by a phone call: exploring designs for lowering the impact of call notifications for smartphone users. Proc. CHI '14, 3045-3054.

4. Cho, H., Rybski, P.E., Bar-Hillel, A., and Zhang, W. (2012). Real-time pedestrian detection with deformable part models. In IEEE Intelligent Vehicles Symp.: 1035–1042.

5. Curry, D., Meyer, J., and Jones, A. (2013). Driver Distraction: Are We Mistaking a Symptom for the Problem?. SAE Technical Paper 2013-01-0439.

6. Cutrell, E. B., Czerwinski, M., and Horvitz, E. (2000). Effects of instant messaging interruptions on computing tasks. Proc. CHIEA '00, 99-100.

7. Espinosa, F., Jimenez, J.A., Santiso, E., Gardel, A., Perez, D., Casanova, J. and Santos, C. (2011). Design and implementation of a portable electronic system for vehicle-driver-route activity measurement. Measurement 44 (2): 326-337.

8. Governors Highway Safety Association. Distracted driving: What research shows and what states can do. Technical Report. July 2011.

9. Green, P. (2004). Driver Distraction, Telematics Design, and Workload Managers: Safety Issues and Solutions in SAE publication P-387. Society of Automotive Engineers, Inc., Pennsylvania, USA, pp. 165-180

10. Haapalainen, E., Kim, S., Forlizzi, J., Dey, A. (2010). Psycho-physiological Measures for assessing Cognitive Load. In: Proc. Ubicomp '10, 301–310.

11. Hong, J., Margines, B., Dey, A.K. (2014) A Smartphone-based Sensing Platform to Model Aggressive Driving Behaviors, Proc. CHI '14, 4047-56.

12. Illinois. Rules of the Road. (2014). See Chapter 11 Safe Driving Tips for details on the "three-second rule".

13. Iqbal, S.T., Horvitz, E., Ju, Y.C., and Mathews, E. (2011). Hang on a sec!: effects of proactive mediation of phone conversations while driving. Proc. CHI '11, 463-472.

14. Kim, S., Hong, J., Li, K.A., Forlizzi, J., and Dey, A.K. (2012). Route Guidance Modality for Elder Driver Navigation. Pervasive '12 , 179-196.

15. Klauer, S.G., Guo, F., Sudweeks, J. and Dingus, T.A. (2010). An Analysis of Driver Inattention Using a Case-Crossover Approach 100-Car Data: Final Report (Report No. DOT HS 811 334). NHTSA.

16. Kun, A.L., Shyrokov, A., and Heeman, P.A. (2013). Interactions between human-human multi-threaded dialogues and driving. Pers Ubiquit Comput. 17(5): 825-834.

17. Larson, R., and Csikszentmihalyi, M. (1983). The experience sampling method". New Directions for Methodology of Social and Behavioral Science. 15, 41-56.

18. Lee, J. D., Moeckli, J., Brown, T. L., Roberts, S. C., Schwarz, C., Yekhshatyan, L., Nadler, E., Liang, Y., Victor, T., Marshall, D., & Davis, C. (2013). Distraction Detection and Mitigation Through Driver Feedback. (Report No. DOT HS 811 547A). NHTSA.

19. Monk, C., Trafton, J., and Boehm-Davis, D. (2008). The effect of interruption duration and demand on resuming suspended goals, Journal of Experimental Psychology: Applied, 14 (4): 299-313.

20. Mulligan, C. B., Schneider, B., and Wolfe, R. (2000). Time use and population representation in the Sloan Study of Adolescents. Unpublished manuscript, Alfred P. Sloan/University of Chicago Center for the Study of Working Families.

21. Riener, A., Chalfoun, P., and Claude Frasson, C. (2014). The Potential of Subliminal Information Displays to Change Driver Behavior. Presence 23(1): 51-70.

22. Salvucci, D.D. (2010) On reconstruction of task context after interruption. Proc. CHI '10, 89-92.

23. Schneegass, S., Pfleging, B., Broy, N., Heinrich, F., and Schmidt, A. (2013). A data set of real world driving to assess driver workload. Proc. Automotive UI '13, 150-7.

24. Sherin, K.M., Lowe, A.L., Harvey, B.J., Leiva, D.F., Malik, A., Matthews, S., Suh, R., and the ACPM Prevention Practice Committee. (2014). Preventing Texting While Driving. Am J Prev Med. (in press).

25. Simon, H.A. (1971). Designing organizations for an information rich world. In Computers, Communications, and the Public Interest: 37-72.

26. Solovey, E.T., Zec, M., Perez, E.A.G., Reimer, B. and Mehler, B. (2014). Classifying driver workload using physiological and driving performance data: two field studies. Proc. CHI '14, 4057-4066.

27. Trafton J. G., Altmann E. M., Brock D. P., and Mintz F. E. (2003). Preparing to resume an interrupted task: Effects of prospective goal encoding and retrospective rehearsal, IJHCS, 58 (5): 583-603.