DAISY Data Analysis and Information SecuritY Lab

Detecting Driver Phone Use Leveraging Car Speakers

Presenter: Yingying Chen

Jie Yang†, Simon Sidhom†, Gayathri Chandrasekaran∗ , Tam Vu∗ , Hongbo Liu†,Nicolae Cecan∗, Yingying Chen†, Marco Gruteser∗, Richard P. Martin∗

†Dept. of ECE, Stevens Institute of Technology ∗ WINLAB, Rutgers University

WINLAB IABNovember 30, 2011

Cell Phones Distract Drivers

2

Cell phone as a distraction in 2009 on U.S. roadways18% of fatalities in distraction-related crashes involved reports

of a cell phone995 fatalities24,000 injuries

Source: “Distracted Driving 2009” National Highway Traffic Safety Administration Traffic Safety Facts, 2009

Talking on Hand-held Cell Visual — Eyes off road Cognitive — Mind off driving

Texting on Hand-held Cell Manual — Hands off wheel Visual — Eyes off road Cognitive — Mind off driving

81% of drivers admit to talking on

phone while driving

18% of drivers admit to texting

while driving

Cell Phones Distract Drivers

3

Do hands-free devices solve the problem?

Minds off driving.

Real-world accidents indicated that hands-free and handheld users are as likely to be involved in accidents

Cognitive load distract driver!

J. Caird, C. Willness, P. Steel, and C. Scialfa. A meta-analysis of the effects of cell phones on driver performance. Accident Analysis & Prevention,40(4):1282–1293, 2008.P. Treffner and R. Barrett. Hands-free mobile phone speech while driving degrades coordination and control. Transportation Research Part F: Traffic Psychology and Behaviour, 7(4-5):229–246, 2004.

Cell Phone Distraction: What’s Being Done?

Law Several States ban handheld phone use

Technology Hard blocking: radio jammer, blocking phone calls, texting, chat … Soft interaction

Routing incoming calls to voicemail, Delaying incoming text notifications Automatic reply to callers

4

Automatic Reply: “I’m driving right now; will get back with you!”

Current Apps that actively prevent cell phone use in vehicle ONLY detect the phone is in vehicle or not!

What’s Being Done? - Is a Cell Phone in a Moving Vehicle ?

5

GPS Handover Signal Strength Car’s speedometer

The Driver-Passenger Challenge

6

I am a passenger!I want to make a phone call.

38% of automobile trips include passengers !

Source: National highway traffic safety administration: Fatality analysis reporting system

Our Basic Idea

7

An Acoustic Ranging ApproachNo need of dedicated infrastructure

Car speakers Bluetooth

Classifying on which car seat a phone is being used No need for localization or fingerprinting

Exploiting symmetric positioning of speakers

Symmetric positioning of speakersPhone connecting with head unit

How Does It work?

8

AudioHead Unit

S1

Left

S2

Right S3

S4

Time of Arrival - Absolute ranging: clock synchronization unknown processing delays

Travel Time: T1

Travel Time: T2

Relative time difference: T2 – T1No clock synchronization Need to distinguish signal from S1 and S2

S1, S2 emit signal simultaneously

Insert a fixed time interval ∆tbetween two channels S1 always come first S2 always come second

No need of signal identifier! No interference from different speakers!

How Does It work?

9

AudioHead Unit

∆t1 - ∆t > 0 => Closer to Left Speaker (S1 )

∆t1 - ∆t < 0 => Closer to Right Speaker (S2 )

∆t

∆t1

S1

Left

S2

Right

t1

t2

t’1t’2

∆t2 - ∆t > 0 => Closer to Front Speaker (S1, S2)

∆t2 - ∆t < 0 => Closer to Back Speaker (S3, S4)

S3

Rear Right

S4

Rear Left

∆t2

- = ?

∆t

∆t1

Walkthrough of the detection system

10

Emit beep signal Record signal Filtering Signal Detection

Relative Ranging

∆t1 - ∆t

Location Classification

Driver v.s.

non-Driver

Walkthrough of the detection system

11

Channel 1 Channel 2

∆t

Beep signal: two channels

High frequency beep Robust to noise:

engine, tire/road, conversation, music Unobtrusiveness

Close to human’s hearing limit

Beep Length: 400 samples (i.e., 10 ms)

∆t: 10,000 samples

Emit beep signal Record signal Filtering Signal Detection

Relative Ranging

∆t1 - ∆t

Location Classification

Driver v.s.

non-Driver

Beep signal design Consider two challenges:

Background noise and unobtrusiveness

Increasing frequency22 kHz

0

engine, tire/road

1 kHz300Hz 3.4kHz

conversation

50Hz 15kHz

Music Beep Frequency Range

Walkthrough of the detection system

12

Recorded signal

Signal distortion:Heavy multipath in-carBackground noiseReduced microphone sensitivity

Emit beep signal Record signal Filtering Signal Detection

Relative Ranging

∆t1 - ∆t

Location Classification

Driver v.s.

non-Driver

Where is the beep signal?

Walkthrough of the detection system

Signal after Filtering

13

Emit beep signal

Filter out background noiseNoise mainly located below 15kHzBeep signal frequency is above 15kHz

Emit beep signal Record signal Filtering Relative Ranging

∆t1 - ∆t

Location Classification

Driver v.s.

non-Driver

STFT FilterMoving window size m: 32 samples

Beep signal

Signal Detection

Walkthrough of the detection system

14

Signal Detection

Threshold td:

Based on noise: μ + 3σ 99.7% confidence level of noise

Robust window W:

Reduce false detection40 samples

Emit beep signal Record signal Filtering Signal Detection

Relative Ranging

∆t1 - ∆t

Location Classification

Driver v.s.

non-Driver

Estimate noise mean and

standard deviation: (μ , σ)

Threshold td

Signal Detected

Robust window

W Change-point detection Identifying the first arriving beep

signal that deviates from the noise

Walkthrough of the detection system

15

∆t: Predefined fixed time interval between two beep sounds

∆t1: Calculated time difference of arrival

based on signal detection ∆t1 - ∆t: Relative ranging -> cell phone to two speakers

Time difference ∆t1:Measured by sample

counting

Emit beep signal Record signal Filtering Signal Detection

Relative Ranging

∆t1 - ∆t

Location Classification

Driver v.s.

non-Driver∆t1 - ∆t

∆t1 - ∆t

Walkthrough of the detection system

16

Emit beep signal Record signal Filtering Signal Detection

Relative Ranging

∆t1 - ∆t

∆t1 - ∆t > 0 => Left Seats (Driver Side)

∆t1 - ∆t < 0 => Right Seats

∆t2 - ∆t > 0 => Front Seats

∆t2 - ∆t < 0 => Rear Seats

With two-channel audio system:

With four-channel audio system: relative ranging from the 3rd or/and 4th channels: ∆t2

Location Classification

Driver v.s.

non-Driver

Driver v.s.

non-Driver

Driver v.s. Passenger

Automobile trips:83.5%: driver only or plus one front passenger;8.7%: a passenger behind driver seat.

Testing positions

Different number of occupantsDifferent noise conditions

Highway Driving 60MPH + music playing + w/o window opened Phones at front seats only

Stationary Varying background noise: idling engine + conversation

Experimental Scenarios

17

Driver’s Control Area

Phones and Cars

Phones

Cars

18

Android Developer Phone 2

• Bluetooth radio• 16-bit 44.1kHz sampling rate• 192 RAM• 528MHz MSM7200

processor

iPhone 3G

• Bluetooth radio• 16-bit 44.1kHz sampling rate• 256 RAM • 600 MHz Cortex

A8processor

Honda Civic Si Coupe

• Bluetooth radio• Two channel audio system• two front and two rear

speakers• Interior dimension

Car I: 175 x 183 cm Car II: 185x 203cm

Acura sedan

ADP2,Civic Iphone 3G, Acura

Highway Driving

ADP2,Civic Iphone 3G, Acura

0

10

20

30

40

50

60

70

80

90

100

Un-calibratedCalibrated

Det

ectio

n Ac

cura

cyResults: Driver v.s. Passenger Phone Use

19

Results

4 channel, all seats2 channel, front seats

Automobile trips:83.5%: driver only or plus one front passenger;8.7%: a passenger behind driver seat.

Results: Position Accuracy

20

Cup-holder v.s. co-driver left

Conclusions

Limitations Phone is inside a full bag or under heavy winter coat Driver places the phone on an empty passenger seat Probabilistic nature of our approach – not intend for enforcement actions

Enabled a first generation system of detecting driver phone use through a smartphone app

Practical today in all cars with built-in Bluetooth Leveraging car speakers – without additional hardware Computationally feasible on off-the-shelf smartphones

Demonstrated the viability of distinguishing between driver’s and passenger’s phone use within the confines of the existing hands-free audio infrastructure

Validated with two kinds of phones and in two different cars Classification accuracy of over 90%, and around 95% with some

calibrations

21

Thank You!

&

Questions

22

http://personal.stevens.edu/~ychen6/ yingying.chen@stevens.edu

DRIVESAFELY

TALK & TEXT LATER

Challenges in Acoustic Approach

Unobtrusiveness The sounds emitted by the system should not be perceptible to

the human ear, so that it does not annoy or distract the vehicle occupants.

Robustness to Noise and MultipathNoise: Engine noise, tire and road noise, wind noise, and music

or conversationsMultipath: A car is a small confined space creating a challenging

heavy multipath scenarioComputational Feasibility on Smartphones

Standard Smartphone platforms should be able to execute the signal processing and detection algorithms with sub-second runtimes.

23

Beep Signal Design

Unobtrusiveness: high frequency beeps Close to the limits of human perception, at about 18 kHz At the edge of the phone microphone frequency response curve

24

Frequency Sensitivity Gap

Detecting Beep Arrival Time

Hard to detect the beep signal in time domain Heavy multipath in-car environments The use of a high frequency beep signal leads to distortions due to the reduced

microphone sensitivity in this range

Idea: detecting the first strong signal in beep frequency band Filtering: applying STFT in each moving window to extracting beep signal energy

at beep signal frequency band Signal Detection: Identifying the first arriving beep signal that deviates from the

noise

25

Detecting First Arriving signal

Problem formulation Let {X1, ...,Xn} be a sequence of recorded audio signal by the mobile phone over n

time point. Based on the observed sequence up to time point t {X1, ...,Xt}, Identifying:

Xt is noise Xt is beep signal

Idea: Change-point detection Estimate the mean value of the noise μ using the recorded signal at first beginning For {X1, ...,Xt}, calculate accumulative sum of difference between the signal Xi and

noise mean:

Sk = max {Sk-1, 0}, where S0 = 0. Given the threshold td, Xt is beep signal if

Sk > td, ………, Sk+m > td, where m is a robust window

26

Detecting First Arriving signal

Illustration

27

Estimate Noise Mean and

standard deviation: (μ , σ)

Threshold td

Robust window

W

Signal Detected

Results: Left v.s. Right Classfication

28

A B C D E

Results of Relative Ranging

29

Experimental set up Line of sight in-car environments -> heavy multipath Music playing at 60dB and 80dB, respectively -> moderate noise and loud noise

Correlation based method Chirp signal -> robust to moderate noise Signal detection: correlating chirp signal with recorded signal

Metric Successful ranging ratio: ranging error less than 10cm

Computational Complexity

30

Bounded by the length of the audio signal needed for analysis STFT: O(nm logm),

m is the STFT window = 32, n is the number of samples analyzed = 1000 samples/beep sound

Signal detection algorithm: O(n)

Run Time ADP2 with Jtransforms library Average processing time:

0.5 second for two-channel system 1 second for four-channel system