session 48 johan karlsson
TRANSCRIPT
ALR-JKAR/Jan2011/Transportforum - 1 Copyright Autoliv Inc., All Rights Reserved
Fordonsstrategisk Forskning och Innovation FFI – D4SF
Driver Drowsiness and Distraction Detection bySensor Fusion
D4SFJohan Karlsson, AutolivTransportforum 2011
ALR-JKAR/Jan2011/Transportforum - 2 Copyright Autoliv Inc., All Rights Reserved
Fordonsstrategisk Forskning och Innovation FFI – D4SF
Overview
�Background�Goals
� Drowsiness detection� (Distraction detection)
�Method� Data collection� Training/optimization of classifier� Sensor fusion
�Results� Reference – ground truth� Improvement by (f)using parallel detectors
ALR-JKAR/Jan2011/Transportforum - 3 Copyright Autoliv Inc., All Rights Reserved
Fordonsstrategisk Forskning och Innovation FFI – D4SF
Driver Drowsiness detection
�Drowsy driving is a road safety problem- drowsiness contributing in 10-30% of accidents (Anund & Patten 2010)
�What can be done?� Commercial fleet traffic
� Fatigue Risk Management� Work time regulation� Detection and warning
� Privately owned vehicles� Detection and warning
�Detection?� Detection systems offered as option from several OEMs� So far, performance is far from ideal...
ALR-JKAR/Jan2011/Transportforum - 4 Copyright Autoliv Inc., All Rights Reserved
Fordonsstrategisk Forskning och Innovation FFI – D4SF
Target and Goals
�Different indicators exist- ’Physiology’ measures - blink duration etc. - Driving performance measures - lane keeping measures- Environment measures - time of day, traffic, road type
(previous sleep possible in commercial fleet vehicles??)
� Various indicators have different strengths and weaknesses
�Improve performance by fusing data from multiple indicators
� The fusion algorithm shall show an improvement in:- Overall performance- Reduced number of faulty detections- Increased number of correct detections
+ +
+
+
–
Specificity
–+Indicator C
+ ++ +Fusion
+–Indicator B
++Indicator A
AvailabilitySensitivity
ALR-JKAR/Jan2011/Transportforum - 5 Copyright Autoliv Inc., All Rights Reserved
Fordonsstrategisk Forskning och Innovation FFI – D4SF
� Data collection
� Relevant vehicle data
� Speed, lane position, SW angle, pedals etc.
� Video based gaze direction, eyelid opening, head pos
� KSS value every 5 minute
� EEG, EOG and EMG
� Video recordings (road scenery and cabin)
� In total: 43 drivers have completed 3 drives each
� Procedure: Each driver drove three times during one day (day, evening and night). Trip duration 80-90 minutes
Data collection�On-road tests were conducted with governmental approval (N2007/5326/TR) and ethical approval by Regional ethics approval board (EPN 142-07 T34-09).
ALR-JKAR/Jan2011/Transportforum - 6 Copyright Autoliv Inc., All Rights Reserved
Fordonsstrategisk Forskning och Innovation FFI – D4SF
Test Route
Road RV34Mostly 9 m width Driving lane width 3,75 m Speed limit - mostly 90 km/h
Numbers on map are Yearly day traffic volume in January 2002
We know of only a few similar studies performed on public roads
90 minute driving, approx 115 km distance
Rested safety driver –dual command
ALR-JKAR/Jan2011/Transportforum - 7 Copyright Autoliv Inc., All Rights Reserved
Fordonsstrategisk Forskning och Innovation FFI – D4SF
Ground Truth – KSS
+ Simple to collect+ Simple to understand – immediately ready for analysis
- Training needed for participants- Some offset for inexperienced participants?
KSS Description in Swedish Verbal description
1 extremt pigg extremely alert
2 mycket pigg very alert
3 pigg alert
4 ganska pigg rather alert
5 varken pigg eller sömnig neither alert nor sleepy
6 första tecknen på sömnighet some signs of sleepiness
7 sömnig, ej ansträngande vara vaken sleepy, but no effort to keep alert
8 sömnig, viss ansträngning vara vaken sleepy, some effort to keep alert
9 mycket sömnig, ansträngande vara vaken, kämpar mot sömnen
very sleepy, great effort to keep alert, fighting sleep
ALR-JKAR/Jan2011/Transportforum - 8 Copyright Autoliv Inc., All Rights Reserved
Fordonsstrategisk Forskning och Innovation FFI – D4SF
Blink duration (AS): Mean blink duration
Lane keeping variability (Lane): Variability in Steering wheel position or Lane Position. e.g. using Generic Variability Indicator (Sandberg 2008) .
Time-of-day (TPM): Expected drowsiness with regard to time of day (circadian rythm)
* Each indicators has several parameters that needs to be tuned for optimal performance
))1(( pxxz ii δδ −+−=
ki
N
ii zzw
NG ||)(
1
1∑
=
=
)()( 11)(
RRLL zR
zL
e
c
e
czw βαβα −−−− +
++
=
GVI (Sandberg 2008)
Short Blink Long Blink
400 ms
200 msOpen
Closed
OpeningClosed
Closing
Amplitude
Example indicators of driver sleepiness
ALR-JKAR/Jan2011/Transportforum - 9 Copyright Autoliv Inc., All Rights Reserved
Fordonsstrategisk Forskning och Innovation FFI – D4SF
Video examples
Video examples
ALR-JKAR/Jan2011/Transportforum - 10 Copyright Autoliv Inc., All Rights Reserved
Fordonsstrategisk Forskning och Innovation FFI – D4SF
� SVM (Support Vector Machine):
� Machine learning method using data from field tests to calculate “best fit” function between indicator values and ground truth (KSS rating scale)
� Indicator parameters optimized simultaneously with training of SVM
� Data sets for SVM training and validation are from separate drivers. Thus, validation is done on truly “never-before-seen” data.
Sensor fusion
Indicator A
Indi
cato
r B
Drowsy data
Alert data
Goal: Find the maximum margin hyperplane
ALR-JKAR/Jan2011/Transportforum - 13 Copyright Autoliv Inc., All Rights Reserved
Fordonsstrategisk Forskning och Innovation FFI – D4SF
Evaluation Method� Assuming a binary classification,
alert or drowsy� Performance is the mean value of
sensitivity and specificity� Performance is related to the
proportion of the time where the algorithm is correct
DrowsyNon-
Drowsy
Detect A(hit)
B(false hit)
Non-Detect
C(miss)
D(correct reject)
Sum A + C B + D
2
yspecificitysensitiviteperformanc
DB
Dyspecificit
CA
Aysensitivit
+=
+=
+=
KS
S
Ground truth
Alg
orith
m o
utpu
t
KSS = ground truth
Binary Algo output
Ground truth cutoff
Time
ALR-JKAR/Jan2011/Transportforum - 14 Copyright Autoliv Inc., All Rights Reserved
Fordonsstrategisk Forskning och Innovation FFI – D4SF
Example of results from sensor fusion
0.84 (0.89)0.76 (0.81)0.80 (0.85)Blink + Steer + Circ.
0.92 (0.88)0.68 (0.68)0.80 (0.78)Blink + Lane + Circ.
0.96 (0.95)0.36 (0.32)0.66 (0.64)Blink
0.83 (0.87)0.77 (0.79)0.80 (0.83)Blink + Circadian
SpecSensFitnessModel
First figure is training data performancesecond figure is test data performance � Decision every 1 minute
� KSS >= 7 � drowsy� KSS < 7 � alert
ALR-JKAR/Jan2011/Transportforum - 15 Copyright Autoliv Inc., All Rights Reserved
Fordonsstrategisk Forskning och Innovation FFI – D4SF
Fulfillment of goals
� The fusion algorithm shall show an improvement in:
- Improved performance true
- Increased number of correct detections true
- Reduced number of faulty detections (?)
Clearly improved overall performance
– Minor differences between different combinations
ALR-JKAR/Jan2011/Transportforum - 16 Copyright Autoliv Inc., All Rights Reserved
Fordonsstrategisk Forskning och Innovation FFI – D4SF
Summary
�Controlled experiment on public roads� 43 drivers so far�What is ideal performance?
� Method developed with focus on mathematical performance� Most important goal is to have relevant warnings
�More data is needed: � Different road types� Different conditions (weather, drive duration etc.)� Different driver types (age, cultural differences etc.)
ALR-JKAR/Jan2011/Transportforum - 17 Copyright Autoliv Inc., All Rights Reserved
Fordonsstrategisk Forskning och Innovation FFI – D4SF
Thank you for you attention!