identification of driver s drowsiness using … · identification of driver’s drowsiness using...

1

IDENTIFICATION OF DRIVER’S DROWSINESS USING DRIVING INFORMATION AND EEG∗∗∗∗

Marcel Ji�ina1, Petr Bouchner, Stanislav Novotný2

1Institute of Computer Science, Academy of Sciences of the Czech Republic,

Pod Vodarenskou vezi 2, Prague 8, 182 07, Czech Republic 2Joint Laboratory of System Reliability, Department of Control Engineering and Telematics

Faculty of Transportation Sciences, Czech Technical University in Prague,

Konviktská 20, Prague 1, 110 00, Czech Republic

Contents Introduction............................................................................................................................2 Simulation environment..........................................................................................................2

Simulators ..........................................................................................................................2 Testing track.......................................................................................................................3 Data collected.....................................................................................................................4

Data processing and selection .................................................................................................4 Driving data........................................................................................................................4 EEG data ............................................................................................................................5 Electrodes and frequency bands selection ...........................................................................5 Filter used...........................................................................................................................5 Feature selection.................................................................................................................8 Correlation analysis ............................................................................................................8 Classifier ..........................................................................................................................12 IINC algorithm description ...............................................................................................12

Results .................................................................................................................................14 Threshold values...............................................................................................................14 Lane Departure.................................................................................................................14 Reaction time and drowsiness...........................................................................................14 Classification/discrimination ............................................................................................14 Drowsy/wakeful state recognition.....................................................................................14 Long/short reaction time recognition ................................................................................16

Conclusion ...........................................................................................................................20 Abstract: This report summarizes the first results with identification of sleepy state in drivers. The driving information as the deviation from the centerline of road and the steering wheel position as well as two-point EEG was used. The process consists of preprocessing data, in fact a transformation into form proper for classification, and a classification into one of two classes, wakefulness and drowsiness. Results show that it is possible to distinguish these two states with relatively large error, which possibly can be tackled by the use of proper methodology.

∗ Final version published: M. Ji�ina, S. Novotný, P. Bouchner: Identification of driver’s drowsiness using driving information and EEG. In: Driver-Car Interaction and Interface - Book of Proceedings 2009. Praha: Ústav Informatiky AV �R, v.v.i. : Bouchner, P. - Novák, M. (ed.), 2009. ISBN 978-80-87136-05-8, pp. 11-19, 2009, Praha.

2

Introduction Driver’s drowsiness is a very dangerous phenomenon causing approximately seven per cent of all car accidents. Drowsiness is analyzed or identified by lot of approaches mostly based on EEG analysis, and sometimes on analysis of driver’s behavior. However, for a practical application there is need to distinguish between wakeful and drowsy state of the driver directly in the car or lorry.

Here we show that data collected in car and EEG information from two electrodes together with a proper procedure could be sufficient for drowsy state identification.

In controlled experiment we measured driver’s behavior by variables as position of steering wheel, use of the throttle, position of car with respect to central line of a road etc. At the same time EEG was recorded together with reaction time. There were two groups of drivers, wakeful ones, and drivers after serious sleep deprivation. We found that it is possible to distinguish these groups using a proper classifier with some rather large error.

Our results demonstrate that two electrodes EEG together with driving information can distinguish between wakeful and drowsy drivers in some extend.

Results obtained lead to idea of practical use of approach described combined with a procedure, which takes into account a time development of driver’s tiredness. There are questions whether on-line EEG measurement is necessary and what driving characteristics can show that driver is too tired to continue his ride. The visual information is one of the most important human operator’s (car driver) information, which man usually use during the process of control. Common presented proportion between visual and other acquired information is 90% to 10%. From this point of view is quite logical inference that if we will use eye-tracking for monitoring of driver’s eye-movements and gaze than in consequence we will be able (on base of those information) to rating more effectively such characteristics and parameters as are: driver’s performance and vigilance; available and process visual information; actual area of interest; ergonomic of various systems and so on.

Simulation environment

Simulators The simulator we used for this experiment is steady based and fully interactive, composed of a fully equipped cockpit of a higher-middle class European car with automatic gear-shifting (Fig. 1). It is equipped with a system of measuring devices and an in-car video recording system. The field of view is 100º with no active back mirrors. A description of the simulator and its usage can be found in [7] or [8].

3

Fig. 1: Driver’s view from the simulator cockpit

Testing track

From the driver’s point of view, most of the ride seems to be mostly straight. A very light curvature was chosen so that the drivers would have to pay attention to driving all the time. If not, they go out of their lane. The next pictures show a top view on a new testing track (Fig. 2). The track contains traffic lights approximately each 200 m in such an arrangement that they are always one or more successive in lights in a driver’s view.

There is no traffic, (only parked cars around) no crossroads and no other driving situations need to be solved.

The tasks, which the driver is supposed to solve, are as follows:

1. Keeping the lane

2. Keeping the speed

3. Watching the traffic lights

4. Reacting on red signal with immediate pushing on the brake pedal

Such an arrangement forces the tested driver to solve only primary driving tasks and we can rely that his/her reaction time on red signal is not influenced by other factors. It is believed to reliably testify his/her vigilance level. Further improper fulfillment of one of the above listed tasks gives evidence of poor driver’s attention, which is caused (in our experiment) by fatigue.

4

-2500

-2000

-1500

-1000

-500

0

500

1000

1500

-3000 -2500 -2000 -1500 -1000 -500 0 500

Fig. 2: The testing ring

Data collected The driving record is sampled each approximately 12 milliseconds. The next table (Tab. 2) shows outputs, which we measured and recorded during the experiment.

Table 1: Measured and recorded data

Technical data Human related data

Subjective data

Trajectory and geometrically ideal path Actual velocity vector

Steering wheel and pedals Video recording of the scenery and experimentees

EEG ECG

Reaction time

Self rating

Data processing and selection

Driving data As mentioned above, individual data samples correspond to one stop on the red light. We use data from last ten seconds before red light appears. The amount of data collected in this time interval is reduced so, that only sample for time 0, time 1 second before red light, time 2 seconds before red light, and so on until 9 seconds before red light is used. For each time there are seven features available as follows:

5

1. Deviation of center of car from the center of the road

2. Velocity [m/s]

3. Steering wheel angle (-1000 till 1000)

4. Throttle (0 till 1000)

5. Brake (0 till 1000)

6. Pneu. tire (1 – left front) out of road band (”pneu1”)

7. Pneu. tire (2 – right front) out of road band (”pneu2”)

For ten times moments it gives 70 features. Together with EEG data from two electrodes (see later) there are total 72 features for each sample. To each sample a class wakeful/drowsy is associated.

EEG data

Electrodes and frequency bands selection Studies Faber et al. (2008) [2] and Faber (2008) [3] show that for identification of wakeful/drowsy states it is optimal to compare energies in delta and alpha frequency bands of EEG. According these references it holds:

a) In EEG all frequencies above 15 Hz carry no useful information.

b) There are two bands: 0,5-3,5 Hz (or 1 - 4 Hz) (delta) that can be divided into two sub bands optionally, and 8 – 13 Hz (alpha).

c) The total energy in these bands varies rather boldly with tiredness and with subject the driver is thinking about. Once there is “lower band energy > higher band energy” and then vice versa. It should suffice to trace these changes.

d) The first problem is that thinking about can be related to driving or it need not and then it is a dangerous state.

e) The second problem is that proper places on the driver’s scull may be different in different people and even may change in the one person. Up to now unwritten standard are two places above ears, namely T3 and T4 (electrodes according to standard international notation). Here we use electrodes T3 and O1, i.e. the left side above ear and the back of scull as Coufal and Šebesta (2005) [l].

f) Final algorithm is: use ratio alpha/delta [2]

Based on these findings we use energy ratio for bands 0.5-4 Hz and 8-13 Hz for electrodes T3 and O1 thus getting two numbers representing EEG measurements.

Filter used Original EEG sampling frequency 256 Hz was reduced to 64 Hz by averaging four successive measurements.

6

We use digital FIR (Finite Impulse Response) band pass filters of order 32 for frequency bands above and integrated energy (the signal value squared) in these bands for 10 seconds. The ratio of these integrated energies for each of both electrodes gave numbers representing the EEG measurements as mentioned above.

For the digital filter design see TechOnline (2008) [6]. We adopted java source code available there into C++.

Table 2: Coefficients of the FIR (Finite Impulse Response filter), order=32 rate=32 fp1=0.5 fp2=4 COEFFICIENTS: a, b[0] -0.0198944 0 a, b[1] -0.0361238 0 a, b[2] -0.045036 0 a, b[3] -0.0407448 0 a, b[4] -0.0245066 0 a, b[5] -0.00505861 0 a, b[6] 0.00536449 0 a, b[7] -0.0023309 0 a, b[8] -0.0281349 0 a, b[9] -0.0610019 0 a, b[10] -0.0825256 0 a, b[11] -0.0750258 0 a, b[12] -0.0304529 0 a, b[13] 0.0442263 0 a, b[14] 0.128105 0 a, b[15] 0.193879 0 a, b[16] 0.21875 0 a, b[17] 0.193879 0 a, b[18] 0.128105 0 a, b[19] 0.0442263 0 a, b[20] -0.0304529 0 a, b[21] -0.0750258 0 a, b[22] -0.0825256 0 a, b[23] -0.0610019 0 a, b[24] -0.0281349 0 a, b[25] -0.0023309 0 a, b[26] 0.00536449 0 a, b[27] -0.00505861 0 a, b[28] -0.0245066 0 a, b[29] -0.0407448 0 a, b[30] -0.045036 0 a, b[31] -0.0361238 0 a, b[32] -0.0198944 0

COEFFICIENTS after normalization: a, b[0] -0.0173159 0 a, b[1] -0.0314418 0 a, b[2] -0.0391989 0 a, b[3] -0.0354639 0 a, b[4] -0.0213303 0 a, b[5] -0.00440297 0 a, b[6] 0.0046692 0 a, b[7] -0.00202879 0 a, b[8] -0.0244884 0 a, b[9] -0.0530954 0 a, b[10] -0.0718294 0 a, b[11] -0.0653017 0 a, b[12] -0.0265059 0 a, b[13] 0.0384941 0 a, b[14] 0.111502 0 a, b[15] 0.168751 0 a, b[16] 0.190398 0 a, b[17] 0.168751 0 a, b[18] 0.111502 0 a, b[19] 0.0384941 0 a, b[20] -0.0265059 0 a, b[21] -0.0653017 0 a, b[22] -0.0718294 0 a, b[23] -0.0530954 0 a, b[24] -0.0244884 0 a, b[25] -0.00202879 0 a, b[26] 0.0046692 0 a, b[27] -0.00440297 0 a, b[28] -0.0213303 0 a, b[29] -0.0354639 0 a, b[30] -0.0391989 0 a, b[31] -0.0314418 0 a, b[32] -0.0173159 0

7

-60

-50

-40

-30

-20

-10

0

0 2 4 6 8 10 12 14 16

Frequency [Hz]

Gai

n [d

B]

Fig. 3: The frequency characteristic of filter for theta (0.5-4 Hz) band (horizontally frequency in Hz, vertically attenuation in dB)

Table 3: Coefficients of the FIR (Finite Impulse Response filter) of order=32 rate=32 fp1=8 fp2=13 COEFFICIENTS: a, b[0] 0 0 a, b[1] 0.0330101 0 a, b[2] -0.0210058 0 a, b[3] -0.000470447 0 a, b[4] -0.0187566 0 a, b[5] 0.0345828 0 a, b[6] 0.0121809 0 a, b[7] -0.0647749 0 a, b[8] 0.0397888 0 a, b[9] 0.00766346 0 a, b[10] 0.020302 0 a, b[11] -0.0512423 0 a, b[12] -0.0562696 0 a, b[13] 0.210168 0 a, b[14] -0.14704 0 a, b[15] -0.141466 0 a, b[16] 0.3125 0 a, b[17] -0.141466 0 a, b[18] -0.14704 0 a, b[19] 0.210168 0 a, b[20] -0.0562696 0 a, b[21] -0.0512423 0 a, b[22] 0.020302 0 a, b[23] 0.00766346 0 a, b[24] 0.0397888 0 a, b[25] -0.0647749 0 a, b[26] 0.0121809 0 a, b[27] 0.0345828 0 a, b[28] -0.0187566 0 a, b[29] -0.000470447 0 a, b[30] -0.0210058 0 a, b[31] 0.0330101 0 a, b[32] 0 0

COEFFICIENTS after normalization: a, b[0] 0 0 a, b[1] 0.0307627 0 a, b[2] -0.0195757 0 a, b[3] -0.000438418 0 a, b[4] -0.0174796 0 a, b[5] 0.0322284 0 a, b[6] 0.0113516 0 a, b[7] -0.0603649 0 a, b[8] 0.03708 0 a, b[9] 0.00714172 0 a, b[10] 0.0189198 0 a, b[11] -0.0477537 0 a, b[12] -0.0524386 0 a, b[13] 0.195859 0 a, b[14] -0.137029 0 a, b[15] -0.131835 0 a, b[16] 0.291225 0 a, b[17] -0.131835 0 a, b[18] -0.137029 0 a, b[19] 0.195859 0 a, b[20] -0.0524386 0 a, b[21] -0.0477537 0 a, b[22] 0.0189198 0 a, b[23] 0.00714172 0 a, b[24] 0.03708 0 a, b[25] -0.0603649 0 a, b[26] 0.0113516 0 a, b[27] 0.0322284 0 a, b[28] -0.0174796 0 a, b[29] -0.000438418 0 a, b[30] -0.0195757 0 a, b[31] 0.0307627 0 a, b[32] 0 0

8

-60

-50

-40

-30

-20

-10

0

0 2 4 6 8 10 12 14 16

Frequency [HZ]

Gai

n [d

B]

Fig. 4: The frequency characteristic of filter for theta (8-13 Hz) band (horizontally frequency in Hz, vertically attenuation in dB)

Feature selection In data there are total 72 driving characteristics (features) for each measurement of the reaction time, i.e. for each stop on the red sign. Two of 72 features represent EEG. This number is not too large from the point if view of advanced classifiers but it is known that excessive nearly uncorrelated features may reduce quality of classification. For it, we first analyzed how large is the influence of a feature on the class wakeful/drowsy by the use of simple correlation analysis.

Correlation analysis In correlation analysis we computed correlation coefficients between each of 72 features and class wakeful/drowsy (in fact, number 1 and -1 for one and the other class). We studied two factors the correlation coefficient and the differences among correlation coefficients for different time intervals before red light. The second one informs whether values in different time intervals prior red light differ giving some new information or no.

Brake usage, and “pneu1” From correlation analysis it follows apparently no influence of the brake usage – there are small values of the correlation coefficient (max. 0.1042) and small their differences. Also from the other point of view the brake information usage has no sense as all measurements were organized so that braking is not necessary before red light appears. In spite of that we found brake usage in three cases from our 361 measurements.

9

For “pneu1”, i.e. the ridings up of the left front tire over the left road margin hold nearly exactly the same as for brake usage. The largest correlation coefficient (0.1327) appears just when red light appears. We use the last value.

Distance from the centerline of the road “dist” We found rather small influence, the correlation coefficient grows in successive time intervals and the largest value (0.1716) corresponds to the time when red light appears. Differences are rather small, so we use the last value only.

Throttle

Correlation coefficients grow in successive time intervals and the largest value (0.2284) corresponds to the time when red light appears similarly as in the distance from the center line. We use last (and largest) three values that correspond to correlation coefficients 0.0756, 0.1610, and 0.2284, respectively.

The ridings up of the right front tire over the right road margin “Pneu2” This variable appears important with the largest correlation coefficient 0.2168 in the time when red light appears. We use the last three values.

Velocity

The velocity appears very important as it has also large correlation coefficient (0.3461) for all times prior the red light appears. Large correlation coefficient simple says the faster the worse. It testifies a known psychological fact that tired driver tries unconsciously suppressing his or hers tiredness by faster speed. As there is nearly the same velocity during the last ten seconds before the red light appears we use one the value of velocity only.

Steering wheel position It appears that there is rather large correlation coefficient (0.2386) for last moment before red light appears and very low correlation coefficient (0.007822) ten seconds before the red light appears. As there is exist a hypothesis that the steering wheel position may be important factor we use all ten values measured.

Results of the correlation analysis and feature selection Detailed results of correlation analysis are presented in Tab. 5. Lines in Tab. 5 are sorted according to the kind if feature and then according to the absolute value of the correlation coefficient between each feature and the class.

10

Table 4: Correlation between feature and class wakeful/drowsy

Feature order number

Correlation coefficient

Kind of feature

Abs. correlation coefficient

Difference of abs. correlation coefficient for the same kind of feature

Comment

61 0.074949 brake 0.074949 0 26 0.074949 brake 0.074949 0 54 0.074949 brake 0.074949 0 47 0.080951 brake 0.080951 -0.006 33 0.084861 brake 0.084861 -0.00391 12 0.091007 brake 0.091007 -0.00615 19 0.100778 brake 0.100778 -0.00977 68 0.100778 brake 0.100778 0 40 0.102504 brake 0.102504 -0.00173

5 0.104202 brake 0.104202 -0.0017 apparently no influence - small values and small differences

8 -0.01965 dist 0.01965 29 -0.03161 dist 0.031615 -0.01196 43 0.034461 dist 0.034461 -0.00285 22 -0.04215 dist 0.042154 -0.00769 36 0.057494 dist 0.057494 -0.01534 15 0.088046 dist 0.088046 -0.03055 1 0.105835 dist 0.105835 -0.01779

64 0.161846 dist 0.161846 -0.05601 57 0.162959 dist 0.162959 -0.00111

50 0.171636 dist 0.171636 -0.00868 small influence - the one last left

71 0.297366 eeg1 0.297366 72 0.280952 eeg2 0.280952 25 -0.00292 throttle 0.002918 11 -0.00933 throttle 0.009326 -0.00641 32 -0.0193 throttle 0.019298 -0.00997 39 0.027748 throttle 0.027748 -0.00845 18 -0.02872 throttle 0.028724 -0.00098 4 -0.04343 throttle 0.043425 -0.0147

46 -0.05733 throttle 0.057327 -0.0139 53 -0.07562 throttle 0.075615 -0.01829 67 -0.16103 throttle 0.161031 -0.08542

60 -0.2284 throttle 0.228398 -0.06737 large influence of last three values

62 0 pneu1 0 20 -0.07412 pneu1 0.074125 -0.07412 55 -0.07412 pneu1 0.074125 -4.7E-16 27 -0.08961 pneu1 0.089613 -0.01549 34 -0.08961 pneu1 0.089613 0 13 -0.08961 pneu1 0.089613 -1.8E-16 41 -0.08961 pneu1 0.089613 -1.9E-16 48 -0.10512 pneu1 0.105121 -0.01551 69 -0.12911 pneu1 0.129107 -0.02399

6 -0.13273 pneu1 0.132729 -0.00362 small influence - small values and differences

35 -0.07779 pneu2 0.07779 49 -0.11422 pneu2 0.114221 -0.03643 21 -0.12582 pneu2 0.12582 -0.0116 28 -0.16047 pneu2 0.160471 -0.03465 14 -0.16047 pneu2 0.160471 0 56 -0.18414 pneu2 0.184144 -0.02367 63 -0.18414 pneu2 0.184144 0 70 -0.18414 pneu2 0.184144 -2.2E-16 42 -0.19213 pneu2 0.192127 -0.00798

7 -0.21679 pneu2 0.216792 -0.02466 large influence of last three values

11

Summarizing final selection of features from Tab. 5 and analysis above we have after elimination of unnecessary features total 31 to 34 features. Selected 33 and 34 features are shown in Tab. 6 and differ in variable “dist” that is omitted or present. The case of 31 and 32 features arises by deleting two features corresponding to EEG. These combinations should show necessity of usage of variables those are rather difficult to measure in road conditions.

Table 5: Features selected for experiments

33 Features 34 Features

No. Feature Seconds before No. Feature

Seconds before

1 steer. wheel 9 1 steer. wheel 9

2 throttle 9 2 throttle 9

3 pneu2 9 3 pneu2 9



6 pneu2 8 6 pneu2 8



9 pneu2 7 9 pneu2 7



12 pneu2 6 12 pneu2 6



15 pneu2 5 15 pneu2 5



18 pneu2 4 18 pneu2 4



21 pneu2 3 21 pneu2 3



24 pneu2 2 24 pneu2 2


12


27 pneu2 1 27 pneu2 1

28 velocity 0 28 dist 0

29 steer. wheel 0 29 velocity 0

30 Throttle 0 30 steer. wheel 0

31 pneu2 0 31 Throttle 0

32 eeg1 (T4) 0 32 pneu2 0

33 eeg2 (O1) 0 33 eeg1 0

34 eeg2 0

Classifier We used advanced classifier according to M. Ji�ina and M Ji�ina, jr. (2008) [4], [5]. The classifier is based on theory of correlation dimension and uses implicitly a local non-integer measure of correlation dimension. In the end, the algorithm (IINC – Inverted Indexes of Neighbors Classifier) is very simple as follows:

IINC algorithm description Let samples of the learning set (i.e. all samples without respect to the class) be sorted according to their distances from the query point x. Let indexes be assigned to these points so that 1 is assigned to the nearest neighbor, 2 to the second nearest neighbor etc.

Let us compute sums:

�==

=N

ci

iN

xS)0(10

0 11

)( and �==

=N

ci

iN

xS)1(11

1 11

)( ,

i.e. sums of reciprocals of the indexes of samples from class c = 0 and from class c = 1. N0 and N1 are numbers of samples of class 0 and class 1, respectively, N0 + N1 = N.

The estimate of probability that point x belongs to class 0 is:

)()()(

)|0(ˆ10

0

xSxSxS

xcp+

==

and similarly the estimate of probability that point x belongs to class 1 is:

13

)()()(

)|1(ˆ10

1

xSxSxS

xcp+

==

.

When some discriminant threshold � is chosen then if p(c = 1| x) � � we decide that point x is of class 1 else it is of class 0. This is the same procedure as in other classification approaches where the output is estimation of probability (naive Bayes) or any real valued variable (neural networks). The value of the threshold can be optimized with respect to the minimal classification error. Default value of the discriminant threshold here is � = 0.5.

The IINC algorithm is illustrated in Fig. 5. In the figure at the left hand side a data space with data points is depicted. The black point is a query point, i.e. the point, which class we are looking for. The other points are points of the learning set, here of two classes, red and green. We assign numbers (indexes) to these points so that 1 is assigned to the nearest neighbor, 2 to the second nearest neighbor etc. without regard to class (Fig. 5 at the right hand side). Then we compute reciprocals of these indexes, see Tab. 7. For all points of the learning set it forms harmonic series (Tab. 7, the row Reciprocals). Finally, probability that there is a red at the query point is equal to the sum of elements of harmonic series for red points only (Tab. 7, row Rec. of red only) that is 1.713. This number divided by the sum of whole finite harmonic series (Tab. 7, the row Reciprocals) that is 3.251. This ratio gives estimate 0.527 of the probability that the query point is red and, similarly, probability of green is estimated as 0.473. Using threshold � = 0.5 we decide that the query point is red.

1 2

3

5 4

6

7

8 10

9

11 13

12 14

Fig. 5: Data space for two-class classification problem

Table 6: Reciprocals of neighbor’s indexes for the example above

14

Results

Threshold values

Lane Departure The lane departure appeared at least once during ten measurements before red light in 73 cases from total 361 red light stops (20.2 %). That happened in eight cases to wakeful drivers and in 18 cases when a short reaction time has been measured without respect to wakeful or drowsy state of the driver. It was found that from these 18 cases six happened to wakeful drivers. It is apparent that lane departure is a rather hard criterion and shows more a general inattentiveness than the risk of long reaction time. On the other hand 24 % of drivers with short reaction time rode up but only 11 % of wakeful drivers. This means that it gives rather good information about reaction time and drowsiness of the driver.

Reaction time and drowsiness From data at hand on can find interesting thresholds:

• If the reaction time is shorter than 0.58 s, the driver is (surely) wakeful.

• If the reaction time is longer than 1.6 s, the driver is (surely) drowsy.

• Unfortunately the first case holds for 13 (3.6 %), and the second case for 41 (11 %) of all our 361 measurements.

Classification/discrimination For evaluation of quality of classification, i.e. the discrimination between classes wakeful/drowsy we use well-known ROC curve as shown in Figs. 6 and 7. In these figures on the vertical axis there is efficiency or sensitivity, i.e. the ratio of wakeful drivers correctly recognized as wakeful ones. The one minus efficiency gives so-called error of the first kind. On the horizontal axis there is the ratio of drowsy drivers erroneously recognized as wakeful ones. It is so-called error of the second kind. The 1 minus this error is sometimes called the specificity.

In the figures there is also an important line for Q=1 that means that the statistical significance of the data set before and after classification is the same. If the curve goes up this line, data after classification are statistically better than before.

Drowsy/wakeful state recognition

In Fig. 6 the (best) results with data that contains EEG information are depicted.

In Fig. 7 it is seen that data without EEG information are rather bad lying below Q=1 curve. Even ROC curve for EEG information only lies below Q=1 line in substantial part. Best results are obtained for selected features with EEG information included.

15

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1drowsy erroneously recognized as wakeful

wak

eful

rec

ogni

zed

as w

akef

ul

34noBraPn19Di9ve

33noBraPn1Dis9ve

2eeg

Q=1

Fig. 6: ROC curves for IIN classifier and data sets with different feature selection all containing EEG information

Explanation to legend: 34noBraPn19Di9ve means 34 features selected that does not contain information about braking, “pneu1”, 9 values of the distance “dist” and 9 values of the velocity. Similarly, 33noBraPn1Dis9ve differs from the previous case in deletion of all ten values of variable “dist”. The 2eeg means the use of information from both EEG points used.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

drowsy erroneously recognized as wakeful

wak

eful

rec

ogni

zed

as w

akef

ul

31noBraPn1Dis9veEeg

32noBraPn19Di9veEeg

2eeg

Q=1

Fig. 7: ROC curves for IIN classifier and different data sets with different feature selection NOT containing EEG information

16

The line denoted “2eeg” is here for comparison. Explanation to legend: 32noBraPn19Di9veEeg means 32 features selected that does not contain information about braking, “pneu1”, 9 values of the distance “dist”, 9 values of the velocity, and information from both EEG points. Similarly, 31noBraPn1Dis9veEeg differs from the previous case in deletion of all ten values of variable “dist”. The 2eeg means the use of information from both EEG points used.

Long/short reaction time recognition In this part we discriminate individual events of stopping when red light appears according to driver’s true reaction time without view to the drowsy/wakeful state of the driver.

We tested also data with “gap”, i.e. the use of the learning set containing cases of clearly long and clearly short reaction time, i.e. without “medium” cases. All data not used in the learning set form the testing set.

We also considered two riding up variables “pneu1” and “pneu2”. There is a consideration that riding up from the limits given by the left and right road margin is a rather graceless infraction of driving rules. It is caused either by drowsiness or by general disregard. Thus it should be recorded anytime.

Data preparation We have total 362 samples. Two samples with longest reaction time were omitted. So, we have 360 samples at hand. We generated new value M(v), “MAX” for each of variables of the same kind denoted by v. The new value is a maximum of absolute values of ten values of a variable before red light appears and a sign of the last of these values multiplies the maximum found. Thus this variable has a sign. It holds

)(max)()(9...0

0 ii

vvsignvM=

=

where vi is a value of corresponding variable i seconds before the red light appears. This function is used for all variables selected as shown in Tab. 8. Two exceptions form variables eeg1 and eeg2 that are copied only.

17

Table 7: The description of files used for log/short reaction time recognition. Note that the number in name of file denotes dimension

File names Variables used

tst4RTmax-pnbree.txt maxims of all variables; deleted “pneu1”, “pneu2”, brake and eeg

lrn4RTmax-pnbree.txt

tst6RTmax-pnbr.txt maxims of all variables; deleted “pneu1”, “pneu2, and brake

lrn6RTmax-pnbr.txt

lrn7RTmax-ee.txt maxims of all variables; deleted eeg

tst7RTmax-ee.txt

tst9RTmax.txt maxims of all variables.

lrn9RTmax.txt

tst38RT-ee.txt variables with correlation coefficient >0.0605; deleted eeg

lrn38RT-ee.txt

tst40RT.txt variables with correlation coefficient >0.0605

lrn40RT.txt

tst2eegRT.txt eeg only

lrn2eegRT.txt

tst72RT.txt all 72 variables

lrn 72RT.txt

Similar data sets were generated with a “gap” in the learning set (see above).

We compute with learning sets denoted lrn… and test with corresponding data sets denoted tst… Then we interchanged these data sets using data denoted as tst… for learning and corresponding data sets denoted lrn… for testing. Thus we can join results and have more, i.e. 360 samples for ROC evaluation.

18

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Fast found as slow

Slo

w fo

und

as s

low

9RTmax

40RT

2eegRT

72RT

Q=1

Fig. 8: ROC curves for wakeful/drowsy data that has been split according to reaction time into two classes Slow, i.e. reaction time longer than 900 ms, and Fast, i.e. reaction time shorter than 900 ms

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Fast found as slow

Slo

w fo

und

as s

low

7RTmax-ee

4RTmax-pnbree

38RT-ee

2eegRT

Q=1

Fig. 9: ROC curves for wakeful/drowsy data that has been split according to reaction time into two classes Slow, i.e. reaction time longer than 900 ms, and Fast, i.e. reaction time shorter than 900 ms. Data

19

without use of EEG information. The line corresponding to the use of EEG information only is shown here for comparison

In Figs. 8 and 9 one can see at the first sight that classification is of similar quality as in case of drowsy/wakeful classification. In both cases one needs to use EEG information to get significantly better results.

Long/short reaction time with “gap”

Compared to previous data we removed 100 samples from central part of region of reaction time making a “gap” in values of reaction times. Thus we got 260 samples that were split into learning set and part of testing set. The learning data contains 130 samples whereas to the testing set 100 samples mentioned above were added so that the testing set contains all remaining data, i.e. 230 samples. There is no reaction time “gap” in testing set.

Note that now we cannot interchange learning and testing set to duplicate number of testing samples. We have only 230 testing samples.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1Fast found as slow

Slo

w fo

und

as s

low

9RTmaxB

6RTmax-pnbrB

2eegRTB

Q=1

Fig. 10: Data with the “gap”. The ROC curves for wakeful/drowsy data that has been split according to reaction time into two classes Slow, i.e. reaction time longer than 900 ms, and Fast, i.e. reaction time shorter than 900 ms using a “gap” in reaction times in the learning set. The line corresponding to the use of EEG information only is shown here for comparison

20

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1

Fast found as slow

Slo

w fo

und

as s

low

7RTmax-eeB

4RTmax-pnbreeB

2eegRTB

Q=1

Fig. 11: Data with the “gap”. The ROC curves for wakeful/drowsy data that has been split according to reaction time into two classes Slow, i.e. reaction time longer than 900 ms, and Fast, i.e. reaction time shorter than 900 ms using a “gap” in reaction times in the learning set. The line corresponding to the use of EEG information only is shown here for comparison

In Figs. 10 and 11 one can see that results seem to be generally worse than in case of data without “gap” (Figs. 8 and 9).

Conclusion We evaluated two basic approaches to microsleep identification based on the use of easily measurable driving data together with the simplest measurements of EEG activity. An important issue is the use of anonymous data that means that our results are driver-independent. One of the approaches deals with separation of drowsy and wakeful states of a driver. This approach is related to possible microsleep identification or identification of states just prior microsleep. The approach does not identify long reaction times as such. The other approach differentiates between fast and slow driver’s reactions without consideration his or hers wakeful or drowsy state. This approach identifies a general state of sleepiness or any other case of lower vigilance of a driver. An experiment to use the learning data with a “gap”, i.e. without samples of medium reaction time length, in this case revealed that this is not a good idea. Results are generally worse than when complete data are used for learning.

As assumed, results are generally better with the use of EEG data. This fact seems to be generally valid even that the use of EEG information only gives unacceptable results. Apparently, combined driving information data together with EEG data seems to be necessary.

Comparing Fig. 6 and Fig. 8 we see that the task of slow/fast classification is very similar to the task of drowsy/wakeful classification. This is rather strange and analysis of detailed shows that drowsy drivers can have very short reaction times, shorter than 0.75 s, and wakeful

21

drivers can have very long reaction times, longer than 1.5 s. As mentioned before, there are two simple rules. If the reaction time is shorter than 0.58 s, the driver is wakeful. If the reaction time is longer than 1.6 s, the driver is (surely) drowsy. Unfortunately the first case holds for 13 (3.6 %), and the second case for 41 (11 %) of all our 361 measurements.

In Fig. 6 it is seen that even the best line goes not too high above Q=1 curve. It reveals rather bad classification but at least some one. On the other hand, if a ROC curve would go much above this line, there would remain the first kind error (drowsy recognized as wakeful), and the second kind error (wakeful recognized as drowsy) of unacceptable size for practical use.

Nevertheless, there are several interesting points on ROC curves.

Note line 31noBraPn1Dis9veEeg in Fig. 7, which holds for data without EEG. In more detailed look it can be seen that for threshold 0.5 there is 100 % recognition of wakeful state and 14.6 % recognition of drowsy state (see the right upper corner of the graph). This is rather low level of warning, but on the other hand a wakeful driver will never be troubled by drowsy state signalization.

On the same line in Fig. 7 there is a point with coordinates 0.7, 0.85 that means identification of 30 % drowsy states at the cost of troubling wakeful driver in 15 %. Conclusion

22

References

[1] D. Coufal, V. Šebesta: GUHA method supported analysis of EEG signals and their spectrograms, Neurodynamic and Neuroinformatic Studies (Second book on Micro-Sleeps), M. Novák (Ed.), Neural Network World - edice monografií, pp. 158-181, 2005

[2] J. Faber et al.: Vigilance and hypnagogium of drivers by EEG analysis, Neural Network World, Vol. 18, No. 2, pp. 89-104, 2008

[3] J. Faber: Personal communication, 2008 [4] M. Ji�ina and M Ji�ina, jr.: Classifier Based on Inverted Indexes of Neighbors, Technical

Report No. V-1034, Institute of Computer Science, Academy of Sciences of the Czech Republic, November, 2008

[5] M. Ji�ina and M Ji�ina, jr.: A device for stating a value of a control variable, Patent pending under number PV 2008-245; Z 7576 submitted on 22 April 2008 to the INDUSTRIAL PROPERTY OFFICE, Antonína �ermáka 2a, 160 68 Prague, Czech Republic, 2008b

[6] TechOnline: http://www.dsptutor.freeuk.com/IIRFilterDesign/IIRFiltDes102.html, java source code: http://www.dsptutor.freeuk.com/IIRFilterDesign/iirfd102.zip, 2008

[7] Bouchner, P.: VR Simulation Device as a Support for Research in Driver’s Micro-sleeps and Attention Decrease Detection, TRANSTEC 2004, Greece Athens, 2004, pp. 65-72.

[8] Bouchner, P., Novotný, S.: System with driving simulation device for HMI measurements, 9th WSEAS Circuits, Systems, Communications and Computers Multiconference, Greece, Athens, 2005

[9] Brandejský T.: Title: The application of behavioral features model to road traffic modeling and analysis, Neural Network World, Volume 18, Issue 1, Pages: 45-53, 2008

identification of driver s drowsiness using … · identification of driver’s drowsiness using...

Documents