This content has been downloaded from IOPscience. Please scroll down to see the full text.

Download details:

IP Address: 202.185.114.222

This content was downloaded on 05/09/2014 at 11:19

Please note that terms and conditions apply.

HMM based automated wheelchair navigation using EOG traces in EEG

View the table of contents for this issue, or go to the journal homepage for more

2014 J. Neural Eng. 11 056018

(http://iopscience.iop.org/1741-2552/11/5/056018)

Home Search Collections Journals About Contact us My IOPscience

HMM based automated wheelchairnavigation using EOG traces in EEG

Fayeem Aziz, Hamzah Arof, Norrima Mokhtar and Marizan Mubin

Department of Electrical Engineering, University of Malaya, 50603 Kuala Lumpur, Malaysia

E-mail: ahamzah@um.edu.my

Received 29 January 2014, revised 6 July 2014Accepted for publication 7 August 2014Published 4 September 2014

AbstractThis paper presents a wheelchair navigation system based on a hidden Markov model (HMM), whichwe developed to assist those with restricted mobility. The semi–autonomous system is equipped withobstacle/collision avoidance sensors and it takes the electrooculography (EOG) signal traces from theuser as commands to maneuver the wheelchair. The EOG traces originate from eyeball and eyelidmovements and they are embedded in EEG signals collected from the scalp of the user at three differentlocations. Features extracted from the EOG traces are used to determine whether the eyes are open orclosed, and whether the eyes are gazing to the right, center, or left. These features are utilized as inputs toa few support vector machine (SVM) classifiers, whose outputs are regarded as observations to anHMM. The HMM determines the state of the system and generates commands for navigating thewheelchair accordingly. The use of simple features and the implementation of a sliding window thatcaptures important signatures in the EOG traces result in a fast execution time and high classificationrates. The wheelchair is equipped with a proximity sensor and it can move forward and backward inthree directions. The asynchronous system achieved an average classification rate of 98% when testedwith online data while its average execution time was less than 1 s. It was also tested in a navigationexperiment where all of the participants managed to complete the tasks successfully without collisions.

Keywords: human–machine interface, electroencephalography, autonomous wheelchair, reha-bilitation, eye gaze tracking, hidden Markov model, support vector machine

1. Introduction

Over the years, various forms of human–machine interface(HMI) systems have been developed for different purposes.Recent advancements in signal processing have enabled HMIusers to issue commands in the forms of voice, motion ofbody parts, electromyography (EMG) [1, 2], electro-oculography (EOG) [3–5], electroencephalography (EEG)[6, 7] or a combination of input signals [8, 9]. HMIs usingvoice and visual cues, such as hand gestures, have beensuccessfully used by people with Parkinson disease, quad-riplegia, amputated or missing limbs, and motor skill dis-orders [5]. However, these HMIs are not so useful for patientssuffering from amyotrophic lateral sclerosis (ALS), motorneuron diseases, or Guillain–Barre syndrome. Since theycannot perform voluntary muscle movements with ease, EEGand EOG signals are the only convenient means for them tointeract with HMI systems [10].

In this study, EOG traces embedded in EEG signals ofalpha and delta rhythm are used to navigate a motorizedwheelchair. Features extracted from these signals are used asinputs to a few support vector machine (SVM) classifiers todetermine the eyelid position and gaze direction of the eyes,which are regarded as observations to an HMM. Based on theobservations and transition probabilities, the HMM dictatesthe direction that the wheelchair should go. The wheelchaircan move forward and backward, in a total of six directions.The asynchronous system was tested by a number of parti-cipants to execute real time instructions. It was also used tonavigate along two routes where all of the participants man-aged to complete the tasks successfully without collisions.

2. Background study

Neuromuscular disorders affect the nerves that control thevoluntary muscles [11, 12]. When the neurons are affected,

Journal of Neural Engineering

J. Neural Eng. 11 (2014) 056018 (16pp) doi:10.1088/1741-2560/11/5/056018

1741-2560/14/056018+16$33.00 © 2014 IOP Publishing Ltd Printed in the UK1

communication between the nervous system and the musclesdeteriorates, thereby weakening the muscles progressively[13, 14]. People with severe motor disabilities can stillmaintain an acceptable quality of life with the use of HMIsthat do not take signals that require muscle control as inputs.A semi-autonomous wheelchair is an example of a HMIwhere the user interacts with the system to navigate, bothindoors and outdoors. Examples of semi-autonomouswheelchairs with different levels of manual assistance arelisted in table 1.

EEG is an electrical brain signal that is recorded from thesurface of the scalp using electrodes. EEG signals have beenutilized by researchers from diverse fields to study epilepsy,apnea, coma, and various brain or neural functions and dis-orders. EEG signals have also been used to help those withdisabilities translate their intentions into computer commandsvia a HMI. Several useful derivatives of EEG signals, such asP300 wave [15–17], mu and beta rhythms and steady-statevisual evoked potentials (SSVEP), have been investigated[18–20]. EOG is an electrical signal or potential differencebetween retina (negative pole) and cornea (positive pole) ofthe eye. The induced voltage can be measured in a vertical orhorizontal direction using two sets of electrodes placed at theouter canthi. The EOG signal can vary from 50 to 3500 μV,with a frequency below 100 Hz. The positive pulse is gen-erated when the cornea moves towards positive electrodes,and vice versa. Besides noise, EOG signals may also containartifacts from EEG, EMG, electrocardiography (ECG), headmovement, and luminance. Likewise, EEG signals may alsocontain noise and traces of other signals, like EOG. Theadvantage of using the EOG trace in the EEG signals overtapping the EOG signal at outer canthi directly is that the usercan have an unobstructed view since there are no electrodesplaced around the eyes.

Blinking and horizontal eye movement signals can betraced from dipolar artifacts in EEG signals [31]. The spectralamplitude of the trace in the alpha response is higher whenthe eyes are closed than when they are open [32]. Therefore,the eyelid position can be inferred from the trace obtainedfrom the O2 occipital region. By contrast, the low frequencyartifacts extracted from the delta rhythm of the frontal lobe arerelated to the horizontal eyeball movement. Hence, the user’sgaze direction can be inferred from the frontal leads of F9 andF10 [33]. Another important cue extracted from the deltarhythm trace is blinking. Since both the natural and inten-tional blink exhibit the same cue, a double blink is usedinstead.

In our experiments, the electrodes are placed at F9, F10and O2 sites, as shown in figure 1. In the experiment, theearlobe is used as reference and FPz is taken as the ground.

3. Methodology

The O2, F9 and F10 signals are sampled at 256 samples persecond (Hz). The EOG trace in the O2 signal is obtained bybandpass filtering the signal with a Butterworth filter, with apassband of 8–13 Hz. The F9 and F10 signals are filtered with

a Chebyshev lowpass filter, with a cutoff frequency of 4 Hz.Then, for each filtered signal, a window of 128 samples isanalyzed every 0.5 s.

3.1. Open or closed eye classification

The alpha trace of channel O2 exhibits a higher amplitude ofsignal fluctuation when the eyes are closed than when they areopen, as shown in figure 2. The common amplitude of thesignal is around 5–10 uV when the eyes are open and20–40 μV when they are shut.

From the alpha trace, two simple features can be calcu-lated to distinguish closed from open eyes, which are: var-iance and central tendency measurement (CTM). Thevariance (σ2) of the 128 samples in the 0.5 s window is givenby

∑δ μ= −=N

x1

( ) (1)i

N

i2

1

2

where N is 128 and μ is the mean of the samples.CTM is simply a 2D plot of sample difference. It is a plot

of [x(n+ 2)− x(n+ 1)] against [x(n+ 1)− x(n)], which displaysthe rate of variability in a time series. From this plot, the ratioof the samples that fall inside a circle of radius r over the totalnumber of samples is calculated. Let N be the total number ofpoints in a window frame. Excluding the end points, there areN-2 points in the CTM plot. The CTM ratio is defined as

⎧⎨⎪⎪

⎩⎪⎪

δ =

+ − +

+ + − <

()i

if x i x i

x i x i r( )

1 [ ( 2) ( 1)]

[ ( 1) ( )]

0 otherwise

(2)

2

2 0.5

∑δ=− =

−

niCTM ratio

1

2( ) (3)

i

n

1

2

We find that the CTM ratio of the alpha trace is smallerduring closed eye than open eye, as shown in figure 3. In ourexperiment, the radius of the circle (r) was fixed at 2. Bothvariance and CTM are used together to determine whether theeyes are closed or open.

3.2. Gaze direction classification

In our system, only three gaze directions are used, they are:left, center and right. As the eyeballs move from one directionto another, voltage levels of the F9 and F10 traces changes inthe opposite directions. For example, when the eye gaze shiftsfrom center to left, the F9 trace decreases but the F10 traceincreases by the same amount. The same phenomenon isobserved when the gaze moves from right to center. But whenthe eye gaze moves from center to right, or from left to center,the F9 trace increases while the F10 trace decreases.

The eye gaze can also shift directly from left to right, orthe reverse. When this happens, a higher jump in voltagelevels in F9 and F10 traces is observed, as shown in figure 4.This is probably due to a full distance shift by the eyeballsgoing from corner to corner as compared to a half distance

2

J. Neural Eng. 11 (2014) 056018 F Aziz et al

Table 1. Examples of semi-autonomous wheelchairs.

HMI system Navigation type Control method Safety sensors Specialty

Nav-Chair [21, 22] Semi-autonomous Joystick Ultrasonic sensor Avoid obstaclesBerman autonomous [23] Semi-autonomous Joystick Sonar sensor Avoid hazardsSENARIO [24] Semi and full autonomous Voice and goal to goal Multiple types of sensor Control without handOMNI [25] Semi/full autonomous Joystick and goal to goal Multiple types of sensor Can be run in high or low level autonomous navigationMAid [26] Semi-autonomous Joystick Motion sensor Crowd avoidanceWheesly [27] Semi-autonomous EOG Proximity and sonar sensor Command system can change by the need of the userVAHM [28] Semi/full autonomous Joystick and goal to goal Ultrasonic sensor Combination of high and low level autonomous navigationSmart wheelchair [29, 30] Semi-autonomous EEG, P-300 wave No sensors Run by mental taskBIDIM-EOG [4] Semi-autonomous EOG No sensors Fully controlled by eye movement

3

J.NeuralE

ng.11

(2014)056018

FAziz

etal

Figure 1. The positions of the electrodes and samples of their trace signals.

40

20

0

0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000

-20

-40

Sample

Amplitude

Open EyeOpen EyeClose Eye

Figure 2. Alpha signal during closed and open eye.

10

8

6

4

2

0

-2

-4

-6

-8

-10-10 -10

10

10 10

8

6

4

2

0

-2

-4

-6

-8

-10-5 0 5 -5 0 5

X(n+2-X(n+1)

X(n+2-X(n+1)

X(n+1)-X(n) X(n+1)-X(n)

(a) (b)

Figure 3. (a) CTM plot of closed eye alpha with CTM ratio = 0.24. (b) CTM plot of open eye alpha with CTM ratio = 0.89.

4

J. Neural Eng. 11 (2014) 056018 F Aziz et al

shift from center to corner (or corner to center). To accentuatethe effect of the opposite voltage level movements in F9 andF10 traces caused by gaze shift, we subtract F10 from F9 andextract two features from the difference signal, which are: thevoltage level rise and average first derivative. The derivativeof the ith sample is defined as

⎧⎨⎪⎪

⎩⎪⎪

=

− <

+ < < −

− > −

+

− +

−

d

x x if i a

x x if a i n a

x x if i n a

( )/2 (4)i

i a i

i a i a

i i a

where n = 128 and a = 5 in our experiment. Then the averageof the first derivatives (f) is calculated as

∑==

fn

d1

. (5)i

n

i

1

The two features are fed to a series of SVM classifiersthat classify a gaze shift into five classes, which are:unchanged, half right, half left, full right and full left. Full

right is the corner to corner gaze shift from left to right, whilefull left is the corner to corner gaze shift from right to left.Half right and half left are semi gaze shift to the right and left,respectively. Half right represents either a center to right orleft to center gaze shift since they both produce the samevoltage level change. Similarly, half left represents either acenter to left or right to center gaze shift. Knowing the eyegaze direction prior to half left or half right gaze shift allowsthe system to determine the final gaze direction.

3.3. Double blink detection classification

When the user blinks his eyes, intentionally or otherwise, ashort positive pulse or spike is temporarily detectable in boththe F9 and F10 signals. Therefore, adding F9 and F10 signalsshould accentuate the blink signal and attenuate the effect ofhorizontal eye movement, as shown in figure 5. Since naturaland intentional blinks generate an identical pulse in the addedsignal, they are indistinguishable. Thus, a double blink isinstead used as one of the instructions in the system. Since a

Figure 4. Samples of voltage jumps caused by gaze shifts (a) from center to left, (b) from center to right, (c) from right to left and (d) from leftto right.

Figure 5. (a) EEG signal pattern of F9 and F10 channels. (b) Added signal of F9 and F10.

5

J. Neural Eng. 11 (2014) 056018 F Aziz et al

double blink consists of two close blinks, two spikes or pulsesshould be observed within two consecutive window frames.Variance and average first derivative can be extracted fromthe sum signal (F9 + F10) to detect the presence of a doubleblink.

3.4. Sliding window

While monitoring the alpha trace, a conventional windowframe might capture an interval when the eyes are just aboutto open or shut. During this short period, the signal is intransition from low amplitude to high, or vice versa. In thiscase, the samples in the window are a mixture of those withlow and high amplitudes, as shown in figure 6. Consequently,the variance obtained neither represents open nor closed eyecondition. Therefore, a sliding window is used to adjust theposition of the frame, so that it will either contain samples ofclosed or open eye exclusively.

First, the absolute values of the samples in the windoware divided into eight adjoining subintervals where eachsubinterval contains 16 samples. Then, the average of samplesin each subinterval is calculated and the averages of the firstand last subinterval are compared. If the absolute differenceof the two averages is more than a specified threshold, thewindow is shifted to the right by 16 samples. The process isrepeated until their difference is less than the specifiedthreshold. Once this condition is met, the variance and CTMof the samples in the whole window can be calculated.

For the sum and difference signals of the F9 and F10traces, the position of the window is also adjusted so that animportant cue will be at the center of the window, as illu-strated in figure 7. This step is necessary to obtain a highclassification accuracy. First, the samples in the window aredivided into eight adjoining subintervals, where each sub-interval contains 16 samples. Then, the average first deriva-tive of samples in each subinterval is calculated and theaverages from the first and last subinterval are compared. Iftheir absolute difference is more than a specified threshold,the window is shifted to the right by 16 samples. The processis repeated until the difference is less than the threshold. Oncethis condition is met, the variance, VLD and AFD of thesamples are calculated.

3.5. SVM classifier

An SVM is a binary classifier that is useful for segregating theinput features into two classes. For closed or open eye clas-sification, using the alpha trace features only a single SVM issufficient. This is also true for the double blink detectionusing features from the sum signal. However, to classify thegaze shifts using features from the difference signal, fourSVMs are needed. The first SVM determines whether thegaze is shifted or not. If the gaze is shifted, then the secondSVM decides whether the shift is to the left or right. If thegaze is shifted to the right, then the third SVM will furtherclassify it into full or half shift. Similarly, if the gaze is shiftedto the left, then the fourth SVM will determine whether theleft shift is half or full. The classification process is depictedin figure 8. Using the outcome of this classification and theinitial gaze direction (prior state), the HMM will decide thecurrent gaze direction (current state) of the system and theappropriate command for the wheelchair.

3.6. Hidden Markov model (HMM)

The system is designed to allow the wheelchair to moveforward and backward in three different directions. It shouldbe able to stop as well. However, there are only a limitednumber of distinct horizontal eyeball movements and they areinsufficient to move the wheelchair in six different directions.Another problem is that, left to center and center to right gazeshifts produce the same voltage change in the difference tracesignal. This is also true for right to center and center to leftgaze shifts. Thus, we resort to using an HMM to tackle thischallenge.

An HMM is a statistical model where the state is notdirectly visible. In our case, the state is left, middle, or right.The transition from one state to another is governed by thefeatures extracted from the alpha and delta traces. The prob-ability of the current state of the model P(St|Model), isobtained by multiplying the transition probability of theprevious state to current state P(St|St − 1) and the probability ofdetecting the current observation given the current state P(Ot|St). The gaze direction that maximizes the product of theprobabilities is chosen as the current state, as stated in the

Figure 6. (a) Conventional window, (b) sliding window capturing close eye in alpha rhythm.

6

J. Neural Eng. 11 (2014) 056018 F Aziz et al

following equations.

= −( ) ( ) ( )P S Model P S S P O S. (6)t t t t t1

= ( )( )S P S Modelarg max . (7)t t

It is assumed that the previous state is known. At the begin-ning (t = 0), when the previous state does not exist yet, theinitial probability P(π) is used to replace the transition prob-ability P(St|St − 1). Details on the fundamental and imple-mentation of HMM are available in [34]. The hidden Markovmodel is shown in figure 9.

3.7. The overall system

The system is designed to have two modes of operation,namely: the ready and run mode. Both modes contain threestates, which are right, middle and left, as depicted infigure 10. In both modes, the state of the system follows thegaze direction of the eyes. The O2, F9 and F10 traces areanalyzed every 0.5 s. The features extracted from these sig-nals are used to inspect the condition of the eyes, whetherthey are closed or open, the occurrence of double blink, andgaze shift.

The ready mode allows the system to be executed atsoftware level only so that the user can train and check thesystem functionality while the wheelchair is stationary. This

mode also allows the user to select either forward or back-ward as the direction the wheelchair will go when it enters runmode. For run mode, both the software and hardware of thesystem are fully functional. In this mode, the wheelchair willmove according the gaze direction. It can move straight, turnleft or turn right, in a forwards or backwards direction. It stopsif the user closes his eyes or double blinks.

When the system is turned on, it is assigned to the middlestate of the ready mode by default. The state should follow thegaze direction of the eyes at all time. If the state of the systemfails to follow the gaze shift correctly, then an error occurs.The system is designed to correct minor errors automaticallybut inform the user when a major error is detected. A minorerror occurs when the state is correct but the command signalis wrong. For instance, the current state is middle but thecommand asks the system to move the state from left to right.In this case, the system will just move the state from center toright. A major error occurs when the state is wrong and acommand cannot be executed at all. For example, the state isright and the command is to move the state from left to right.In this case, the system cannot execute the command becausethe state is already right. Thus, the system will alert the user tomove his gaze to the center and reset its state to the middle.For whatever reason, the user can also reset the systemmanually by closing their eyes for at least 3 s. This is abackup measure to correct a major error if the system fails to

Figure 7. (a) Conventional window capturing part of a ramp in difference signal. (b) Sliding window capturing the full ramp. (c)Conventional window capturing part of a pulse in sum signal. (d) Sliding window capturing the full pulse.

7

J. Neural Eng. 11 (2014) 056018 F Aziz et al

correct it automatically. The wheelchair is also equipped witha proximity sensor to detect nearby objects and avoidcollisions.

Once the user is familiar with the system, they may wantto command the wheelchair to move. The first step is to

choose the direction that the wheelchair will go in whenentering the run mode, either forward or backward. The movedirection is chosen by shifting the state to the right or left, forforward or backward direction, respectively. Then, the usershould maintain their gaze in that direction for at least 3 s or 6consecutive frames to lock the chosen direction. The user canchange the move direction by gazing in the opposite directionand maintaining their gaze for at least 3 s. But if the userwants to unlock the move direction, they simply have to shuttheir eyes for at least 3 s. This action unlocks the movedirection and returns the system to the middle state of readymode. The user is expected to set their gaze to the centerwhen opening their eyes so that their gaze will match with thedefault state. Once a move direction is selected, and the useris ready to move the wheelchair, they should blink twice tosend the system into run mode immediately.

Upon entering the run mode, the user is given 3 s todirect his gaze to where he wants to go before it is taken as the

Figure 8. Flow of classification process.

Figure 9. Hidden Markov model.

8

J. Neural Eng. 11 (2014) 056018 F Aziz et al

first instruction to move the wheelchair. Each instruction willbe executed for at least 2 s to maintain the stability of themoving wheelchair and avoid command overcrowding.Another restriction is that the wheelchair will stop beforeturning or changing direction. And finally, only a gradualchange of direction is allowed. An abrupt change of directionfrom left to right, or vice versa, will be executed with anintermediate step of going straight for 2 s. For instance, if auser wishes to change the direction of the wheelchair fromright to left, then the wheelchair will change direction gra-dually from left to straight first before turning left. This is toprevent the wheelchair from toppling over. At all times, thewheelchair moves at 5 km h−1. If a user wishes to stop thewheelchair, all they have to do is to close their eyes for atleast 0.5 s, so that the system will terminate whatever com-mand it is executing and stop. This action makes the systemexit the run mode and sends it to the middle state of readymode without unlocking its move direction. If the user wishesto re-enter the run mode, they have to double blink as before.In ready mode, the user can also select a new move direction,reset the system or rest.

It should be noted that in run mode a double blink cue isnot used because detecting a double blink is more time con-suming since it occupies two consecutive windows. The lastpoint to note is that in a span of 2 s, only one command iskept for the next move while the wheelchair is executing thecurrent instruction. If the user produces multiple instructionswithin that interval, only the latest is kept as the next com-mand to be executed.

4. Experiments

The EEG signals are recorded using the g-USBAMP toolkitfrom Guger Technology at a sampling rate of 256 Hz. Silverelectrodes are placed at five different locations on the scalp

according to the standard International 10–20 system. Thelocations are F9, F10, O2, with the earlobe as reference andFPz as ground. These locations are selected since they containthe strongest EOG trace in a filtered signal with minimumnoise. The horizontal gaze movement and blink are inter-preted from the F9 and F10 channels. Closed and opened eyeconditions are inferred from the O2 channel. The recordedsignals are analyzed using the LabView software fromNational Instrument. The window size is 0.5 s since it is thesmallest possible for capturing a blink pulse.

The experiments are conducted in three sessions, namely:offline, online, and navigation sessions. In the offline session,the user is exposed to the system functionality and data arecollected to train the SVMs and HMM, as well as to deter-mine the values of thresholds needed for sliding windowadjustment. This is followed by an online session where theuser is asked to move the wheelchair according to randomlychosen instructions. Lastly, a real time navigation sessionwherein the user is asked to navigate the wheelchair alongtwo routes is performed. Details of the experiments in eachsession are described in the following section.

4.1. Offline session

The objectives of the training session are to check the systemcondition and record the alpha and delta signals for calcu-lating thresholds and training the SVMs and HMM. Thissession is conducted in ready mode and thus the wheelchair isstationary. A series of instructions are issued to the user andthe time allocated for each instruction is 5 s. Then, a 5 s breakis given before the next instruction is issued. Turn left, turnright, go straight, double blink and close eye are the fivepossible instructions that are randomly assigned. The parti-cipant is expected to respond within 2 s after the instruction isannounced. The change in states and modes are shown by theLED display on the user interface. If any of the instructionsare wrongly or not executed, then the instruction is repeated.Altogether, 100 instructions must be executed by each parti-cipant and the output signals recorded.

4.2. Online session

The online session is conducted in run mode and thus thewheelchair will move according to the user’s commands.First, each participant is instructed to lock the move directionto forward and enter the run mode. In run mode, there areonly four possible instructions, which are: turn left, turn right,go straight and close eye. In total, 20 instructions are assignedto each participant, five from each command. The instructionsare randomly assigned every 5 s without break. Upon hearinga command, the participant is expected to perform itaccordingly. A command is considered successfully executedif the user is able to move the wheelchair correctly in lessthan 2 s.

Next, the participant is instructed to change the movedirection to backwards. Then, 20 instructions are assigned toeach participant randomly, five from each command asbefore. Throughout this session the changes in the state are

Figure 10. The modes and states of the system.

9

J. Neural Eng. 11 (2014) 056018 F Aziz et al

also displayed on the user interface panel by LED lights.After this, the participant is given 30 min to tinker with thesystem freely to gain more familiarity before the navigationsession commences.

4.3. Navigation session

Two routes are designed to allow the user to navigate in anindoor environment, as shown in figure 11. The passage widthis in the range of 4.5 m to 1.5 m. The first route is shorter andits total optimal length is 13.5 m with four checkpoints of A1,B1, C1 and D1. The second route has a total optimal length of54.4 m with seven checkpoints from A2 to G2. The participantsare responsible for maneuvering the wheelchair to pass throughthese check points, starting from the first point to the last point,and then return to the first point. However, when the partici-pants reach the dead end points of D1 (in figure 11(a)) and G2

(in figure 11(b)) they are expected to reverse the wheelchairbackwards to exit the tight end and only make a u-turn at pointC1 and F2 respectively. Each participant is given three attemptsto repeat the task and at the end of the session they are asked toanswer a set of questionnaires regarding their experience.

4.4. Hardware implementation

From the result of the classifier, a command to the wheelchairis identified and issued. The system checks whether it is thesame command that is currently being executed. If it is thesame, then the execution of the current command is pro-longed. If it is different, then a new instruction in the form ofa digital command is sent to a digital to analog module thatconverts the digital signal to an analog voltage level. Themodule used in our experiment is a NI9264 from NationalInstrument. The analog output is then sent to a motor con-troller that controls the right and left motors of the wheelchair,as shown in figure 12(a). The controller will switch on theright, left, or both motors if the command is turn left, turnright, or move forward, respectively. If a shut eye is detected,both motors will stop and the system will exit run mode andenter ready mode at a middle state. The mode, move direction

and current state are all displayed in front of the user by ledlights, as shown in figure 12(c).

5. Results and discussion

5.1. Offline and online session results

A total of 20 healthy subjects participated in the offline andonline sessions. They were mainly undergraduate and post-graduate students who had no neurological impairments ornystagmus. Other data on the subjects are given in table 2. Inthe offline session, the participants were required to performdouble blink, eye closed, and horizontal saccades to the leftand right. Altogether, 2000 signals were recorded from theparticipants. Features extracted from the EOG traces wereused to train the SVMs and calculate the thresholds necessaryfor the sliding window operation.

This is followed by the online session where participantswere instructed to execute the four commands in forward andbackward directions. Throughout the session, each of the fourcommands was performed 100 times in the forward and thenbackward direction. A summary of the online tests is pre-sented in the confusion matrix of table 3. From the total of800 commands issued, 786 are correctly executed, so that thetotal execution rate is 98.25%. It can be observed that the gostraight and close eye commands are perfectly classified andexecuted. Six turn right and eight turn left commands werewrongly executed. Out of these errors, eight commands were

Figure 11. The navigation route with checkpoints (a) route 1. (b) route 2.

Table 2. Average height, weight, gender and corrective lenses of thesubjects according to age groups.

Agegroup

Avgheight(cm)

Avgweight(kg) Male Female

Wearinglenses

20-23 167.33 62.33 2 3 224-27 169.02 73.14 8 2 428-31 171.33 65.52 2 1 032-35 152.50 61.20 0 2 2

10

J. Neural Eng. 11 (2014) 056018 F Aziz et al

not executed within 2 s and the remaining six errors are cornerto corner gaze shifts that were wrongly interpreted as cornerto center shift. This is mainly due to the user failing to shifthis/her gaze from corner to corner rapidly. This generatesweaker pulses in F9 and F10, which are similar to the onesgenerated by corner to center gaze shift. Factors that maycause this error are fatigue, inattention, and distraction.Another possible cause is that the wheelchair movementmight lead to head or body movement, which could result inunwanted EEG or EOG artifacts. The artifacts might in turnaffect the accuracy of command estimation.

A comparison with similar wheelchair navigation studiesis given in table 4. Although the classification rate achievedby Rebsamen et al [16] is slightly higher, the execution timeof their system is considerably higher than 1 s. On top of this,our system features backward movement, which is notavailable in these other systems. Furthermore, they employeda P300 synchronous system, which requires the user to followthe system pace.

5.2. Navigation session results

Five participants with the highest score in the online sessionwere selected to perform the navigation tests. In this session,

the participants were free to use as many commands asnecessary to control the position and direction of the wheel-chair along the paths. Then, the performances of the wheel-chair system and the participant were evaluated according tothe following criteria:

(1) Task success: completion of the navigation through theend of path.

(2) Path length: distance in meters traveled to accomplishthe task.

(3) Time: time in seconds taken to accomplish the task.(4) Path optimality ratio: ratio of the path length to the

optimal path (the optimal path was 13.5 m for route 1 and54.4 m for route 2, see figure 5).

(5) Time optimality ratio: ratio of the time taken to theoptimal time to complete the task (the optimal time wasapproximately 9.7 s for route 1 and 39 s for route 2 basedon maximum and rotational velocities of 1.39 m s−1 and0.4 rad s−1).

(6) Collisions: total number of collisions during the tasks. Acollision is not considered to b a failure as long as thesystem is able to continue with a new command, orrequires a brief intervention from the supervisor.

(7) Mean velocity: mean velocity in meter per secondsduring motion.

Figure 12. (a) Diagram of the integrated system. (b) A subject undergoing the experiment. (c) Print screen of the user interface.

11

J. Neural Eng. 11 (2014) 056018 F Aziz et al

Table 3. Confusion matrix for system errors in direction estimation.

Commands Actions Right-middle Left-middle Middle-right Left-right Middle-left Right-left Close eye Not executed

Go straight Right-middle 100 0 0 0 0 0 0 0Left-middle 0 100 0 0 0 0 0 0

Turn right Middle-right 0 0 98 0 0 0 0 2Left-right 0 3 0 96 0 0 0 1

Turn left Middle-left 0 0 0 0 97 0 0 3Right-left 3 0 0 0 0 95 0 2

Stop Close eye 0 0 0 0 0 0 100 0

12

J.NeuralE

ng.11

(2014)056018

FAziz

etal

Table 4. Comparison between this study and other HMI system used for wheelchair navigation.

HMI system Classifier Classification accuracy (%) Time (s)

Huang et al (2012) [35] Imagery Synchronous Mahalanobis linear distance (MLD) 84.3 60.4Kus et al (2012) [36] Imagery Asynchronous MLD 74.84 8.81Choi et al (2013) [37] Imagery + SSVEP+ P300 Asynchronous SVM, CCA and Bayesian 84.4 – 91 27.2Long et al (2012) [38] Imagery + P300 Asynchronous LDA 83.10 6Rebsamen et al (2010) [16] P300 Synchronous SVM 99.78 6Iturrate et al (2009) [39] P300 Synchronous Stepwise linear discriminant analysis (SWLDA) 94 5.40Diez et al (2011) [40] SSVEP Asynchronous Fourier transform 81.68 4.48Parini et al (2009) [41] SSVEP Asynchronous LDA 97.5 2.38Postelnicu (2012) [5] EOG Asynchronous Fuzzy logic 95.63 2.6Proposed study EEG+EOG Asynchronous SVM and HMM 98 0.68

13

J.NeuralE

ng.11

(2014)056018

FAziz

etal

All of the subjects succeeded in navigating the wheel-chair along the routes without collision. The results of thecorresponding metrics are summarized in table 5. The averagedistances travelled by the participants were 15.76 meters and75.42 meters for route 1 and route 2, respectively. The pathoptimality ratios were low (1.17 for route 1 and 1.35 for route2), which suggests that the participants were able to travelclose to the designated paths. The average times to traversethe routes were 15.04 and 85.67 s for route 1 and route 2,respectively. On average, the ratios of the optimal times were1.55 for route 1 and 2.05 for route 2. The participants took alonger time to complete the second route because it has moreturns and longer narrow passages. The actual time is higherthan the optimal time because it includes mode changing,

command selection, and computational time. The meanvelocity of the wheelchair was 1 m s−1 for the first route. Forthe second route, the participants had a lower velocity of0.88 m s−1. The maximum speed was limited to 1.3 m sec−1

for safety reasons. In general, a narrow performance gapobserved among the participants signifies the system’susability, and consistency to navigate and maneuver thewheelchair in open and small spaces.

5.3. Participant experience evaluation

At the end of the navigational session, the participants weregiven a set of questionnaires to express their navigationalexperience in four metrics of average workload, learnability,confidence and difficulty, as presented in figure 13. The

Table 5. Metrics to evaluate the navigation performance.

Task 1 Task 2

Factors min max mean std Min max mean std

Task success 1 1 1 0 1 1 1 0Path length 13.87 17.14 15.76 1.17 55.46 95.57 75.42 19.27Time 12.66 17.86 15.04 1.93 67.52 117.06 85.67 18.84Path optimally ratio 1.03 1.27 1.17 0.09 1.02 1.76 1.35 0.32Time optimally ratio 1.30 1.84 1.55 0.20 1.52 2.99 2.05 0.51Collisions 0 0 0 0 0 0 0 0Mean velocity 0.90 1.12 1.00 0.10 0.64 1.09 0.88 0.18

Figure 13. The assessment of the participants’ navigational experience as (a) work load (b) learnability (c) confidence (d) difficulty. The scaleof 0–5 (least to the most) was used to express the metrics.

14

J. Neural Eng. 11 (2014) 056018 F Aziz et al

metrics were measured on a scale between 0 and 5, from theleast to the most. The workload metrics denotes the effort toaccomplish the tasks. All of the participants agreed that ittook a higher workload to complete route 2 than route 1because route 2 has more turns and a narrower passage thanroute 1. Participant P3 and P5 recorded the most effortrequired to operate the wheelchair in both routes. The learn-ability metrics indicate that the participants adapted to controlthe navigation system based on their experience from theprevious attempt, thus increasing their confidence level toperform the task from route 1 to route 2. As for difficultymetrics, all of the participants agreed that route 2 is moredifficult than route 1.

A t-test was performed to gauge the influence of routeson the level of difficulty. The null hypothesis stipulates thatthere is no significant difference between the means of diffi-culty for route 1 and route 2. The result of the test clearlydisproves the null hypothesis and shows that the difference inmean difficulty of the routes is significant with (Mean = 3.2 vs4.4, t(4) =−3.21, p = 0.016).

6. Conclusions

In this paper, EOG traces embedded in EEG signals are usedto drive a wheelchair navigation system. A sliding window isused to capture important signatures in the EOG tracesaccurately without partial loss of information. Extracted fea-tures are used to infer the eye condition (i.e. whether the eyeis open or closed) and gaze direction using a number of SVMsand an HMM. The HMM determines the state of the systemand generates the commands that move the wheelchair. Thewheelchair can be steered in six directions forwardly andbackwardly. Two different modes of operation are madeavailable to ensure safety and convenience to the user. Theasynchronous system achieved an average classification rateof 98% in an online test, with an average execution time ofless than 1 s.

An actual navigation session was performed whereinfive participants undertook two navigation tasks by maneu-vering the wheelchair through two designated routes. Theparticipants managed to complete the tasks without colli-sions. In this session, the usability of the backward move-ment proved useful when the wheelchair was trapped intight dead ends with no space to make a turnaround. Agood average accuracy of 98% for route 1 and 97% for route2 shows that the system is easy to use and stable. Thissystem is still in the development phase and, therefore, hasnot been tested with real patients who suffer a severe motordisorder.

Acknowledgments

This research is fully funded by High Impact Research GrantScheme (D000016-16001) under Ministry of Higher Educa-tion, Malaysia.

References

[1] Ratwani R M and Trafton J G 2011 A real-time eye trackingsystem for predicting and preventing postcompletion errorsHum.–Comput. Interact. 26 205–45

[2] Freire Bastos T, Ferreira A, Cardoso Celeste W,Cruz Calieri D, Sarcinelli Filho M and de la Cruz C (ed)2009 Silla de ruedas robótica multiaccionada inteligente concapacidad de comunicación. Proc. III Int. Congr. onDomotics, Robotics and Remote-Assistance for All(Barcelona, Spain May 2009) pp 45–52

[3] Usakli SG A B, Aloise F, Vecchiato G and Babiloni F 2010The use of electrooculogram for efficient human computerinterfaces Comput. Intell. Neurosci. 2010 5

[4] Barea R, Boquete L, Bergasa L M, Lopez E and Mazo M 2003Electro-oculographic guidance of a wheelchair using eyemovements codification Int. J. Robot. Res. 22 641–52PMID: ISI:000185206200012

[5] Postelnicu C C, Girbacia F and Talaba D 2012 EOG-basedvisual navigation interface development Expert Syst. Appl.doi:10.1016/j.eswa.2012.03.007

[6] Escolano C, Antelis J and Minguez J (ed) 2009 Humanbrain-teleoperated robot between remote places Proc. IEEEInt. Conf. on Robotics and Automation (ICRA ’09)pp 4430–7

[7] Millán J R and Carmena J M 2010 Invasive or noninvasive:understanding brain-machine interface technology IEEEEng. Med. Biol. 29 16

[8] Hoppenot P and Colle E 2002 Mobile robot command byman–machine co-operation–application to disabled andelderly people assistance J. Intell. Robot. Syst. 34 235–52

[9] Perrin X, Chavarriaga R, Colas F, Siegwart R and Millán JdR2010 Brain-coupled interaction for semi-autonomousnavigation of an assistive robot Robot. Auton. Syst. 581246–55

[10] Deng L Y, Hsu C L, Lin T C, Tuan J S and Chang S M 2010EOG-based human–computer interface system developmentExpert Syst. Appl. 37 3337–43

[11] Rowland L P and Shneider N A 2001 Amyotrophic lateralsclerosis New Engl. J. Med. 344 1688–700 PMID: 11386269

[12] Emery A E H 1991 Population frequencies of inheritedneuromuscular diseases—a world survey NeuromuscularDisorders 1 19–29

[13] Coers C, Telerman-Toppet N and Gerard J-M 1973 Terminalinnervation ratio in neuromuscular disease: II. disorders oflower motor neuron, peripheral nerve, and muscle Arch.Neurol. 29 215

[14] Abresch R T, Carter G T, Jensen M P and Kilmer D D 2002Assessment of pain and health-related quality of life inslowly progressive neuromuscular disease Am. J. HospicePalliative Med. 19 39–48

[15] Jin J, Sellers E W and Wang X Y 2012 Targeting an efficienttarget-to-target interval for P300 speller brain-computerinterfaces Med. Biol. Eng. Comput. 50 289–96 PMID:ISI:000302263100009

[16] Rebsamen B et al 2010 A brain controlled wheelchair tonavigate in familiar environments IEEE Trans. Neural Syst.Rehabil. 18 590–8 PMID: ISI:000285057800002

[17] Yu T Y, Li Y Q, Long J Y and Gu Z H 2012 Surfing theinternet with a BCI mouse J. Neural. Eng. 9 036012 PMID:ISI:000304526300012

[18] Allison B Z, Brunner C, Altstatter C, Wagner I C,Grissmann S and Neuper C 2012 A hybrid ERD/SSVEPBCI for continuous simultaneous two dimensional cursorcontrol J. Neurosci. Meth. 209 299–307 PMID:ISI:000308897900005

[19] Ng K B, Bradley A P and Cunnington R 2012 Stimulusspecificity of a steady-state visual-evoked potential-based

15

J. Neural Eng. 11 (2014) 056018 F Aziz et al

brain-computer interface J. Neural. Eng. 9 036008 PMID:ISI:000304526300008

[20] Hwang H J, Lim J H, Jung Y J, Choi H, Lee S W and Im C H2012 Development of an SSVEP-based BCI spelling systemadopting a QWERTY-style LED keyboard J. Neurosci.Meth. 208 59–65 PMID: ISI:000305865600008

[21] Levine S P, Bell D A, Jaros L A, Simpson R C, Koren Y andBorenstein J 1999 The NavChair assistive wheelchairnavigation system IEEE Trans. Rehabil. Eng. 7 443–51

[22] Simpson R C and Levine S P 1999 Automatic adaptation in theNavChair assistive wheelchair navigation system IEEETrans. Rehabil. Eng. 7 452–63

[23] Röfer T and Lankenau A 2000 Architecture and applications ofthe Bremen autonomous wheelchair Inf. Sci. 126 1–20

[24] Katevas N I et al 1997 The autonomous mobile robotSENARIO: a sensor aided intelligent navigation system forpowered wheelchairs IEEE Robot. Autom. Mag. 4 60–70

[25] Hoyer H et al (ed) 1995 The omni wheelchair with highmanoeuvrability and navigational intelligence RehabilitationTechnology, Proc. 2nd TIDE Congress (Brussels: IOS Press)

[26] Prassler E, Scholz J and Fiorini P 2001 A robotics wheelchairfor crowded public environment IEEE Robot. Autom. Mag. 838–45

[27] Yanco H A 1998 Wheelesley: a robotic wheelchair system:indoor navigation and user interface Assistive Technologyand Artificial Intelligence (Berlin: Springer) pp 256–68

[28] Bourhis G and Agostini Y 1998 The VAHM robotizedwheelchair: system architecture and human-machineinteraction J. Intell. Robot. Syst. 22 39–50

[29] Millán J, Galán F, Vanhooydonck D, Lew E, Philips J andNuttin M (ed) 2009 Asynchronous non-invasive brain-actuated control of an intelligent wheelchair Proc. Annu. Int.Conf. IEEE/EMBS (EMBC 2009) pp 3361–4

[30] Rebsamen B et al (ed) 2006 A brain-controlled wheelchairbased on P300 and path guidance Proc. 1st IEEE/RAS-EMBS Int. Conf. on Biomedical Robotics andBiomechatronics (BioRob 2006) pp 3361–4

[31] Fatourechi M, Bashashati A, Ward R K and Birch G E 2007EMG and EOG artifacts in brain computer interface systems:A survey Clin. Neurophysiol. 118 480–94

[32] Kirkup L, Searle A, Craig A, McIsaac P and Moses P 1997EEG-based system for rapid on-off switching without priorlearning Med. Biol. Eng. Comput. 35 504–9

[33] Marzano C et al 2007 Slow eye movements and subjectiveestimates of sleepiness predict EEG power changes duringsleep deprivation Sleep 30 610–6

[34] Rabiner L R 1989 A tutorial on hidden Markov models andselected applications in speech recognition Proc. IEEE 77257–86

[35] Huang D D, Qian K, Fei D Y, Jia W C, Chen X D and Bai O2012 Electroencephalography (EEG)-based brain-computerinterface (BCI): A 2-D virtual wheelchair control based onevent-related desynchronization/synchronization and statecontrol IEEE Trans. Neural Syst. Rehabil. 20 379–88 PMID:ISI:000304550600017

[36] Kus R, Valbuena D, Zygierewicz J, Malechka T,Graeser A and Durka P 2012 Asynchronous BCI based onmotor imagery with automated calibration andneurofeedback training IEEE Trans. Neural Syst. Rehabil.20 823–35 PMID: ISI:000311357400009

[37] Choi B, Jo S A and Low-Cost E E G 2013 System-basedhybrid brain-computer interface for humanoid robotnavigation and recognition PLOS One 8 e74583

[38] Long J Y, Li Y Q, Wang H T, Yu T Y, Pan J H and Li F A2012 Hybrid brain computer interface to control thedirection and speed of a simulated or real wheelchair IEEETrans. Neural Syst. Rehab. Eng. 20 720–9 PMID:WOS:000308462600011

[39] Iturrate I, Antelis J M, Kubler A and Minguez J A 2009Noninvasive brain-actuated wheelchair based on a P300neurophysiological protocol and automated navigation IEEETrans. Robot. 25 614–27 PMID: ISI:000266878300014

[40] Diez P F, Mut V A, Perona E M A and Leber E L 2011Asynchronous BCI control using high-frequency SSVEPJ. Neuroeng. Rehabil. 8 39 PMID: ISI:000293704300001

[41] Parini S, Maggi L, Turconi AC and Andreoni G A 2009Robust and self-paced BCI system based on a four ClassSSVEP paradigm: algorithms and protocols for a high-transfer-rate direct brain communication Comput. Intell.Neurosci. 2009 864564

16

J. Neural Eng. 11 (2014) 056018 F Aziz et al