towards development of a robust asynchronous eeg-bci using ... · accuracy (+7%) and the...

Towards Development of a Robust Asynchronous EEG-BCI usingError-Related Potentials

by

Rozhin Yousefi

A thesis submitted in conformity with the requirementsfor the degree of Doctor of Philosophy

Graduate Department of Biomedival EngineeringUniversity of Toronto

c© Copyright 2019 by Rozhin Yousefi

Abstract

Towards Development of a Robust Asynchronous EEG-BCI using Error-Related Potentials

Rozhin Yousefi

Doctor of Philosophy

Graduate Department of Biomedival Engineering

University of Toronto

2019

The primary goal of brain-computer interface (BCI) research is to provide a means of communication

for individuals with severe motor impairments. For BCIs to be widely adopted by patients, they need

to have a reliable accuracy. One method to achieve this goal is to automatically correct erroneous

classifications by exploiting error-related potentials (ErrPs). The first objective of this research is to

explore how does the introduction of error-based autocorrection impact the accuracy of a multi-class

active BCI based on non-motor imagery (MI) cognitive tasks. Another unavoidable step toward making

BCIs clinically viable is to develop self-paced BCIs that users can control whenever they intend to.

The second objective of this research is to investigate the impact of ErrP-based autocorrection on the

accuracy of self-paced BCIs based on non-MI cognitive tasks. The first two studies were designed to

answer the first research question. In the first study, participants performed multiple iterations of five

different cognitive tasks. To simulate errors, a random subset of 20% of the trials were followed by

incorrect feedback. An average area under curve of 0.83 was reached for the detection of ErrPs which

confirmed the presence of ErrPs in such BCIs. The second study explored the effect of ErrP-guided error

correction in an online three-class active BCI (idle state and two personally selected cognitive tasks).

ErrP-based error correction modestly but significantly improved the average online task classification

accuracy (+7%) and the information transfer rate (+0.9 bits/min) of the BCI across participants. The

third study was designed to answer the second research question. Using a self-paced EEG-BCI based on

cognitive tasks, the possibility of improving the BCI performance using ErrPs was investigated. The BCI

continuously analyzed the EEG data and displayed real-time feedback as soon as it detected a cognitive

task. Then, the BCI analyzed the EEG data after the feedback onset to detect ErrPs. The average of

post-error correction success rate across participants improved significantly compared to the pre-error

correction value (+7%). The findings of these studies support the addition of ErrP-informed correction

to maximize the accuracy of cue-based and self-paced BCIs based on non-MI cognitive tasks.

ii

Acknowledgements

My PhD journey has been a truly life-changing experience for me and it would not have been possible

to do without my supervisor’ guidance, Dr. Tom Chau. I would like to express my sincere gratitude for

his constant support, encouragement, and kind advice through this stage of my life. I will always be

grateful for accepting me as your student, helping me to put behind my unfortunate experiences at the

beginning of my PhD, and turning it into most rewarding experience of my academic life.

I would like to thank my committee members, Dr. Anne-Marie Guerguerian, Dr. Tilak Dutta, and

Dr. Babak Taati for their continuous support, valuable suggestions, and helpful questions that have

made this work stronger. I would also like to thank Dr. Gernot Muller-Putz, who graciously agreed to

serve in my final PhD exam committee.

Thanks to all members of the PRISM lab, you have made graduate school such a welcoming and

stimulating experience.

Thanks to all those who volunteered their time to participate in my experiments. None of this would

have been possible without you.

Many thanks go out to my mom, dad, and brother for their constant support and encouraging me

along the way. Special thanks to my sister, Zhino, for always being there for me.

Finally, to my wonderful spouse and my best friend, Alborz, thank you for everything you have done

for me. Words fail to express my appreciation for your love and affection.

iii

Contents

1 Introduction 1

1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Research questions and objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.3 Outline of the thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Background 4

2.1 Brain-computer interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.1.1 EEG as BCI Modality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1.2 BCI types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.1.3 Cue-based versus self-paced BCIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.1.4 Functional activity induced by cognitive tasks . . . . . . . . . . . . . . . . . . . . . 7

2.2 Limitations of current BCIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.2.1 Error-related potentials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.3 Literature review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.3.1 Active BCI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.3.2 ErrP-based error correction in BCI . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.3.3 BCI brain-switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3 Exploiting error-related potentials in cognitive task-based BCI 11

3.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.3 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.3.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.3.2 Instrumental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.3.3 Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.3.4 Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.3.5 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.4.1 Error Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.4.2 Tasks Classification Accuracies: Pre- and Post-Error Correction . . . . . . . . . . . 20

3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.5.1 Presence of ErrP in a cognitive task-based BCI paradigm . . . . . . . . . . . . . . 21

3.5.2 The morphology of the ErrP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.5.3 Value of ErrP-based error correction . . . . . . . . . . . . . . . . . . . . . . . . . . 24

iv

3.5.4 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4 Online detection of error-related potentials in multi-class cognitive task-based BCIs 26

4.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.3 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28


4.3.2 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.3.3 Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.3.4 Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.3.5 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4.3.6 Performance Metric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.5.1 ErrP-based Error Correction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.5.2 Error-related Potentials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

4.5.3 Task Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

4.5.4 Limitations and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

5 Robust Asynchronous Brain-switch using ErrP-based Error Correction 40

5.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

5.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

5.3 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

5.3.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42


5.3.3 Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

5.3.4 Experimental Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

5.3.5 Analysis of Training Session Data . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

5.3.6 Analysis of Test Session Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5.3.7 Performance Assessment Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5.4.1 Offline Training Session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5.4.2 Online Test Session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

5.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

5.5.1 Asynchronous Protocol Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

5.5.2 BCIs as Brain-switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

5.5.3 Adding Error-correction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

5.5.4 ErrP Waveforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

5.5.5 Limitations and Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

6 Conclusion 52

6.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

6.1.1 Investigating ErrPs in more realistic settings and scenarios . . . . . . . . . . . . . 54

6.1.2 Considering collecting more data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

v

6.1.3 Exploring multi-choice asynchronous brain switches and the effect of ErrP-guided

error correction in their performance . . . . . . . . . . . . . . . . . . . . . . . . . . 55

6.1.4 Validating the findings with different age groups as well as potential BCI users . . 55

6.2 Peer-reviewed publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Bibliography 57

vi

List of Tables

3.1 Error-related potential classification results: area under the curve (AUC), true positive

rate (TPR) and false positive rate (FPR). . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.2 Binary task classification accuracies (%) between each mental task and idle state pre- and

post-error correction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.3 ITR (bits/min) values of binary task classifications between each mental task and idle

state pre- and post-error correction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.4 Ternary task classification accuracies (%) between pairs of mental tasks and idle state

pre- and post-error correction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.5 ITR (bits/min) values of ternary task classifications between each mental task and idle

state pre and post ErrP-based error correction. . . . . . . . . . . . . . . . . . . . . . . . . 24

4.1 Tasks selected by participants. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.2 Achieved online accuracy (ACC), true positive rate (TPR), and false positive rate (FPR)

for ErrP classifiers are reported for every three consecutive blocks and the entire session.

The upper limit of the 95% confidence interval of chance (ULC) for every participant is

also reported (using cumulative binomial distribution). . . . . . . . . . . . . . . . . . . . . 32

4.3 Online pre- and post-error correction ternary task classification accuracies for two cogni-

tive tasks and unconstrained rest/idle are reported for every three consecutive blocks and

the entire session. Online information transfer rates (ITRs) are reported as well. . . . . . 33

4.4 An overview of online error detection studies. . . . . . . . . . . . . . . . . . . . . . . . . . 35

5.1 The mental task selected by each participant. . . . . . . . . . . . . . . . . . . . . . . . . . 43

5.2 Leave-one-out cross validation results of the intention classifier (task vs. idle) for each

participant: true positive rate (TPR), true negative rate (TNR), and accuracy (ACC).

The cognitive task was defined as the positive class while idle was defined as the negative

class. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

5.3 Leave-one-out cross validation results of the ErrP classifier (error-present vs. error-

absent): true positive rate (TPR), true negative rate (TNR), and accuracy (ACC). The

presence of error was defined as the positive class while the absence of error was considered

the negative class. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

5.4 Online performance of the asynchronous BCI: pre- and post-error correction success rate

(SR), hit rate (HR), and positive predictive value (PPV). . . . . . . . . . . . . . . . . . . 48

vii

List of Figures

2.1 The BCI cycle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3.1 EEG channels (shaded) considered in this study. Channel location and nomenclature were

determined according to the 10-10 international system. . . . . . . . . . . . . . . . . . . . 14

3.2 The user interface. Each icon corresponded to one task. The icon in the middle with no

text represented idle state. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.3 Trial timing and user interface. (i) The required task was highlighted for 5 s. In this

example, the target is the leftmost icon. (ii) Participant performed the required task for

5 s. The EEG signals recorded during this time interval were used for task classification.

(iii) Simulated feedback was displayed for 2 s. Data from this period of time were analyzed

for the presence of ErrP. (iv) Participant rested for 3 s and prepared for the next trial. . . 16

3.4 Average event-related potentials for correct (dotted line) and error (dashed line) trials for

participant #10 at AFz, Fz and FCz. The solid line is the difference between correct and

error responses. Note that in error trials, the second peak is larger and occurs later (630

ms) than in correct trials (600 ms). Time 0 marks the beginning of the visual feedback. . 19

3.5 The top row depicts the grand average ERPs across all participants for correct (dotted line)

and error (dashed line) trials along with their difference (error minus correct; solid line) at

AFz, Fz and FCz. The bottom row shows the grand average of the difference waveforms

along with the corresponding standard error (shaded region) across participants. Time 0

marks the beginning of the visual feedback. . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.6 The topographic maps of the average of the (a) correct trials, (b) error trials, and (c)

their difference for participant #10 at 330, 400, and 700 ms from feedback onset. . . . . . 21

3.7 Boxplots of binary classification accuracies across participants pre- and post-error correction. 23

3.8 Boxplots of ternary classification accuracies across participants pre- and post-error cor-

rection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

viii

4.1 Trial interface: (a) The required task was highlighted for 4 s by changing the text font

color from gray to red. (b) The color of the icon’s bow-tie changed to magenta to indicate

that the participant should start the task. The participant performed the required task

for 5 s. Then, the color of the bow-tie changed back to gray. The 5 s of EEG data

were used for task classification. Then, real-time feedback was displayed. At this point,

depending on the output of the classifier, the timing followed either (c) or (d). (c) In

the first case, the task classifier output was correct, so the icon representing the required

task remained unchanged while the other two icons fell. (d) In the second case, the task

classifier output was incorrect as the icon representing the required task fell. EEG data

after feedback onset (from time 0 to 1.4 s) was analyzed to detect error. (e,g) No error was

detected so the participant was notified with the following message: “No Error!”. (f,h)

Error was detected so the BCI changed its decision. The participant was notified with

following message: “Error! BCI changed its decision to x”, where x refers to the output

of the binary task classifier and can be “task 1”, “task2”, or “rest”. “task 1” referred to

the task written on the left icon and “task 2” referred to the task written on the right icon. 37

4.2 Pipeline for training the online task classifier for the nth online block, n = 1, . . . , 9. . . . . 38

4.3 Pipeline for tuning the online ErrP classifier for the nth online block, n = 1, . . . , 9. . . . . 38

4.4 The grand average of responses across all participants for correct (red line) and error (blue

line) trials along with their corresponding standard error (shaded region) at FCz. Time

0 marks the beginning of the visual feedback. . . . . . . . . . . . . . . . . . . . . . . . . . 39

4.5 Boxplots of online ternary task classification accuracies across participants pre- and post-

error correction. Post-error correction accuracies for blocks 1-9 were improved significantly

compared to pre-error correction values (p=0.002, Wilcoxon Signed-Rank test). . . . . . . 39

5.1 The interface during the training session. The top panel depicts a no-error trial (the icon

falls upon task performance) where as the bottom panel is an error trial (the icon falls

although the user remains idle). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

5.2 The timing of trials in the training session. The idle and task interval times, tidle and

ttask, are drawn from discrete uniform distributions, U(a, b). . . . . . . . . . . . . . . . . . 45

5.3 The grand average of ErrP waveforms across all participants for error and no-error trials

along with their corresponding error bars (shaded region). Time 0 marks the beginning

of the visual feedback. The vertical dashed red lines demarcate the segment which was

used for classification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

ix

Abbreviations

ECOG Electrocorticography

EEG Electroencephalography

ERP Event-related potential

ErrP Error-related potential

fMRI Functional magnetic resonance imaging

fNIRS Functional near-infrared spectroscopy

MEG Magnetoencephalography

SSEP Steady-state evoked potential

SSVEP Steady-state visual evoked potential

x

Chapter 1

Introduction

1.1 Motivation

Individuals with certain neuromuscular disorders such as amyotrophic lateral sclerosis, brain stem stroke,

and spastic quadriplegic cerebral palsy have limited or no means of communication. Most assistive

technologies developed for these individuals rely on some form of voluntary muscle control. However,

patients who present as locked-in do not have any reliable voluntary motion. One alternative means of

communication for these patients is a brain-computer interface since they retain cognitive capabilities

[4, 172].

Brain-computer interfaces (BCIs) provide a communication channel between humans and external

devices by directly using brain signals, without the involvement of muscles. BCIs translate a user’s

intention into command signals on a computer to control a communication device. A BCI cycle starts

with measuring modulations of the user’s brain activities. There are a variety of unique signal patterns

that can be measured and utilized to control a communication device [171, 164, 115, 4]. There are various

measurement modalities to collect changes in brain signals such as functional near-infrared spectroscopy

(fNIRS) [166, 114], functional magnetic resonance imaging (fMRI) [170], and electroencephalography

(EEG) [107]. EEG is the most common modality used in BCI, given that the associated instrumentation

is inexpensive and portable and provides millisecond temporal resolution. In this thesis, EEG modality

was used.

EEG-BCIs can be categorized into three types of active, reactive, and passive in terms of the mental

activities they use for control [180]. Active and reactive BCIs detect user’s intention for control while

passive BCIs typically measure mental states of the brain such as fatigue or stress [113]. Reactive

BCIs require an external stimulus to generate specific brain signals that are shown to be detectable,

such as P300 and steady-state visual evoked potential (SSVEP) [6]. In active BCIs, users intentionally

modulate their brain activity by performing specific tasks that have been shown to produce detectable

signal patterns, such as motor imagery (MI) [173, 127], imagined speech [149], or mental arithmetic [32].

In this study, we used non-MI active tasks, and the reason was threefold. First, active BCIs are

needless of constantly attending external cues, which makes them more practical for most asynchronous

paradigms. Secondly, it has been shown that motor imagery and motor execution involve similar brain

regions [106]. This makes use of these tasks for some asynchronous systems, such as brain switches,

not practical as voluntary movements can interfere with the control of the system. We note that MI

1

Chapter 1. Introduction 2

tasks are viable and intuitive choices for some applications of BCI, such as controlling prosthetic arms

or legs [110, 111]. Third, it has been shown that “BCI illiteracy”, inability to control MI-based BCIs, is

present in 15 ∼ 30% of population [12]. Cognitive tasks can be investigated as a potential replacement

for this group. Besides, MI can be hard or impossible to perform by individuals with congenital motor

impairments [23, 24, 22]. Nonetheless, the performances of most asynchronous BCIs developed to this

date, regardless of their activation tasks, are not sufficient for daily use adoption.

For EEG-BCIs to become widely adopted among severely disabled individuals, certain challenges

must be addressed. First, BCI’s performance must be improved. The performance of BCIs is not ideal,

and they require training and recalibration prior to and during operation. One possible way to improve

performance while remaining training time is to introduce automatic error correction. It is possible to

design an EEG-BCI that can detect its mistakes and change its decision accordingly [142]. Besides,

such BCIs can learn from their mistakes and consequently approach their optimal performance [17, 145].

One error correction mechanism for EEG-BCIs is based on the error-related potential (ErrP), which is

produced when the user realizes that the choice made by the BCI is incorrect [41]. Our lab and others

have shown that ErrP can be detected in EEG signals with a reliable accuracy [18, 182].

Another shortcoming of most EEG-BCIs developed to date is that they operate in a synchronous

mode (paced by an external cue). In such systems, users perform mental tasks and hence make selections

according to the pace dictated by the BCI. A solution for this problem is to use asynchronous (self-paced)

BCIs, which can be activated whenever the user performs a specific mental task and hence, does not

require an external cue. However, this type of BCI is more difficult to implement [98]. For instance, an

activation mental task should be very specific so that no other brain activity produces similar patterns

(and hence falsely activates the BCI), while at the same time, the mental task should not be difficult to

perform so that a user in urgent need (to activate the BCI and communicate) can still perform it. In such

a paradigm, detecting mistakes (such as accidental activations of the BCI) can help the performance of

asynchronous BCIs and increase their practicality.

1.2 Research questions and objectives

The overall objective of this thesis was to develop a reliable asynchronous EEG-based brain-computer

interface based on non-motor active tasks by incorporating error-related potentials. The purpose was

to advance one step toward a more practical means of communication for the main target population

of BCI research, namely, individuals who present as locked-in. To achieve the goal of this research, the

following questions were investigated:

I. Which non-motor cognitive tasks are more discriminant against unconstrained rest (also known

as idle state)? This information will be important for BCIs designed to work in an asynchronous

mode.

II. Is single-trial classification of ErrPs feasible in a multi-class, non-motor active EEG-BCI?

III. How does the introduction of ErrP-based autocorrection impact the accuracy of a multi-class,

active EEG-BCI?

IV. How accurately can non-motor cognitive tasks be detected in an asynchronous BCI?

Chapter 1. Introduction 3

V. How does the introduction of ErrP-based autocorrection impact the accuracy of an asynchronous

EEG-BCI?

To answer these questions, the following objectives were defined:

I. We built a multi-class active EEG-BCI based on non-motor cognitive tasks. The tasks included

unconstrained rest to determine which cognitive tasks combination provided most discriminant

information against unconstrained rest.

II. The presence of ErrPs was investigated in a multi-class non-motor active EEG-BCI, and the

feasibility of single-trial ErrP classification was explored. The theoretical added value of ErrP-

based error correction to the multi-class EEG BCI, if any, was computed.

III. To determine the benefits of incorporating ErrP-based error correction, a multi-class active EEG-

BCI was designed, and online performance of the BCI was compared pre- and post-error correction.

IV. An asynchronous EEG-BCI based on non-motor active tasks was developed.

V. ErrP-based error correction was added to an asynchronous EEG-BCI and the performance was

measured pre- and post-error correction.

1.3 Outline of the thesis

Following this introductory chapter, chapter 2 presents a brief review of different types of BCIs, EEG

modality, and background literature on asynchronous BCIs and BCIs with ErrP-based error correction.

To address the above objectives, three studies were completed during the course of this thesis.

The first study was designed to answer the objectives I and II. Results of the first study, which

focused on the offline classification of EEG during non-motor cognitive tasks and detection of ErrPs,

are presented in chapter 3. This chapter is reproduced from the following published article: Yousefi R,

Sereshkeh AR, Chau T. “Exploiting error-related potentials in cognitive task based BCI”. Biomedical

Physics and Engineering Express. 2018 Dec 20;5(1):015023.

The second study, designed to answer objective III, is detailed in chapter 4. This study focused on

the online classification of cognitive tasks in a ternary task EEG-BCI and classification of ErrPs following

provided real-time feedback. This chapter is reproduced from the following published article: Yousefi R,

Sereshkeh AR, Chau T. “Online detection of error-related potentials in multi-class cognitive task-based

BCIs”. Brain-Computer Interfaces (2019): 1-12.

Chapter 5 presents the results of the third study and concentrates on objectives IV and V, which

was to explore the potential benefits of adding ErrP-based error correction to a cognitive task-based

asynchronous BCI in an online paradigm. This chapter is reproduced from the following article that

has been submitted for publication: Yousefi R, Sereshkeh AR, Chau T. “Development of a robust

asynchronous brain-switch using ErrP-based error correction”. Journal of Neural Engineering, Under

Review.

Finally, the main contributions of this research and suggested directions for future works are sum-

marized in chapter 6.

Chapter 2

Background

2.1 Brain-computer interfaces

A brain-computer interface is a system that can identify certain brain activity patterns. A measurement

modality is used to record the user’s brain activity and transmits the data to a computer. The computer

interprets the data and generates a corresponding command to control external devices such as moving a

robot’s hand, controlling a wheelchair or guiding Pacman through a maze. In more details, a BCI system

comprises five consecutive stages as shown in 2.1: (i) signal acquisition, (ii) preprocessing including

digitization and filtering to remove artifacts and noise, (iii) data analysis including feature extraction

and classification, (iv) the control interface executing the commands, and (v) the feedback obtained by

the user about the success or failure of his/her effort to control the system.

Figure 2.1: The BCI cycle.

In the pre-processing stage, it is essential to identify data contaminated with artifacts as muscular

movement. Contaminated data can be either discarded or artifacts removal techniques can be applied

4

Chapter 2. Background 5

to remove artifact components [40]. In the feature extraction step, discriminant features are identified.

These features can be from different domains such as time, wavelet, or frequency, depending on the

data. Besides, as the number of data in BCI is typically low, the number of features must also be of

a low dimension. Common feature reduction techniques such fast correlation-based filter solution [177]

and sequential floating forward selection [132] are sometimes used to reduce the number of features.

Classification methods, including support vector machine (SVM), linear discriminant analysis (LDA),

and neural network (NN) are among the most commonly used techniques in BCI [95, 94]. Finally, the

control interface stage translates the results of data classification into control commands such as moving

a cursor or spelling a word.

2.1.1 EEG as BCI Modality

There are various signal acquisition modalities used in BCI [134]. Selecting which modality to use

depends on the application. The most common noninvasive modality is EEG [107], which measures elec-

trical voltage from the surface of the scalp. Another technique that measures the electrical signal from

the neural activity is electrocorticography (ECoG) [52]. This modality is invasive and requires surgery

to implant electrodes on the cortex. However, compared to EEG, ECoG is less sensitive to artifacts

and provides a higher signal to noise ratio (SNR). Another non-invasive technique called magnetoen-

cephalography (MEG) measures magnetic fields induced by electrical field fluctuations of neural activity

over time [101]. Compared to EEG, MEG has a higher spatial resolution and is less sensitive to noise but

is more expensive and not portable. fNIRS [166, 114] and fMRI [170] are two non-invasive techniques

which measure hemodynamic response in the active brain area. Active neurons demand more glucose

and oxygen compared to inactive neurons, which result in an increase in oxyhemoglobin concentration.

An fNIRS system emits near-infrared light from the surface of the scalp to the brain and measures the

reflected light. The attenuation level of the reflected light is affected by fluctuations in oxyhemoglobin

and deoxyhemoglobin concentrations of the outer cortical layer. Hemodynamic responses are delayed in

nature, which limits their applications. In addition, fMRI is bulky and expensive and has a low temporal

resolution. So, overall, fMRI and MEG are not viable modalities for daily BCI use by individuals with

disabilities. In this thesis, we employed EEG for data collection during the experiments.

Brain electrical signals arise from the flow of ions in neurons when transmitting information. Cortical

pyramidal neurons contribute significantly to the generation of the EEG. During signals transmission,

neurons become electrical dipoles. The signals may be excitatory where net voltage increases or inhibitory

where net voltage decreases [133].

Hans Berger, a German psychiatrist, created an electroencephalogram for the first time by attaching

electrodes to the scalp to measure electrical activity in human brains in 1924. In general, an electroen-

cephalogram consists of a cap placed around the head with a number of electrodes to measure EEG.

The measured EEG signal of each electrode is the differential voltage between that electrode and ground

(GND) electrode minus the voltage measured from the reference (REF) electrode.

EEG Electrodes can be categorized into two major types based on their development technology.

The first type called ‘wet’ electrodes as they require a conductive gel to be applied between them and

the skin to lower the impedance to the range of 10 kΩ. However, applying gel can take a long time,

and the signal quality drops as the gel dries out by time. The other type of electrodes are called ‘dry’,

and they have the advantage of being gel-free and easier to montage, but they are more expensive and

provide less signal to noise ratio [55].


At each moment, measured brain EEG signals originate from numerous simultaneous activities.

However, certain activities have signature signal patterns which can be detected and used for BCI

control.

2.1.2 BCI types

BCIs can be categorized into three types, namely, active, reactive, or passive, according to the mental

activities they employ to interpret the user’s state of mind and intention [180].

Active

Active BCIs, also referred to as endogenous BCIs, require the user to consciously perform mental tasks

such as motor imagery (MI), word generation, and mental arithmetic. Users are able to control the BCI

by performing these tasks as each task is typically associated with a certain command.

Active tasks deployed in BCIs are either motor-related (e.g. motor imagery [126], kinesthetic at-

tention [100], readiness potentials [181]) or cognitive (e.g. word generation [32], music imagery [54],

imagined speech [149]). Motor imagery (MI) is the most common mental task for the development

of active BCIs [43, 161, 139, 75, 178, 109, 167, 99, 173]. However, for individuals with congenital or

long-term motor impairments, MI can be impossible or difficult to perform [22, 23, 24]. It has also

been shown that approximately 15-30% of BCI users (even without any motor impairments) are unable

to control an MI BCI [12]. An alternative to be explored for these individuals is to use non-MI tasks

[32, 34, 33, 29, 149, 148, 100] which we will refer to as cognitive tasks.

Passive

Passive BCIs do not require an external stimulus or intentional brain activity by the user. Passive

BCIs simply monitor spontaneously occurring background brain activity, which therefore can provide

information on the user’s mental state such as fatigue, attention, or frustration [112, 113].

Reactive

Reactive BCIs, also referred to as exogenous BCIs, involve using external stimuli to elicit specific brain

responses. In other words, users of this type of BCIs do not control their brainwaves consciously but

instead react to external cues. Two common reactive signals in BCI include P300 and steady-state

evoked potentials (SSEPs) [6]. P300 occurs in response to an infrequent stimulus, and SSEPs occur

when the stimulus is oscillating at a specific frequency. The stimulus for generating P300 and SSEPs

can be visual [183, 174, 187], auditory [85, 46, 116], somatosensory [14, 15], or a combination of them

[175, 86].

Actives BCIs may require a lengthy period of user training before achieving proficiency, especially

for naiive BCI users [121] but reactive BCIs require minimal training. Active BCIs also typically do

not achieve high accuracy when the number of classes exceeds two and the number of training sessions

is small. Hence, active BCIs typically have a lower information transfer rate compared to reactive

BCIs. However, in reactive BCIs, the user must continuously attend to external stimuli, which may

induce fatigue [115] and reduces the practicality of systems for certain applications. One way to exploit

the advantages of both types (higher ITR of reactive BCIs and being needless of constantly attending

external cues in active BCIs) is to combine them [128]. For example, a brain-switch (an asynchronous


BCI designed to detect one brain pattern) can be developed based on an active task to turn on a fast

reactive BCI (such as a p300 speller).

2.1.3 Cue-based versus self-paced BCIs

Most existing BCIs are cue-based, also known as synchronous, which means users’ brain activity is

analyzed in specific time intervals. In other words, the BCI dictates to the users by external cues when

they can interact with the system. This makes analyzing the users’ data and determining their intentions

less challenging as activation time intervals are pre-determined.

Self-paced or asynchronous BCIs offer a more natural mode of communication, allowing users to

control the timing of the BCI. In an asynchronous BCI, brain signals need to be continuously analyzed

to isolate the brain activities associated with intentional control (IC) from any other ongoing brain

activity. From numerous simultaneous functions happening in our brain to non-stationary nature of

these activities result in many elusive challenges complicating the development of asynchronous systems.

This means the BCI must be able to distinguish the IC periods from every other non-control (NC)

mental states. The wide variety of NC mental states are typically referred to as idle state.

2.1.4 Functional activity induced by cognitive tasks

The human brain is a distributed processing system. Most functional brain imaging studies have focused

on finding specific brain regions involved in cognitive tasks. But a newer perspective suggests that brain

functions must be studied in a network paradigm which suggests any cognitive task activates different

brain regions which are connected as a network [103].

Mental 3D figure rotation - It has been confirmed that parietal areas are involved in mental

rotation of visual images including superior parietal cortex (predominantly on the right side), posterior

parietal cortex (which is part of spatial attention network), transverse occipital sulcus, intraparietal

sulcus (IPs) and adjacent regions [179, 160, 158].

Mental math - Studies on brain regions involved in arithmetic tasks have demonstrated activities in

prefrontal and parietal cortex. Specifically, the angular gyrus of the inferior parietal cortex and inferior

prefrontal cortex are activated during the mental numerical calculation. They also have shown that

lesions to prefrontal, frontoparietal, and thalamus may lead to impaired arithmetic reasoning [135, 102,

28].

Mental counting - Ordered verbal concepts, such as months, weekdays, and numbers activate

intraparietal sulcus while other non-ordered verbal do not [70, 152]. In [70], Ischebeck et al. showed that

IPS activation was similar for both numbers and months, which means that the IPS activation might

not be modulated by the quantity information carried by numbers.

Mental phonemic fluency - We refer to phonemic fluency task as word generation. Many studies

have investigated various types of fluency tasks. Phenomic fluency tasks are shown to activate left frontal

lobe and left temporal lobe, specifically left inferior frontal gyrus and insula, left rolandic operculum and

the left middle frontal gyrus [67, 11].

Unconstrained rest - It has been shown that the brain is always active even during resting state.

The active brain regions during unconstrained rest or mind-wandering form a network called default

mode network (DMN). It was called the default mode since it was assumed to become deactivated

during tasks. However, recent studies show that DMN plays a role in some other certain tasks such as


thinking about past and future [53].

2.2 Limitations of current BCIs

There are still several challenges that need to be addressed before widespread adoption of BCIs. First,

BCIs must be development sufficiently user-friendly, so non-BCI experts can set them up easily and

quickly. EEG is portable and relatively inexpensive compared to other modalities. As technology

advances and dry wireless EEG electrodes are becoming more available, EEG-BCIs are getting closer to

being practical for daily usage. In this thesis, we used dry EEG system to move one step forward toward

practicality.

Secondly, the user must be able to control the BCI whenever they intend to, which means a practical

BCI must work in an asynchronous mode.

Third, brain signals are nonstationary, and thus change over time, which can be the result of changes

in mental strategy and mental state. This makes training a reliable BCI more challenging. One partial

solution is to collect a sufficient amount of data on different days to train the BCI and continue updating

it.

Forth, BCIs must be reliable in understanding the user’s intentions. However, the performance level

of most BCIs developed to this date is not sufficient for wide adoption. Much effort has been devoted

to improving the performance of BCIs. Recent improvements in the quality of recorded EEG signals

(e.g., signal-to-noise ratio) [55] and classification algorithms [95, 94] have enhanced BCI performance,

but further improvements are required. Another approach is to detect the BCI’s errors and correct them.

Brain reacts when someone commits an error [37] or observes an error by others [165] or a computer [41]

and generates certain signal patterns called error-related potentials (ErrPs). The feasibility of detecting

these signals and using them to enhance BCI’s performance has been proven [18].

2.2.1 Error-related potentials

The typical success rate of a BCI in recognizing user’s intentions is less than 100%. Error-related

potentials, brain signals elicited when someone perceives that a mistake is made, can be used to correct

BCI’s behavior. Falkenstein et al. and Gehring et al. are among the first researchers to investigate the

presence of ErrPs in the early 1990s [37, 48]. There are several variants of ErrPs based on the source of

the error, including response ErrP, feedback ErrP, observation ErrP, and, interaction ErrP [42].

Response ErrPs occur when a person commits an error during a rapid ipsative or memory task

[122]. Feedback ErrPs are generated when the user makes a mistake in a reinforcement learning task

and may be unaware of the mistake unless she/he is explicitly informed of the error using feedback

[59]. Observation ErrPs occur when an individual is monitoring the performance of another person or

a computer and observes a mistake being made [165]. It has been shown that another type of ErrPs,

called interaction ErrPs, are elicited when a BCI makes an error in interpreting the user’s intention, and

this error is perceived by the user [41].

The feasibility of single-trial detection of ErrPs with reliable accuracy has been previously shown in

many studies [18, 9, 43, 185]. Schalk et al. first demonstrated that error-related potentials could be

detected in a motor imagery BCI [142]. The detected ErrPs have been used to improve BCI performance

via two mechanisms: (i) error correction, where detected mistakes trigger a change to the output [13, 93,

137]; and (ii) classifier adaptation, where the BCI classifier is gradually updated based on the detection


of ErrPs [182, 8]. These mechanisms have been investigated in the context of MI-based active BCIs

[43, 83, 8] and reactive BCIs that harness evoked potentials such as the P300 [26, 20, 159] and SSVEPs

[155, 183].

Different types of ErrP waveforms have some characteristics in common. First, the error-related

negativity (ERN) appears over the frontocentral cortex. It has been demonstrated that ERN can be

detected with the same performance after several months without recalibration, suggesting stability of

ERN over time [42]. ERN is most likely generated in the anterior cingulate cortex (ACC) [163] with

a peak latency varying between 50ms to 250 ms from the error onset [18]. Iturrate et al. investigated

ErrPs in three tasks with different difficulty levels [71]. They suggested that some features of ErrP

waveforms such as ERN latency varies for the different tasks while the amplitude and overall waveform

shape remain similar. ERN is followed by a centroparietal positive peak, which has been shown to be

modulated by error awareness [64].

2.3 Literature review

The general purpose of this research was to develop a robust asynchronous EEG-BCI by employing

error-related potentials. Previous studies that have investigated active BCIs developed asynchronous

brain-switches or adopted error-related potentials to improve BCI performance are reviewed here.

2.3.1 Active BCI

Non-motor imagery active tasks have received little attention in BCI. Since the first step of this research

is to develop a multi-class BCI based on active tasks, relevant publications are reviewed here. In one

of the first reports on active tasks, published in 1990, Keirn and Aunon collected EEG data during five

different active mental tasks and were able to classify the tasks pairwise with a minimum accuracy of

85% [76, 77]. These five tasks included complex math problem solving, geometric figure rotation, mental

letter composing, visual counting, and baseline measurement. The power and asymmetry ratio from

four frequency bands (delta, theta, alpha, and beta) were extracted from the EEG data, and a Bayes

quadratic model was used for classification.

Over the next two decades, several other studies investigated various types of feature extraction and

classification methods on the EEG data collected by Keirn and Aunon [57, 119, 7, 120]. To name a

few, Palaniappan et al. applied a fuzzy ARTMAP neural network on power spectral density (PSD) and

autoregressive (AR) features of the EEG data to choose the three best mental tasks from five available

ones [120]. They showed that these best triplets of mental tasks are different for each subject because

each subject had a different way of performing each mental task. Also, an improvement of about 5%

was reported in some of the binary classification problems [120].

In another effort by Palaniappan, spectral power and asymmetry ratio features from the gamma band

(24-37 Hz) were extracted and added to the features from the lower frequency bands which resulted in

improved accuracies compared to the original obtained ones [119]. Different types of support vector

machines (SVM) have also been investigated on the same dataset, which led to improvements in some

of the classification problems [89, 57]. It should be noted that some of the attempts in investigating

new techniques on this dataset led to lower accuracies compared to previous work [7]. In conclusion,

developing a reliable BCI using non-motor imagery tasks appears feasible, given an appropriate signal

processing pipeline.


Some of the other non-motor imagery active mental tasks which have been used for BCI include

navigation imagery [32], auditory recall [32], phone imagery [32], and word generation [91]. Also, some

studies have investigated more intuitive mental tasks such as speech imagery of vowels [27], syllables [35]

or complete words [168] and reported higher than chance average accuracies.

2.3.2 ErrP-based error correction in BCI

Schalk et al. first demonstrated that error-related potentials are generated when an error is made by a

machine in a BCI context [142]. Feasibility of single-trial detection of error-related potentials is shown

in many studies. It has been suggested that ErrPs can be used to improve BCI performance in two ways:

(i) to correct the erroneous decisions made by BCI where the BCI reverts the command before being

fully executed, (ii) to learn from the mistakes by recalibrating classifying algorithms [13, 93, 137]. So far,

most studies have investigated ErrP as a corrective signal in either motor imagery [8, 9, 43, 83] or P300

based BCIs [26, 20, 159, 155, 183]. The results of these studies demonstrated that BCI performance

was improved by incorporating ErrP. One essential note is that both sensitivity and specificity of ErrP

detection and the ensuing corrective mechanism are important. Even with a high detection accuracy of

ErrP, information transfer rate might decrease.

2.3.3 BCI brain-switches

Throughout the past decade, asynchronous systems have received increasing attention among the BCI

community. The most common modality for developing asynchronous BCIs is EEG. The first study on

asynchronous EEG-BCI was performed by Mason and Birch in 2000 [98]. They designed a brain-switch

based on the finger movements. A brain-switch is designed to detect one brain pattern and is used as

a binary switch. They achieved an average area under the curve (AUC) of 79 %. Most asynchronous

studies have explored MI tasks [66, 161, 87, 16, 88, 139, 125, 49]. Some of these studies demonstrated

the feasibility of asynchronous MI-related BCIs in motor-impaired patients [69, 125, 88]. Several studies

have focused on using SSVEP [31, 128] and P300 signals [129, 90, 144] to develop an asynchronous BCI.

There are also several papers that have developed asynchronous BCIs using a combination of MI and

non-MI tasks [32, 25, 34, 47, 33, 105, 29]. These studies showed that cognitive tasks such as 3D object

rotation, mathematical operations, word generation, spatial navigation, and auditory recall could be

classified from idle state with acceptable accuracies.

Chapter 3

Exploiting error-related potentials in

cognitive task-based BCI

This chapter is reproduced from the following published article: Rozhin Yousefi, Alborz Rezazadeh

Sereshkeh, and Tom Chau. ”Exploiting error-related potentials in cognitive task based BCI. “Biomedical

Physics & Engineering Express 5.1 (2018): 015023. Hence, there is some material that is repeated from

Chapter 1 and 2 (e.g. parts of the literature review). The final published article can be found in the

following link: https://iopscience.iop.org/article/10.1088/2057-1976/aaee99/meta.

3.1 Abstract

Brain-computer interfaces (BCIs) can make mistakes in recognizing user intention. It has been well-

established that for reactive and motor imagery (MI) BCIs, an error-related potential (ErrP) occurs

subsequent to such mistakes, and can be used to improve BCI performance. However, the presence

of ErrPs in active BCIs based on non-MI cognitive tasks has not been confirmed. In this study, we

attempted to elicit ErrPs in a BCI based on non-MI mental tasks. Twelve typically developed young

adults participated in two sessions each. Participants performed multiple iterations of five different

mental tasks (mental arithmetic, counting, word generation, figure rotation and idle state) to “knock

down” one of 4 targets on a graphical interface (each mental task was associated with a different target).

To simulate errors, a random subset of 20% of the trials were followed by incorrect feedback (i.e., the

wrong target fell). Our findings confirmed the presence of an interaction ErrP, with a negative peak

at ∼180 ms, followed by two positive peaks, respectively, at ∼400 and ∼630 ms post-feedback onset.

The classification of mental tasks and error versus non-error trials were both performed using a pseudo-

online paradigm where the last quarter of trials were used for testing. For binary (task and idle) and

ternary (2 tasks and idle) classification, across-participant average accuracies of 76%±12 and 63%±12,

respectively, were attained. An average area under curve (AUC) of 0.83 was reached across participants

for the detection of ErrPs. After applying ErrP-based error correction, the average binary and ternary

classification accuracies of mental tasks improved by 9% and 14%, respectively. Our findings support

the addition of ErrP detection and ErrP-informed correction to maximize the accuracy of BCIs based

on cognitive tasks.

11

Chapter 3. Exploiting error-related potentials in cognitive task-based BCI 12

3.2 Introduction

Individuals with conditions such as amyotrophic lateral sclerosis, and those living with the effects of

brain stem stroke, may have limited or no means of communication. Most assistive technologies rely

on extant voluntary movements, which are elusive in those who present as locked-in. One alternative

means of communication for these individuals is a brain-computer interface (BCI). BCIs provide a

communication channel between the brain and external devices, circumventing the need for muscular

activity. Electroencephalography (EEG) is the most common modality used in BCIs; the associated

instrumentation is generally portable, and offers sub-second temporal resolution.

Much effort has been devoted to improving the performance of BCIs [138]. Some BCIs employ

a verification step to reduce error rates, but the communication rate is reduced as a consequence. An

alternative approach is to detect errors using certain brain signals called error-related potentials (ErrPs).

Based on the source of the error, a response, feedback, observation, or interaction ErrP may be

observed. Response ErrPs occur when a person commits an error during a rapid ipsative or memory

task. Feedback ErrPs are generated when the user makes a mistake in a reinforcement learning task

and may be unaware of the mistake unless she/he is explicitly informed of the error using feedback.

Observation ErrPs occur when an individual is monitoring the performance of another person or a

computer and observes a mistake being made. It has been shown that another type of ErrPs, called

interaction ErrPs, are elicited when a BCI makes an error in interpreting the user’s intention and this

error is perceived by the user [42].

The feasibility of single-trial detection of ErrPs with reliable accuracy has been previously shown

in many studies [18, 9, 43, 185]. Schalk et al. first demonstrated that error-related potentials can be

detected in a motor imagery (MI) BCI [142]. The detected ErrPs have been used to improve BCI

performance via two mechanisms: (i) error correction, where detected mistakes trigger a change to the

output [13, 93, 137]; and (ii) classifier adaptation, where the BCI classifier is gradually updated based

on the detection of ErrPs [182, 8]. These mechanisms have been investigated in the context of MI-based

active BCIs [43, 83, 8] and reactive BCIs that harness evoked potentials such as the P300 [26, 20, 159]

and steady-state visually evoked potential [155, 183].

Although reactive BCIs often have higher information transfer rates than active BCIs, the user

must continuously attend to external stimuli, which may induce fatigue [115]. In active BCIs, the user

consciously performs a task such as MI, word generation or mental arithmetic to control the BCI. Some

individuals with congenital or long-term motor impairments may have difficulty eliciting a significant

response in the motor cortex [23, 24, 22]. In addition, a lengthy period of user training may be required

before achieving proficiency with a motor-based BCI [121]. An alternative is to use non-MI cognitive

tasks [32, 34, 33, 29, 149, 148].

Excluding BCIs controlled by actual movements [78, 42], the study of ErrP in active BCIs has been

limited to those of the motor imagery variety. Ferrez and Millan achieved an average detection rate

of correct versus error trials of 84.7% and 78.8%, for two participants, respectively. By incorporating

ErrP-based error correction, the BCI error rate was reduced from 30% to 7% [43]. Kreilinger et al.

showed that combining ErrP and MI improved classification accuracies by 11% on average in three

participants [83] and confirmed the presence of ErrPs during a continuous feedback paradigm [82]. In a

recent study, Bhattacharyya et al. reported ErrP classification accuracies of 80% or higher and showed

that the inclusion of ErrP-based error correction led to higher performance in controlling a robotic arm

[10]. Collectively, these studies confirm the value of ErrP-driven error correction in active, MI-based


BCIs.

It has been shown that many factors affect the morphology of ErrP waveforms such as the user’s

level of engagement during the task [60], as well as the cognitive workload of the task and the feedback

paradigm [71]. Schmidt et al. measured ErrPs in a P300-based center speller and a calibration speller

(controlled via key press) while the interface and the feedback were kept identical [145]. They found

that although the ErrPs for both cases exhibited similarities in terms of the general shape, latencies and

amplitudes of error-related negativity and positivity were notably different. As the impact of variations

in BCI mental tasks on ErrPs has not been adequately evaluated, it cannot be assumed that a single

classifier can be trained to reliably detect ErrPs elicited in different mental task and feedback paradigms.

The main goal of this study was to investigate the possibility of using a participant-dependent

classifier for single-trial detection of error potentials subsequent to five different non-MI cognitive tasks.

To the best of our knowledge, this is the first study to investigate whether machine-discernible ErrP is

elicited subsequent to errors committed by a non-MI cognitive task-driven BCI. We also explored the

impact of incorporating ErrP-based error correction on BCI performance using accuracy and information

transfer rate (ITR) [171]. Finally, it should be noted that the measurements of this study were all

performed using dry EEG electrodes.

3.3 Methods

3.3.1 Participants

This study was approved by the research ethics boards of Holland Bloorview Kids Rehabilitation Hospital

and the University of Toronto. Twelve typically developed adults (6 male) with normal or corrected-to-

normal vision were recruited for this study (mean age: 28 ± 2 years). Participants attended two sessions

on separate days. They were seated in front of a computer screen about 70 cm away and were asked to

relax and refrain from excessive blinking or moving during mental tasks periods.

3.3.2 Instrumental Setup

Brain activity was recorded using a 32 channel dry EEG system (actiCAP Xpress Twist, Brain Products,

Germany). The locations of the electrodes are shown in figure 3.1. Reference and ground electrodes

were placed on the left and the right earlobe, respectively. The sampling rate for data acquisition was

set to 1000 Hz.

3.3.3 Tasks

The following five mental tasks were chosen for this study: (I) Backward counting from 10 at a constant

pace; (II) Mental arithmetic - iterative mental subtraction of a single-digit number from a 2-digit number

(both self-selected), e.g. 94-7=87, 87-7=80, 80-7=73, etc.; (III) Word generation - thinking of as many

words as possible that begin with a certain letter (self-selected prior to the trial); (IV) Figure rotation -

visualizing the rotation of a particular three-dimensional object (self-selected) around an axis; and, (V)

Idle state - rest while allowing normal thought processes to occur and refraining from performing any of

the defined mental tasks.


Figure 3.1: EEG channels (shaded) considered in this study. Channel location and nomenclature weredetermined according to the 10-10 international system.

3.3.4 Protocol

Each participant completed two sessions on separate days. Each session lasted approximately 90 minutes

including setup time. In each session, participants completed 180 trials (36 trials per task). During

each trial, five icons, each corresponding to one task, were displayed on the monitor (figure 3.2). At

the beginning of each trial, the colour of the target icon changed from white to yellow (figure 4.1(i)),

indicating the task the participant was to perform. The sequence of targets was pseudo-randomized.

Participants had 5 s to prepare themselves for the upcoming task (e.g. choose the two numbers for

the mental arithmetic task, choose the letter for the word generation task, or choose the 3D object for

the figure rotation task). Then, participants were visually cued (the bowtie of the target icon changed

from white to magenta in colour) to perform the mental task (figure 4.1(ii)) for 5 s. Afterward, visual

feedback (all icons except one fell down) was displayed for two seconds (figure 4.1(iii)). In correct trials,

the remaining icon was that corresponding to the mental task performed. Participants were told that the

feedback was the real-time result of classifying their recorded EEG signals whereas in fact, the feedback

was randomly generated to be erroneous in 20% of the trials [83, 17, 8]. This contrived mismatch

between the completed task and the feedback was deliberately created to evoke error-related potentials.

Pseudo-randomization of targets and the randomization of feedback mitigated the potential of any order

effects [42].

3.3.5 Data Analysis

Preprocessing

The EEG data from the two sessions were combined for each participant. The data were treated using

a 1 to 40 Hz bandpass finite impulse response (FIR) filter which is the recommended bandwidth for this

dry EEG system [51]. The data were then partitioned into a training set and a test set. The first 75%

of trials (all trials of session I and the first half of session II) were placed in the training set and the rest

in the test set to emulate an online evaluation (i.e. pseudo-online).


Figure 3.2: The user interface. Each icon corresponded to one task. The icon in the middle with no textrepresented idle state.

Ocular and blink artefacts were removed by applying independent component analysis (ICA) and

the ADJUST algorithm [108]. Only the training subset of EEG data (i.e., data from session I and the

first half of session II) was used for calculating ICA weights.

Error-related Potential Detection

1.3 s of EEG signal post-feedback onset was epoched for ErrP detection. The data were filtered using

an FIR bandpass filter with a passband from 1 to 12 Hz (in addition to the aforementioned filtering and

artifact removal). This bandwidth has been shown to capture ErrPs [29]. The first 100 ms was averaged

and subtracted from the signal to remove the baseline offset. The signal was then downsampled by a

factor of 20. The downsampled signal was used as the feature vector. Other common feature extraction

techniques, including power spectral density, the Yule-Walker autoregressive (AR) model, and discrete

wavelet transform (DWT) features were investigated. However, using the downsampled signal as a

feature vector resulted in the highest cross-validation (CV) performance on the training set.

As error potentials were elicited in only 20% of the trials, the data for training the error detector

was imbalanced between classes (i.e., there were more trials without error potentials). Hence, instead of

accuracy, the area under the curve (AUC) of the receiver operating characteristic (ROC) was selected

as the evaluation metric for the ErrP classifier.

Various algorithms including Support Vector Machine (SVM), Linear Discriminant Algorithm (LDA),

and regularized LDA (rLDA) classifiers were tested on the training set. We also explored different

feature selection techniques (namely, Fast Correlation-Based Filter [177] and Sequential Forward Floating

Selection [132] with the training set. Regularized LDA without feature selection yielded the highest

average CV classification AUC on the training set. The regularization parameter (γ) was optimized

for each participant by maximizing the average leave-one-out cross-validation (LOOCV) AUC on the

training set for different values of γ over the range of [0, 1.0] in 0.1 increments. Using the optimal γ, an

rLDA classifier was trained. All the reported metrics were computed on the test set.

Task Classification

All possible binary (each task and idle state) and ternary (each pair of tasks and idle state) combinations

of tasks were considered. An asynchronous BCI must allow users to freely mind wander when not


Figure 3.3: Trial timing and user interface. (i) The required task was highlighted for 5 s. In thisexample, the target is the leftmost icon. (ii) Participant performed the required task for 5 s. The EEGsignals recorded during this time interval were used for task classification. (iii) Simulated feedback wasdisplayed for 2 s. Data from this period of time were analyzed for the presence of ErrP. (iv) Participantrested for 3 s and prepared for the next trial.

intending to control the BCI. Hence, all classification problems included an idle state. Any task that

can be differentiated from an idle state may be a candidate control task for an asynchronous BCI for a

given participant.

The 5 s task period of each trial was epoched. Spectral power values were calculated from 1 to

40 Hz in 1 Hz increments, which resulted in 40 features per EEG channel. Other common types of

EEG features including DWT and AR features were investigated on the training set. However, features

generated using spectral power values yielded the best average CV performance on the training set.

Various feature selection and classification algorithms were tested on the training set and again,

rLDA resulted in the highest mean CV accuracy (averaged across participants and all different task

combinations). The regularization parameter (γ) was optimized for each participant using LOOCV on

the training set. Using the selected γ, an rLDA classifier was trained. All the reported metrics were

calculated on the test set.

Theoretical Improvement of the BCI System

The goal of detecting ErrPs was to correct wrong decisions made by the BCI. Improvement in BCI

performance depends on the sensitivity and specificity of ErrP detection. In this section, we explain how

we calculated the expected improvement in the task classifier after endowing it with ErrP-based error

correction.

A binary task classifier, without any error correction, can be represented by the following confusion


matrix:Predicted

Task1 Task2

ActualTask1 C11 C12

Task2 C21 C22

(3.1)

where the matrix entries are counts of the classifier’s predicted label (Task1 or Task2) for each trial.

Since C11 + C22 is the total number of trials with the correctly predicted labels, the accuracy of this

classifier (ACCpre) can be calculated as:

ACCpre =C11 + C22

C11 + C12 + C21 + C22(3.2)

The error detection classifier can also be represented by a confusion matrix:

Predicted

Error No Error

ActualError TPe FNe

No Error FPe TNe

(3.3)

where the matrix entries are the counts of true positive (TPe), false positive (FPe), true negative (TNe),

and false negative (FNe) error detections.

For binary task classification with error correction, trials that are classified incorrectly (C12 + C21)

using the task classifier (corresponding to the count of actual errors in the confusion matrix for the

error detector, i.e., TPe + FNe), will be corrected if the decision of the error detector is correct (TPe).

However, some of the trials that are classified correctly using the task classifier (TNe + FPe) will

be rendered incorrect if the error detector mistakenly flags an error (FPe). Hence, the new binary

classification accuracy after error correction can be calculated as follows:

ACCpost =C12 + C21∑2i=1

∑2j=1 Cij

× TPeTPe + FNe

+C11 + C22∑2i=1

∑2j=1 Cij

× TNeTNe + FPe

(3.4)

A ternary task classifier, without any error correction, can be represented by the following confusion

matrix:

Predicted

Task1 Task2 Task3

Actual

Task1 C11 C12 C13

Task2 C21 C22 C23

Task3 C31 C32 C33

(3.5)

The ternary classification accuracy is calculated as follows:

ACCpre =

∑3i=1 Cii∑3

i=1

∑3j=1 Cij

(3.6)

After the addition of error correction, the error classifier may correctly detect the presence of ErrP

(TPe) following the trials that were wrongly classified using the task classifier (TPe + FNe). However,

in the ternary case, there are two possible ways to revise each identified misclassification. Hence, only


some of the detected misclassifications will be changed to the correct class. We estimated this fraction

using the elements of the task classifier confusion matrix in Equation (3.5).

To elucidate, consider an example. C12 represents the number of Task1 trials which are classified as

Task2 by the ternary task classifier. If the error classifier correctly detects error following any of C12

trials in an online paradigm, the classifier decision can be changed to the class with the second highest

probability score. An estimation of the conditional probability that the correct class (i.e., Task1) will be

predicted given that Task2 is not the correct class, is given by C11

C11+C13. The same logic can be applied to

all the other instances of incorrect classifications (i.e., Cij , i 6= j). Hence, the new ternary classification

accuracy after error correction can be calculated as follows:

ACCpost =(∑3

i=1

∑3j=1,j 6=i

∑3k=1,k 6=i,j Cij ×

Cii

Cii+Cik∑3i=1

∑3j=1 Cij

)× TPeTPe + FNe

+

∑3i=1 Cii∑3

i=1

∑3j=1 Cij

× TNeTNe + FPe

(3.7)

Performance Metrics

To evaluate the task classifier, accuracy and information transfer rate were computed. Information

transfer rate, sometimes referred to as bit-rate, is a common performance metric in BCI [171]. It takes

both accuracy and time into account. However, it is a theoretical metric derived from mutual information

which does not depend on the BCI implementation [26]. The following formula was used to calculate

ITR in terms of bits per minutes.

ITR =60

T× (log2N + P log2 P + (1− P ) log2

1− PN − 1

) (3.8)

where T represents the time required to communicate one command in seconds, N is the number of

classes and P is the classification accuracy in the range of [0,1].

To evaluate the error detector, we computed area under the curve, true positive rate and false

positive rate. These metrics are conventionally invoked in signal detection and have been applied in the

characterization of error potential detectors [183, 185, 184].

3.4 Results

3.4.1 Error Detection

Table 4.2 shows the area under the ROC curve (AUC) for each participant for the test set. An average

AUC of 0.83 was achieved across all participants. Table 4.2 also presents the true positive rate (TPR)

and false positive rate (FPR) achieved for each participant on their respective test set.

To find these optimum values for TPR and FPR for each participant, first, an ROC curve was plotted

using the training set for three hundred different discrimination thresholds. To generate each point, TPR

and FPR values were calculated by applying LOOCV on the training set. Then, post-error correction

accuracies (i.e. after applying ErrP-based error correction on all possible binary LOOCV classification

accuracies of task vs. idle on the training set) were computed using equation 3.4 for all points on the

ROC. The discrimination threshold which maximized the mean cross-validated binary task classification


Table 3.1: Error-related potential classification results: area under the curve (AUC), true positive rate(TPR) and false positive rate (FPR).

Participant # AUC TPR FPR

1 0.81 0.33 02 0.95 0.84 0.063 0.72 0.40 0.084 0.84 0.68 0.115 0.82 0.72 0.106 0.79 0.75 0.117 0.95 0.95 0.158 0.92 0.89 0.149 0.85 0.78 0.1210 0.83 0.61 0.1211 0.72 0.61 0.0812 0.78 0.78 0.11

AVG±STD 0.83 ±0.08

accuracy (averaged over all pairwise accuracies) was chosen as optimum on a per participant basis. In

the event of a tie, the discrimination threshold with the lower FPR was selected. The reported TPR and

FPR values in table 4.2 were calculated over the test set using the selected discrimination threshold.

Figure 3.4: Average event-related potentials for correct (dotted line) and error (dashed line) trials forparticipant #10 at AFz, Fz and FCz. The solid line is the difference between correct and error responses.Note that in error trials, the second peak is larger and occurs later (630 ms) than in correct trials (600ms). Time 0 marks the beginning of the visual feedback.

The average event-related potentials (ERPs) are demonstrated in figure 3.4 for correct trials, error

trials and their difference for one participant, at three different recording locations of AFz, Fz, and FCz.

A small deflection and a positive peak are present at about 180 and 400 ms after feedback onset in both

correct and error trials. There is also a second positive peak in both correct and error trials. However,

the amplitude and latency of this second peak are noticeably greater in error trials. The design of the

experimental protocol, particularly the method of feedback, is known to affect the morphology of ERPs

[73, 185, 175]. The difference waveform (error minus correct) consists of a small negativity at ∼ 330 ms

followed by a large positivity at ∼ 700 ms.

The first row of figure 3.5 depicts the grand average ERPs across all participants for correct and error

trials along with their difference at AFZ, Fz, and FCz. The bottom row of figure 3.5 shows the grand

average of the difference waveforms along with the corresponding standard error across participants. As


Figure 3.5: The top row depicts the grand average ERPs across all participants for correct (dotted line)and error (dashed line) trials along with their difference (error minus correct; solid line) at AFz, Fz andFCz. The bottom row shows the grand average of the difference waveforms along with the correspondingstandard error (shaded region) across participants. Time 0 marks the beginning of the visual feedback.

depicted, the first 800 ms of the difference waveform was relatively consistent among participants.

Figure 3.6 portrays the topographic maps of averaged error and correct trials along with their differ-

ence for one participant at 330, 400, and 700 ms from feedback onset. These were the time points where

the most prominent peaks appeared in the difference waveforms. From these maps, we can see that the

positivity at 400 ms occurs in the fronto-central region while the subsequent positivity appears over the

centro-parietal region.

3.4.2 Tasks Classification Accuracies: Pre- and Post-Error Correction

Classification accuracies for the binary classifiers (i.e. each cognitive task vs. idle) are reported in

table 3.2 for all participants. Post-error correction accuracies for each participant were calculated using

equation 3.4. The upper limit of the 99% confidence interval of the corresponding chance estimates was

64% using the binomial cumulative distribution [21]. In seven cases, as marked with † in the table,

pre-error correction accuracies did not surpass the chance level while all post-error correction accuracies

exceeded chance. There were six cases where applying error correction reduced accuracy as marked with

‡ in the table. When averaged across participants, post-error correction accuracies for two out of four

pairs of tasks increased compared to the pre-error correction accuracies (p<0.05, Wilcoxon Signed-Rank

test) as depicted in figure 3.7. For further assessment of the ErrP-based error correction, ITRs were

computed as explained in section 3.3.5. The time (T) required for completion of a command without

error-correction was computed as follows: 5 s (for performing the task)+ 0.5 s (time between end of the


Figure 3.6: The topographic maps of the average of the (a) correct trials, (b) error trials, and (c) theirdifference for participant #10 at 330, 400, and 700 ms from feedback onset.

task and feedback onset) + 0.5 s (time for completion of feedback). Error-correction added 1.3 s to the

aforementioned time. The pre- and post-error correction values of ITR for all the binary tasks classifiers

are provided in table 3.3.

Table 4.3 summarizes the results for ternary (each pair of cognitive tasks and idle state) classification

before and after error correction. Post-error correction accuracies were calculated using equation 3.7.

There were two cases, as marked with † in the table, where the pre-error correction accuracies did

not surpass the chance level of 44% (p=0.01 using cumulative binomial distribution [21]), while all

post-correction accuracies exceeded chance (table 4.3). There were also two cases where the addition

of error correction led to a decrease in accuracy (marked with ‡ in the table). As shown in figure 3.8,

when averaged across participants, ErrP-based error correction increased ternary classification accuracies

(p<0.05, Wilcoxon Signed-Rank test). For all the ternary tasks classifiers, the pre- and post-error

correction values of ITR are provided in table 3.5.

3.5 Discussion

3.5.1 Presence of ErrP in a cognitive task-based BCI paradigm

Our findings confirm the presence of an ErrP following erroneous feedback in an active BCI based on

non-MI cognitive tasks, and that the ErrP can be detected on a single-trial basis. After combining data


Table 3.2: Binary task classification accuracies (%) between each mental task and idle state pre- andpost-error correction.

Participant Arithmetic Counting Word Gen. Figure Rot.# Pre Post Pre Post Pre Post Pre Post

1 79 86 63† 75 88 92 87 912 97 94‡ 87 93 86 93 92 933 67 75 53† 68 87 85 57† 704 64 82 70 83 64 81 69 825 64 84 56† 82 78 86 75 866 68 85 77 86 79 86 72 857 72 88 70 88 75 88 78 878 97 86‡ 72 87 96 86‡ 81 879 70 85 56† 84 86 87 75 8610 97 87‡ 78 82 80 83 95 87‡11 56† 78 61† 80 72 83 72 8312 83 87 72 86 75 86 94 88†

AVG 76 85 68 83 81 86 79 85STD 14 5 10 6 9 3 11 6

p< 0.05∗ No Yes No Yes† Accuracies which did not exceed the chance level.‡ The post-error correction accuracies which were lower than theircorresponding pre-error correction accuracies.∗ p-values were corrected with Holm-Bonferroni method.

Table 3.3: ITR (bits/min) values of binary task classifications between each mental task and idle statepre- and post-error correction.

Participant Arithmetic Counting Word Gen. Figure Rot.# Pre Post Pre Post Pre Post Pre Post1 2.59 3.42 0.49 1.55 4.71 4.91 4.43 4.632 8.06 5.53 4.43 5.21 4.16 5.21 5.98 5.213 0.85 1.55 0.03 0.79 4.43 3.21 0.14 0.984 0.57 2.63 1.19 2.81 0.57 2.45 1.07 2.635 0.57 3.01 0.10 2.63 2.40 3.42 1.89 3.426 0.96 3.21 2.22 3.42 2.59 3.42 1.45 3.217 1.45 3.87 1.19 3.87 1.89 3.87 2.40 3.648 8.06 3.42 1.45 3.64 7.58 3.42 2.99 3.649 1.19 3.21 0.10 3.01 4.16 3.64 1.89 3.4210 8.06 3.64 2.40 2.63 2.78 2.81 7.14 3.6411 0.10 1.97 0.35 2.29 1.45 2.81 1.45 2.8112 3.42 3.64 1.45 3.42 1.89 3.42 6.73 3.87

AVG 2.99 3.26 1.28 2.94 3.22 3.55 3.13 3.43STD 3.19 1.00 1.28 1.13 1.89 0.81 2.36 1.04

p< 0.05∗ No Yes No No∗ p-values were corrected with Holm-Bonferroni method.

from both sessions for each participant, an average AUC of 0.83 was achieved for ErrP detection which

is comparable to that reported in existing literature. Zeyl and Chau reported an average AUC of 0.81

in a visual P300 speller [185]. An accuracy of 80% was achieved for ErrP detection by Bhattacharyya

et la. in a motor imagery BCI in one study [9] and 93% in another [10]. They attributed their high

accuracy in [10] to protracted training times of at least 60 hours.

Also noteworthy is the fact that error potentials were detected with dry electrodes in this study.

While literature has cited better signal-to-noise ratio with wet rather than dry electrodes [19], the latter

requires less preparation time and arguably offers better comfort to users due to the lack of gel. The ErrP

detection accuracies with the dry electrodes were comparable to findings reported in the aforementioned

studies, all of which deployed gel-based measurements [185, 9, 10].


Figure 3.7: Boxplots of binary classification accuracies across participants pre- and post-error correction.

Table 3.4: Ternary task classification accuracies (%) between pairs of mental tasks andidle state pre- and post-error correction.

Arithmetic Arithmetic Arithmetic Counting Counting Word Gen.Participant + + + + + +

# Counting Word Gen. Figure Rot. Word Gen. Figure Rot. Figure Rot.Pre Post Pre Post Pre Post Pre Post Pre Post Pre Post

1 72 80 67 76 81 86 57 69 69 74 75 782 72 88 74 90 85 91 74 89 80 90 90 933 47 69 49 68 53 69 44 62 45 64 56 914 52 69 41† 55 46 64 47 70 56 72 50 685 52 69 56 73 57 72 54 72 59 77 63 776 57 71 59 77 61 78 62 79 60 76 62 757 59 83 54 74 63 82 56 78 63 83 63 808 69 86 89 86‡ 81 86 72 86 61 76 78 829 57 71 69 80 61 74 67 79 50 69 67 7810 70 80 83 83 87 84‡ 69 77 70 74 81 8211 45 71 54 73 48 72 51 73 43† 71 48 7212 65 78 69 76 83 78 61 74 74 76 76 81

AVG 60 76 64 76 67 78 60 76 61 76 67 78STD 10 7 14 9 15 8 10 7 11 7 13 7

p< 0.05∗ Yes Yes Yes Yes Yes Yes† Accuracies which did not exceed the chance level.‡ The post-error correction accuracies which were lower than their corresponding pre-error correctionaccuracies.∗ p-values were corrected with Holm-Bonferroni method.

3.5.2 The morphology of the ErrP

The overall morphology of the ErrPs was similar to that of observation and interaction error potentials

reported by [71, 72] and [118]. The morphological resemblance to observation error potentials may

be due the fact that, in this study, feedback was shown to the participants 0.5 s after they stopped

performing the required tasks, and hence had greater opportunity to“monitor” the computer’s response.

Literature has reported that these two types of ErrPs share very similar morphologies [78, 118]. There

were however departures from literature in terms of latencies, which are expected given the different

experimental protocols used to elicit error responses [72]. The latencies in this study were longer than

most of those reported in literature, which is likely attributable to the gradual falling of the icon over


Table 3.5: ITR (bits/min) values of ternary task classifications between each mental taskand idle state pre and post ErrP-based error correction.

Arithmetic Arithmetic Arithmetic Counting Counting Word Gen.Participant + + + + + +

# Counting Word Gen. Figure Rot. Word Gen. Figure Rot. Figure Rot.Pre Post Pre Post Pre Post Pre Post Pre Post Pre Post

1 4.50 5.45 3.40 4.52 6.93 7.07 1.69 3.14 3.82 4.09 5.24 4.972 4.50 7.69 4.98 8.35 8.25 8.70 4.98 8.01 6.63 8.35 10.16 9.443 0.58 3.14 0.75 2.96 1.18 3.14 0.35 2.03 0.42 2.32 1.55 3.504 1.06 3.14 0.18 1.17 0.50 2.32 0.58 3.32 1.55 3.69 0.85 2.965 1.06 3.14 1.55 3.89 1.69 3.69 1.30 3.69 1.98 4.74 2.64 4.746 1.69 3.50 1.98 4.74 2.30 4.97 2.47 5.21 2.14 4.52 2.47 4.307 1.98 6.22 1.30 4.09 2.64 5.96 1.55 4.97 2.64 6.22 2.64 5.458 3.82 7.07 9.75 7.07 6.93 7.07 4.50 7.07 2.30 4.52 6.05 5.969 1.69 3.50 3.82 5.45 2.30 4.09 3.40 5.21 0.85 3.14 3.40 4.7410 4.04 5.45 7.57 6.22 8.98 6.50 3.82 4.74 4.04 4.09 6.93 5.9611 0.42 3.50 1.30 3.89 4.50 3.69 0.95 3.89 0.29 3.50 4.50 3.6912 3.01 4.97 3.82 4.52 7.57 4.97 2.30 4.09 4.98 4.52 5.50 5.70

AVG 2.36 4.73 3.37 4.74 4.48 5.18 2.32 4.61 2.64 4.48 4.33 5.12STD 1.53 1.64 2.90 1.88 3.07 1.91 1.54 1.67 1.92 1.55 2.63 1.67

p< 0.05∗ Yes No No Yes Yes No∗ p-values were corrected with Holm-Bonferroni method.

0.5 s, resulting in a delayed perception of the BCI decision.

The ERP waveforms for error and correct trials generally consisted of an initial negativity around

180 ms followed by two positive peaks, the first prior to 400 ms and a second, broader peak at 600 ms or

shortly thereafter. The initial negative peak was similar in magnitude for both error and correct trials

and could be a component of a motion-onset evoked potential resulting from the initiation of the visual

feedback (i.e., falling icon). Its magnitude and latency is similar to the N2 component of motion-onset

evoked potentials reported elsewhere [56]. The initial negative peak could also be a feedback-related

negativity, which has been previously observed in both correct and error trials in a P300 speller [183].

The first positive peak may be a combination of a movement-evoked positivity and a surprisal re-

sponse, as seen in [74] where the authors elicited simultaneous motion-onset and P300 potentials. The

earlier peak on average in the correct trials may be attributable to the motion of the falling non-target

icons (i.e., correct feedback) occurring in the periphery of the participant’s visual field as peripheral

stimuli are known to elicit faster responses than perifoveal stimuli [58]. The second, wider positive peak

around 600 ms was generally unique to error trials and is a response characteristic previously associ-

ated with observing erroneous feedback [183, 145]. The prominence of this latter positivity over the

centro-parietal region is in line with the finding in a previous ErrP study [184].

3.5.3 Value of ErrP-based error correction

From the classification accuracies reported in tables 3.2 and 4.3, it can be seen that cognitive task perfor-

mance varied across individuals which is consistent with literature [24]. For the majority of participants,

regardless of cognitive task, the introduction of error correction significantly increased binary task classi-

fication accuracy. The instances where accuracy was compromised by error correction were those where

the original binary task classification accuracy was already very high (≥ 95%). In a practical system,

error correction would be invoked judiciously; task classifiers that perform well would typically not be

endowed with error correction. For the ternary case, all but one specific task triplet (arithmetic, word

generation and idle) for one participant (P8) did not benefit from the introduction of error correction.


Figure 3.8: Boxplots of ternary classification accuracies across participants pre- and post-error correction.

This participant-ternary task combination already achieved 89% accuracy prior to error correction. In

sum, the combination of task classifier and ErrP detection improved the accuracies of the binary and

ternary classifier by 9% and 14% on average, respectively. This finding supports the continued integration

of ErrP-based error correction in cognitive task-based BCI systems.

For a comprehensive evaluation of a BCI, both speed and accuracy must be taken into account. ITR

includes both of these factors. The average values of ITR after ErrP-based error correction for the binary

classifiers changed by 0.64 (bits/min). However, this change was only significant for one out of the four

classifiers. For the ternary classifiers, the average ITR increased by 1.56 (bits/min). The improvement

was significant for three out of six classifiers. The authors suggest the use of error correction, even in

cases where accuracy is significantly enhanced but ITR remains unchanged, given the speed-accuracy

trade-off. An accuracy enhancement can potentially reduce the user’s frustration when controlling a

BCI.

3.5.4 Future work

For future studies, the authors suggest implementing the proposed BCI in an online paradigm (with

real-time feedback of the initial task decision, error detection decision, and updated task decision) to

fully assess the performance of cognitive task-based, active BCIs with ErrP-based error correction. Also,

additional sessions (i.e. more trials) may improve both the task classification and error detection. In

this study, the regularization parameter was optimized separately for each participant and for each

classification problem. Additional sessions and extra trials may enhance the generalizability of the

classification models and mitigate the need for such customization. Last but not least, evaluating the

performance of this BCI on individuals who present as locked-in is necessary prior to clinical translation.

Chapter 4

Online detection of error-related

potentials in multi-class cognitive

task-based BCIs


Sereshkeh, and Tom Chau. “Online detection of error-related potentials in multi-class cognitive task-

based BCIs.” Brain-Computer Interfaces (2019): 1-12. Hence, there is some material that is repeated

from Chapter 1 and 2 (e.g. parts of the literature review). The final published article can be found in

the following link: https://www.tandfonline.com/doi/abs/10.1080/2326263X.2019.1614770.

4.1 Abstract

One method for improving the accuracy and hence the rate of communication of a brain-computer in-

terface (BCI) is to automatically correct erroneous classifications by exploiting error-related potentials

(ErrPs). The merit of such a correction scheme has been demonstrated in both active (e.g., motor im-

agery) and reactive (e.g., P300) BCIs. Here, we investigated the effect of ErrP-guided error correction in

a three-class, active BCI based on cognitive rather than motor imagery tasks using electroencephalogra-

phy (EEG). Ten able-bodied adults participated in three sessions of data collection. For each participant,

a ternary BCI differentiated among idle state and two personally selected cognitive tasks (e.g., mental

arithmetic, counting, word generation, and figure rotation). Real-time feedback of the BCI decision was

displayed to the participant following each task. EEG data after feedback onset were used to detect

ErrPs and correct the BCI’s output in the case of detected errors. ErrP-based error correction modestly

but significantly improved the average online task classification accuracy (+7%) as well as the informa-

tion transfer rate (+0.9 bits/min) of the ternary BCI across participants. Our findings support further

study of ErrPs in active BCIs based on cognitive tasks.

26

Chapter 4. Online detection of error-related potentials in multi-class cognitive task-based BCIs27

4.2 Introduction

Brain-computer interfaces (BCIs) can provide a channel of communication for individuals who are not

able to communicate conventionally [4, 172]. Changes in the brain signal patterns during different mental

tasks can be measured by a BCI and translated into meaningful commands or messages. Different

modalities can be used to measure brain activities. Electroencephalography (EEG), which measures the

electrical activity of the brain, is the most common modality for BCI development [115].

BCIs can be categorized based on the type of mental activities invoked [180]. Reactive BCIs use

the brain activity elicited in reaction to an external stimulus. Two common examples of reactive BCIs

are P300 spellers [74, 96] and steady-state visual evoked potential (SSVEP) [187] BCIs. Reactive BCIs

require minimal training and can provide a relatively high information transfer rate (ITR) [115]; however,

the user must continuously attend to external stimuli which can be easily fatiguing.

Active BCIs use self-regulation of brain activity without external stimuli. They require longer training

times compared to reactive BCIs, especially for naıve BCI users, and provide a relatively lower ITR [115].

However, this category of BCI is more conducive to self-paced control where users dictate the timing of

interaction.

Motor imagery (MI) is the most common mental task for the development of active BCIs [167, 99, 173].

However, for individuals with congenital or long-term motor impairments, MI can be impossible or

difficult to perform [22, 23, 24]. It has also been shown that approximately 15-30% of BCI users (even

without any motor impairments) are unable to control an MI BCI [12]. One potential alternative is to

use non MI-based mental tasks, such as mental arithmetic, word generation, and figure rotation, which

we will refer to as cognitive tasks.

One reason for the lower ITR of active compared to reactive BCIs is the low accuracy when the

number of classes exceeds two and the number of training sessions is small. Recent improvements in

the quality of recorded EEG signals (e.g., signal-to-noise ratio) [55] and classification algorithms [95, 94]

enhanced BCI performance, but further improvement is required before BCIs become practical and

widely adopted. Using error-related potentials (ErrPs) to detect BCI mistakes has been investigated to

enhance BCI performance [18].

Monitoring or committing of errors elicits discernible electrical signals in medial frontal and central

brain regions [37, 165]. The morphology of these error-related potentials (ErrPs) depends on many

factors such as the experimental protocol [73, 71] and frequency of the error [17]. Errors made by a BCI

are known to evoke ErrPs [142]. Many studies have demonstrated the feasibility of single-trial detection

of BCI-induced ErrPs [18, 82, 184, 183]. Detected ErrPs can be used to correct an erroneous command,

prevent its execution, or to promote error-based learning [18]. ErrP-based error correction has been

explored in different contexts including reactive BCIs (P300 [183] and SSVEP [157]), continuous feedback

[82, 30, 156], human-robot interaction [140, 79] and MI-based active BCIs [43, 83], yielding in most cases,

improved decoding of user intention.

One of the defining criteria of any BCI is providing real-time feedback [123]. The importance of

real-time feedback in BCI training has frequently been cited in the literature [117, 50, 150]. Closing the

human-machine feedback loop can enhance the neural response. For instance, Kaiser et al. showed that

the provision of long-term training with real-time visual feedback to low BCI performers (< 70%) in an

MI-BCI paradigm over ten sessions led to an increase in cortical hemodynamic activity and beta band

activity [75]. It has also been shown that long-term training with feedback can preserve activation in

the sensorimotor cortex of patients with spinal cord injury [36]. Surprisingly, despite the evidence of


the effectiveness of real-time feedback and the extensive number of online reactive or MI-based active

BCIs, there has been a paucity of studies of active BCIs with non-MI tasks (which may be necessary for

individuals with a disinclination to MI). Most of non-MI studies have collected their data in an offline

paradigm. Among the limited non-MI BCI studies, most have exploited the hemodynamic response,

measured by functional near-infrared spectroscopy (fNIRS), and have explored a combination of cognitive

and MI active tasks to control the BCI [146, 104, 65].

In this paper, we present a ternary BCI based on non-MI active tasks, which is capable of correcting

its mistakes by detecting ErrPs. After two offline training sessions with five cognitive tasks (mental

arithmetic, word generation, mental counting, figure rotation and unconstrained rest/idle), participants

were asked to choose their top two preferences (other than idle) for the online session. The variety of

cognitive tasks increased the likelihood of finding tasks for each user which produced distinctive brain

signal patterns. To the best of our knowledge, this is the first report of the integration of a non-MI

active BCI with ErrPs (generated by actual BCI feedback). Also, unlike previous ErrP studies, all the

measurements were made using dry rather than gel-based EEG electrodes. Dry electrodes have lower

signal-to-noise ratio, but require less preparation time, making them an appealing option for practical

use where user time, attention and patience may be limited [19].

4.3 Methods


EEG was recorded using a dry EEG system (actiCAP Xpress Twist, Brain Products, Germany) at 32

different locations according to the 10/10 international system: FP1, FP2, AFz, AF3, AF4, F1, Fz, F2,

F7, F8, FT7, FT8, T7, T8, FC5, FC1, FCz, FC2, FC6, C1, Cz, C2, CP5, CP1, CPz, CP2, CP6, Pz,

P3, P4, POz, and Oz. Reference and ground electrodes were placed on the left and the right earlobes,

respectively. The EEG sampling rate was set to 1000 Hz.

4.3.2 Participants

Ten adults without any known disabilities with normal or corrected-to-normal vision participated in this

study (5 male, mean ± standard deviation age of 29 ± 3 years). All participants were fluent in English.

None of the participants had any reported history of neurological disorders. Participants were asked to

refrain from drinking alcoholic or caffeinated beverages at least 3 hours before each session. The study

was approved by the research ethics boards of the University of Toronto and the Holland Bloorview Kids

Rehabilitation Hospital. All participants provided informed, written consent. Participants attended two

offline sessions and one online session on separate days, each approximately 90 minutes in length. The

time between offline sessions varied from 7 days to a month. The online sessions were collected about

six months after offline sessions.

4.3.3 Tasks

The following popular BCI mental tasks were chosen for this study: (I) Mental arithmetic: Participants

were asked to mentally subtract a single-digit number from a 2-digit number (both self-selected per trial),

and then iterate, each time subtracting the same single-digit number from the previous result (e.g., 94-

7=87, 87-7=80, 80-7=73, etc.); (II) Backward counting: Participants were asked to count backward from


10 at a constant pace; (III) Word generation: Participants were asked to mentally name as many words

as possible that begin with a letter chosen by the participants randomly before the trial started; and, (IV)

Figure rotation: Participants were asked to visualize the rotation of a three-dimensional object (freely

chosen per trial) around an axis. In addition to these tasks, we also considered an idle/unconstrained

rest state where participants allowed normal thought processes to occur but refrained from performing

any of the defined mental tasks.

During the offline sessions, participants performed all the aforementioned tasks. Prior to the online

session, participants chose two of the tasks based on their personal preference, as listed in table 4.1.

Participants were informed of their offline performance but they were free to choose any task combination

among the six possibilities [176]. Interestingly, all participants chose a task combination that was among

their top three in terms of accuracy. Then, a participant-specific ternary BCI was developed based on

the participant’s preferred tasks and idle state, using the data collected during the offline sessions.

Table 4.1: Tasks selected by participants.

Participant # Task 1 Task 2P1 Word generation Figure rotationP2 Mental arithmetic Figure rotationP3 Word generation Figure rotationP4 Backward counting Word generationP5 Mental arithmetic Word generationP6 Word generation Figure rotationP7 Mental arithmetic Figure rotationP8 Mental arithmetic Word generationP9 Mental arithmetic Word generationP10 Mental arithmetic Word generation

4.3.4 Protocol

Offline Sessions

In each offline session, participants completed 36 trials per task (a total of 180 trials). At the beginning

of each trial, participants were cued with the trial type (i.e., one of the mental tasks or idle). After 4

s of trial type presentation, participants were cued to start performing the mental task or idling for 5

s. At the end of the mental task period, participants were shown visual feedback. In 20% of the trials,

evenly distributed across different tasks, the feedback was configured to randomly display the wrong

task, i.e., one that was different from that which the participant performed. Since participants were told

that the feedback was the result of real-time classification of their recorded EEG, the mismatch between

the detected task and the actual mental task they performed (in 20% of trials) was intended to evoke

error-related potentials. The details of the experimental design, as well as the obtained classification

results for offline sessions, were previously reported [176].

Online Session

Online sessions consisted of 135 trials divided into nine blocks of 15 trials. After each block, which

lasted approximately 5 min, participants were able to rest and start the next block when ready. For

each participant, a BCI was built based on three tasks (their two preferred cognitive tasks and the idle

state). The detailed timeline of each trial and the interface designed for this experiment are shown in


figure 4.1.

Each task was displayed with an icon with the name of the corresponding task. At the beginning

of the trial, the participant was notified of the trial type (i.e., one of the two mental tasks or idle) as

the relevant icon was highlighted for 4 s by changing the text font color from gray to red (figure 4.1a).

Then, the participant performed the task for 5 s. The bow-tie color on the icon was changed to magenta

to cue the participant (figure 4.1b). The 5 s of task data were analyzed in real-time as explained in

section 4.3.5. Visual feedback was displayed to inform the participant of the classification result. A

correct classification would be indicated when the icon corresponding to the performed mental activity

remained upright while the others fell (figure 4.1c). If an icon other than the one associated with the

required task fell, it meant task classification was not correct (i.e. wrong feedback). After the feedback

onset, 1.4 s of EEG data were tested for the presence of any error-related signals. If an error was not

detected, a message, “No Error!”, was shown (figures 4.1e and 4.1g). If an error was detected, the

task data were re-classified using a binary classifier based on the two remaining tasks to determine the

second-best choice. The participant was informed of the new BCI output with a message (figures 4.1f

and 4.1h).

There was a score board below the icons to increase the engagement of the participants. They

received one point if the task classification result was correct, but they would lose that point if an error

was detected in the same trial. If the task classification result was wrong, they would not receive any

point. But if an error was detected and the result of the task reclassification was correct, they received

one score. So, to summarize, at the end of a trial, they would receive one point if the end result was

correct. If it was not, they would not get a point. The scores were added up until the end of each block.

4.3.5 Data Analysis

Task Classifier

Prior to an online session, a ternary task classifier and three binary task classifiers (for all possible pairs

of tasks) were trained based on the data from the offline sessions for each participant. 72 examples

per task per participant were available from the offline sessions. Offline task data were filtered using

an FIR filter (order 96) between 1 to 40 Hz. Then, artifacts including eye blinks, ocular movement,

and discontinuities were removed by applying independent component analysis (ICA) and the ADJUST

algorithm [108].

To extract features, power spectral density (PSD) estimates of task data were calculated for all

channels. The area under the PSD in 1 Hz bins from 1 to 40 Hz served as the associated band power

estimate and yielded 40 features per channel. Classifiers were trained using regularized linear discrim-

inant analysis (rLDA). The regularization parameter (γ) was optimized over the range of 0 to 1 in 0.1

increments using a leave-one-out cross-validation (LOOCV) on the data from the offline sessions. During

each participant’s online session, their respective classifier was retrained after every block (15 trials) to

incorporate same day data but γ, being time-consuming to optimize, was kept constant.

At the beginning of each participant’s online session, 3 min of EEG baseline was collected. ICA

weights were computed using this baseline segment. Then, contaminated components with artifacts

were identified using the ADJUST algorithm. During each trial, the 5 s task interval was epoched and

filtered using an FIR filter (order 96) between 1 to 40 Hz. Then, the PSD features described above were

extracted from each trial. Finally, the ternary task classifier was applied and the output was shown to


the participants. Figure 4.2 summarizes the steps of training the task classifier.

ErrP Classifier

Prior to each participant’s online session, an ErrP classifier was trained using the participant’s data from

the offline sessions. There were 72 trials with error potentials and 288 trials without for each participant

as the error rate was fixed at 20%. For each trial, the 1.4 s of EEG signal after feedback onset, referred

to as feedback data, was epoched and filtered using an FIR bandpass filter with a passband from 1 to

12 Hz. ICA and ADJUST algorithm were applied to identify artifact components. 100 ms of feedback

data was averaged and subtracted from the feedback data to remove the baseline offset.

The actual time samples were used as features by downsampling the data by a factor of 20, which

resulted into 70 features per channel. An rLDA was used to train the ErrP classifier. The regularization

parameter (γ) was chosen for each participant by calculating the average LOOCV accuracy for different

values over the range of 0 to 1 in 0.1 increments and selecting the value which yielded the highest

accuracy.

Figure 4.3 outlines the ErrP classifier. Each participant’s ErrP classifier was also retrained after

every block (15 trials) with the same γ. During each participant’s online session, after displaying a

visual feedback of the task classifier output, 1.4 s of the EEG data were analyzed to find the presence

of ErrP. The same preprocessing and feature extraction steps (listed above) were applied, except that

the ICA weights were calculated from the 3 min EEG baseline recorded at the beginning of the online

session. After classifying the the data, if the output score of the classifier was higher than a specified

decision threshold, that trial would count as an error.

A rLDA classifier generates an score between 0 to 1 for each class which indicates the probability of

the input belonging to that class. Then, the score is compared with a decision threshold (default value

of 0.5 in a binary classifier) to predict the class of the input. However, in this study, the default value

did not always yield the best online results for ErrP classification in pilot testing. Hence, this decision

threshold was optimized after every online block. The threshold used for the first block was equal to

1 which means the classifier only detected error if the output score was 1. The reason for choosing

this threshold for the ErrP classifier was to minimize the number of false positives at the beginning of

the online session (when there is still no same-day data to train the classifier) to prevent frustration of

participants. We made this decision following the pilot sessions, where we noticed that without having

same day data, the ErrP classifier was highly susceptible to false positives (i.e., changing the correctly

predicted label of the task classifier to an incorrect label). For completeness, true positives were counted

when an error potential was detected following an erroneous task classification.

As demonstrated in figure 4.3, to optimize the decision threshold prior to block n, LOOCV was

performed for every trial in the previous online blocks. To elaborate, for each of the (n− 1)× 15 online

trials, a classifier was trained using all the available data (offline sessions plus the online data) for the

given participant, except the trial under consideration. Then, the held-out trial was classified using

the trained classifier. Classifications scores for the online trials were computed for different decision

thresholds and the post-error correction accuracy was computed. The decision threshold was varied over

the range of 0 to 1 (300 values evenly spaced between 0 and 1) and the value yielding the maximum post-

error correction accuracy was selected for use in the subsequent block. The threshold value fluctuated

across its full range according to the participant’s most recent performance.


4.3.6 Performance Metric

In addition to accuracy, information transfer rate (ITR) was estimated. ITR is a common BCI perfor-

mance metric as it takes both accuracy and time into account [171]. ITR was calculated in bits per

minutes using the following equation:

ITR =60

T× (log2N + P log2 P + (1− P ) log2

1− PN − 1

) (4.1)

where P represents the classification accuracy in the range of [0,1], T is the time required to communicate

one command in seconds, and N is the number of classes.

In this study, N=3. The time (T) required for completion of a command without error-correction

was calculated as follows: 5 s (for performing the task) + 0.5 s (time between end of the task and

feedback onset) + 0.5 s (time for completion of feedback). EEG data used for error-correction were 1.4

s in duration and started from feedback onset. Since the complete presentation of feedback required 0.5

s, the error-correction phase added 0.9 s (1.4 s - 0.5 s) to the aforementioned time.

4.4 Results

For each participant, the online performance of their ErrP classifier is reported in table 4.2. Accuracies

are reported for the entire session as well as for every three consecutive blocks to reveal potential per-

formance changes in the classifier, which was retrained on a block-by-block basis. An average accuracy

of 0.74 was achieved across all blocks for all participants. The chance level for error detection accura-

cies varied across participants as they had different task classification accuracies. Chance levels were

calculated using the binomial cumulative distribution and are shown in table 4.2.

Table 4.2: Achieved online accuracy (ACC), true positive rate (TPR), and false positive rate (FPR) forErrP classifiers are reported for every three consecutive blocks and the entire session. The upper limitof the 95% confidence interval of chance (ULC) for every participant is also reported (using cumulativebinomial distribution).

Participant Blocks 1-3 Blocks 4-6 Blocks 7-9 Block 1-9ACC TPR FPR ACC TPR FPR ACC TPR FPR ACC ULC TPR FPR

P1 0.78 0.59 0.11 0.89 0.67 0.06 0.89 0.78 0.08 0.85 0.80 0.70 0.08P2 0.58 0.11 0.08 0.87 0.50 0.03 0.87 0.59 0.03 0.78 0.76 0.33 0.04P3 0.51 0.69 0.74 0.56 0.88 0.85 0.51 0.45 0.44 0.53† 0.61 0.69 0.66P4 0.62 0.61 0.36 0.73 0.72 0.25 0.71 0.77 0.37 0.69 0.68 0.69 0.34P5 0.62 0.11 0.00 0.78 0.09 0.00 0.73 0.08 0.00 0.71† 0.75 0.07 0.00P6 0.67 0.40 0.12 0.73 0.25 0.09 0.84 1.00 0.23 0.75 0.72 0.56 0.15P7 0.82 0.78 0.15 0.89 0.63 0.05 0.86 0.89 0.14 0.86 0.80 0.76 0.11P8 0.82 0.67 0.10 1.00 1.00 0.00 0.93 0.92 0.06 0.92 0.78 0.84 0.05P9 0.47 0.35 0.29 0.71 0.91 0.48 0.71 0.96 0.57 0.63† 0.64 0.69 0.47P10 0.71 0.58 0.14 0.64 0.54 0.21 0.62 0.58 0.35 0.66 0.58 0.56 0.24

Mean 0.66 0.49 0.21 0.78 0.62 0.20 0.77 0.70 0.23 0.74 – 0.59 0.21SD 0.12 0.24 0.21 0.13 0.29 0.27 0.13 0.28 0.20 0.12 – 0.23 0.21

† Accuracies which did not exceed the chance level.

Figure 4.4 displays the grand averaged responses of correct and error trials across all participants at

channel FCz. The shaded regions denote the standard error for the averaged trials. Time 0 marks the

beginning of the visual feedback.


The averaged error waveform had a very small deflection at about 300 ms (294 ± 33 ms being the

mean and standard deviation of the latency across trials) after feedback onset. There were also two

positive peaks present at about 500 ms (496 ± 52 ms) and 800 ms (818 ± 90 ms) in both correct and

error trials but their amplitudes differed.

Online ternary task classification accuracies are presented for pre- and post-error correction in table

4.3. The results are shown for every three consecutive blocks (45 trials) as well as for the entire session

for all participants. The pre-correction accuracies changed significantly in blocks 4-6 (p=0.006) and

7-9 (p=0.002) compared to blocks 1-3, according to a 2-sided Wilcoxon Signed-Rank test. The upper

limit of the 95% confidence interval of chance was 44% and 40% for three blocks and the entire session,

respectively. Online accuracies that did not surpass chance are marked with † in table 4.3. Post-error

correction accuracies were increased compared to pre-error correction accuracies except in three cases

which are marked in the table (‡). The addition of ErrP-based error correction improved the average

accuracy from 60% to 67%(p=0.002, Wilcoxon Signed-Rank test). ITR values for the entire online

session for each participant are provided in the last column of table 4.3. The information transfer rate

was also significantly improved from 2.70 to 3.59 (bits/min) (p=0.006, Wilcoxon Signed-Rank test). The

boxplots of pre- and post-error correction accuracies are plotted in figure 4.5 for different groupings of

blocks as well as for the entire session.

Table 4.3: Online pre- and post-error correction ternary task classification accuracies for two cognitivetasks and unconstrained rest/idle are reported for every three consecutive blocks and the entire session.Online information transfer rates (ITRs) are reported as well.

ParticipantAccuracies ITR (bits/min)

Blocks 1-3 Blocks 4-6 Blocks 7-9 Blocks 1-9 Blocks 1-9

Pre-errorco

rrection

accuracies

P1 0.62 0.80 0.80 0.74 4.98P2 0.58 0.78 0.73 0.70 4.04P3 0.42† 0.45 0.51 0.46 0.50P4 0.31† 0.44† 0.42† 0.39† 0.10P5 0.58 0.76 0.73 0.69 3.82P6 0.56 0.73 0.67 0.65 3.01P7 0.60 0.82 0.80 0.74 4.98P8 0.67 0.76 0.73 0.72 4.50P9 0.31† 0.51 0.47 0.43 0.29P10 0.47 0.42† 0.58 0.49 0.75

Mean 0.51 0.65 0.64 0.60 2.70STD 0.13 0.17 0.14 0.14 2.05

Post-errorco

rrection

accuracies

P1 0.73 0.84 0.89 0.82 6.30P2 0.58 0.82 0.87 0.76 4.78P3 0.44† 0.45 0.51 0.47 0.50P4 0.40† 0.53 0.60 0.51 0.83P5 0.62 0.78 0.73 0.71 3.71P6 0.64 0.69‡ 0.78 0.70 3.51P7 0.76 0.86 0.82 0.81 6.03P8 0.76 1.00 0.87 0.88 8.14P9 0.31† 0.51 0.60 0.47 0.50P10 0.62 0.56 0.56‡ 0.58 1.60

Mean 0.59 0.70 0.72 0.67 3.59STD 0.16 0.18 0.14 0.15 2.71

† Accuracies which did not exceed the chance level.‡ The post-error correction accuracies which were lower than their corresponding pre-errorcorrection accuracies.


4.5 Discussion

4.5.1 ErrP-based Error Correction

Several studies have investigated the impact of incorporating ErrP-based error detection in BCIs based

on P300 [9, 26, 20, 155], code-modulated visually evoked potential [157], and motor imagery [9, 43, 82,

83, 142, 10], confirming the feasibility of error detection on a single-trial basis [18]. When an error is

detected following a trial, at least three different strategies can be employed. The first strategy is to

change the BCI output to the second most probable outcome upon error detection [20, 155, 183]. In

this strategy, the post-error correction accuracy is limited by the accuracy of the ErrP classifier. The

second strategy, which has been adopted by most studies, is to discard and repeat the trials detected

as errors [9, 26, 43, 83]. Compared to the first strategy, the discard and repeat approach may lead to

a higher post-error correction accuracy given that classification is less constrained by the performance

of the ErrP classifier. However, trial repetitions prolong the time required to transmit information and

hence may reduce ITR. The third strategy uses ErrP data to update the BCI classifier [18]. For example,

in binary cases, ErrP detection can be used to identify misclassifications and provide new labels for the

data. Then, in a semi-supervised manner, these labeled samples can be used to update the classifier

parameters [93]. In this study, we deployed the first strategy to enhance an online active BCI.

For comparison purposes, studies that performed online error correction in active BCIs [9, 43, 83, 10]

are summarized in table 4.4. All of these studies applied the second error correction strategy (i.e.

discarding the wrong trials) and used motor imagery tasks for BCI development. Ferrez and Millan

showed that applying error correction for two participants reduced the error rate from 32% to 7% which

subsequently tripled the bit rate [43]. Kreilinger et al. demonstrated an improvement in accuracy of 11%

post-error correction averaged over three participants [83] but ITR values were not reported. Given the

modest number of participants in these studies, the significance of these results cannot be ascertained.

Bhattacharyya et al. [9, 10] conducted two studies of ErrP-based error correction reporting higher ErrP-

corrected performance in controlling a robotic arm. However, they did not report pre- and post-correction

accuracies and ITR values, but rather robot targeting metrics to estimate the improvement. Nonetheless,

these studies collectively indicate potential value of ErrP-based error correction in active, MI-based BCIs.

In this study, the average TPR and FPR values of 0.59±0.23 and 0.21±0.21 were achieved for error

detection. These average values are within the range of those reported in existing literature (see table

4.4). Bhattacharyya et al. [10] obtained higher error potential detection, which can be attributed to their

higher BCI task classification accuracy, making errors more rare and hence eliciting stronger and more

easily detectable brain responses. Their high BCI performance was attributable to the extensive user

training time (60-80 hours). The BCI’s task classification error rate inversely relates to the performance

of the ErrP classifier (i.e. higher BCI error rate tends to suppress the ErrP classification accuracy [17]).

Hence, in general, it can be expected that when a BCI performs poorly, the performance of the associated

error detector diminishes as well. This relationship between the performance of the task classifier and

error detector may explain, in part, the below-chance error detection for P3 and P9; their BCI accuracies

were among those of the bottom three participants.


Table 4.4: An overview of online error detection studies.

Study TPR FPR

Proposed study 0.59 0.21Bhattacharryya et al. [9]∗ - -Ferrez & Millan [43] 0.78 0.15Kreilinger et al. [83] 0.53 0.11Bhattacharyya et al. [10] 0.94 0.04∗ Did not report FPR and TPR values.

4.5.2 Error-related Potentials

Although ErrP waveforms reported in literature share some similar characteristics (such as the presence of

a negative and positive peak), there can be many variations as well. The ErrP waveform morphology may

vary among participants [1] and according to feedback design [72]. Even for the same participant and

feedback design, the BCI error rate may cause variations in the characteristics of the ErrP waveforms [17].

Feedback design determines the mental process required to assess the BCI’s performance [1] and

thus impacts ErrP signal morphology. It is argued that the error-related negativity and sometimes the

positivity are affected by the detectability of an error (in other words, the conspicuity of the difference

between correct and error feedback) [38].

As shown in figure 4.4, there was a small initial negative deflection in error trials and two prominent

positive peaks in both correct and error trials. The positivities had greater amplitude in error trials.

The waveform characteristics are similar to observations in some studies [17, 72, 118] while different to

others [183]. For example, two positive peaks observed by this study and [17, 72, 118] somewhat form

one broader peak because of their bandwidth and latency. But, these two peaks are distinct from each

other in [183]. These waveform differences arise from variations across studies in error portrayal by the

interface (i.e. the feedback design) and error comprehension by the user (i.e. the user-specific mental

processing of error occurrence).

The depicted shaded error bars in figure 4.4 arise from both inter- and intra-participant response

variations. The differences may be attributable to the fact that classification error rate was different for

each individual. It has been confirmed that a less frequent error elicits a larger amplitude (c.f. oddball

paradigm) [17]. Even the error rate for each task varied for one participant resulting in a task-specific

differences between error-related potentials. Furthermore, it has been suggested that the error-related

negativity is correlated with a participant’s cognitive modeling of correct and erroneous feedback [141],

which may have also contributed to inter-participant variation in the ErrP waveform.

4.5.3 Task Classification

The cognitive tasks selected for this study are among the most common used in BCIs while the achieved

online ternary accuracies with error correction (table 4.3) are on par with previously reported offline

ternary EEG studies. Articles on active tasks can be categorized according to the mental tasks invoked:

motor imagery [167, 99, 173, 142], cognitive tasks [77, 61, 63], and combinations of MI and cognitive

tasks [186, 57, 119, 5, 25, 68, 32, 34, 33, 29, 39, 104, 65, 44, 45, 143]. At the time of writing, only

two cognitive EEG studies [75, 36] were conducted online but did not report online task classification

accuracies but rather the time required to reach the targets. The remainder of the studies were con-

ducted offline. Keirn et al. [77] collected data from seven participants performing six different cognitive


tasks. Several studies subsequently applied various feature extraction and classification techniques to

this dataset [61, 63, 186, 57, 119, 5], achieving a wide range of task classification accuracies. Notably,

Dyson et al. reported an average offline binary classification accuracy of 68% between an idle state and

a mental task (one of motor imagery, auditory imagery, mental arithmetic or navigation imagery) [34]

while Agarwal et al. [2] achieved an average offline ternary classification accuracy of 69% in discriminat-

ing among left/right MI and word generation. Our task classifier reached comparable ternary accuracies

online.

The significant improvement of task classification accuracies in blocks 4-6 and 7-9 compared to

blocks 1-3 (figure 4.5) supports the practice of supplementing the training set with same-day data as

brain signals are non-stationary and may change from day to day [115].

4.5.4 Limitations and Future Work

To the best of our knowledge, the active BCI developed in this study is the first online ternary BCI based

on cognitive tasks with ErrP-based automatic error correction. Our results are comparable with those

in literature in terms of multi-class task classification and ErrP detection accuracies. Future research

should include more training sessions and continuous updating of the classifier. We also suggest that

the proposed paradigm be investigated in a functional and asynchronous context, such as controlling a

robot or wheelchair, to ascertain its feasibility as a practical interface.


Figure 4.1: Trial interface: (a) The required task was highlighted for 4 s by changing the text font colorfrom gray to red. (b) The color of the icon’s bow-tie changed to magenta to indicate that the participantshould start the task. The participant performed the required task for 5 s. Then, the color of the bow-tiechanged back to gray. The 5 s of EEG data were used for task classification. Then, real-time feedbackwas displayed. At this point, depending on the output of the classifier, the timing followed either (c)or (d). (c) In the first case, the task classifier output was correct, so the icon representing the requiredtask remained unchanged while the other two icons fell. (d) In the second case, the task classifier outputwas incorrect as the icon representing the required task fell. EEG data after feedback onset (from time0 to 1.4 s) was analyzed to detect error. (e,g) No error was detected so the participant was notifiedwith the following message: “No Error!”. (f,h) Error was detected so the BCI changed its decision. Theparticipant was notified with following message: “Error! BCI changed its decision to x”, where x refersto the output of the binary task classifier and can be “task 1”, “task2”, or “rest”. “task 1” referred tothe task written on the left icon and “task 2” referred to the task written on the right icon.


Figure 4.2: Pipeline for training the online task classifier for the nth online block, n = 1, . . . , 9.

Figure 4.3: Pipeline for tuning the online ErrP classifier for the nth online block, n = 1, . . . , 9.


Figure 4.4: The grand average of responses across all participants for correct (red line) and error (blueline) trials along with their corresponding standard error (shaded region) at FCz. Time 0 marks thebeginning of the visual feedback.

Figure 4.5: Boxplots of online ternary task classification accuracies across participants pre- and post-error correction. Post-error correction accuracies for blocks 1-9 were improved significantly compared topre-error correction values (p=0.002, Wilcoxon Signed-Rank test).

Chapter 5

Robust Asynchronous Brain-switch

using ErrP-based Error Correction


Sereshkeh, and Tom Chau. “Development of a Robust Asynchronous Brain-switch using ErrP-based

Error Correction”. Journal of Neural Engineering, Under Review.

5.1 Abstract

The ultimate goal of most researches in brain-computer interface (BCIs) is to provide individuals with

severe motor impairments with a communication channel that they can control whenever they intend

to. Unavoidable step toward reaching this goal is to develop an asynchronous BCI, a system that allows

users to reliably control it in a self-paced manner independently of a cue stimulus. However, due to

technical difficulties, despite many years of research, asynchronous BCIs have been explored in only

a small percentage of BCI studies. Also, the performance of most asynchronous BCIs developed to

this date are not adequate for daily use adoption. In this paper, to address this issue, we presented

an asynchronous BCI using electroencephalography (EEG) and based on non-motor imagery cognitive

tasks and investigated the possibility of improving its performance using error-related potentials (ErrP).

Ten able-bodied adults attended two sessions of data collection each, one for training and one for testing

the proposed BCI. For each participant, an asynchronous BCI differentiated among idle state and a

personally selected cognitive tasks (mental arithmetic, word generation or figure rotation). The BCI

continuously analyzed the EEG data and displayed real-time feedback as soon as it detected a cognitive

task. Then, the BCI analyzed the EEG data after the feedback onset using an ErrP classifier to correct

erroneous classifications. The average of post-error correction success rate across participants improved

significantly from 78 ± 11% to 85 ± 12% compared to the pre-error correction value (p < 0.05). Our

findings support the addition of ErrP-correction to maximize the performance of asynchronous BCIs.

5.2 Introduction

Brain-computer interfaces (BCIs) measure brain activities and translate them to meaningful commands.

The main goal of BCI research is to provide a communication channel for individuals with severe motor

40

Chapter 5. Robust Asynchronous Brain-switch using ErrP-based Error Correction 41

impairments who are not able to communicate through conventional ways [124].

Ideally, it is desirable for a BCI to allow the users to reliably control the system whenever they

intend to and without having to wait for a cue from the system. This type of BCI is called self-paced or

asynchronous [115]. However, due to technical difficulties, most BCIs developed to date are cue-based

or synchronous which means that the system dictates the time of interactions.

In asynchronous BCIs, the challenge is to detect the user’s intention for control without knowing its

onset. Hence, the brain activity must be continuously analyzed to identify intentional control (IC) from

non-control (NC) states [97]. In this paper, we refer to the umbrella of NC states as idle. NC states vary

greatly as there are many simultaneous ongoing processes in the brain. These variations make training a

robust classifier more challenging compared to synchronous setups, which usually include only a limited

number of mental states.

BCIs can be categorized into three types of active, reactive and passive in terms of the mental

activities they use for control [180]. Active and reactive BCIs detect user’s intention for control while

passive BCIs typically measure mental states of the brain such as fatigue or stress [113]. Reactive

BCIs require an external stimulus to generate specific brain signals that are shown to be detectable,

such as P300 and steady-state visual evoked potential (SSVEP) [6]. In active BCIs, users intentionally

modulate their brain activity by performing specific tasks that have been shown to produce detectable

signal patterns, such as motor imagery (MI) [173, 127], imagined speech [149], or mental arithmetic [32].

Reactive BCIs typically have higher information transfer rate (ITR) [171] compared to active BCIs;

however, requiring stimuli for operation limits their functionality. For instance, developing an asyn-

chronous BCI solely based on reactive brain responses such as P300 [129] or SSVEP [31] requires the

constant presence of stimuli, which reduces the practicality of such asynchronous BCIs for everyday use

by the target population. One way to exploit the advantages of both types (higher ITR of reactive BCIs

and being needless of constantly attending external cues in active BCIs) is to combine them [128]. For

example, a brain-switch (an asynchronous BCI designed to detect one brain pattern) can be developed

based on an active task to turn on a fast reactive BCI (such as a p300 speller). In conclusion, in order

to fill the gap between laboratory BCIs and practical BCIs for everyday use, there is a definite need for

accurate asynchronous BCIs without any dependency on external cues.

There have been several efforts in developing asynchronous active BCIs. Most of these studies have

investigated motor related tasks, such as motor execution [66], motor movement intention [69, 92], and

motor imagery [161, 87, 16, 88, 62]. Few asynchronous studies have used non-MI tasks [47, 105, 39] such

as sound imagery [154], conceptual perception [81], and emotion imagery [3].

In this study, we used non-MI active tasks to develop an asynchronous brain-switch, using electroen-

cephalography (EEG). Participants could select one of three mental tasks including word generation,

arithmetic, and figure rotation to control their BCI. The reason that non-MI tasks were chosen was three-

fold. First, non-MI tasks have not thoroughly been studied in the context of asynchronous paradigm.

Secondly, it has been shown that motor imagery and motor execution involve similar brain regions [106].

This makes use of these tasks for some asynchronous systems, such as brain switches, not practical as

voluntary movements can interfere with the control of the system. We note that MI tasks are viable

and intuitive choices for some applications of BCI such as controlling prosthetic arms or legs [110, 111].

Third, it has been shown that “BCI illiteracy”, inability to control MI-based BCIs, is present in 15 ∼30% of population [12]. Cognitive tasks can be investigated as a potential replacement for this group.

Besides, MI can be hard or impossible to perform by individuals with congenital motor impairments


[23, 24, 22]. Nonetheless, the performances of most asynchronous BCIs developed to this date, regardless

of their activation tasks, are not sufficient for daily use adoption.

The problem of improving the performance of BCIs has been approached from various directions such

as improving measurement modalities [115, 164], artifact removal [40], and classification pipeline [95].

Another approach, which has gained many attractions over the last decade, is to detect BCI mistakes

when they occur [41, 83]. It has been shown that the brain reacts when someone commits an error [37]

or observes an error by others or a computer [165]. This reaction produces a specific pattern of signals

which can be reliably detected using EEG [48]. These EEG signals are called error-related potentials

(ErrPs). Feasibility of single-trial detection of ErrPs have been shown in many BCI studies [18], including

synchronous active [43] and reactive BCI [183], but it has been very limited for asynchronous BCIs [10].

Specifically, to the best of our knowledge, it has not been investigated in the context of asynchronous

brain-switches.

The contribution of this study is threefold. First, we investigate the feasibility of a user-specific asyn-

chronous brain-switch that uses EEG signals to continuously detect intentional control events. Secondly,

we explore the possibility of single-trial detection of ErrPs in this asynchronous paradigm. Finally, we in-

vestigate the impact of error-correction on the performance of the developed asynchronous brain-switch.

5.3 Methods

5.3.1 Participants

Ten adults (six females) without any reported history of neurological disorders participated in this

study (mean ± standard deviation age of 29 ± 3 years). All participants were fluent in English with

normal or corrected-to-normal vision. The study was approved by the research ethics boards of the

University of Toronto and the Holland Bloorview Kids Rehabilitation Hospital. All participants provided

informed written consent. Participants attended two sessions of data collection on the same day, each

approximately 50 minutes in length including setup time. Participants were provided a 15 to 20 minute

break between the two sessions.


A 32-channel dry EEG system (actiCAP Xpress Twist, Brain Products, Germany) was used for EEG

measurement at the following locations according to the 10/10 international system: FP1, FP2, AFz,

AF3, AF4, F1, Fz, F2, F7, F8, FT7, FT8, T7, T8, FC5, FC1, FCz, FC2, FC6, C1, Cz, C2, CP5, CP1,

CPz, CP2, CP6, Pz, P3, P4, POz, and Oz. Reference and ground electrodes were placed on the left and

the right earlobes, respectively. The EEG sampling rate was set to 1000 Hz.

5.3.3 Tasks

Three mental tasks, namely, mental arithmetic, word generation and figure rotation were chosen for this

study, based on our previous work [176] that confirmed their separability from an idle state. At the

beginning of each training session, the participant selected one of the tasks based on her/his personal

preference to control the BCI. The selected tasks are listed in table 5.1.

(I) Arithmetic - Participants were asked to mentally subtract a single-digit number from a 2-digit

number (both self-selected), and then iterate, each time subtracting the same single-digit number from


the previous result (e.g., 69-8=61, 61-8=53, 53-8=45, and so on). (II) Word generation - Participants

were asked to mentally name as many words as possible that begin with a certain self-selected letter.

(III) Figure rotation - Participants were asked to visualize the rotation of a particular three-dimensional

object of their choosing (e.g., a chair) around an axis.

Table 5.1: The mental task selected by each participant.

Participant # Task Participant # Task

1 Arithmetic 6 Arithmetic2 Figure Rotation 7 Word Generation3 Figure Rotation 8 Arithmetic4 Arithmetic 9 Figure Rotation5 Figure Rotation 10 Arithmetic

5.3.4 Experimental Protocol

Training Session

The first session of data collection was dedicated to gathering training data. This session included

60 trials. The interface displayed to participants during trials is shown in figure 5.1. During a trial,

participants were instructed to allow normal thought processes to occur until they observed a cue (the

color of the bow-tie on the icon in the middle of the screen changed from gray to magenta as shown in

figure 5.1). They were instructed to start performing their selected mental task after the cue appeared

and continue until the icon fell. The onset of the cue was randomly chosen, between 10 to 16 s after the

beginning of the trial, to simulate an asynchronous setup. The duration of the mental task (from the

onset of the cue until the icon’s fall) was also chosen randomly, between 5 to 12 s as shown in figure 5.2.

These trials will herein be referred to as no-error trials.

We did not perform any real-time analysis during the training session. However, in order to generate

and collect error-related potentials, participants were told that their EEG data were being continuously

analyzed in real-time and whenever the BCI detected the task (even if it was before the onset of the

task cue), the icon in the middle of the screen would fall. During 30% of the trials, chosen randomly, the

icon fell during the idle state, while the color of the bow-tie was still gray 5.1. Since the participant had

not started the task execution, he/she would perceive icon-falling as an error, and hence, ErrPs were

evoked. These trials will be referred to as error trials.

Test Session

After the training data collection session for each participant, an intention classifier (task vs. idle)

and an ErrP classifier (error vs. no error) were developed, as outlined in section 5.3.5. The ensuing

test session included 50 trials. Each trial was a maximum of 30 s long. The interface was similar to

that of the training session, except that participants were not prompted with a cue to begin the task.

Rather, participants could start performing the task whenever they desired. They also had the option

of remaining idle for the entire length of the trial.

The BCI continuously analyzed the EEG data using the intention classifier. To elaborate, after 5 s

from the beginning of the trial, the BCI started analyzing the EEG data from the last 5 s, once every

second. As soon as the BCI detected the task, feedback was presented, i.e., the icon fell. Then, the BCI


Figure 5.1: The interface during the training session. The top panel depicts a no-error trial (the iconfalls upon task performance) where as the bottom panel is an error trial (the icon falls although the userremains idle).

analyzed the EEG data after the feedback onset using the ErrP classifier. If an error was detected, the

icon would return. Whether the BCI detected an error or not, the trial would end at this point.

If the BCI detected no task after 30 s, the trial would end. If a participant decided to perform the

mental task in a trial (rather than remaining idle for the entire length of the trial), he/she was required

to start the task at least 5 s before the end of the trial, i.e., no later than 25 s after the beginning of

the trial. To assist participants with timing, the text (showing the name of the task) on the icon would

disappear at 25 s. Participants were instructed to not start the task after that point as the remaining

time would be less than 5 s, and not sufficient for intention classification.

After the end of each trial, participants reported whether the BCI detected the presence of the task

correctly or not. Incorrect detection include cases where the participant performed the task, but the icon

did not fall, and cases where the icon fell before the participant started the mental task. For trials where

the BCI detected the task and the icon fell (whether it was correct or not), participants also needed to

report if the BCI detected the error correctly or not.

5.3.5 Analysis of Training Session Data

Artifact Removal

First, the raw EEG data were filtered using a finite impulse response (FIR) filter (order 96) between 1 to

40 Hz, and the EEG data for all training trials were extracted. Then, independent components analysis

(ICA) was applied and the ADJUST algorithm [108] was deployed to identify components associated

with eye blinking, ocular movement, and generic discontinuities [108]. After removing these artifacts,

the remaining components were used to reconstruct the EEG data. These ICA weights were also used


Figure 5.2: The timing of trials in the training session. The idle and task interval times, tidle and ttask,are drawn from discrete uniform distributions, U(a, b).

to remove artifacts during the online test session.

Feature Extraction

From each trial, idle and task segments (as described in 5.3.4) were extracted. Note that 30% of the

trials contained only idle segments (i.e., no task). Each segment was further decomposed into 5 s epochs

with 1 s overlap. For example, consider a trial where the user performed the task between 10 and 17 s.

The idle segment would span from 0 to 10 s, consisting of six 5 s, overlapping epochs (0 to 5 s, 1 to 6

s,. . . , 5 to 10 s). Likewise, the task segment would consist of three 5 s, overlapping epochs (10 to 15 s,

11 to 16 s, and 12 to 17 s). This segmentation approach yielded ∼580 epochs of idle and ∼180 epochs

of task per participant.

All task and idle epochs were downsampled by a factor 10. Then, discrete wavelet transform (DWT)

coefficients were calculated using the Daubechies 5 (db5) wavelet with four levels of decomposition.

The root-mean-square (RMS) of the coefficients of each decomposition level were used as features for

classification. These four levels represent the following pseudo-frequency ranges: 40-25 Hz, 25-12.50 Hz,

12.5-6.75 Hz, 6.75-3.38 Hz. A total of 128 DWT features (32 EEG electrodes × 4 DWT features) were

generated from each epoch.

For the ErrP analysis, in all trials, 900 ms of data after the feedback onset were extracted. The

first 200 ms of each epoch was averaged and subtracted from the whole epoch to remove the baseline

offset. Then, the data from 200 to 900 ms were extracted and labeled as error-free or containing error

depending on whether the feedback was correct or wrong. Since, wrong feedback was displayed in 30%

of the trials, for each participant, there were 18 and 42 epochs of error and no-error data, respectively.

The same types of DWT features as described above (for the idle and task epochs) were extracted from


each error and no-error epoch.

Intention Classification

All the task and idle trials of the training session were used to build an intention classifier using a

regularized linear discriminant analysis (rLDA) model. As the classifier was built between the training

and test sessions on the same day, in the interest of time, the regularization parameter (γ) was kept

constant at a value of 0.1.

In general, given an unlabeled input, a binary rLDA classifier calculates a score for each class (where

the scores sum to unity). The score can be considered the probability that the input example belongs to

a given class. The classifier assigns to the unlabeled input, the label of the class with a score greater than

0.5. However, this default threshold is not necessarily optimal in terms of sensitivity and specificity. To

optimize the decision threshold, θ, as a hyperparameter for the intention classifier, leave-one-out cross-

validation (LOOCV) was applied to the training data with a γ of 0.1. Then, classification outcomes of

the intention classifier (subscripted with I) were tabulated as true positive (TPI), false positive (FPI),

true negative (TNI), and false negative (FNI) values for θ ∈ 0.1, 0.2, . . . , 0.9. The threshold, θ∗I which

yielded the highest F-score (equation 5.2) using LOOCV was selected for use in the test session (section

5.3.6).

θ∗I = arg maxθI

F (θI) (5.1)

where

F (θI) =2

TPI(θI) + FPI(θI)

TPI(θI)+TPI(θI) + FNI(θI)

TPI(θI)

(5.2)

Error Potential Classification

Similar to intention classification, rLDA was used for ErrP classification. Since the number of training

examples for the ErrP classification was approximately one order of magnitude lower than that available

for the intention classifier, it was possible, time-wise, to optimize both the regularization parameter γe

and the decision threshold θe, where the subscript denotes the error potential classifier. To determine

optimum values, we followed the procedure below:

• For a given θe, γe ∈ 0.1, 0.2, . . . , 0.9, TPe, FPe, TNe, and TPe were computed via LOOCV with

the ErrP training data. For notational convenience, the explicit dependence of each classification

outcome on θe and γe is not shown.

• The classification outcomes of the combined system (subscripted with C), i.e., the combination of

ErrP and intention classifiers, were tabulated as:

TPC = TPI × TNe

TNe+FPe

TNC = TNI + FPI × TPe

TPe+FNe

FPC = FPI × TPe

TPe+FNe

FNC = FNI + TPI × FPe

FPe+TNe

• F-scores were calculated using the above outcomes.


• The hyperparameter pair (γ∗e and θ∗e) which achieved the highest F-score and the ErrP classifier

using γ∗e and trained with all available ErrP trials were retained for the online test session.

5.3.6 Analysis of Test Session Data

During the test session, EEG signals were continuously analyzed with a window size of 5 s. The window

was moved forward every 1 s. For each 5 s epoch, signals were filtered using the same FIR filter (1 to 40

Hz) as in the training session. Then, the ICA weights calculated in the training session were deployed

to remove artifacts using the ADJUST algorithm. Feature extraction was performed as in the training

phase. Features were classified with the intention classifier developed in the training phase. The classifier

returned the scores for each class. The scores were compared with the decision threshold, θ∗I , optimized

for the intention classifier (as explained above 5.3.5), yielding the classifier’s output (task or idle). If

the epoch was classified as idle, then the trial resumed, and after 1 s, the next 5 s epoch was analyzed.

However, if the epoch was classified as a task, the participant received visual feedback (the icon in the

middle of the screen fell, as described in section 5.3.4).

For error detection, 900 ms of EEG signals post-feedback onset were epoched to determine the

presence of an error. After filtering and removing artifact (as above), baseline offset was removed, and

features were calculated (section 5.3.5). Features were fed to the previously trained ErrP classifier and

scores for the error-present and error-absent classes were compared to the optimized threshold, θ∗e , for

the ErrP classifier. If no error was detected, no action ensued. If an error was detected, the fallen icon

was restored, thereby notifying the participant of the corrective action.

5.3.7 Performance Assessment Metrics

In the training session, the label of every epoch was known. Hence, it was possible to report true positive

rate (TNR) and true negative rate (TNR) on an epoch-by-epoch basis. However, in the online test of

the asynchronous BCI, the exact time of task onset was unknown as participants were free to initiate

activity at will. Hence, it was not possible to label each epoch. Instead, we defined three other metrics to

evaluate the BCI’s performance. These metrics were calculated on trial-by-trial basis rather than at the

level of individual epochs: (i) Success rate - The number of trials classified according to the participant’s

intention divided by the total number of trials. (ii) Hit rate (HR): The number of positive trials, i.e.,

where participants actually performed the mental task, that were correctly identified as positive over

the total number of actual positive trials. (iii) Positive predictive value (PPV): Positive trials that were

correctly identified as positive over the total number of trials predicted as positive.

5.4 Results

5.4.1 Offline Training Session

Table 5.2 lists the LOOCV results of the intention classifier for each participant in terms of true positive

rate (TPR), true negative rate (TNR), and accuracy (ACC) for all participants. The TNR (i.e., detection

of idle) for all participants was high, ranging from 0.92 to 1. In contrast, TPR (i.e., detection of task)

varied greatly, from 0.50 to 0.99 among participants.

Table 5.3 presents LOOCV accuracies of the ErrP classifiers for every participant as well as the

corresponding TPR and TNR values.


Table 5.2: Leave-one-out cross validation results of the intention classifier (task vs. idle) for eachparticipant: true positive rate (TPR), true negative rate (TNR), and accuracy (ACC). The cognitivetask was defined as the positive class while idle was defined as the negative class.

Participant # 1 2 3 4 5 6 7 8 9 10 Mean std

TPR 0.84 0.99 0.62 0.98 0.93 0.83 0.50 0.74 0.90 0.85 0.82 0.16TNR 0.99 0.97 0.93 1.00 0.99 0.94 0.92 0.92 0.99 0.97 0.96 0.03ACC 0.95 0.99 0.85 0.99 0.97 0.91 0.82 0.88 0.96 0.94 0.93 0.06

Table 5.3: Leave-one-out cross validation results of the ErrP classifier (error-present vs. error-absent):true positive rate (TPR), true negative rate (TNR), and accuracy (ACC). The presence of error wasdefined as the positive class while the absence of error was considered the negative class.


TPR 0.83 0.83 0.61 0.72 0.72 0.89 0.94 0.57 0.83 1.00 0.79 0.14TNR 0.86 0.93 0.91 0.95 0.81 0.95 1.00 0.83 0.98 1.00 0.92 0.07ACC 0.85 0.90 0.82 0.88 0.78 0.93 0.98 0.75 0.93 1.00 0.88 0.08

5.4.2 Online Test Session

Table 5.4 reports the online performance of the BCIs in terms of success rate, hit rate, and precision

for every participant. Without considering the error correction mechanism, successful trials include two

different scenarios: (1) trials where the BCI detected the task correctly (i.e., after the participant started

performing the task); (2) trials during which the participant decided not to perform the mental task,

and the BCI correctly classified the trial as no-control, i.e., every 5 s interval of the trial (with 1 s steps)

was correctly labeled as idle.

The online error detection step produced two different scenarios post-intention classification: (1)

The intention classifier detected the mental task correctly, but the ErrP classifier detected an error and

changed the predicted label (task or idle) to the wrong one; (2) The participant did not perform the

task, and the intention classifier mistakenly detected the task, but the ErrP classifier detected the error

and corrected the output. The trade-off between these two scenarios determined whether the proposed

error correction technique improved the trial success rate or not.

The average post-error correction success rate of 85 ± 11% was achieved across participants which

was 7% higher than the average pre-error correction value (p=0.047, Wilcoxon Signed-rank).

As shown in table 5.4, the positive predictive value (PPV) was significantly increased by 9% (p=0.016,

Wilcoxon Singed-rank).

Table 5.4: Online performance of the asynchronous BCI: pre- and post-error correction success rate(SR), hit rate (HR), and positive predictive value (PPV).


Pre-error correctionSR 0.68 0.84 0.56 0.66 0.86 0.80 0.74 0.92 0.86 0.88 0.78 0.12HR 0.83 1.00 0.75 1.00 1.00 0.85 0.76 0.91 1.00 0.98 0.91 0.10PPV 0.80 0.80 0.57 0.66 0.83 0.93 0.97 1.00 0.84 0.87 0.83 0.13

Post-error correctionSR 0.82 0.96 0.62 0.90 0.86 0.76 0.74 0.92 0.96 0.92 0.85 0.11HR 0.78 0.97 0.70 0.94 1.00 0.79 0.76 0.91 0.94 0.97 0.88 0.11PPV 1.00 0.97 0.63 0.91 0.83 0.95 0.97 1.00 1.00 0.93 0.92 0.11

Figure 5.3 illustrates the grand average of all error and no-error trials from both training and test


Figure 5.3: The grand average of ErrP waveforms across all participants for error and no-error trialsalong with their corresponding error bars (shaded region). Time 0 marks the beginning of the visualfeedback. The vertical dashed red lines demarcate the segment which was used for classification.

sessions across over all participants along with their error bars.

5.5 Discussion

5.5.1 Asynchronous Protocol Design

An ideal asynchronous BCI must perform purely based on the user’s intention without the user deciding

in advance or being told by the system when to start intentional control. However, most previous

asynchronous BCIs do not meet this condition because knowledge of the exact time intervals of intentional

control is needed for reporting typical assessment metrics such as TP and FP rates.

In many of the reported asynchronous BCI studies, a cue was informed participants what task to

perform and when to perform it [161, 16, 39]. In [80], Koo et al. asked their participants to decide the

task and when they would like to perform it, prior to a trial. Subsequently, during the trial, a timer

was shown to the participants. One can argue that the timer was essentially a synchronous cue, as

participants needed to start the task at a specific time, albeit determined pre-trial.

Other studies have tried to avoid cues. For example, several studies designed a goal-oriented protocol

where participants had to reach a target, such as going through a maze [87, 136] or controlling a

wheelchair [47, 178, 162], a virtual avatar [88, 169] or a robot [16]. Hence, without telling the participants

specifically what to do, their performance could be measured.

Finally, in some asynchronous BCI studies, participants were asked to press a button before or after

the mental task period. Song and Sepulveda [154] displayed a circular progress bar during each trial

to help their participants with timing. Participants were given 30 s and they were free to perform the


activation task for 3 s at anytime during that period. They were required to press a button when they

finished the task. In [109], Muller-Putz et al. told participants that they had 180 s for each trial and

that they were free to start a foot imagery task anytime during that period, but were required to press

a button few seconds before they intended to start. A beep sound was played as a feedback whenever

the task was detected.

In our study, we tried to minimize any type of motor movement or distraction and designed a protocol

according to the aforementioned definition of an ideal asynchronous BCI. In other words, in our study,

the beginning of intentional control was unknown. We considered this a worthwhile compromise for the

opportunity to study the feasibility of a real asynchronous brain-switch. Prior asynchronous BCI studies

have used different metrics for performance assessment. TP and FP rates [161, 109], mean time to reach

a target [87, 16, 169, 162], and information transfer rate [87]. In this study, the BCI’s performance was

evaluated based on hit rate, precision, and success rate on a trial-by-trial basis (also known as event-by

event basis as described by [161]).

5.5.2 BCIs as Brain-switches

The performance of the asynchronous BCI was comparable to that of previous EEG brain-switches.

Mason and Birch developed a brain-switch based on finger movement. They achieved offline TP rates

in the range of 0.38 to 0.81 and FP rates in the range of 0.03 to 0.12 over five participants [98]. In

[153], Solis-Escalante et al. developed a MI-based asynchronous BCI that was trained based on motor

execution data. Six out of nine participants obtained offline TP rates between 0.50 to 0.88 and the

average FPR was 0.09±0.04. Muller-Putz et al. reported hit rates between 0.69 and 0.8 and precision

values between 0.75 and 0.93 for a single-channel MI-based asynchronous system [109]. In this study, we

achieved comparable results. The average hit rate and precision of 91% and 83% was obtained before

applying error-correction.

5.5.3 Adding Error-correction

In [10], Bhattacharyya et al. confirmed that adding error detection improved the performance of their

asynchronous BCI. Error-correction increased the success rate (the number of times of successful target

attainment over the total number of attempts) from 55 % to 86 %. Our study also showed that adding

ErrP-based error detection significantly improved the average trial success rate by 7% from 78% to

85% over participants. These results suggest the feasibility and benefits of adding error detection to

asynchronous BCIs.

There is an imperative difference between the effect of error correction in asynchronous and syn-

chronous BCIs. In synchronous BCIs, error correction can be used to correct the results of all classes.

However, for example, in an asynchronous brain-switch, error detection is only possible in the case of

false positives (e.g., when the BCI makes the mistake of detecting IC). Hence, ErrP-based error detection

can potentially correct false positives while cannot detect and correct false negatives.

5.5.4 ErrP Waveforms

In this study, whenever the activation task was detected, visual feedback was shown to the participant

in the form of a falling icon. However, participants’ reactions were different based on whether the error

response was detected correctly or not as shown in figure 5.3. The dip at around 300 ms in both error and


no-error waveforms can be attributed to the motion-onset visual evoked potentials (mVEP) due to the

falling icon [84]. Although participants were constantly looking at the icon in the center, the mVEP was

stronger in no-error cases. This can be explained by the fact that during no-error trials, their attention

level was higher as they were actively trying to knock down the icon. We postulate that the no-error

waveform shape is a P300 response from the visual feedback. It comprises two positive peaks at ∼ 500

ms and ∼ 700 ms. It has been confirmed that P300 waveforms have two sub-components called P3a and

P3b which originate from attention and memory-related storage activities, respectively [130]. The error

waveform shares similar morphological characteristics with other error waveforms reported in literature,

such as the outcome error observed in [156] and error in a virtual moving square experiment [72].

The latencies of mVEP and P300 components were about 200 ms longer than reported in literature.

This delay might be attributable to the fact that feedback was gradual, and the icon fell over 500 ms

and it took participants about a 200 ms to notice the fall.

5.5.5 Limitations and Future Directions

For future studies, the authors suggest the use of additional sessions as greater data samples may

enhance both intention and ErrP classifiers’ performance. In fact, with more sessions, one may gain

insight about the maximum achievable classifier performance. Also, calibrating the thresholds and

regularization parameters for both classifiers after one or more online blocks may potentially improve

the results.

Future research can also explore the effect of increasing the number of classes in the brain switch,

e.g., a two-choice switch. Such a BCI could be used by clinical populations to activate an assistive device

or a call bell for assistance. The effect of error correction on a multi-class asynchronous BCI can also be

investigated.

Finally, prior to any clinical translation, the results herein should be replicated with different age

groups, and more importantly, individuals who present as locked-in.

Chapter 6

Conclusion

This thesis explored the possibility of detecting error-related potential in synchronous and asynchronous

BCIs based on non-motor imagery cognitive tasks and investigated the extent that error detection and

correction can improve BCI performance. The major contributions of this thesis are as follows:

1. Investigated the presence of ErrPs in an EEG study with five non-MI cognitive tasks (mental arith-

metic, counting, word generation and figure rotation and idle state) and estimated the potential im-

provement in EEG classification of cognitive tasks by exploiting ErrPs. To simulate errors, a random

subset of 20% of the trials were followed by incorrect feedback. The classification of mental tasks and

error versus non-error trials were both performed using a pseudo-online paradigm where the last quarter

of trials were used for testing.

I. Our findings confirmed the presence of an interaction ErrP, with a negative peak at 180 ms, followed

by two positive peaks, respectively, at 400 and 630 ms post-feedback onset. There were however

departures from literature in terms of latencies, which are expected given the different experimental

protocols used to elicit error responses [1]. The latencies in this study were longer than most of

those reported in literature, which can be attributed to the gradual feedback, resulting in a delayed

perception of the BCI decision.

II. For binary (task and idle) and ternary (2 tasks and idle) classification, across-participant average

accuracies of 76%± 12 and 63%±12, respectively, were attained. Dyson et al. reported an average

offline binary classification accuracy of 68% between an idle state and a mental task (one of motor

imagery, auditory imagery, mental arithmetic or navigation imagery) [34] while Agarwal et al. [2]

achieved an average offline ternary classification accuracy of 69% in discriminating among left/right

MI and word generation. Our task classifier reached comparable ternary accuracies online.

III. An average area under curve (AUC) of 0.83 ± 0.08 was reached across participants for the detection

of ErrPs. The results are comparable to that reported in existing literature. Zeyl and Chau

reported an average AUC of 0.81 in a visual P300 speller [185]. An accuracy of 80% was achieved

for ErrP detection by Bhattacharyya et la. in a motor imagery BCI in one study [9] and 93% in

another [10]. They attributed their high accuracy in [10] to protracted training times of at least

60 hours.

IV. After applying ErrP-based error correction, the average binary and ternary classification accuracies

of mental tasks improved by 9% and 14%, respectively.

52

Chapter 6. Conclusion 53

V. Our findings supported the addition of ErrP detection and ErrP-informed correction to maximize

the accuracy of BCIs based on cognitive tasks.

2. Designed and tested a three-class active online EEG-BCI based on cognitive rather than MI

tasks and investigated the effect of ErrP-based error correction, i.e., to automatically correct erroneous

classifications by exploiting ErrPs. To the best of our knowledge, this was the first report of exploiting

ErrPs to improve the performance of an active BCI based on non-MI cognitive tasks.

I. For each participant, a ternary BCI differentiated among idle state and two personally selected

cognitive tasks (e.g., mental arithmetic, counting, word generation, and figure rotation). An

average ternary classification accuracy of 60%±14 was obtained across participants, with nine

out of ten participants surpassing the chance level.

II. For ErrP classification, an average TPR of 0.59 ± 0.23 and FPR of 0.21 ± 0.21 was obtained across

participants with seven out of ten participants surpassing the chance level. These average values

are within the range of those reported in existing literature [10, 9, 83, 43]. Bhattacharyya et al. [10]

obtained higher error potential detection, which can be attributed to their higher BCI task classi-

fication accuracy, making errors more rare and hence eliciting stronger and more easily detectable

brain responses. Their high BCI performance was attributable to the extensive user training time

(60-80 hours). The BCI’s task classification error rate inversely relates to the performance of the

ErrP classifier (i.e. higher BCI error rate tends to suppress the ErrP classification accuracy [17]).

Hence, in general, it can be expected that when a BCI performs poorly, the performance of the

associated error detector diminishes as well.

III. ErrP-based error correction modestly but significantly improved the average online task classifica-

tion accuracy (+7%) as well as the information transfer rate (+0.9 bits/min) of the ternary BCI

across participants. Ferrez and Millan showed that applying error correction for two participants

reduced the error rate from 32% to 7% which subsequently tripled the bit rate [43]. Kreilinger et

al. demonstrated an improvement in accuracy of 11% post-error correction averaged over three

participants [83] but ITR values were not reported. Given the modest number of participants in

these studies, the significance of these results cannot be ascertained. Bhattacharyya et al. [9, 10]

conducted two studies of ErrP-based error correction reporting higher ErrP-corrected performance

in controlling a robotic arm. However, they did not report pre- and post-correction accuracies and

ITR values, but rather robot targeting metrics to estimate the improvement. Nonetheless, these

studies collectively indicate potential value of ErrP-based error correction in active, MI-based BCIs.

3. Designed and tested an asynchronous BCI using EEG and based on non-motor imagery cognitive

tasks and investigated the possibility of improving its performance using ErrPs. To the best of our

knowledge, this was the first report on exploiting ErrPs to improve the performance of an asynchronous

brain switch.

I. Without using ErrP, an average success rate of 0.78 ± 0.12 and a positive precision value of 0.83 ±0.13 was reached across participants. The performance of the asynchronous BCI was comparable

to that of previous EEG brain-switches. Mason and Birch developed a brain-switch based on finger

movement. They achieved offline TP rates in the range of 0.38 to 0.81 and FP rates in the range

of 0.03 to 0.12 over five participants [98]. In [153], Solis-Escalante et al. developed a MI-based


asynchronous BCI that was trained based on motor execution data. Six out of nine participants

obtained offline TP rates between 0.50 to 0.88 and the average FPR was 0.09±0.04. Muller-Putz

et al. reported hit rates between 0.69 and 0.8 and precision values between 0.75 and 0.93 for a

single-channel MI-based asynchronous system [109].

II. For ErrP classification, an average TPR of 0.79 ± 0.14 and TNR of 0.92 ± 0.07 was obtained

across participants.

III. The average of post-error correction success rate and positive precision value across participants

both improved significantly to 0.85 ± 0.11 and 0.92 ± 0.11 compared to the pre-error correction

values (p < 0.05 using the Wilcoxon signed-rank test). In [10], Bhattacharyya et al. confirmed that

adding error detection improved the performance of their asynchronous BCI. Error-correction in-

creased the success rate (the number of times of successful target attainment over the total number

of attempts) from 55 % to 86 %. Our study also showed that adding ErrP-based error detection

significantly improved the average trial success rate by 7% from 78% to 85% over participants.

These results suggest the feasibility and benefits of adding error detection to asynchronous BCIs.

IV. Our findings supported the addition of ErrP-correction to improve the performance of asynchronous

BCIs.

6.1 Future Work

6.1.1 Investigating ErrPs in more realistic settings and scenarios

All studies within this dissertation were conducted in a quiet room to minimize the effects of distraction

and help participants maximize their focus on task performance. However, such environment is not

necessarily representative of environments of regular BCI use such as homes, classrooms, or outdoor,

where there is an abundance of distractions. For future studies, presence and effect of ErrPs in BCIs

used in such realistic settings and scenarios can be investigated. Future research can also investigate

ErrPs when BCIs are used to control an assistive device such as a robot or wheelchair. It has been shown

in literature that different experimental protocols and feedback design can affect the characteristics of

ErrP waveforms [73]. Hence, the reliability of ErrP detection and its potential in improving the BCI

performance can be explored and optimized for various assistive devices. Note that the assistive device

does not need to necessarily be controlled by the user’s brain signals. In other words, ErrPs potentially

can be exploited to improve the performance of non-BCI assistive devices which make mistakes.

6.1.2 Considering collecting more data

For future studies, we suggest the use of additional sessions and collecting more data. Increasing the

number of trials can potentially enhance classifier performance in various ways. First, a more effective

mathematical model can be trained for classification using more data. For example, in all of the studies

in this thesis, the regularization parameter was optimized separately for each participant and for each

classification problem. Additional sessions and extra trials may enhance the generalizability of the

classification models and mitigate the need for such customization. Secondly, users’ brains can also

learn from the BCI decisions and feedback which may result to a better performance [147]. Our findings

in Chapter 4 showed improvement in the ErrP classification performance in later test blocks compared to


earlier one, which is possibly due to the increased training data. Additional sessions would shed further

light on the achievable ErrP classifier performance and robustness. Additionally, upon the availability

of enough trials, more sophisticated classification models, e.g., deep learning networks, can potentially

be trained and tested.

6.1.3 Exploring multi-choice asynchronous brain switches and the effect of

ErrP-guided error correction in their performance

In this thesis, we explored a binary asynchronous brain switch and demonstrated that exploiting ErrP

can significantly improve its performance. For future studies, we suggest investigating multi-choice

asynchronous brain switches, where more than one cognitive task can be used to activate the BCI. Using

ErrP-guided error correction in such setting can potentially improve the BCI performance as it can be

used not only to correct false positives, but also to correct the BCI decisions when the presence of an

activation (i.e., a cognitive task) against the idle state is detected correctly but the type of the cognitive

tasks is detected erroneously.

6.1.4 Validating the findings with different age groups as well as potential

BCI users

The results presented in this thesis are promising. However, they are yet ready for clinical translation.

The findings herein must be first replicated with individuals from different age groups, and more im-

portantly, individuals who present as locked-in, as the main target population of BCI research. It has

been previously shown that the characteristics of some brain responses, such as P300, can be different in

various age groups [131]. All the studies in this thesis were conducted on young adults. Future research

can investigate the changes in the characteristics of ErrPs and their detectability in BCIs based on active

tasks in the pediatric and elderly populations. To date, the sample size and number of BCI studies on

candidate BCI users, individuals who present as locked-in and with limited or no voluntary movement,

have been limited [107, 151]. Testing ErrP-guided error correction in active BCIs with different clinical

populations is required to determine what modifications will be required upon their eventual clinical

implementation.

6.2 Peer-reviewed publications

Three peer-reviewed journal articles have resulted from this work. They are listed as follows:

1. Yousefi, R., Sereshkeh, A.R. and Chau, T., 2018. Exploiting error-related potentials in cognitive

task based BCI. Biomedical Physics and Engineering Express, 5(1), p.015023.

2. Yousefi, R., Sereshkeh, A.R. and Chau, T., 2019. Online detection of error-related potentials

in multi-class cognitive task-based BCIs. Brain-Computer Interfaces, Brain-Computer Interfaces

(2019): 1-12..

3. Yousefi, R., Sereshkeh, A.R. and Chau, T., 2019., Development of a Robust Asynchronous Brain-

switch using ErrP-based Error Correction, Journal of Neural Engineering, Under Review.


In addition to these, two other peer-reviewed journal articles were completed during my Ph.D., but

are not part of my thesis. They are listed as follows:

1. Sereshkeh, A.R., Yousefi, R., Wong, A.T. and Chau, T., 2018. Online classification of imagined

speech using functional near-infrared spectroscopy signals. Journal of neural engineering, 16(1),

p.016005.

2. Sereshkeh, A.R., Yousefi, R., Wong, A.T. and Chau, T., 2019. Development of a ternary hybrid

fNIRS-EEG brain-computer interface based on imagined speech, Brain-Computer Interfaces, Under

review

Bibliography

[1] Mohammad Abu-Alqumsan, Christoph Kapeller, Christoph Hintermuller, Christoph Guger, and

Angelika Peer. Invariance and variability in interaction error-related potentials and their conse-

quences for classification. Journal of neural engineering, 14(6):066015, 2017.

[2] Saurabh Kumar Agarwal, Saatvik Shah, and Rajesh Kumar. Classification of mental tasks

from EEG data using backtracking search optimization based neural classifier. Neurocomputing,

166:397–403, 2015.

[3] Md Abu Baker Siddique Akhanda, Shaon Md Foorkanul Islam, and Md Mostafizur Rahman.

Detection of cognitive state for brain-computer interfaces. In 2013 International Conference on

Electrical Information and Communication Technology (EICT), pages 1–6. IEEE, 2014.

[4] Brendan Z Allison, Elizabeth Winter Wolpaw, and Jonathan R Wolpaw. Brain–computer interface

systems: progress and prospects. Expert review of medical devices, 4(4):463–474, 2007.

[5] Hafeez Ullah Amin, Aamir Saeed Malik, Rana Fayyaz Ahmad, Nasreen Badruddin, Nidal Kamel,

Muhammad Hussain, and Weng-Tink Chooi. Feature extraction and classification for eeg signals

using wavelet transform and machine learning techniques. Australasian physical & engineering

sciences in medicine, 38(1):139–149, 2015.

[6] Setare Amiri, Ahmed Rabbi, Leila Azinfar, and Reza Fazel-Rezai. A review of p300, ssvep, and

hybrid p300/ssvep brain-computer interface systems. In Brain-Computer Interface Systems-Recent

Progress and Future Prospects. InTech, 2013.

[7] Charles W Anderson, James N Knight, Tim O’Connor, Michael J Kirby, and Artem Sokolov.

Geometric subspace methods and time-delay embedding for eeg artifact removal and classification.

IEEE Transactions on Neural Systems and Rehabilitation Engineering, 14(2):142–146, 2006.

[8] Xavier Artusi, Imran Khan Niazi, Marie-Francoise Lucas, and Dario Farina. Performance of a sim-

ulated adaptive bci based on experimental classification of movement-related and error potentials.

IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 1(4):480–488, 2011.

[9] Saugat Bhattacharyya, Amit Konar, and DN Tibarewala. Motor imagery, p300 and error-related

eeg-based robot arm movement control for rehabilitation purpose. Medical & biological engineering

& computing, 52(12):1007–1017, 2014.

[10] Saugat Bhattacharyya, Amit Konar, and DN Tibarewala. Motor imagery and error related po-

tential induced position control of a robotic arm. IEEE/CAA Journal of Automatica Sinica,

4(4):639–650, 2017.

57

Bibliography 58

[11] J Matthijs Biesbroek, Martine JE van Zandvoort, L Jaap Kappelle, Birgitta K Velthuis, Geert Jan

Biessels, and Albert Postma. Shared and distinct anatomical correlates of semantic and phonemic

fluency revealed by lesion-symptom mapping in patients with ischemic stroke. Brain Structure and

Function, 221(4):2123–2134, 2016.

[12] Benjamin Blankertz, Claudia Sanelli, Sebastian Halder, Eva-Maria Hammer, Andrea Kubler,

Klaus-Robert Muller, Gabriel Curio, and Thorsten Dickhaus. Predicting bci performance to study

bci illiteracy. 2009.

[13] Julie Blumberg, Jorn Rickert, Stephan Waldert, Andreas Schulze-Bonhage, Ad Aertsen, and

Carsten Mehring. Adaptive classification for brain computer interfaces. In Engineering in Medicine

and Biology Society, 2007. EMBS 2007. 29th Annual International Conference of the IEEE, pages

2536–2539. IEEE, 2007.

[14] Christian Breitwieser, Vera Kaiser, Christa Neuper, and Gernot R Muller-Putz. Stability and

distribution of steady-state somatosensory evoked potentials elicited by vibro-tactile stimulation.

Medical & biological engineering & computing, 50(4):347–357, 2012.

[15] Anne-Marie Brouwer and Jan BF Van Erp. A tactile p300 brain-computer interface. Frontiers in

neuroscience, 4:19, 2010.

[16] Yongwook Chae, Jaeseung Jeong, and Sungho Jo. Toward brain-actuated humanoid robots: asyn-

chronous direct control using an eeg-based bci. IEEE Transactions on Robotics, 28(5):1131–1144,

2012.

[17] Ricardo Chavarriaga and Jose del R Millan. Learning from eeg error-related potentials in noninva-

sive brain-computer interfaces. IEEE transactions on neural systems and rehabilitation engineering,

18(4):381–388, 2010.

[18] Ricardo Chavarriaga, Aleksander Sobolewski, and Jose del R Millan. Errare machinale est: the

use of error-related potentials in brain-machine interfaces. Frontiers in neuroscience, 2014.

[19] JM Clements, EW Sellers, DB Ryan, K Caves, LM Collins, and CS Throckmorton. Applying

dynamic data collection to improve dry electrode system performance for a p300-based brain–

computer interface. Journal of neural engineering, 13(6):066018, 2016.

[20] Adrien Combaz, Nikolay Chumerin, Nikolay V Manyakov, Arne Robben, Johan AK Suykens, and

Marc M Van Hulle. Towards the detection of error-related potentials and its integration in the

context of a p300 speller brain–computer interface. Neurocomputing, 80:73–82, 2012.

[21] Etienne Combrisson and Karim Jerbi. Exceeding chance level by chance: The caveat of theoretical

chance levels in brain signal classification and statistical assessment of decoding accuracy. Journal

of neuroscience methods, 250:126–136, 2015.

[22] Massimiliano Conson, Simona Sacco, Marco Sara, Francesca Pistoia, Dario Grossi, and Luigi

Trojano. Selective motor imagery defect in patients with locked-in syndrome. Neuropsychologia,

46(11):2622–2628, 2008.

[23] Steven C Cramer, Lindsey Lastra, Michael G Lacourse, and Michael J Cohen. Brain motor system

function after chronic, complete spinal cord injury. Brain, 128(12):2941–2950, 2005.

Bibliography 59

[24] Steven C Cramer, Elizabeth LR Orr, Michael J Cohen, and Michael G Lacourse. Effects of

motor imagery training after chronic, complete spinal cord injury. Experimental brain research,

177(2):233–242, 2007.

[25] Eleanor Curran, Peter Sykacek, Maria Stokes, Stephen J Roberts, Will Penny, Ingrid Johnsrude,

and Adrian M Owen. Cognitive tasks for driving a brain-computer interfacing system: a pilot

study. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 12(1):48–54, 2004.

[26] Bernardo Dal Seno, Matteo Matteucci, and Luca Mainardi. Online detection of p300 and error

potentials in a bci speller. Computational intelligence and neuroscience, 2010:11, 2010.

[27] Charles S DaSalla, Hiroyuki Kambara, Makoto Sato, and Yasuharu Koike. Single-trial classification

of vowel speech imagery using common spatial patterns. Neural networks, 22(9):1334–1339, 2009.

[28] Stanislas Dehaene, Nicolas Molko, Laurent Cohen, and Anna J Wilson. Arithmetic and the brain.

Current opinion in neurobiology, 14(2):218–224, 2004.

[29] Jose del R Millan, Josep Mourino, Marco Franze, Febo Cincotti, Markus Varsta, Jukka Heikkonen,

and Fabio Babiloni. A local neural classifier for the recognition of eeg patterns associated to mental

tasks. IEEE transactions on neural networks, 13(3):678–686, 2002.

[30] Catarina Lopes Dias, Andreea I Sburlea, and Gernot R Muller-Putz. Masked and unmasked

error-related potentials during continuous control and feedback. Journal of neural engineering,

15(3):036031, 2018.

[31] Pablo F Diez, Vicente A Mut, Enrique M Avila Perona, and Eric Laciar Leber. Asynchronous bci

control using high-frequency ssvep. Journal of neuroengineering and rehabilitation, 8(1):39, 2011.

[32] M Dyson, Francisco Sepulveda, and John Q Gan. Localisation of cognitive tasks used in eeg-based

bcis. Clinical Neurophysiology, 121(9):1481–1493, 2010.

[33] Matthew Dyson, Francisco Sepulveda, and John Q Gan. Mental task classification against the idle

state: a preliminary investigation. In Engineering in Medicine and Biology Society, 2008. EMBS

2008. 30th Annual International Conference of the IEEE, pages 4473–4477. IEEE, 2008.

[34] Matthew Dyson, Francisco Sepulveda, John Q Gan, and Stephen J Roberts. Sequential classifica-

tion of mental tasks vs. idle state for eeg based bcis. In Neural Engineering, 2009. NER’09. 4th

International IEEE/EMBS Conference on, pages 351–354. IEEE, 2009.

[35] Michael D’Zmura, Siyi Deng, Tom Lappas, Samuel Thorpe, and Ramesh Srinivasan. Toward eeg

sensing of imagined speech. In International Conference on Human-Computer Interaction, pages

40–48. Springer, 2009.

[36] Christian Enzinger, Stefan Ropele, Franz Fazekas, Marisa Loitfelder, Faton Gorani, Thomas

Seifert, Gudrun Reiter, Christa Neuper, Gert Pfurtscheller, and Gernot Muller-Putz. Brain motor

system function in a patient with complete spinal cord injury following extensive brain–computer

interface training. Experimental brain research, 190(2):215–223, 2008.

[37] Michael Falkenstein, Joachim Hohnsbein, Jorg Hoormann, and Ludger Blanke. Effects of cross-

modal divided attention on late erp components. ii. error processing in choice reaction tasks.

Electroencephalography and clinical neurophysiology, 78(6):447–455, 1991.

Bibliography 60

[38] Michael Falkenstein, Jorg Hoormann, Stefan Christ, and Joachim Hohnsbein. Erp components on

reaction errors and their functional significance: a tutorial. Biological psychology, 51(2-3):87–107,

2000.

[39] Farhad Faradji, Rabab K Ward, and Gary E Birch. Toward development of a two-state brain–

computer interface based on mental tasks. Journal of neural engineering, 8(4):046014, 2011.

[40] Mehrdad Fatourechi, Ali Bashashati, Rabab K Ward, and Gary E Birch. Emg and eog artifacts

in brain computer interface systems: A survey. Clinical neurophysiology, 118(3):480–494, 2007.

[41] Pierre W Ferrez and Jose del R Millan. You are wrong!—automatic detection of interaction errors

from brain waves. In Proceedings of the 19th international joint conference on Artificial intelligence,

number EPFL-CONF-83269, 2005.

[42] Pierre W Ferrez and Jose del R Millan. Error-related eeg potentials generated during simulated

brain–computer interaction. IEEE transactions on biomedical engineering, 55(3):923–929, 2008.

[43] Pierre W Ferrez and Jose del R Millan. Simultaneous real-time detection of motor imagery and

error-related potentials for improved bci accuracy. In Proceedings of the 4th international brain-

computer interface workshop and training course, number CNBI-CONF-2008-004, pages 197–202,

2008.

[44] Elisabeth VC Friedrich, Christa Neuper, and Reinhold Scherer. Whatever works: a systematic

user-centered training protocol to optimize brain-computer interfacing individually. PloS one,

8(9):e76214, 2013.

[45] Elisabeth VC Friedrich, Reinhold Scherer, and Christa Neuper. The effect of distinct mental

strategies on classification performance for brain–computer interfaces. International Journal of

Psychophysiology, 84(1):86–94, 2012.

[46] Adrian Furdea, Sebastian Halder, DJ Krusienski, Donald Bross, Femke Nijboer, Niels Birbaumer,

and Andrea Kubler. An auditory oddball (p300) spelling system for brain-computer interfaces.

Psychophysiology, 46(3):617–625, 2009.

[47] Ferran Galan, Marnix Nuttin, Eileen Lew, Pierre W Ferrez, Gerolf Vanacker, Johan Philips, and

J del R Millan. A brain-actuated wheelchair: asynchronous and non-invasive brain–computer

interfaces for continuous control of robots. Clinical Neurophysiology, 119(9):2159–2169, 2008.

[48] William J Gehring, Brian Goss, Michael GH Coles, David E Meyer, and Emanuel Donchin. A

neural system for error detection and compensation. Psychological science, 4(6):385–390, 1993.

[49] Tao Geng, Matthew Dyson, Chun SL Tsui, and John Q Gan. A 3-class asynchronous bci controlling

a simulated mobile robot. In Engineering in Medicine and Biology Society, 2007. EMBS 2007.

29th Annual International Conference of the IEEE, pages 2524–2527. IEEE, 2007.

[50] Rodolphe Gentili, Cheol E Han, Nicolas Schweighofer, and Charalambos Papaxanthis. Motor

learning without doing: trial-by-trial improvement in motor performance during mental training.

Journal of neurophysiology, 104(2):774–783, 2010.

[51] Brain Products GmbH. actiCAP Xpress Twist Operating Instructions. Brain Products, 2016.

Bibliography 61

[52] B Graimann, G Townsend, JE Huggins, A Schlogl, SP Levine, and G Pfurtscheller. A comparison

between using ecog and eeg for direct brain communication. Proceedings of the EMBEC05, 2005.

[53] Michael D Greicius, Ben Krasnow, Allan L Reiss, and Vinod Menon. Functional connectivity in

the resting brain: a network analysis of the default mode hypothesis. Proceedings of the National

Academy of Sciences, 100(1):253–258, 2003.

[54] L Großberger, M Hohmann, J Peters, and M Grosse-Wentrup. Investigating music imagery as a

cognitive paradigm for low-cost brain-computer interfaces. In Graz BCI Conf. 2017, pages 160–164,

2017.

[55] Christoph Guger, Gunther Krausz, Brendan Z Allison, and Guenter Edlinger. Comparison of dry

and gel based electrodes for P300 brain–computer interfaces. Frontiers in neuroscience, 6:60, 2012.

[56] Fei Guo, Bo Hong, Xiaorong Gao, and Shangkai Gao. A brain–computer interface using motion-

onset visual evoked potential. Journal of neural engineering, 5(4):477–485, 2008.

[57] Lei Guo, Youxi Wu, Lei Zhao, Ting Cao, Weili Yan, and Xueqin Shen. Classification of mental

task from eeg signals using immune feature weighted support vector machines. IEEE Transactions

on Magnetics, 47(5):866–869, 2011.

[58] Donald J Hagler. Visual field asymmetries in visual evoked responses. Journal of Vision, 14(14):13–

13, 2014.

[59] Greg Hajcak, Jason S Moser, Clay B Holroyd, and Robert F Simons. The feedback-related negativ-

ity reflects the binary evaluation of good versus bad outcomes. Biological psychology, 71(2):148–154,

2006.

[60] Greg Hajcak, Jason S Moser, Nick Yeung, and Robert F Simons. On the ern and the significance

of errors. Psychophysiology, 42(2):151–160, 2005.

[61] M Hariharan, Vikneswaran Vijean, R Sindhu, P Divakar, A Saidatul, and Sazali Yaacob. Classifi-

cation of mental tasks using stockwell transform. Computers & Electrical Engineering, 40(5):1741–

1749, 2014.

[62] Bashar Awwad Shiekh Hasan and John Q Gan. Hangman bci: An unsupervised adaptive self-paced

brain–computer interface for playing games. Computers in biology and medicine, 42(5):598–606,

2012.

[63] Mounia Hendel, Abdelkader Benyettou, and Fatiha Hendel. Hybrid self organizing map and proba-

bilistic quadratic loss multi-class support vector machine for mental tasks classification. Informatics

in Medicine Unlocked, 4:1–9, 2016.

[64] Robert Hester, John J Foxe, Sophie Molholm, Marina Shpaner, and Hugh Garavan. Neural mech-

anisms involved in error processing: a comparison of errors made with and without awareness.

Neuroimage, 27(3):602–608, 2005.

[65] Enrique Hortal, Daniel Planelles, A Costa, Eduardo Ianez, Andres Ubeda, Jose Maria Azorın, and

Eduardo Fernandez. Svm-based brain–machine interface for controlling a robot arm through four

mental tasks. Neurocomputing, 151:116–121, 2015.

Bibliography 62

[66] Wei-Yen Hsu. Continuous eeg signal analysis for asynchronous bci application. International

journal of neural systems, 21(04):335–350, 2011.

[67] Gina F Humphreys, Paul Hoffman, Maya Visser, Richard J Binney, and Matthew A Lambon

Ralph. Establishing task-and modality-dependent dissociations between the semantic and default

mode networks. Proceedings of the National Academy of Sciences, 112(25):7857–7862, 2015.

[68] Eduardo Ianez, Jose Marıa Azorın, Andres Ubeda, Jose Manuel Ferrandez, and Eduardo

Fernandez. Mental tasks-based brain–robot interface. Robotics and Autonomous Systems,

58(12):1238–1245, 2010.

[69] Jaime Ibanez, Jose Ignacio Serrano, M Dolores del Castillo, Juan Alvaro Gallego, and E Rocon.

Online detector of movement intention based on eeg—application in tremor patients. Biomedical

Signal Processing and Control, 8(6):822–829, 2013.

[70] Anja Ischebeck, Stefan Heim, Christian Siedentopf, Laura Zamarian, Michael Schocke, Christian

Kremser, Karl Egger, Hans Strenge, Filip Scheperjans, and Margarete Delazer. Are numbers

special? comparing the generation of verbal materials from ordered categories (months) to numbers

and other categories (animals) in an fmri study. Human brain mapping, 29(8):894–909, 2008.

[71] Inaki Iturrate, Ricardo Chavarriaga, Luis Montesano, Javier Minguez, and Jose R del Millan.

Latency correction of error potentials between different experiments reduces calibration time for

single-trial classification. In Engineering in Medicine and Biology Society (EMBC), 2012 Annual

International Conference of the IEEE, pages 3288–3291. IEEE, 2012.

[72] Inaki Iturrate, Ricardo Chavarriaga, Luis Montesano, Javier Minguez, and JdR Millan. Latency

correction of event-related potentials between different experimental protocols. Journal of neural

engineering, 11(3):036005, 2014.

[73] Inaki Iturrate, Luis Montesano, and Javier Minguez. Task-dependent signal variations in eeg error-

related potentials for brain–computer interfaces. Journal of neural engineering, 10(2):026024, 2013.

[74] Jing Jin, Brendan Z Allison, Xingyu Wang, and Christa Neuper. A combined brain–computer

interface based on p300 potentials and motion-onset visual evoked potentials. Journal of neuro-

science methods, 205(2):265–276, 2012.

[75] Vera Kaiser, Gunther Bauernfeind, Alex Kreilinger, Tobias Kaufmann, Andrea Kubler, Christa

Neuper, and Gernot R Muller-Putz. Cortical effects of user training in a motor imagery based

brain–computer interface measured by fNIRS and EEG. Neuroimage, 85:432–444, 2014.

[76] Zachary A Keirn and Jorge I Aunon. Man-machine communications through brain-wave processing.

IEEE Engineering in Medicine and biology magazine, 9(1):55–57, 1990.

[77] Zachary A Keirn and Jorge I Aunon. A new mode of communication between man and his

surroundings. IEEE transactions on biomedical engineering, 37(12):1209–1214, 1990.

[78] Su Kyoung Kim and Elsa Andrea Kirchner. Handling few training data: Classifier transfer between

different types of error-related potentials. IEEE Transactions on Neural Systems and Rehabilitation

Engineering, 24(3):320–332, 2016.

Bibliography 63

[79] Su Kyoung Kim, Elsa Andrea Kirchner, Arne Stefes, and Frank Kirchner. Intrinsic interactive rein-

forcement learning–using error-related potentials for real world human-robot interaction. Scientific

reports, 7(1):17562, 2017.

[80] Bonkon Koo, Hwan-Gon Lee, Yunjun Nam, Hyohyeong Kang, Chin Su Koh, Hyung-Cheul Shin,

and Seungjin Choi. A hybrid nirs-eeg system for self-paced brain computer interface with online

motor imagery. Journal of neuroscience methods, 244:26–32, 2015.

[81] Nataliya Kosmyna, Franck Tarpin-Bernard, and Bertrand Rivet. Operationalization of conceptual

imagery for bcis. In 2015 23rd European Signal Processing Conference (EUSIPCO), pages 2726–

2730. IEEE, 2015.

[82] Alex Kreilinger, Christa Neuper, and Gernot R Muller-Putz. Error potential detection during con-

tinuous movement of an artificial arm controlled by brain–computer interface. Medical & biological

engineering & computing, 50(3):223–230, 2012.

[83] Alex Kreilinger, Christa Neuper, Gert Pfurtscheller, and Gernot R Muller-Putz. Implementation of

error detection into the graz-brain-computer interface, the interaction error potential. In European

Conference for the Advancement of Assistive Technology, pages 195–199, 2009.

[84] Miroslav Kuba and Zuzana Kubova. Visual evoked potentials specific for motion onset. Documenta

Ophthalmologica, 80(1):83–89, 1992.

[85] Andrea Kubler, Adrian Furdea, Sebastian Halder, Eva Maria Hammer, Femke Nijboer, and Boris

Kotchoubey. A brain–computer interface controlled auditory event-related potential (p300) spelling

system for locked-in patients. Annals of the New York Academy of Sciences, 1157(1):90–100, 2009.

[86] Rafa l Kus, Tomasz Spustek, Magdalena Zieleniewska, Anna Duszyk, Piotr Rogowski, and Piotr

Suffczynski. Integrated trimodal ssep experimental setup for visual, auditory and tactile stimula-

tion. Journal of neural engineering, 14(6):066002, 2017.

[87] Rafal Kus, Diana Valbuena, Jaroslaw Zygierewicz, Tatsiana Malechka, Axel Graeser, and Piotr

Durka. Asynchronous bci based on motor imagery with automated calibration and neurofeedback

training. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 20(6):823–835,

2012.

[88] Robert Leeb, Doron Friedman, Gernot R Muller-Putz, Reinhold Scherer, Mel Slater, and Gert

Pfurtscheller. Self-paced (asynchronous) bci control of a wheelchair in virtual environments: a

case study with a tetraplegic. Computational intelligence and neuroscience, 2007, 2007.

[89] Xiaoou Li, Xun Chen, Yuning Yan, Wenshi Wei, and Z Jane Wang. Classification of eeg signals

using a multiple kernel learning support vector machine. Sensors, 14(7):12784–12802, 2014.

[90] Yuanqing Li, Jinyi Long, Tianyou Yu, Zhuliang Yu, Chuanchu Wang, Haihong Zhang, and Cuntai

Guan. A hybrid bci system for 2-d asynchronous cursor control. In Engineering in Medicine and

Biology Society (EMBC), 2010 Annual International Conference of the IEEE, pages 4205–4208.

IEEE, 2010.

[91] Cheng-Jian Lin and Ming-Hua Hsieh. Classification of mental task from eeg data using neural

networks based on particle swarm optimization. Neurocomputing, 72(4):1121–1130, 2009.

Bibliography 64

[92] Dong Liu, Weihai Chen, Kyuhwa Lee, Ricardo Chavarriaga, Fumiaki Iwane, Mohamed Bouri,

Zhongcai Pei, and Jose del R Millan. Eeg-based lower-limb movement onset decoding: Continuous

classification and asynchronous detection. IEEE Transactions on Neural Systems and Rehabilita-

tion Engineering, 26(8):1626–1635, 2018.

[93] Alberto Llera, Marcel AJ van Gerven, Vicenc Gomez, Ole Jensen, and Hilbert J Kappen. On

the use of interaction error potentials for adaptive brain computer interfaces. Neural Networks,

24(10):1120–1127, 2011.

[94] Fabien Lotte, Laurent Bougrain, Andrzej Cichocki, Maureen Clerc, Marco Congedo, Alain Rako-

tomamonjy, and Florian Yger. A review of classification algorithms for EEG-based brain–computer

interfaces: a 10 year update. Journal of neural engineering, 15(3):031005, 2018.

[95] Fabien Lotte, Marco Congedo, Anatole Lecuyer, Fabrice Lamarche, and Bruno Arnaldi. A review

of classification algorithms for eeg-based brain–computer interfaces. Journal of neural engineering,

4(2):R1, 2007.

[96] Perrin Margaux, Maby Emmanuel, Daligault Sebastien, Bertrand Olivier, and Mattout Jeremie.

Objective and subjective evaluation of online error correction during P300-based spelling. Advances

in Human-Computer Interaction, 2012:4, 2012.

[97] SG Mason, A Bashashati, M Fatourechi, KF Navarro, and GE Birch. A comprehensive survey of

brain interface technology designs. Annals of biomedical engineering, 35(2):137–169, 2007.

[98] Steven G Mason and Gary E Birch. A brain-controlled switch for asynchronous control applications.

IEEE Transactions on Biomedical Engineering, 47(10):1297–1307, 2000.

[99] Dennis J McFarland, William A Sarnacki, and Jonathan R Wolpaw. Electroencephalographic (eeg)

control of three-dimensional movement. Journal of neural engineering, 7(3):036007, 2010.

[100] Filip Melinscak, Luis Montesano, and Javier Minguez. Asynchronous detection of kinesthetic

attention during mobilization of lower limbs using eeg measurements. Journal of neural engineering,

13(1):016018, 2016.

[101] Jurgen Mellinger, Gerwin Schalk, Christoph Braun, Hubert Preissl, Wolfgang Rosenstiel, Niels

Birbaumer, and Andrea Kubler. An meg-based brain–computer interface (bci). Neuroimage,

36(3):581–593, 2007.

[102] V Menon, SM Rivera, CD White, GH Glover, and AL Reiss. Dissociating prefrontal and parietal

cortex activation during arithmetic processing. Neuroimage, 12(4):357–365, 2000.

[103] Vinod Menon. Large-scale brain networks in cognition: Emerging principles. Analysis and function

of large-scale brain networks, pages 43–54, 2010.

[104] Jd R Millan, Frederic Renkens, Josep Mourino, and Wulfram Gerstner. Noninvasive brain-actuated

control of a mobile robot by human eeg. IEEE Transactions on biomedical Engineering, 51(6):1026–

1033, 2004.

[105] Jdel R Millan and Josep Mourino. Asynchronous bci and local neural classifiers: an overview of

the adaptive brain interface project. IEEE Transactions on Neural Systems and Rehabilitation

Engineering, 11(2):159–161, 2003.

Bibliography 65

[106] Kai J Miller, Gerwin Schalk, Eberhard E Fetz, Marcel den Nijs, Jeffrey G Ojemann, and Rajesh PN

Rao. Cortical activity during motor execution, motor imagery, and imagery-based online feedback.

Proceedings of the National Academy of Sciences, 107(9):4430–4435, 2010.

[107] Saba Moghimi, Azadeh Kushki, Anne Marie Guerguerian, and Tom Chau. A review of eeg-based

brain-computer interfaces as access pathways for individuals with severe disabilities. Assistive

technology : the official journal of RESNA, 25(2):99–110, 2013.

[108] Andrea Mognon, Jorge Jovicich, Lorenzo Bruzzone, and Marco Buiatti. Adjust: An automatic

eeg artifact detector based on the joint use of spatial and temporal features. Psychophysiology,

48(2):229–240, 2011.

[109] Gernot R Muller-Putz, Vera Kaiser, Teodoro Solis-Escalante, and Gert Pfurtscheller. Fast set-up

asynchronous brain-switch based on detection of foot motor imagery in 1-channel eeg. Medical &

biological engineering & computing, 48(3):229–233, 2010.

[110] Gernot R Muller-Putz, Reinhold Scherer, Gert Pfurtscheller, and Rudiger Rupp. Eeg-based neu-

roprosthesis control: a step towards clinical practice. Neuroscience letters, 382(1-2):169–174, 2005.

[111] Gernot R Muller-Putz, Reinhold Scherer, Gert Pfurtscheller, and Rudiger Rupp. Brain-

computer interfaces for control of neuroprostheses: from synchronous to asynchronous mode of

operation/brain-computer interfaces zur steuerung von neuroprothesen: von der synchronen zur

asynchronen funktionsweise. Biomedizinische Technik, 51(2):57–63, 2006.

[112] Andrew Myrden and Tom Chau. Effects of user mental state on eeg-bci performance. Frontiers in

human neuroscience, 9:308, 2015.

[113] Andrew Myrden and Tom Chau. A passive eeg-bci for single-trial detection of changes in mental

state. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2017.

[114] N Naseer and K Hong. fnirs-based brain-computer interfaces: a review. Name: Frontiers in Human

Neuroscience, 9(3), 2015.

[115] Luis Fernando Nicolas-Alonso and Jaime Gomez-Gil. Brain computer interfaces, a review. Sensors,

12(2):1211–1279, 2012.

[116] Seiji Nishifuji, Atsushi Matsubara, Yuya Sugita, Akira Iwata, Hirotaka Nakamura, and Hitoshi

Hirano. Eyes-closed brain computer interface using modulation of steady-state visually evoked

potential and auditory steady-state response. In 2017 56th Annual Conference of the Society of

Instrument and Control Engineers of Japan (SICE), pages 1138–1143. IEEE, 2017.

[117] Lars Nyberg, Johan Eriksson, Anne Larsson, and Petter Marklund. Learning by doing versus

learning by thinking: an fmri study of motor and mental training. Neuropsychologia, 44(5):711–

717, 2006.

[118] Jason Omedes, Andreas Schwarz, Gernot R Muller-Putz, and Luis Montesano. Factors that affect

error potentials during a grasping task: toward a hybrid natural movement decoding bci. Journal

of neural engineering, 2018.

Bibliography 66

[119] Ramaswamy Palaniappan. Utilizing gamma band to improve mental task based brain-computer

interface design. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 14(3):299–

303, 2006.

[120] Ramaswamy Palaniappan, Raveendran Paramesran, Shogo Nishida, and Naoki Saiwaki. A new

brain-computer interface design using fuzzy artmap. IEEE Transactions on Neural Systems and

Rehabilitation Engineering, 10(3):140–148, 2002.

[121] Cheolsoo Park, David Looney, Naveed ur Rehman, Alireza Ahrabian, and Danilo P Mandic. Classi-

fication of motor imagery bci using multivariate empirical mode decomposition. IEEE Transactions

on neural systems and rehabilitation engineering, 21(1):10–22, 2013.

[122] Lucas C Parra, Clay D Spence, Adam D Gerson, and Paul Sajda. Response error correction-a

demonstration of improved human-machine performance using real-time eeg monitoring. IEEE

transactions on neural systems and rehabilitation engineering, 11(2):173–177, 2003.

[123] Gert Pfurtscheller, Brendan Z Allison, Gunther Bauernfeind, Clemens Brunner, Teodoro Solis Es-

calante, Reinhold Scherer, Thorsten O Zander, Gernot Mueller-Putz, Christa Neuper, and Niels

Birbaumer. The hybrid BCI. Frontiers in neuroscience, 4:3, 2010.

[124] Gert Pfurtscheller, Doris Flotzinger, and Joachim Kalcher. Brain-computer interface—a new com-

munication device for handicapped persons. Journal of Microcomputer Applications, 16(3):293–299,

1993.

[125] Gert Pfurtscheller, Gernot R Muller-Putz, Jorg Pfurtscheller, and Rudiger Rupp. Eeg-based asyn-

chronous bci controls functional electrical stimulation in a tetraplegic patient. EURASIP Journal

on Applied Signal Processing, 2005:3152–3155, 2005.

[126] Gert Pfurtscheller and Christa Neuper. Motor imagery and direct brain-computer communication.

Proceedings of the IEEE, 89(7):1123–1134, 2001.

[127] Gert Pfurtscheller, Christa Neuper, Alois Schlogl, and Klaus Lugger. Separability of eeg signals

recorded during right and left motor imagery using adaptive autoregressive parameters. IEEE

transactions on Rehabilitation Engineering, 6(3):316–325, 1998.

[128] Gert Pfurtscheller, Teodoro Solis-Escalante, Rupert Ortner, Patricia Linortner, and Gernot R

Muller-Putz. Self-paced operation of an ssvep-based orthosis with and without an imagery-based

“brain switch:” a feasibility study towards a hybrid bci. IEEE transactions on neural systems and

rehabilitation engineering, 18(4):409–414, 2010.

[129] Andreas Pinegger, Josef Faller, Sebastian Halder, Selina C Wriessnegger, and Gernot R Muller-

Putz. Control or non-control state: that is the question! an asynchronous visual p300-based bci

approach. Journal of neural engineering, 12(1):014001, 2015.

[130] John Polich. Updating p300: an integrative theory of p3a and p3b. Clinical neurophysiology,

118(10):2128–2148, 2007.

[131] John Polich, Christine Ladish, and Tim Burns. Normal variation of p300 in children: age, memory

span, and head size. International Journal of Psychophysiology, 9(3):237–248, 1990.

Bibliography 67

[132] Pavel Pudil, Jana Novovicova, and Josef Kittler. Floating search methods in feature selection.

Pattern recognition letters, 15(11):1119–1125, 1994.

[133] Dale Purves. Neuroscience. Scholarpedia, 4(8):7204, 2009.

[134] Rabie A Ramadan and Athanasios V Vasilakos. Brain computer interface: control signals review.

Neurocomputing, 223:26–44, 2017.

[135] Susana M Rivera, AL Reiss, Mark A Eckert, and Vinod Menon. Developmental changes in mental

arithmetic: evidence for increased functional specialization in the left inferior parietal cortex.

Cerebral cortex, 15(11):1779–1790, 2005.

[136] Ricardo Ron-Angevin, Francisco Velasco-Alvarez, Salvador Sancha-Ros, and Leandro da Silva-

Sauer. A two-class self-paced bci to control a robot in four directions. In 2011 IEEE International

Conference on Rehabilitation Robotics, pages 1–6. IEEE, 2011.

[137] SA Roset, HF Gonzalez, and JC Sanchez. Development of an eeg based reinforcement learning

brain-computer interface system for rehabilitation. In Conference proceedings:... Annual Interna-

tional Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in

Medicine and Biology Society. Conference, volume 2013, pages 1563–1566, 2013.

[138] Rudiger Rupp. Challenges in clinical applications of brain computer interfaces in individuals with

spinal cord injury. Frontiers in Neuroengineering, 7:38, 2014.

[139] Elnaz Banan Sadeghian and Mohammad Hassan Moradi. Continuous detection of motor imagery

in a four-class asynchronous bci. In Engineering in Medicine and Biology Society, 2007. EMBS

2007. 29th Annual International Conference of the IEEE, pages 3241–3244. IEEE, 2007.

[140] Andres F Salazar-Gomez, Joseph DelPreto, Stephanie Gil, Frank H Guenther, and Daniela Rus.

Correcting robot mistakes in real time using EEG signals. In 2017 IEEE International Conference

on Robotics and Automation (ICRA), pages 6570–6577. IEEE, 2017.

[141] Ben D Sawyer, Waldemar Karwowski, Petros Xanthopoulos, and PA Hancock. Detection of error-

related negativity in complex visual stimuli: a new neuroergonomic arrow in the practitioner’s

quiver. Ergonomics, 60(2):234–240, 2017.

[142] Gerwin Schalk, Jonathan R Wolpaw, Dennis J McFarland, and Gert Pfurtscheller. Eeg-based

communication: presence of an error potential. Clinical neurophysiology, 111(12):2138–2144, 2000.

[143] Reinhold Scherer, Josef Faller, Elisabeth VC Friedrich, Eloy Opisso, Ursula Costa, Andrea Kubler,

and Gernot R Muller-Putz. Individually adapted imagery improves brain-computer interface per-

formance in end-users with disability. PloS one, 10(5):e0123727, 2015.

[144] F Schettini, F Aloise, P Arico, S Salinari, D Mattia, and Febo Cincotti. Self-calibration algo-

rithm in an asynchronous p300-based brain–computer interface. Journal of neural engineering,

11(3):035004, 2014.

[145] Nico M Schmidt, Benjamin Blankertz, and Matthias S Treder. Online detection of error-related

potentials boosts the performance of mental typewriters. BMC neuroscience, 13(1):19, 2012.

Bibliography 68

[146] Larissa C Schudlo and Tom Chau. Towards a ternary NIRS-BCI: single-trial classification of verbal

fluency task, stroop task and unconstrained rest. Journal of neural engineering, 12(6):066008, 2015.

[147] Tanja Schultz, Michael Wand, Thomas Hueber, Dean J Krusienski, Christian Herff, and Jonathan S

Brumberg. Biosignal-based spoken communication: A survey. IEEE/ACM Transactions on Audio,

Speech, and Language Processing, 25(12):2257–2271, 2017.

[148] Alborz Rezazadeh Sereshkeh, Robert Trott, Aurelien Bricout, and Tom Chau. Eeg classification

of covert speech using regularized neural networks. IEEE/ACM Transactions on Audio, Speech,

and Language Processing, 25(12):2292–2300, 2017.

[149] Alborz Rezazadeh Sereshkeh, Robert Trott, Aurelien Bricout, and Tom Chau. Online eeg classi-

fication of covert speech for brain–computer interfacing. International journal of neural systems,

27(08):1750033, 2017.

[150] Alborz Rezazadeh Sereshkeh, Rozhin Yousefi, Andrew T Wong, and Tom Chau. Online classifi-

cation of imagined speech using functional near-infrared spectroscopy signals. Journal of neural

engineering, 16(1):016005, 2018.

[151] Stefano Silvoni, Ander Ramos-Murguialday, Marianna Cavinato, Chiara Volpato, Giulia Cisotto,

Andrea Turolla, Francesco Piccione, and Niels Birbaumer. Brain-computer interface in stroke: a

review of progress. Clinical EEG and Neuroscience, 42(4):245–252, 2011.

[152] H Moriah Sokolowski, Wim Fias, Chuka Bosah Ononye, and Daniel Ansari. Are numbers grounded

in a general magnitude processing system? a functional neuroimaging meta-analysis. Neuropsy-

chologia, 105:50–69, 2017.

[153] Teodoro Solis-Escalante, Gernot Muller-Putz, Clemens Brunner, Vera Kaiser, and Gert

Pfurtscheller. Analysis of sensorimotor rhythms for the implementation of a brain switch for

healthy subjects. Biomedical Signal Processing and Control, 5(1):15–20, 2010.

[154] YoungJae Song and Francisco Sepulveda. A novel onset detection technique for brain–computer

interfaces using sound-production related cognitive tasks in simulated-online system. Journal of

neural engineering, 14(1):016019, 2017.

[155] Martin Spuler, Michael Bensch, Sonja Kleih, Wolfgang Rosenstiel, Martin Bogdan, and Andrea

Kubler. Online use of error-related potentials in healthy users and people with severe motor

impairment increases performance of a p300-bci. Clinical Neurophysiology, 123(7):1328–1337, 2012.

[156] Martin Spuler and Christian Niethammer. Error-related potentials during continuous feedback:

using eeg to detect errors of different type and severity. Frontiers in human neuroscience, 9:155,

2015.

[157] Martin Spuler, Wolfgang Rosenstiel, and Martin Bogdan. Online adaptation of a c-vep brain-

computer interface (bci) based on error-related potentials and unsupervised learning. PloS one,

7(12):e51077, 2012.

[158] Georgios A Tagaris, Wolfgang Richter, Seong-Gi Kim, Giuseppe Pellizzer, Peter Andersen, Kamil

Ugurbil, and Apostolos P Georgopoulos. Functional magnetic resonance imaging of mental rotation

Bibliography 69

and memory scanning: a multidimensional scaling analysis of brain activation patterns. Brain

Research Reviews, 26(2-3):106–112, 1998.

[159] Hiromu Takahashi, Tomohiro Yoshikawa, and Takeshi Furuhashi. Reliability-based automatic

repeat request with error potential-based error correction for improving p300 speller performance.

In International Conference on Neural Information Processing, pages 50–57. Springer, 2010.

[160] Barbara Tomasino and Raffaella I Rumiati. Effects of strategies on mental rotation and hemispheric

lateralization: neuropsychological evidence. Journal of cognitive neuroscience, 16(5):878–888, 2004.

[161] George Townsend, Bernhard Graimann, and Gert Pfurtscheller. Continuous eeg classification

during motor imagery-simulation of an asynchronous bci. IEEE Transactions on Neural Systems

and Rehabilitation Engineering, 12(2):258–265, 2004.

[162] Chun Sing Louis Tsui, John Q Gan, and Huosheng Hu. A self-paced motor imagery based brain-

computer interface for robotic wheelchair control. Clinical EEG and neuroscience, 42(4):225–229,

2011.

[163] Markus Ullsperger and D Yves Von Cramon. Subprocesses of performance monitoring: a dis-

sociation of error processing and response competition revealed by event-related fmri and erps.

Neuroimage, 14(6):1387–1401, 2001.

[164] Marcel Van Gerven, Jason Farquhar, Rebecca Schaefer, Rutger Vlek, Jeroen Geuze, Anton Nijholt,

Nick Ramsey, Pim Haselager, Louis Vuurpijl, Stan Gielen, et al. The brain–computer interface

cycle. Journal of neural engineering, 6(4):041001, 2009.

[165] Hein T van Schie, Rogier B Mars, Michael GH Coles, and Harold Bekkering. Modulation of activity

in medial frontal and motor cortices during error observation. Nature neuroscience, 7(5):549, 2004.

[166] A Villringer, J Planck, C Hock, L Schleinkofer, and U Dirnagl. Near infrared spectroscopy (nirs):

a new tool to study hemodynamic changes during activation of brain function in human adults.

Neuroscience letters, 154(1-2):101–4, May 1993.

[167] A Vuckovic and F Sepulveda. A four-class bci based on motor imagination of the right and

the left hand wrist. In Applied Sciences on Biomedical and Communication Technologies, 2008.

ISABEL’08. First International Symposium on, pages 1–4. IEEE, 2008.

[168] Li Wang, Xiong Zhang, Xuefei Zhong, and Yu Zhang. Analysis and classification of speech imagery

eeg for bci. Biomedical signal processing and control, 8(6):901–908, 2013.

[169] Po T Wang, Christine E King, Luis A Chui, An H Do, and Zoran Nenadic. Self-paced brain–

computer interface control of ambulation in a virtual reality environment. Journal of neural

engineering, 9(5):056016, 2012.

[170] Nikolaus Weiskopf, Klaus Mathiak, Simon W Bock, Frank Scharnowski, Ralf Veit, Wolfgang

Grodd, Rainer Goebel, and Niels Birbaumer. Principles of a brain-computer interface (bci) based

on real-time functional magnetic resonance imaging (fmri). IEEE transactions on bio-medical

engineering, 51(6):966–70, Jun 2004.

Bibliography 70

[171] Jonathan R Wolpaw, Niels Birbaumer, William J Heetderks, Dennis J McFarland, P Hunter Peck-

ham, Gerwin Schalk, Emanuel Donchin, Louis A Quatrano, Charles J Robinson, and Theresa M

Vaughan. Brain-computer interface technology: a review of the first international meeting. IEEE

transactions on rehabilitation engineering, 8(2):164–173, 2000.

[172] Jonathan R Wolpaw, Niels Birbaumer, Dennis J McFarland, Gert Pfurtscheller, and Theresa M

Vaughan. Brain–computer interfaces for communication and control. Clinical neurophysiology,

113(6):767–791, 2002.

[173] Jonathan R Wolpaw and Dennis J McFarland. Control of a two-dimensional movement signal by a

noninvasive brain-computer interface in humans. Proceedings of the national academy of sciences,

101(51):17849–17854, 2004.

[174] Erwei Yin, Timothy Zeyl, Rami Saab, Tom Chau, Dewen Hu, and Zongtan Zhou. A hybrid brain–

computer interface based on the fusion of p300 and ssvep scores. IEEE Transactions on Neural

Systems and Rehabilitation Engineering, 23(4):693–701, 2015.

[175] Erwei Yin, Timothy Zeyl, Rami Saab, Dewen Hu, Zongtan Zhou, and Tom Chau. An auditory-

tactile visual saccade-independent p300 brain–computer interface. International journal of neural

systems, 26(01):1650001, 2016.

[176] Rozhin Yousefi, Alborz Rezazadeh Sereshkeh, and Tom Chau. Exploiting error-related potentials

in cognitive task based BCI. Biomedical Physics & Engineering Express, 5(1):015023, 2019.

[177] Lei Yu and Huan Liu. Feature selection for high-dimensional data: A fast correlation-based filter

solution. In Proceedings of the 20th international conference on machine learning (ICML-03),

pages 856–863, 2003.

[178] Yang Yu, Yadong Liu, Jun Jiang, Erwei Yin, Zongtan Zhou, and Dewen Hu. An asynchronous

control paradigm based on sequential motor imagery and its application in wheelchair navigation.

IEEE Transactions on Neural Systems and Rehabilitation Engineering, 26(12):2367–2375, 2018.

[179] Jeffrey M Zacks. Neuroimaging studies of mental rotation: a meta-analysis and review. Journal

of cognitive neuroscience, 20(1):1–19, 2008.

[180] Thorsten O Zander and Christian Kothe. Towards passive brain–computer interfaces: applying

brain–computer interface technology to human–machine systems in general. Journal of neural

engineering, 8(2):025005, 2011.

[181] Elias Abou Zeid, Alborz Rezazadeh Sereshkeh, and Tom Chau. A pipeline of spatio-temporal

filtering for predicting the laterality of self-initiated fine movements from single trial readiness

potentials. Journal of Neural Engineering, 13(6):066012, 2016.

[182] Timothy Zeyl. Adaptive brain-computer interfacing through error-related potential detection. PhD

thesis, University of Toronto, 2016.

[183] Timothy Zeyl, Erwei Yin, Michelle Keightley, and Tom Chau. Adding real-time bayesian ranks to

error-related potential scores improves error detection and auto-correction in a p300 speller. IEEE

Transactions on Neural Systems and Rehabilitation Engineering, 24(1):46–56, 2016.

Bibliography 71

[184] Timothy Zeyl, Erwei Yin, Michelle Keightley, and Tom Chau. Improving bit rate in an auditory

bci: Exploiting error-related potentials. Brain-Computer Interfaces, 3(2):75–87, 2016.

[185] Timothy Zeyl, Erwei Yin, Michelle Keightley, and Tom Chau. Partially supervised p300 speller

adaptation for eventual stimulus timing optimization: target confidence is superior to error-related

potential score as an uncertain label. Journal of neural engineering, 13(2):026008, 2016.

[186] Li Zhang, Wei He, Chuanhong He, and Ping Wang. Improving mental task classification by adding

high frequency band information. Journal of medical systems, 34(1):51–60, 2010.

[187] Danhua Zhu, Jordi Bieger, Gary Garcia Molina, and Ronald M Aarts. A survey of stimulation

methods used in ssvep-based bcis. Computational intelligence and neuroscience, 2010:1, 2010.

towards development of a robust asynchronous eeg-bci using ... · accuracy (+7%) and the...

Documents