using a low-cost electroencephalograph for task classification in hci research johnny c. lee...

Using a Low-Cost Electroencephalograph for Task Classification in HCI Research

Johnny C. Lee

Carnegie Mellon University

Desney S. TanMicrosoft Research

UIST 2006, Montreux Switzerland

National Geographic, March 2005 NY Times Magazine, October 16, 2005

Brain-Computer Interfaces (BCI)

A direct technological interface between a brain and computer not requiring any motor output from the user

Example Conferences/Journals with BCI interests:Neural Information Processing Systems (NIPS)IEEE Transactions on Biomedical EngineeringIEEE Transactions on Neural Systems and Rehabilitation Engineering

Why is this relevant to UIST or HCI?

BCI research traditionally focuses on exploratory neuroscience and rehabilitation engineering.

Brain sensing could provide valuable data about:- engagement

- cognitive work load

- surprise

- satisfaction

- frustration

Potentially helpful toContext Sensitive or Evaluation Systems

Values of BCI Values of HCI

VS

To use any means necessary to demonstrate that brain-computer interaction is possible.

To use reasonable means to achieve a practical benefit to many users.

use equipment costing $100K to +$1 million USD

use highly invasive surgical procedures

require hours or days of operant conditioning

remove data from poor performing subjects

It is okay to:

use fairly affordable and accessible equipment

be safe for repeated and extended use

be usable without requiring significant user training

use data from all subjects to evaluate its performance

We’d like to:

Where do we start?

Brain Sensing/Imaging Technologies

MRI

CT

ECoG

SPECT

PET

MEG

fMRI

EROS/fNIR

EEG

Currently Impracticalfor HCI

- Safe, easy, no medical expertise

EEG – Electroencephalograph

the neurophysiological measurement of the electrical activity

of the brain by recording from electrodes placed on the scalp

(skipping the lower level neurophysiology)

- Measures the voltage difference between two locations on the scalp

- Only picks up gross, macroscopic, coordinated, and synchronized firing of neurons near the surface of the brain with perpendicular orientation to the scalp. (thus majority of activity is hidden)

Analogous to holding a thermometer up to the side of a PC case

EEG Devices

Manufacturer: BioSemiChannels: 64-128Cost: ~$30K USD

Manufacturer: EGI SystemsChannels: 128-512Cost: $100K-$250K USD

• Lowest cost FDA approved device• Designed for home and small clinical use.• Only $1500 USD• Specs:

– 2-channels– 8-bit at 4µV resolution– 256 samples/sec

• Has yet to be validated for BCI research work.• If it works, it lowers the entry bar for BCI research.

The Brainmaster

Validating the Device (and ourselves)

1. Validate the device

Can we get useful data from such a low-end device?

2. Validate ourselves

To explore this space, we must be able to collect our own data.

Validating the Device (and ourselves)

Keirn, Z., “A New Mode of Communication Between Man and His Surroundings”, IEEE Transactions on Biomedical Engineering, Vol. 37, No. 12, 1990.

• Data is available for download• Data has not been reproduced in the past 15 years

– Some computational BCI researchers have just used this data.– State of the art does is not a great deal better.

Reproducing the Keirn Data

We adapted procedure from Keirn to better control potential confounds.

3 tasks:• Rest (Baseline): Relaxation and clearing of mind• Math: Mental arithmetic, prompted with “7 times 3 8 5”• Rotation: Mentally rotate an object, prompted with “peacock”

Tasks from the original paper were designed to elicit hemispheric differences.

Experimental Procedure

User is instructed to keep eyes closed, minimize body movement, and not to vocalize part of the tasks.

For each task, a computer driven cue is given: Rest, Math, Rotate

Following Math and Rotate, the experimenter says either the math problem or object


Rest Math Rot

Rot Rest Math

Rest Rot Math

Rot Math Rest

Math Rest Rot

Math Rot Rest

session

task (14 seconds)

trial

Block design adapted from Kiern


Rest Math Rot

Rot Rest Math

Rest Rot Math

Rot Math Rest

Math Rest Rot

Math Rot Rest

Rest Math Rot

Rot Rest Math

Rest Rot Math

Rot Math Rest

Math Rest Rot

Math Rot Rest

Rest Math Rot

Rot Rest Math

Rest Rot Math

Rot Math Rest

Math Rest Rot

Math Rot Rest

3 sessions per subject

Many short tasks prevent correlation with EEG drift


Rest Math Rot

Rot Rest Math

Rest Rot Math

Rot Math Rest

Math Rest Rot

Math Rot Rest

Rest Math Rot

Rot Rest Math

Rest Rot Math

Rot Math Rest

Math Rest Rot

Math Rot Rest

Rest Math Rot

Rot Rest Math

Rest Rot Math

Rot Math Rest

Math Rest Rot

Math Rot Rest

Subjects:8 subjects (3 female)29-58 years of ageAll were cognitively and neurologically healthyAll right handed

EEG Setup

International 10-20 EEG electrode placement system

Two channels placed on P3 and P4 with both references tied to Cz.

Electrodes are held in place using conductive paste.

5-10 minute preparation.

Processing the Data

Data Processing

14 secs

Rot

task (14 seconds)

Data Processing

14 secs

Task Cue

Rot

task (14 seconds)

Data Processing

14 secs

Experimenter Prompt

Rot

task (14 seconds)

Data Processing

14 secs

Task Onset

Rot

task (14 seconds)

Data Processing

14 secs

Performing Task

Rot

task (14 seconds)

Data Processing

14 secs

Performing Task

~4 secs

Rot

task (14 seconds)

Data Processing

10 secs

Performing Task

Rot

task (14 seconds)

Removing time for machine learning

Most machine learning algorithms don’t handle time series data very well.

0 500 1000 1500 2000 2500 3000-30

-20

-10

0

10

20

30

10 seconds


• Divide the 10 seconds into 2 sec windows that overlap by 1 sec• Perform signal processing on each of the 9 windows to get our

“time less” feature set

0 500 1000 1500 2000 2500 3000-30

-20

-10

0

10

20

30

2 secs


• Divide the 10 seconds into 2 sec windows that overlap by 1 sec• Perform signal processing on each of the 9 windows to get our

“time less” feature set

0 500 1000 1500 2000 2500 3000-30

-20

-10

0

10

20

30

2 secs

This provides486 windows

per participant

Signal features for each window

Generic signal features such as mean power, peak frequency, peak frequency amplitude, etc.

Features frequently used in EEG signal analysis.

Common EEG Features

0 500 1000 1500 2000 2500 3000-30

-20

-10

0

10

20

30

0 5 10 15 20 25 30 35 40 45 500

1000

2000

3000

4000

5000

Raw EEG

SpectralPower

Gamma (30Hz-50Hz)

Beta High(20Hz-30Hz)

Beta Low(12Hz-20Hz)

Alpha(8Hz-12Hz)

Theta(4Hz-8Hz)

Delta(1Hz-4Hz)

Feature Processing and Selection

The 39 base features from each window are mathematically combined to create 1521 total features.

We used a feature preparation and selection process similar to [Fogarty CHI’05] to reduce the number of features:

23 features for 3-task classification (486 examples)

16.4 features for pair-wise classification (324 examples)

Baseline Results – 3 cognitive tasks

3 taskMath v. Rotate

Rest v. Math

Rest v. Rotate

user 1 67.9% 83.3% 88.0% 85.8%

user 2 70.6% 82.7% 91.4% 84.3%

user 3 77.6% 88.3% 93.8% 86.7%

user 4 63.6% 69.4% 84.9% 86.7%

user 5 66.5% 91.0% 81.2% 80.9%

user 6 59.3% 80.6% 80.2% 68.5%

user 7 71.4% 87.3% 90.4% 86.7%

user 8 69.8% 87.7% 82.4% 83.6%

Average 68.3% 83.8% 86.5% 82.9%

BayesNet classifier

Chance: 33.3% 50% 50% 50%

0 500 1000 1500 2000 2500 3000-30

-20

-10

0

10

20

30

2 secs 86.5%68.3

%

83.8% 82.9%

We can do better…

0 500 1000 1500 2000 2500 3000-30

-20

-10

0

10

20

30

???

0 500 1000 1500 2000 2500 3000-30

-20

-10

0

10

20

30

Throwing time back in…

“Math”

We can average over temporally adjacent windows

to improve classification accuracy

Averaging with Task Transitions

Task transitions result in conflicting data in averaging window.

High density of transitions will result in lower accuracy.


Fewer task transitions will yield better classification accuracy.


No transitions and averaging over all data will be the even better.

Classification Accuracy with Averaging

3 tasks

Math v. Rot

Rest v. Rot

Rest v. Math

100%

95%

90%

85%

80%

75%

70%

65%

60%

55%

50%

Mean Classification Accuracy vs. Averaging Scenarios (Mental Tasks)

Baseline No Averaging

5 windows with transitions

5 windows no transitions


+5.1 to +15.7%for 3-tasks

Error bars represent standard deviation

So, can we really read minds?

3 tasks

Math v. Rot

Rest v. Rot

Rest v. Math

100%

95%

90%

85%

80%

75%

70%

65%

60%

55%

50%

Mean Classification Accuracy vs. Averaging Scenarios (Mental Tasks)






Possibly not, we might be really detecting subtle motor movements….

Cognitive/Motor “Fabric”

Does this matter to neuroscience? Yes

Does this matter to HCI? Maybe not

Tasks of varying cognitive difficultly are involuntarily coupled with physiological responses, such as minute imperceptible motor activity. [Kramer ’91]

Therefore, it is impossible to completely isolate cognitive activity neurologically intact individuals.

Cognitive/Motor “Fabric”

If motor artifacts are reliably correlated with different types of tasks or engagement, why not use those to help the classifier?

Requiring users to not move is also very impractical.

Non-Cognitive Artifacts detected by EEG:– Blinking– Eye movement– Head movement– Scalpal GSR– Jaw and facial EMG– Gross limb movements– Sensory Response Potentials

Experiment 2 – Game Task

To explore this idea of using non-cognitive artifacts to classify tasks using EEG, we chose a PC-based video game task.

Halo, a PC-based first person shooter game produced by Microsoft Game Studios.

Navigate a 3D environment in an effort to shoot opponents using various weapons.

Relatively high degree of interaction with mouse and keyboard input controls.

Game Tasks

Rest – baseline rest task, relax, fixate eyes on cross hairs on center of screen, do not interact with controls. Game elements do not interact with participant.

Solo – navigate environment, interact with elements in the scene, and collect ammunition. Opponent controlled by expert did not interact with participant.

Play – navigate environment and engage opponent controlled by expert. Expert instructed to play at a level just slightly above skill of participant to optimally challenge them.

Game Experimental Procedure

Setup, design, and procedure was similar to first study.

• Participants had tutorial and practice time with game controls.

• 3 tasks repeated 6 times (counterbalanced)

• Tasks were 24 seconds to allow navigation time.

• Only 2 sessions were run for each participant

• Same 8 participants from first study were run in this study.

• Same data preparation and machine learning procedure.

Results – Game Tasks

3 tasks

Rest v. Play

Rest v. Solo

Solo v. Play

100%

95%

90%

85%

80%

75%

70%

65%

60%

55%

50%

Mean Classification Accuracy vs. Averaging Scenarios (Game Tasks)






93.1%

Conclusion

This experimental design and data processing procedure can be applied to a much wider range of applications/tasks. Our two experiments were just two examples at different ends of a spectrum.

Compelling results can be achieved with low-cost equipment and without significant medical expertise or training.

Non-cognitive artifacts (inevitable in realistic computing scenarios) can be embraced improve classification power.

To make BCI relevant to HCI, we must challenge traditional assumptions and creatively work with its limitations.

Thanks!

Johnny Chung [email protected]

Desney [email protected]

Thanks to MSR and the VIBEGroup for supporting this work.

Cross-user Classifier

3 Cognitive Tasks

40.00%

45.00%

50.00%

55.00%

60.00%

65.00%

70.00%

75.00%

80.00%

85.00%

90.00%

95.00%

100.00%

Base 5 win 5win no edges Known

3class

MathRotate

RestRotate

RestMath

Cross-user Classifier

3 game tasks

50.00%

55.00%

60.00%

65.00%

70.00%

75.00%

80.00%

85.00%

90.00%

95.00%

100.00%

Base 5 win 5win no edges Known

3class

SoloPlay

RestPlay

RestSolo


MRI – only anatomical data

CT – only anatomical data

ECoG

SPECT

PET

MEG

fMRI

EROS/fNIR

EEG




ECoG – highly invasive surgery

SPECT – radiation exposure

PET – radiation exposure

MEG

fMRI

EROS/fNIR

EEG







MEG – extremely expensive

fMRI – extremely expensive

EROS/fNIR

EEG







MEG – extremely expensive

fMRI – extremely expensive

EROS/fNIR – currently expensive, still in infancy

EEG – safe, easy, no medical expertise

Other cool things you can do with an EEG device…

Event Related Potentials (ERP)

• Electrical activity related to or in response to the presentation of a stimulus

• Very well studied• Relatively robust• Used daily in clinical settings to check sensory

mechanisms, typically in infants

• Requires averaging over 30-100 windows synchronized with to see response.

ERP - AEP

ERP: Auditory Evoked Potential • Used in clinics/hospitals to check hearing.• Response to clicks in the ear

Bold Lines = no clicksThin Line = with clicksAEP response

AEP response

ERP - VEP

ERP: Visual Evoked Potential • Focusing on a flashing target, the visual cortex will “resonate” at the

stimulus frequency.

Stimulus Frequency

Harmonics

ERP – Auditory and Visual P300

• Well known/studied potential related to “attention” or “surprise”

• Presented with 2 stimuli and instructed to count one of the stimuli

• Positive response will follow the stimulus of interest.

Side note: EEG as ECG

ECG - Electrocardiogram • placing an electrode on the chest provides a clear measure of

cardiac activity.• translation to BPM is a simple autocorrelation

Heart beats

Single Beat period

0.4

µV

un

its

EEG as EMG

EMG - Electromyography • Measures muscular activity

Wrist relaxation (return to straight position)

Tension holding

Wrist inward contraction (toward inner forearm)

Wrist rest state

NOTE: The magnitude of the spikes seem to be proportional to the acceleration involved with the movement.

0.4

µV

un

its

EEG as Blink Detector

• Electrical activity due to muscle movement involved with eye blinks propagate through the head.

• Similarly, eye movements also affect the EEG recording

0 0.5 1 1.5 2 2.5

x 104

-200

-150

-100

-50

0

50

100

150

200

Blinks Blinks

Task Classification Background

• Previous work is split primarily into two camps:

Operant Conditioning Pattern Recognition




Human learns how the machine works

Machine learns how the human works




Human learns how the machine works

Machine learns how the human works

Relatively newEarly dabbling in the late-80’s

Most work has happened in last 5 years.

Pattern Recognition

Data Collection &

Experimental Design

Signal Processing &

Feature Generation

Machine Learning &

Improving Accuracy

EEG Setup

International 10-20 EEG electrode placement system

Two channels placed on P3 and P4 with both references tied to Cz. Locations selected based on pilot recordings.

Attaching electrodes: Prepare the site with a cleaner, use conductive paste to improve connection and hold electrode in place.

P Paste rinses out with water, non-toxic. 5-10 minute preparation.

using a low-cost electroencephalograph for task classification in hci research johnny c. lee...

Documents

data slide

seconds slide

usd slide

kiern slide

brainmaster slide

eeg drift slide

data processing

keirn data