![Page 1: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/1.jpg)
Brain Computer Interface
for Virtual Reality Control
Christoph Guger
![Page 2: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/2.jpg)
www.gtec.at
GRAZ
Mozart
VIENNA
g.tec
MOZART
Emperor„s castle
Musical
Empress
Elisabeth
![Page 3: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/3.jpg)
www.gtec.at
Research Projects
#) EC project: ReNaChip - Synthetic system integration
Rehabilitation of a discrete sensory motor learning function by a
#) EC project: Sm4all – Smart Home for all
Brain-Computer Interface for smart home control
#) EC project: RGS – Rehabilitation Gaming System
faster recovery from stroke with games
#) EC project: BrainAble
BCI with VR and social networks
#) EC project: Decoder
BCI for locked in patients
#) EC project: CSI - Central Nervous System Imaging
#) EC project: BETTER
BCI for Stroke rehabilitation and rehabilitation robots
#) EC project: VERE –Virtual Embodiment Real Embodiment
Dissolving the boundary between the human body
and surrogate representation in virtual and physical reality.
#) EC project: ALIAS – Adaptable Ambient Living Assistant
Robot system interacting with elderly people providing
cognitive assistance and social interaction and inclusion
![Page 4: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/4.jpg)
www.gtec.at
Our co-operations partners
• University of Barcelona, Spain
Mel Slater, Chris Groenegress
• IDIBAPS, Barcelona, Spain
Mavi Sanchez-Vives
•University College London (UCL), UK
Anthony Steed, Angus Antely,
• University of Technology Graz, Austria
Robert Leeb, Gert Pfurtscheller
• Wadsworth Center, New York, USA
Gerwin Schalk, Eric Sellers
• Tel Aviv University
Matti Mintz
![Page 5: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/5.jpg)
Brain-Computer-Interface (BCI)
“A system for controlling a device e.g. computer, wheelchair or a
neuroprothesis by human intention which does not depend on the
brain‟s normal output pathways of peripheral nerves and muscles”
[Wolpaw et al., 2002].
HCI – Human Computer Interface
DBI – Direct Brain Interface (University of Michigan)
TTD – Thought Translation Device (University of Tübingen)
www.gtec.at
Subject/
Patient
Brain-
Computer
Interface
Device
Feedback
EEG/
ECoG control signal
![Page 6: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/6.jpg)
Some examples of BCI applications
BCI
BCI_
![Page 7: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/7.jpg)
www.gtec.at
MATLAB and Simulink environment
User-System #1
Subject,Patient
Real-time system
Real-time blockset
Custom featureextraction andclassification
MATLAB/Simulink
DAQ board16-32 channel
Biosignal amplifier
Custom hardwaree.g. orthosis
Stimulation unit
User-System #2Personal Area Network (PAN)control unit
biosignals
feedback
![Page 8: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/8.jpg)
technical issues
Influencing components
adaptation
to subject
www.gtec.at
![Page 9: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/9.jpg)
How to record brain activity for BCI ?
Functional imaging techniques: FMRI, SPECT, PET
Magnetencephalogram (MEG, SQUID)
Near Infrared Spectroscopy (NIRS, fNIR)
Electrocorticogram (ECoG)
Electroencephalogram (EEG)
www.gtec.at
![Page 10: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/10.jpg)
Measuring brain electrical activity
Electro-corticogram (ECoG)
closely spaced multi-electrode grids or
strips applied directly to the cortical
surface, electrode diameter ~ 4mm, up
to 500 µV, 1 – 100 Hz
high signal-to-noise ratio, high spacial
and temporal resolution
highly invasive and limited study
opportunities
modified from University of Michigan
www.gtec.at
Electroencephalogram (EEG)
1 – 64 (128) channels, 1 µV – 100 µV, DC up to mV-range, 0 – 40 Hz, low signal-to-noise
ratio, moderate spacial resolution, high temporal resolution
Surface electrodes: 8 ...12 mm, mounted with conductive gel/paste
![Page 11: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/11.jpg)
www.gtec.at
Changes of brain electrical activity and mental
strategies
- Slow cortical potentials (anticipation tasks)
DC-derivation, artifact problem, difficult strategy, feedback method
- Steady-State Evoked potentials (SSVEP, SSSEP)
Flickering light with specific freuqency
- Event-related, non-phase-locked changes of oscillatory activity
ERD/ERS (motor imagery tasks)
Changes of mu-rhythm, alpha activity and beta activity over sensorimotor areas,
imagination of hand- ,foot-, tongue- movements
- Evoked potentials (focus on attention task)
Thalamic gating, various methods of stimulation (visual, tactile, electrical, auditory, ...),
P300
![Page 12: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/12.jpg)
Communication for the ´locked-in´
ALS patient in Germany using a BCI system for communication
Birbaumer, Kübler, Hinterberger,… Tübingen www.gtec.at
![Page 13: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/13.jpg)
www.gtec.at
Changes of brain electrical activity and mental
strategies
- Slow cortical potentials (anticipation tasks)
DC-derivation, artifact problem, difficult strategy, feedback method
- Steady-State Evoked potentials (SSVEP, SSSEP)
Flickering light with specific freuqency
- Event-related, non-phase-locked changes of oscillatory activity
ERD/ERS (motor imagery tasks)
Changes of mu-rhythm, alpha activity and beta activity over sensorimotor areas,
imagination of hand- ,foot-, tongue- movements
- Evoked potentials (focus on attention task)
Thalamic gating, various methods of stimulation (visual, tactile, electrical, auditory, ...),
P300
![Page 14: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/14.jpg)
Physiological Background – why does it work
Imagination of hand movement causes an ERD which is used to classify
the side of movement. The desynchronization occurs in motor and related
areas of the brain. Therefore, for analyzing and classifying ERD-patterns
the electrodes must be placed close to sensorimotor areas.
Left hand
movementRight hand
movement
www.gtec.at
C4
GND
REF
RIGHT
C3
![Page 15: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/15.jpg)
The ``Finger Movement Task``
Brisk movement of right index finger
C3
www.gtec.at
![Page 16: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/16.jpg)
Oscillatory Activity
Paradigm for a simple motor imagery BCI experiment
left
right
Fixation cross CUE
0 2 5 6
beep
7431
motor imagery
8 s
Offline data classification
Recording of 40 trials minimumTRAINING
![Page 17: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/17.jpg)
left
right
Fixation cross CUE
0 2 5 6
beep
7431
FB
8 s
feedback (FB)
classifier
Oscillatory Activity
Paradigm for a simple motor imagery BCI experiment
FEEDBACK
![Page 18: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/18.jpg)
www.gtec.at
Patient Tom (C4/C5 lesion)
![Page 19: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/19.jpg)
Case study: Electrocorticograms, ECoG
www.gtec.atWith permission from Schalk 2007
Direct Brain Interface, rhythmic activity, Albany, USA
www.gtec.at
![Page 20: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/20.jpg)
With permission from Schalk 2007
www.gtec.at
![Page 21: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/21.jpg)
Augmentation of neuronal
population activity using a brain-
computer interface
g.tec's Spike & ECoG Workshop
November 14th, 2010
Kai J. MillerPhysics, Medicine Neurobiology and Behavior, Neural Systems LabUniversity of Washington
![Page 22: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/22.jpg)
Basic spectral changes with movement
“ERD”
![Page 23: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/23.jpg)
Real-time representation of cortical activity
![Page 24: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/24.jpg)
![Page 25: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/25.jpg)
www.gtec.at
Active Electrodes
Advantages:
- No preparation (abrasion) of the skin required
- High signal quality and less 50/60 Hz noise with high impedance
- Reduced artifacts from electrode and cable movements
![Page 26: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/26.jpg)
www.gtec.at
Active versus passive electrodes
Active
Passive
![Page 27: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/27.jpg)
www.gtec.at
EYE MOVEMENTS
Channels closer to the eyes (1 and 4) show higher EOG artefacts than
central and occipital channels.
Both passive and active electrodes show a similar EOG contamination
which is also clear because both pick up the same source signal.
![Page 28: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/28.jpg)
www.gtec.at
BITING
Biting produces EMG contamination almost equally on all channels
No difference between active and passive electrodes because both pick
up the same source signal.
![Page 29: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/29.jpg)
www.gtec.at
CABLE ARTEFACTS
Cable artefacts are produced by touching or shaking the cables.
Active electrodes are almost unaffected while the passive electrodes
show large movement artefacts.
![Page 30: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/30.jpg)
www.gtec.at
ACTIVE HEAD MOVEMENTS
Active head movements produce fewer artefacts with active electrodes
compared to passive ones.
Artefacts for both electrodes can occur because of skin-electrode
movements.
Passive electrodes are mostly affected by the cable movements initiated
by the head movements.
![Page 31: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/31.jpg)
www.gtec.at
PASSIVE HEAD MOVEMENTS
Passive head movements have lower accelerations than active head
movements and therefore the artefacts are smaller and mostly visible
with passive electrodes.
![Page 32: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/32.jpg)
www.gtec.at
BCI AND DRY ELECTRODES
BCI use P300, motor imagery or steady-state visually evoked potentials
(SSVEP) measured with the electroencephalogram (EEG) to control
external devices.
Evoked potentials,
event-related desynchronization,
power spectrum
accuracies
were calculated for dry and gel based electrodes to compare them.
![Page 33: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/33.jpg)
www.gtec.at
![Page 34: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/34.jpg)
www.gtec.at
![Page 35: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/35.jpg)
www.gtec.at
![Page 36: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/36.jpg)
www.gtec.at
![Page 37: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/37.jpg)
www.gtec.at
Discussion:
First dry electrode system that works for motor imagery, SSVEP and
P300 (same accuracies reached for all)
Whole frequency range available: 0.1-40 Hz
First dry electrode system that covers extended 10/20 system on frontal,
central, parietal and occipital sites
More low frequency components in the EEG spectrum below 3 Hz
Careful montage required and more sensitive to surrounding noise
Group studies submitted in March 2011 to Journal of Neural Engineering
and BCI conference in Graz 2011.
![Page 38: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/38.jpg)
Biosignal Analysis and Recording System in VE
• The recording system has to work in noisy environments
• CAVE system: creates a 3D Virtual World
TRIMENSION ReaCTor
3 back projected screens (3m x 2.2m)
1 floor screen projected by ceiling mounted projector
• 3D effect with shutter glasses www.gtec.at
![Page 39: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/39.jpg)
Movie: “Walking through a Virtual City by Thought”
![Page 40: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/40.jpg)
BCI Award 2010
www.gtec.at
![Page 41: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/41.jpg)
www.gtec.at
Categorization of the 10 nominees
Title Control signal Application
fMRI Spikes N200/
P300
SSVEP MI Stroke Spelling/
internet/
art
Algorithm
development
A high speed word spelling BCI
system based on code modulated
visual evoked potentials
X X
Motor imagery-based Brain-Computer
Interface robotic rehabilitation for
stroke
X X
An active auditory BCI for intention
expression in locked-in
X X
Brain-actuated Google search by using
motion onset VEP
X X
Brain Painting - "Paint your way out” X X
Thought Recognition with Semantic
Output Codes
X X
Predictive Spelling with a P300-based
BCI: Increasing Communication Rate
X X
Innovations in P300-based BCI
Stimulus PresentationMethods
X X
Operant conditioning to identify
independent, volitionally-controllable
patterns of neural activity
X X
Neurorehabilitation for Chronic-Phase
Stroke using a Brain-Machine
Interface
X X
Total 1 1 6 2 2 7 1
![Page 42: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/42.jpg)
www.gtec.at
Categorization of the 57 submitted projects
Property Percentage
(N=57)
Property Percentage
(N=57)
Real-time BCI 65.2 Stroke 7.0
Off-line
algorithms
17.5 Spelling 19.3
P300 29.8 Wheelchair/rob
ot
7.0
SSVEP 8.9 Internet/VR 8.8
Motor imagery 40.4 Control 17.5
EEG 75.4 Platform/Techn
ology
12.3
fMRI 3.5
ECoG 3.5
NIRS 1.8
![Page 43: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/43.jpg)
Stroke
![Page 44: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/44.jpg)
Potential Users Worldwide
Cerebral palsy – 16,000,000
Brainstem stroke – 10,000,000
Other stroke – 60,000,000
Spinal cord injury – 5,000,000
Postpolio syndrome – 7,000,000
Amyotrophic lateral sclerosis – 400,000
Multiple sclerosis – 2,000,000
Muscular dystrophy – 1,000,000
Guillain-Barre syndrome – 70,000
New York State Department of Health
Wadswor t h Cent er
![Page 45: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/45.jpg)
Brain Computer Interface (BCI) for
Stroke Rehabilitation – Virtual Reality
• BCI in combination with VR for stoke rehabilitation
– People can be motivated via VR to activate their motor cortex
• Simple bar feedback:
– Patients should try to increase the length of a virtual bar just
by motor imagery tasks
• Rehabilitation games (e.g. RGS – Rehabilitation Gaming System)
– Patients play games activated by the motor imagery BCI
– Patients can activate the virtual arm (picture: RGS project)
![Page 46: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/46.jpg)
Brain Computer Interface (BCI) for
Stroke Rehabilitation – Virtual Reality
• BCI - VR System setup:
EEG cap
www.gtec.at
![Page 47: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/47.jpg)
• Motor imagery-based Brain-Computer Interface robotic rehabilitation
for stroke (Cuntai Guan et al.)
www.gtec.at
Brain Computer Interface (BCI) for
Stroke Rehabilitation – Recent Results
![Page 48: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/48.jpg)
Brain Computer Interface (BCI) for
Stroke Rehabilitation – Recent Results
• Study for comparison between BCI guided rehabilitation and non
BCI guided rehabilitation (MANUS – manual user)
– 26 patients were recruited, and randomized into two groups
(15 in MANUS group, 11 in BCI group). The protocol of the
rehabilitation is summarized as follows
• Patients performed 4 weeks‟ rehabilitation training, 3
sessions per week, and each session lasted around 1
hour
– Clinical evaluation was done at the beginning of the training
(week 0), mid of the training (week 2), end of the training
(week 4) and following-up assessment (week 12).
www.gtec.at
![Page 49: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/49.jpg)
• Clinical evaluation measures various outcomes for patients
– Fugl-Meyer Assessment (FMA) score is reported here as it represents
an overall recovery of motor impairment.
– Patients in MANUS group performed 960 repetitions, while BCI group
only performed 160 repetitions.
www.gtec.at
Brain Computer Interface (BCI) for
Stroke Rehabilitation – Recent Results
![Page 50: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/50.jpg)
P300 – Approach (EEG)
A B C D E F
G H I J K L
M N O P Q R
S T U V W X
Y Z 0 1 2 3
4 5 6 7 8 9
A B C D E F
G H I J K L
M N O P Q R
S T U V W X
Y Z 0 1 2 3
4 5 6 7 8 9
visual stimulationTime (ms)
Amplitude (µV)
51
![Page 51: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/51.jpg)
www.gtec.at
The 6 x 6 matrix speller, single character flash
A B C D E F
G H I J K L
M N O P Q R
S T U V W X
Y Z 0 1 2 3
4 5 6 7 8 9
concentrate on „W“Individual character intensifies for 60ms with 10ms between each
intensification
![Page 52: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/52.jpg)
www.gtec.at
The 6 x 6 matrix speller, single character flash
A B C D E F
G H I J K L
M N O P Q R
S T U V W X
Y Z 0 1 2 3
4 5 6 7 8 9
![Page 53: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/53.jpg)
www.gtec.at
The 6 x 6 matrix speller, single character flash
A B C D E F
G H I J K L
M N O P Q R
S T U V W X
Y Z 0 1 2 3
4 5 6 7 8 9
![Page 54: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/54.jpg)
www.gtec.at
The 6 x 6 matrix speller, single character flash
A B C D E F
G H I J K L
M N O P Q R
S T U V W X
Y Z 0 1 2 3
4 5 6 7 8 9
![Page 55: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/55.jpg)
www.gtec.at
The 6 x 6 matrix speller, single character flash
A B C D E F
G H I J K L
M N O P Q R
S T U V W X
Y Z 0 1 2 3
4 5 6 7 8 9
![Page 56: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/56.jpg)
www.gtec.at
The 6 x 6 matrix speller, single character flash
Target:1
5 µV
Non-target: 1 µV
Letter W Presentation
NON Target
Target
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7-10
-8
-6
-4
-2
0
2
4
6
8
10
time [s]
[µV]
P300
www.gtec.at
![Page 57: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/57.jpg)
A B C D E F
G H I J K L
M N O P Q R
S T U V W X
Y Z 0 1 2 3
4 5 6 7 8 9
Rows- and Columns- Flashing
![Page 58: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/58.jpg)
A B C D E F
G H I J K L
M N O P Q R
S T U V W X
Y Z 0 1 2 3
4 5 6 7 8 9
Rows- and Columns- Flashing
![Page 59: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/59.jpg)
A B C D E F
G H I J K L
M N O P Q R
S T U V W X
Y Z 0 1 2 3
4 5 6 7 8 9
Rows- and Columns- Flashing
![Page 60: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/60.jpg)
A B C D E F
G H I J K L
M N O P Q R
S T U V W X
Y Z 0 1 2 3
4 5 6 7 8 9
Rows- and Columns- Flashing
![Page 61: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/61.jpg)
A B C D E F
G H I J K L
M N O P Q R
S T U V W X
Y Z 0 1 2 3
4 5 6 7 8 9
Rows- and Columns- Flashing
![Page 62: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/62.jpg)
www.gtec.at
Performing real-time BCI experiments
Hands on seminar
P300 based speller Video
![Page 63: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/63.jpg)
BCI goes
www.gtec.at
• BCI for social networking
• P300 based BCI – twitter interface
• Tasks:
• Definition of the users needs
reduced set of full functionality
• Adaptation of P300 interface
![Page 64: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/64.jpg)
www.gtec.at
Twitter Control
![Page 65: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/65.jpg)
www.gtec.at
Japanese version
www.gtec.at
![Page 66: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/66.jpg)
www.gtec.at
intendix - g.tec integrates Brain-Computer Interface (BCI)
technology into patients‟ everyday life
intendiX® is designed to be installed and operated by caregivers/patient‟s family at home
enables the user to sequentially select characters from a keyboard-like matrix on the screen
requires some training but most subjects can use intendiX® after only 10 minutes
a spelling rate of 5 to 10 characters per minute can be achieved by the majority of healthy users
the system can trigger an alarm, let the computer speak the written text, print out or copy the
text into an e-mail or to send commands to external devices
put on the cap, inject a drip of gel into each of the electrodes and start to spell!
intendiX® includes the active electrode system, a portable EEG acquisition system and a
notebook or netbook computer with the intendiX® software installed.
![Page 67: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/67.jpg)
www.gtec.at
Live Experiment:
Smart Home Control XVR and BCI
![Page 68: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/68.jpg)
To use the P300 potential for smart home control we developed:
Virtual Reality apartment in XVR (extreme VR) – open source package
to build VR
The Smart home needed several controllable elements such as TV,
music, windows, doors,...
Special icons to control all the devices inside the apartment
Portable BCI system to be inside the 3 D environment
Reliable and fast BCI system
Requirements
www.gtec.at
![Page 69: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/69.jpg)
Concept
www.gtec.at
![Page 70: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/70.jpg)
Concept
www.gtec.at
Subject
BCI-System
Electrodes and
biosignal amplifier
Signal processing
VR smart home
with controllable
elements
Head-tracker
UDP interface
Switch on the BCI
system Extract the EEG
information
Send commands
Translate BCI
commands
Highly immersive
feedback
![Page 71: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/71.jpg)
www.gtec.at
The Virtual Reality apartment
Designed by Chris Groenegress, Mel Slater
![Page 72: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/72.jpg)
www.gtec.at
The Virtual Reality apartment
Designed by Chris Groenegress, Mel Slater
![Page 73: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/73.jpg)
www.gtec.at
The Virtual Reality apartment
Designed by Chris Groenegress, Mel Slater
![Page 74: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/74.jpg)
www.gtec.at
Control matrix for smart home with special icons
Select music
![Page 75: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/75.jpg)
www.gtec.at
Goto specific position – The Beamer
![Page 76: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/76.jpg)
12 subjects, 8 EEG channels recorded
Fz, Cz, P3, Pz, P4, PO7, Oz, PO8
Referenced to right mastoid, grounded to the forehead
Data recorded with g.MOBIlab+ via Bluetooth
Fa = 256 Hz, bandpass 0.1 – 30 Hz
1st training run -> 7 icons per mask (7)
Application runs -> 23 icons
„Spelling Device“ Application
Single character flash experiment
Total of ~ 2.5 hours incl. electrode montage and instruction of the
subject
Study Design
www.gtec.at
![Page 77: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/77.jpg)
www.gtec.at
Control the Smart Home by your thoughts
In cooperation with the
Virtual Environments and Computer Graphics, UPC, Barcelona
![Page 78: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/78.jpg)
www.gtec.at
Results
- tested with 8, 4 and only 2 flashes per item
- best result: subject 8 with 100 % accuracy for 8 and 4 flashes
- worst result: subject 5 with only 30 % for only 2 flashes
Best and worst selection
![Page 79: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/79.jpg)
www.gtec.at
Comparison of different masks: accuracy and EPs
-> Accuracy depends on arrangement of characters, background,...
and not on number of icons flashing in the matrix
Best accuracy
Worst accuracy
11 µV max15 µV max 8 µV max
![Page 80: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/80.jpg)
The P300 works very well for spelling
This is based on 85 subjects so far !
Discussion
www.gtec.at
Why does the GoTo perform
worse?
![Page 81: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/81.jpg)
What makes the difference?
www.gtec.at
![Page 82: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/82.jpg)
www.gtec.at
Controlling Second Life with the BCI
Allows patients to take part in social networks
Control avatars just by thinking
Patients appear as healthy person
![Page 83: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/83.jpg)
www.gtec.at
Changes of brain electrical activity and mental
strategies
- Slow cortical potentials (anticipation tasks)
DC-derivation, artifact problem, difficult strategy, feedback method
- Steady-State Evoked potentials (SSVEP, SSSEP)
Flickering light with specific frequency
- Event-related, non-phase-locked changes of oscillatory activity
ERD/ERS (motor imagery tasks)
Changes of mu-rhythm, alpha activity and beta activity over sensorimotor areas,
imagination of hand- ,foot-, tongue- movements
- Evoked potentials (focus on attention task)
Thalamic gating, various methods of stimulation (visual, tactile, electrical, auditory, ...),
P300
![Page 84: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/84.jpg)
www.gtec.at
Steady-State Visual Evoked Potentials
(SSVEP)
Frequency of stimulation Brain response
0 ... 2 Hz transient (single) VEP
3 ... 5 Hz undefined response
6 ... 24 Hz SSVEP
![Page 85: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/85.jpg)
www.gtec.at
MethodologyVisually Evoked Potentials (VEP)
single VEP
![Page 86: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/86.jpg)
www.gtec.at
MethodologySteady State Visually Evoked Potentials
(SSVEP)
SSVEP
7
Hz
![Page 87: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/87.jpg)
Steady-State-Evoked Potentials
Higher
Frequency(e.g. 17 Hz)
Lower
Frequency(e.g. 14 Hz)
EEG Power Spectrum EEG Power Spectrum
![Page 88: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/88.jpg)
www.gtec.at
Steady-State Visual Evoked Potentials
(SSVEP)
up to 48 different frequencies possible!
A B
C D
E F
![Page 89: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/89.jpg)
Robot with video camera control
www.gtec.at
Saumitra Dasgupta, Mike Fanton, Jonathan Pham, Mike Willard
Faculty advisors: Deniz Erdogmus (BCI) and Bahram Shafai
(iRobot)
Equipment used: g.USBamp with g.Butterfly electrodes, iRobot
Four checkerboards flicker at:
13Hz (top left: means turn left)
11Hz (top right: means turn right)
9Hz (bottom left: means go forward)
7Hz (bottom right: means stop).
Robots sends video back from webcam via Skype to operator
Zero-order-hold fashion control - until a new/different command
comes the robot continues to perform the latest received
command.
Welch periodogram + SVM classifier of frequencies
![Page 90: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/90.jpg)
+_
less training
up to 48 classes
fast response
up to 40 ... 55 bit/min
reduced responses for
higher frequencies
annoying stimulation
the adaptation problem
SSVEP
![Page 91: Brain Computer Interface for Virtual Reality Control](https://reader030.vdocuments.us/reader030/viewer/2022032817/620619518c2f7b17300493e9/html5/thumbnails/91.jpg)
BCI Award 2010
www.gtec.at