chapter 23 combining bci and virtual reality : scouting virtual worlds
DESCRIPTION
Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds. Introduction. Before a BCI can be used for control purposes, several training sessions are necessary Operant conditioning Feed back, real-time changes to the user Machine learning - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/1.jpg)
Chapter 23
Combining BCI and Virtual Real-ity
: Scouting Virtual Worlds
![Page 2: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/2.jpg)
Before a BCI can be used for control pur-poses, several training sessions are neces-sary
◦ Operant conditioning Feed back, real-time changes to the user
◦ Machine learning Adaptive algorithms to detect brain patterns
Introduction
![Page 3: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/3.jpg)
A technology which allows a user to interact with a computer-simulated environment.
VR (Virtual Reality)
![Page 4: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/4.jpg)
VE (Virtual Environment)◦ Allow users
to be shielded from the outside world to be able to focus on the required mental task
VR (Virtual Reality)
![Page 5: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/5.jpg)
Then, Why VR for BCI?
Feedback
Visualization vs. VE feedback
Ron Angevin et al. (2004)◦ Control group (standard BCI
feedback) reacted faster◦ VR group achieved less error
![Page 6: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/6.jpg)
Basic principle◦ detection & classification of motor-imagery re-
lated EEG patterns
Sensorimotor rhythms are analyzed◦ C3, Cz and Cz
Graz-BCI
![Page 7: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/7.jpg)
VE◦ To let a user become immersed in a 3D scene
Cave◦ Multiprojection stereo-based head-tracked VE sys-
tem
Graz-BCI + VE
![Page 8: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/8.jpg)
3D virtual environment1. Creation of a 3D model of the scene
3D modeling Software Packages Performer Maya
2. Generation of a VR-application that controls and animates the modeled scene
Virtual Research V8 HMD 640 X 480 pixels, refresh rate 60Hz
Vrjuggler + single back-projected wall + shutter glasses
Cave-like system
Graz-BCI + VE
![Page 9: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/9.jpg)
BCI experiments◦ Require a subject in a sitting position
No positional information had to be considered
◦ Rotational information from the tracking system was ignored Rotation should be controlled by the BCI
Graz-BCI + VE
![Page 10: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/10.jpg)
Study 1Roation in a VE by Left- and Right-Hand Motor Imagery
![Page 11: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/11.jpg)
Imagination of left and right hand movement
Subjects◦ 2 male(23, 26 years old), 1 female(28 years old)◦ 7 months
Feedback conditions1. A standard horizontal bar graph on a desktop monitor2. Virtual conference room presented with an HMD3. Virtual pub populated with animated avatars (includ-
ing music and chatter of the avatars)
Experiment
![Page 12: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/12.jpg)
The order of feedback conditions◦ Bar graph → HMD → Cave → HMD → bar graph
Instruction◦ To imagine left or right hand movements depending
on an acoustic cue (single or double beep)
Control◦ Either the length and the orientation of the horizontal
bar graph (in case of the standard BCI feedback)◦ Rotation angle and direction within VR
Experiment
![Page 13: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/13.jpg)
During experiments◦ Cue at second 3◦ Feedback for 4s◦ Screen update (including rotation) 24 times/s◦ One run 40 trials
Experiment
![Page 14: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/14.jpg)
Results
![Page 15: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/15.jpg)
No difference between HMD and Cave
Performed well with VR than bar graph
Results
![Page 16: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/16.jpg)
Study 2Moving Forward in a Virtual Street by Foot Mortor Im-agery
![Page 17: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/17.jpg)
Imagination in the virtual street◦ Right hand movement: to stop◦ Foot movement to move (constant speed)◦ Walking distance is scored
CAM (Cumulative achieved mileage)
Male 23, 28 and 30 years old.
Experiment
![Page 18: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/18.jpg)
Results
![Page 19: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/19.jpg)
Study 3, 4Scouting through a Virtual Apartment
Asynchronous freeSpace Experi-ments
![Page 20: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/20.jpg)
Virtual apartment on a single back-projected stereoscopic wall
Subject could decide freely where to go along predefined pathways (through the corridors or rooms)◦ Turn right, left, or straight)
System automatically guided the subject to the next junction◦ Small map in the bottom right corner of the display
Experiment
![Page 21: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/21.jpg)
Results
![Page 22: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/22.jpg)
Using VR◦ High classification accuracy (low error rate) can be achieved.
Subjects felt more natural in VE compared with BCI experiments with standard feedback
Each subject preferred the Cave experiments to the HMD and both were favored over BCI session on a desktop PC
Motivation seems to improve the BCI performance, but too much excitement might also distract the subject
Despite distraction from auditory and moving visual stimuli in VE, motor imagery and its classification in the ongoing EEG is still possible
Conclusion
![Page 23: Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds](https://reader036.vdocuments.us/reader036/viewer/2022081505/56816724550346895ddbaf51/html5/thumbnails/23.jpg)
Thank you !