The Brain Matters: A 3D Real-TimeVisualization to Examine Brain SourceActivation Leveraging Neurofeedback

Thomas Kosch1, Mariam Hassib1,2, Albrecht Schmidt1

1VIS, University of Stuttgart{firstname.lastname}@vis.uni-stuttgart.de

2Group for Media Informatics, University of Munich{mariam.hassib}@ifi.lmu.de

Figure 1: Left: Participant during the pilot study with the BCI andVR headset. Right: Real-time 3D visualization of the brain sourceactivation during the VR session.

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the Owner/Author. Copyright is held by the owner/author(s).CHI’16 Extended Abstracts, May 07-12, 2016, San Jose, CA, USAACM 978-1-4503-4082-3/16/05.http://dx.doi.org/10.1145/2851581.2892484

AbstractAs Brain-Computer Interfaces become available to the con-sumer market, this provides more opportunities in analyzingbrain activity in response to different external stimuli. Cur-rent output modalities often generate a lot data, such asan electroencephalogram which only displays electrodemeasurements. We introduce a three-dimensional real-timebrain data visualization based on the measured values re-ceived by a brain-computer interface. Instead of visualizingthe collected voltages by electrodes, we calculate a currentdensity distribution to estimate the origin of electrical sourcewhich is responsible for perceived values at electrodes. Un-derstanding where the centers of activation in the brain areallows to better understand the relationship between exter-nal stimuli and brain activity. This could be relevant in thecontext of information presentation for doctors to analyzepathological phenomena. A pilot study was conducted us-ing Virtual Reality as input stimulus. Results indicate visiblechanges in real-time regarding brain activation.

Author KeywordsBrain Data Visualization; Virtual Reality; Neurofeedback;Electroencephalogram

ACM Classification KeywordsH.1.2 [Models and Principles]: User/Machine Systems —Human information processing

IntroductionBrain-Computer Interfaces (BCIs) are a category of phys-iological sensors that are currently making their way intothe consumer market. Traditionally, they have been used inthe medical field to help patients with severe disabilities togain more control over their lives. Electroencephalography(EEG) signals from the brain, gathered through electrodesplaced on or inside the scalp, can be interpreted to allowpatients to move a wheelchair, a cursor [11, 13] or providefeedback to the user in what is called a neurofeedback loop.Neurofeedback training involves interpreting brain signals inreal time and providing the user with feedback about theirstate through different stimuli, for example, using Virtual Re-ality (VR) environments. Virtual Reality in combination withBCIs have been researched and used in several applica-tions ranging from treating brain damage [10], substanceabuse [6] to mindfulness training [1] (see Figure 1).Figure 2: An EEG displaying a

graph for each electrode. Theoutput increases linearly with thenumber of electrodes.

Using new, light-weight and less complex consumer BCI de-vices coupled with a VR system allows us to create adap-tive virtual environments that can change according to themeasured brain signals to increase or decrease activationin certain parts of the brain. This can especially help peo-ple with neuropsychiatric diseases (e.g. depression) andtheir doctors to investigate the brain activation in real-timeduring VR-based neurofeedback sessions. The desire toinvestigate the impact of VR scenes on the brain is high butbecomes complicated because of linearly increasing outputof an EEG over time (see Figure 2). Whereas the measuredbrain activity at certain electrodes is what drives the neu-rofeedback system, the origin of the electrical sources ofactivation inside the brain is also required to analyze whichparts of the brain are affected by the VR scene. The lowresolution electromagnetic tomography algorithm (LORETA)solves this problem for a sufficient number of electrodes ina three-dimensional space [9].

Our work presents a simple and understandable real-timevisualization of the mentioned LORETA algorithm. A sim-ulation is executed to show the correctness of the imple-mentation. Furthermore, a pilot study is conducted to showthat changes regarding brain activation can be observed inreal-time.

Related WorkResearch has already been done in the domain of inves-tigating the impact of VR on the brain and brain data vi-sualizations in general, which divides this section in twocategories: (1) The impact of virtual reality on the brain inmedical treatments and (2) currently existing brain data vi-sualizations.

Baumgartner et al. has shown that neurophysiologicalchanges can be found when using VR [3]. Increased brainactivity at the parietal and frontal lobe could be found. TheEEG analysis of the displayed VR scene was carried outusing LORETA. However, the output cannot be inspected inreal time and the implementation is not interactive. Grealyet al. performed a 4-week intervention program for treat-ing patients with traumatic brain injuries using VR [4]. Sig-nificant cognitive increases could be noticed in terms ofspeech, visual learning and reaction times. Improvementswere quantitatively measurable, but the affected brain partsby this training could not be observed at all. Hoffman usedfunctional magnetic resonance imaging (fMRI) to investi-gate perceived pain when treating burn damages in com-bination with and without VR [5]. The fMRI has shown thatdistraction delivered by VR helps to reduce the amount ofbrain activity in pain-related brain areas. The brain scanspresented in the study still lack of real-time capabilities toinvestigate the augmentation and impact of brain activationthroughout a VR session. A three-dimensional visualizationis presented by the project Glass Brain from the company

Neuroscape Lab1. It enables the visualization of source lo-calization in real-time with interactive features. However, theinner of the visualized brain is hard to investigate since it isfilled with a fixed solid color.

Overall, the presented brain visualization solutions lack ofinteractive features, real-time capabilities or suffer perfor-mance issues. In contrast our work shows a simple visualrepresentation of the activities in the brain which allow hu-man observer to understand the current brain activationin real-time. Currently we investigate this in two context:(1) Providing information to improve medical treatment ofbrain related issues, (2) assessment of reactions to visualstimuli in the context of human-computer interaction. In ourwork we propose a real-time implementation of LORETAincluding a three-dimensional representation enriched withinteractive features to allow more precise investigations ofbrain parts. As a proof of concept, we performed a pilotstudy with an adaptive VR scene which changes accordingto the measured brain activities to achieve a neurofeedbacksession. Changes in brain activation are investigated duringVR sessions in real-time as well as in post-hoc analysis.

Figure 3: Evaluated sLORETAimplementation. Top: Placement oftwo electrical sources in everybrain hemisphere. Bottom:Recognition of both electricalsources by the sLORETAimplementation. Green dotsrepresent electrodes.

SystemThe implementation of our algorithm uses neuromore Stu-dio2 as base platform. Together it builds a system allowingus to retrieve data measured by a BCI device and process-ing it with our implementation in real-time. Furthermore, weare able to create adaptive VR-based neurofeedback ses-sions based on the measured EEG data.

Localizing Electrical SourceWe use a multithreaded implementation of the standardizedLORETA (sLORETA) algorithm for electrical source local-

1www.neuroscapelab.com (last access 02-09-2016)2www.neuromore.com (last access 02-09-2016)

ization [8]. sLORETA allows precise source localization ina three-dimensional space with different settings regardingnoise minimization, voxel density and the usage of a headmodel. The input of the algorithm are current voltages ofextracranial scalp measurements received by the attachedelectrodes. sLORETA then calculates a density distributionthroughout a head and associates each voxel with a magni-tude which describes the current density power at the voxel.Voxels are color-coded depending on the calculated magni-tude ranging from blue (weak) over green (medium) to red(strong).

Visualizing ResultsAfter retrieving electrode measurements, the algorithm be-gins processing the data based on the chosen head modeland number of voxels. We have decided to use a sphericalhead model because it is easier to reproduce and compareresults from other research projects. The number of voxelscan be set during program runtime, while a higher numberof voxels increases the precision of the result at the costof performance. Interactive features like zoom and rotationare supported to ease the analysis of brain source activa-tion. The computation of the visualization algorithm is GPUaccelerated. In contrast to current systems we are able tovisualize the activated areas in real-time for the observer(doctor, researcher) to directly see the reaction inside thebrain.

EvaluationThe implementation is evaluated using the BESA simulatortool3 to verify the correctness of the results. The BESA sim-ulator provides the functionality to place electrical sourceinside a head. These can be exported as csv file, whichallows an import into the sLORETA implementation. The

3www.besa.de/downloads/besa-simulator (last access 02-09-2016)

evaluation showed that the algorithm is working correctlyand can be used in future user studies (see Figure 3).

Figure 4: OpenBCI mounted on a3D printed retainer.

Figure 5: Tunnel traversed duringthe VR session.

Input StimulusOur system uses VR as an input stimulus which enablesus to create controllable simulations of real world situationswhere the reaction of body and mind can be analyzed. Thesimulation can be changed based on the measured elec-trode values to match a VR scene to the cognitive load ofa user. Depending on the result of the EEG analysis, thedifficulty of tasks or environments can be modified for braintraining exercises, for example, to decrease frustration orboredom. Frequent training using VR with suitable param-eters can be used to increase overall brain activity. Treat-ment of neuropsychiatric diseases often aim to increasebrain activity in the frontal lobe, since decreased brain ac-tivity in these areas often are often seen in patients of de-pression, schizophrenia, substance abuse or obsessive-compulsive disorders [12].

StudyA pilot study with eight male participants was conductedto find visible changes of electrical source activation withinthe brain during a neurofeedback session that used VRto change the stimuli. The average age was 27.12 years(SD = 5.19). The participants were healthy and have notparticipated in a bio- or neurofeedback study before. Thestudy took approximately 40 minutes per participant.

ApparatusWe used an OpenBCI4 board with 16-channel support asEEG device for the study. OpenBCI and its electrodes aremounted on a 3D printed retainer which can be worn on thehead (see Figure 4). An Oculus Rift5 was used to display

4www.openbci.com (last access 02-09-2016)5www.oculus.com (last access 02-09-2016)

VR content. The visualization uses 8000 voxels to renderthe results.

MethodThe study is divided into a baseline task, a VR session anda repetition of the baseline task to see visible differencesbefore and after the VR session. The OpenBCI is contin-uously recording EEG data to enable post-hoc analysis.Changes in brain activation are observed in real-time.

The baseline task is conducted to visualize initial sourceactivation. The participant is asked to keep the eyes openfor three minutes, followed by three minutes of keeping theeyes closed. Participants were seated in a dimmed roomfacing a white wall. This is a common task to estimate initialelectrical brain source activation [2].

The VR session displays an infinite tunnel which is tra-versed by the participant for ten minutes (see Figure 5).The traversal speed is influenced by the activation mea-sured by the sum of three electrodes at the frontal lobe.High alpha band magnitudes at these electrodes cause anincreasing traversal speed while low voltages slowdown thetraversal speed. We told the participant to traverse the tun-nel as fast as possible. This training aims to increase brainactivation in the frontal lobe. To achieve this, a signal pro-cessing algorithm is constructed which focuses on frontallobe training (see Figure 7). The electrodes at the positionsF3, F4 and Fz are used to estimate brain activation at thefrontal lobe. The measured electrode values are summedup and processed in a fast Fourier transformation (FFT) toestimate frequencies in the range between 8 Hertz and 13Hertz. The estimated frequency is normalized to the inter-val [0; 1]. The normalized values are used to set the traver-sal speed of the VR tunnel, where 0 causes the slowesttraversal speed and 1 the fastest traversal speed.

Figure 6: Brain source localization of participant 1 during thebaseline task. Left: Brain activation during the eyes-opened taskcan be observed. Right: Brain activation during the eyes-closedtask, which shows higher brain activation.

In the last step, the baseline task is repeated again. This isrequired to detect differences in brain activation before andafter the VR session.

Figure 7: Flow chart of tunnelspeed estimation during the VRsession.

Preliminary ResultsResults have shown that significant electrical brain sourcechanges could be found within all participants. Figure 6shows brain activation of one participant during the eyes-opened and eyes-closed task. Higher brain activation isnoticed during the eyes-closed task. The most active partis the back of the brain which is responsible for visual pro-cessing.

Before and after conducting the VR session, a snapshot ofbrain activation is taken for comparison purposes. Figure8 shows clear differences regarding brain activation be-fore and after conducting the VR session. The brain showshigher brain activation before the session than the base-line task, which could be the effect of excitement since theparticipants have not participated in a VR related neuro-

Figure 8: Brain source localization of participant 1 before andafter conducting the VR session. Left: Localized brain activationbefore the session. Right: Localized brain activation after the VRsession.

feedback session before. After conducting the VR session,higher brain activation was measured. Repetition of thebaseline task shows higher brain activation than in the firstbaseline run (see Figure 9).

DisucssionInvestigating brain source localization has shown that changesare visible and measurable when using VR as stimulus. In-creased brain activity is observed when participants startedclosing their eyes. Previous research has shown that this isa natural effect [7]. The back of the brain is permanently ac-tive since it is responsible for visual processing. Closing theeyes pushes activity in other brain regions responsible forimagination and multi-sensory actions which explains en-hanced electrical source localization during the eyes-closedtask.

Considering the effect of VR as stimulus has shown in-creased brain activity in different brain regions. The cere-bellum and the frontal lobe have shown improvement inbrain activation. These regions are responsible for motion

Figure 9: Baseline task repetition of participant 1 after the VRsession. Left: Brain activation of the eyes-opened task. Right:Brain activation during the eyes-closed task.

sensing and motivation-related thinking. The changes inmotion and speed in the tunnel explain brain activation dif-ferences in the cerebellum, while excitement and the desireto speed up the tunnel could explain enhanced activationsin the frontal lobe.

The repetition of the baseline task has shown sustainablebrain activity after the VR session. This is usually not a pro-longed effect since permanent brain activation changesrequires extensive neurofeedback training. Neurofeedbacksessions have to be repeated several times for long termlasting effects.

The study has shown that real-time brain source localiza-tion can be used to understand the effect of stimuli on aneuropsychological level. Improved therapy of neuropsychi-atric diseases and tailored enhancements of neurofeedbacksessions are possible to increase treatment efficiency.

Conclusion and Future WorkWe present an interactive three-dimensional real-time im-plementation of the sLORETA algorithm which calculates adensity distribution to estimate the electrical origin of brainsource activation. The usage of VR in medical areas hasbeen investigated to estimate neuronal impacts on the brainusing EEG devices. Our implementation of sLORETA isused in a user study to find out if it is possible to see visiblechanges in brain activation during a VR session. The re-sults show that changes in real-time can be observed whenusing VR scenes as input stimulus. The outcomes can beused to optimize medical treatment methods or to identifyneuropsychiatric diseases.

Our visualization proves visible changes regarding brain ac-tivation for different stimuli. Using this as first step, we wantto find out if our visualization can be used to treat and iden-tify neuropsychiatric diseases. Related research shows thatneurofeedback can be used for treatment of brain relateddiseases like, Alzheimer, epilepsy or attention deficit hyper-activity disorders (ADHD). The usage of our brain visualiza-tion as analysis tool by doctors can help to identify upcom-ing diseases or track progression of existing diseases. Ad-ditionally we want to find out if it is possible to modify exist-ing brain activation patterns by using dynamically changingVR scenes to train the brain towards an optimal state. Thedynamic scenes should be able to change based on thevalues received by sLORETA and should not depend by thecurrent electrode measurements. This requires operational-ization of the results received by sLORETA. A quantitativeanalysis of all participants will be possible after substantiat-ing the values calculated by sLORETA.

AcknowledgmentsWe take this opportunity to express our thankfulness toBenjamin Jillich, Manuel Jerger and Patrick Hilsbos for pro-viding us neuromore Studio as their software base platform.We also want to to express our gratitude to Dr. Ashley E.Stewart, Dr. Deborah C. Mash and Matthew Seely for theirprovided knowledge regarding the pilot study.

References[1] L Anatole, Fabien Lotte, Richard B Reilly, Robert

Leeb, Michitaka Hirose, Mel Slater, and others.2008. Brain-computer interfaces, virtual reality, andvideogames. Computer 10 (2008), 66–72.

[2] Robert J Barry, Adam R Clarke, Stuart J Johnstone,Christopher A Magee, and Jacqueline A Rushby.2007. EEG differences between eyes-closed andeyes-open resting conditions. Clinical Neurophysiol-ogy 118, 12 (2007), 2765–2773.

[3] Thomas Baumgartner, Lilian Valko, Michaela Esslen,and Lutz Jäncke. 2006. Neural correlate of spatialpresence in an arousing and noninteractive virtual re-ality: an EEG and psychophysiology study. CyberPsy-chology & Behavior 9, 1 (2006), 30–45.

[4] Madeleine A Grealy, David A Johnson, and Simon KRushton. 1999. Improving cognitive function after braininjury: the use of exercise and virtual reality. Archivesof physical medicine and rehabilitation 80, 6 (1999),661–667.

[5] Hunter G Hoffman. 2004. Virtual-reality therapy. SCI-ENTIFIC AMERICAN-AMERICAN EDITION- 291(2004), 58–65.

[6] Sang Hoon Lee, Doug Hyun Han, Seman Oh, In Ky-oon Lyoo, Young Sik Lee, Perry F Renshaw, andScott E Lukas. 2009. Quantitative electroencephalo-graphic (qEEG) correlates of craving during virtual

reality therapy in alcohol-dependent patients. Phar-macology Biochemistry and Behavior 91, 3 (2009),393–397.

[7] Esther Marx, Angela Deutschländer, Thomas Stephan,Marianne Dieterich, Martin Wiesmann, and ThomasBrandt. 2004. Eyes open and eyes closed as restconditions: impact on brain activation patterns. Neu-roimage 21, 4 (2004), 1818–1824.

[8] Roberto Domingo Pascual-Marqui and others. 2002.Standardized low-resolution brain electromagnetictomography (sLORETA): technical details. MethodsFind Exp Clin Pharmacol 24, Suppl D (2002), 5–12.

[9] Roberto D Pascual-Marqui, Christoph M Michel, andDietrich Lehmann. 1994. Low resolution electromag-netic tomography: a new method for localizing electri-cal activity in the brain. International Journal of psy-chophysiology 18, 1 (1994), 49–65.

[10] F David Rose, Barbara M Brooks, and Albert A Rizzo.2005. Virtual reality in brain damage rehabilitation:review. CyberPsychology & Behavior 8, 3 (2005),241–262.

[11] Kazuo Tanaka, Kazuyuki Matsunaga, and Hua OWang. 2005. Electroencephalogram-based controlof an electric wheelchair. Robotics, IEEE Transactionson 21, 4 (2005), 762–766.

[12] Sibel Tekin and Jeffrey L Cummings. 2002. Frontal–subcortical neuronal circuits and clinical neuropsychia-try: an update. Journal of psychosomatic research 53,2 (2002), 647–654.

[13] Jonathan R Wolpaw, Dennis J McFarland, Gregory WNeat, and Catherine A Forneris. 1991. An EEG-basedbrain-computer interface for cursor control. Electroen-cephalography and clinical neurophysiology 78, 3(1991), 252–259.