full body spatial vibrotactile brain computer interface paradigm
TRANSCRIPT
![Page 1: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/1.jpg)
Full Body Spatial Vibrotactile Brain Computer Interface Paradigm
1
Full Body Spatial Vibrotactile Brain Computer Interface Paradigm
1
Takumi KodamaDepartment of Computer Science
Graduate School of System and Information Engineering Supervisor: Shoji Makino
![Page 2: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/2.jpg)
Introduction - What’s the BCI?
● Brain Computer Interface (BCI)○ Exploits user intentions ONLY using brain responses
2
![Page 3: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/3.jpg)
Introduction - ALS Patients
● Amyotrophic lateral sclerosis (ALS) patients○ Have difficulty to move their muscle by themselves○ BCI could be a communicating tool for them
3
…
… !
![Page 4: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/4.jpg)
Introduction - Research Approach
1, Stimulate touch sensories 2, Classify brain response
AB
A
B
3, Predict user thought
92.0% 43.3%
A B
TargetNon-Target
P300 brainwave response
4
● Tactile (Touch-based) P300-based BCI paradigm○ Predict user’s intentions by decoding P300 responses○ P300 responses are evoked by external (tactile) stimuli
![Page 5: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/5.jpg)
● Previous Tactile P300-based BCI paradigm○ Chest Tactile BCI (for around chest positions) [1]○ Tactile and auditory BCI (for head positions) [2]
Introduction - Previous Researches
5
[1] H. Mori, S. Makino, T. M. Rutkowski, Multi–command chest tactile brain computer interface for small vehicle robot navigation, 2013. [2] H. Mori, et al., “Multi-command tactile and auditory brain computer interface based on head position stimulation,” 2013.
![Page 6: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/6.jpg)
● Previous Tactile P300-based BCI paradigm○ Chest Tactile BCI (for around chest positions) [1]○ Tactile and auditory BCI (for head positions) [2]
Introduction - Previous Researches
6
[1] H. Mori, S. Makino, T. M. Rutkowski, Multi–command chest tactile brain computer interface for small vehicle robot navigation, 2013. [2] H. Mori, et al., “Multi-command tactile and auditory brain computer interface based on head position stimulation,” 2013.
Problems
1. Discrimination of each stimulus pattern2. Application for actual ALS patients
![Page 7: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/7.jpg)
1. Propose a new touch-based BCI paradigm intended for communicating with ALS patients
2. Confirm an effectiveness of the modality by improving stimulus pattern classification accuracies
Introduction - Research Purpose
7
![Page 8: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/8.jpg)
Method - Our Approach
8
● Full-body Tactile P300-based BCI (fbBCI)○ Applies six vibrotactile stimulus patterns to user’s back○ User can take experiment with their body lying down
![Page 9: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/9.jpg)
Method - Four fbBCI experiments
9
Ⅰ. Psychophysical
Ⅱ. EEG online
Ⅲ. SWLDA&SVM
Ⅳ. CNN
Online experiment
Offline experiment(Training one by one)
Offline experiment(Training altogether)
Pre experiment(Without ERP calculation)
![Page 10: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/10.jpg)
Method - Four fbBCI experiments
10
Ⅱ. EEG online
Ⅲ. SWLDA&SVM
Ⅳ. CNN
Online experiment
Offline experiment(Training one by one)
Offline experiment(Training altogether)
Pre experiment(Without ERP calculation)Ⅰ. Psychophysical
![Page 11: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/11.jpg)
Experiment Ⅰ - Psychophysical
11
● Main objective○ To evaluate the fbBCI
stimulus pattern feasibility
● How to ?○ Selecting target stimulus
with button pressing○ EEG electrodes were not
attached on user’s scalp
Button press
No EEG cap
Exciters
Targets presented
![Page 12: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/12.jpg)
Condition Details
Number of users (mean age) 10 (21.9 years old)
Stimulus frequency of exciters 40 Hz
Vibration stimulus length 100 ms
Inter-stimulus Interval (ISI) 400 ~ 430 ms
Number of trials 1 trial
Experiment Ⅰ - Psychophysical
12
● Experimental conditions
![Page 13: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/13.jpg)
Result Ⅰ - Psychophysical
● Correct rate exceeded 95% in each stimulus pattern
13
![Page 14: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/14.jpg)
Method - Four fbBCI experiments
14
Ⅰ. Psychophysical
Ⅱ. EEG online
Ⅲ. SWLDA&SVM
Ⅳ. CNN
Online experiment
Offline experiment(Training one by one)
Offline experiment(Training altogether)
Pre experiment(Without ERP calculation)
![Page 15: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/15.jpg)
Experiment Ⅱ - EEG online
15
● Main objective○ To reveal the fbBCI
classification accuracies● How to ?
○ Selecting target stimulus with ERP intervals
○ Are P300 responses present in ERPs?
EEG cap
EEG amplifier
Targets & Results presented
Exciters
![Page 16: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/16.jpg)
Experiment Ⅱ - EEG online
16
● Experimental conditionsCondition Details
Number of users (mean age) 10 (21.9 years old)
Stimulus frequency of exciters 40 Hz
Vibration stimulus length 100 ms
Inter-stimulus Interval (ISI) 400 ~ 430 ms
Number of trials 1 training + 5 tests
EEG sampling rate 512 Hz
Electrode channels Cz, Pz, C3, C4, P3, P4, CP5, CP6
Classification algorithm SWLDA with BCI2000
![Page 17: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/17.jpg)
● Grand mean ERP intervals in each electrode channel
Result Ⅱ - EEG online
17*Gray-shaded area … significant difference (p < 0.01) between targets and non-targets
![Page 18: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/18.jpg)
Result Ⅱ - EEG online
18
User No. Classification accuracy with SWLDA
1 23.33 %
2 50.0 %
3 43.33 %
4 66.67 %
5 66.67 %
6 53.33 %
7 30.0 %
8 33.33 %
9 93.33 %
10 76.67 %
Average. 53.67 %
![Page 19: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/19.jpg)
Method - Four fbBCI experiments
19
Ⅰ. Psychophysical
Ⅱ. EEG online
Ⅲ. SWLDA&SVM
Ⅳ. CNN
Online experiment
Offline experiment(Training one by one)
Offline experiment(Training altogether)
Pre experiment(Without ERP calculation)
![Page 20: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/20.jpg)
Exp. Ⅲ - Accuracy Refinement
20
● Main objective○ Improvement of classification accuracies
● How to?○ Accuracy comparison
■ Down-sampling (nd = 1, 4 and 16) ①■ Epoch averaging (ne = 1, 5 and 10) ①■ Machine learning algorithms (SWLDA & SVM) ②
① ②
![Page 21: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/21.jpg)
● SWLDA classification accuracies○ BEST: 57.48 % (nd = 4, ne = 1)
Result Ⅲ - Accuracy Refinement
21
Signal decimation (nd)
![Page 22: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/22.jpg)
● Linear SVM classification accuracies
○ BEST: 58.5 % (nd = 16, ne = 10)
Result Ⅲ - Accuracy Refinement
22
Signal decimation (nd)
![Page 23: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/23.jpg)
● Non-linear SVM classification accuracies
○ BEST: 59.83 % (nd = 4, ne = 1)
Result Ⅲ - Accuracy Refinement
23
Signal decimation (nd)
![Page 24: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/24.jpg)
Method - Four fbBCI experiments
24
Ⅰ. Psychophysical
Ⅱ. EEG online
Ⅲ. SWLDA&SVM
Ⅳ. CNN
Online experiment
Offline experiment(Training one by one)
Offline experiment(Training altogether)
Pre experiment(Without ERP calculation)
![Page 25: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/25.jpg)
● Main objective○ More improvement of classification accuracies ○ Achievement of non-training ERP classifications
● How to?○ Feature vectors were transformed into squared input
volume matrices (60 × 60) ⇒ next page○ Evaluate with the classifier model trained by other nine
participated user
Experiment Ⅳ - CNN application
25User 1
1
2 3 4
5 6 7
8 9 10
Classifier model
trained by user 2~10
ERP classification
![Page 26: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/26.jpg)
● Main objective○ More improvement of classification accuracies ○ Achievement of non-training ERP classifications
● How to?○ Feature vectors were transformed into squared input
volume matrices (60 × 60) ⇒ next page○ Evaluate with the classifier model trained by other nine
participated user
Experiment Ⅳ - CNN application
26User 10
10
1 2 3
4 5 6
7 8 9
trained by user 1~9
ERP classification
Classifier model
![Page 27: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/27.jpg)
Experiment Ⅳ - CNN application
27
1. ERP interval elements were deployed in a 20 × 20 squared matrix
2. Matrices generated in each electrode channel and mean of all electrodes were concatenated into a 3 × 3 grid
● Transform feature vectors to input volumes
![Page 28: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/28.jpg)
Experiment Ⅳ - CNN application
● Overview of CNN architecture in fbBCI○ CONV > POOL > CONV > POOL (LeNet)○ (Ix, Iy) … Size of the input volume○ (Ax, Ay) … Size of activation maps
28
MLP
![Page 29: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/29.jpg)
Result Ⅳ - CNN application
29
User No. Non-averaging (ne = 1) SMA
1 97.22 % 100 %
2 30.0 % 100 %
3 72.22 % 100 %
4 86.11 % 100 %
5 94.44 % 100 %
6 88.89 % 100 %
7 86.11 % 100 %
8 100.0 % 100 %
9 100.0 % 100 %
10 41.67 % 100 %
Average. 79.66 % 100 %
![Page 30: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/30.jpg)
● The validity of fbBCI paradigm was confirmed○ Ⅰ. Stimulus pattern correct rate > 95% manually○ Ⅱ. Classification accuracy : 53.67 % by SWLDA○ Ⅲ. 59.83 % by non-linear SVM (nd = 4, ne = 1)○ Ⅳ. 100 % by CNN with classifier model by all user
● To improve QoL for ALS patients with fbBCI in the future○ Conduct experiments in practical conditions○ Implementation of off-line methods to online ERP
classification environments● Hope the series of experimental results will contribute to
developments of tactile P300-based BCI paradigms
Conclusions
30
![Page 31: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/31.jpg)
Journal Article (Lead; 1)
31
1. T. Kodama, K. Shimizu, S. Makino and T.M. Rutkowski, "Comparison of P300--based Brain--computer Interface Classification Accuracy Refinement Methods using Full--body Tactile paradigm," Journal of Bionic Engineering, (invited; submitting), 2017. Invited
![Page 32: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/32.jpg)
1. T.M. Rutkowski, K. Shimizu, T. Kodama, P. Jurica and A. Cichocki, "Brain--robot Interfaces Using Spatial Tactile BCI Paradigms - Symbiotic Brain-robot Applications," in Symbiotic Interaction (vol. 9359 of Lecture Notes in Computer Science), B. Blankertz, G. Jacucci, L. Gamberini, A. Spagnolli and J. Freeman Eds., Springer International Publishing, pp. 132-137, Oct. 2015. doi: 10.1007/978-3-319-24917-9_14
Book chapter (Co; 1)
32
![Page 33: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/33.jpg)
1. T. Kodama, S. Makino and T.M. Rutkowski, "Spatial Tactile Brain-Computer Interface Paradigm Applying Vibration Stimuli to Large Areas of User’s Back," in Proc. the 6th International Brain-Computer Interface Conference, Graz University of Technology Publishing House, pp. Article ID: 032-1-4, Sep. 2014. doi:10.3217/978-3-85125-378-8-32
2. T. Kodama, S. Makino and T.M. Rutkowski, "Spatial Tactile Brain-Computer Interface by Applying Vibration to User’s Shoulders and Waist," in Proc. the 10th AEARU Workshop on Computer Science and Web Technologies (CSWT-2015), University of Tsukuba, pp. 41-42, Feb. 2015. Best Poster Award
3. T. Kodama, K. Shimizu and T.M. Rutkowski, "Full Body Spatial Tactile BCI for Direct Brain-robot Control," in Proc. the Sixth International Brain-Computer Interface Meeting: BCI Past, Present, and Future, Verlag der Technischen Universitaet Graz, pp. 68, May 2016. doi:10.3217/978-3-85125-467-9-68 Student Travel Award
Conference Papers (Lead; 1)
33
![Page 34: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/34.jpg)
4. T. Kodama, S. Makino and T.M. Rutkowski, "Toward a QoL improvement of ALS patients: Development of the Full-body P300-based Tactile Brain--Computer Interface," in Proc. the 2016 AEARU Young Researchers International Conference (AEARU YRIC-2016), University of Tsukuba, pp. 5-8, Sep. 2016.
5. T. Kodama, K. Shimizu, S. Makino and T.M. Rutkowski, "Full–body Tactile P300–based Brain–computer Interface Accuracy Refinement," in Proc. the International Conference on Bio-engineering for Smart Technologies (BioSMART 2016), IEEE Press, pp. 20–23, Dec. 2016. (Extended version invited to the Journal of Bionic Engineering) Best Paper Award Nomination
6. T. Kodama, S. Makino and T.M. Rutkowski, "Tactile Brain-Computer Interface Using Classification of P300 Responses Evoked by Full Body Spatial Vibrotactile Stimuli," in Proc. the Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC 2016), IEEE Press, pp. Article ID: 176, Dec. 2016.
Conference Papers (Lead; 2)
34
![Page 35: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/35.jpg)
7. T. Kodama and S. Makino, "Analysis of the brain activated distributions in response to full-body spatial vibrotactile stimuli using a tactile P300-based BCI paradigm," in Proc. the IEEE International Conference on Biomedical and Health Informatics 2017 (BHI-2017), IEEE Engineering in Medicine and Biology Society, pp. (accepted, in press), Feb. 2017.
8. T. Kodama and S. Makino, "Convolutional Neural Network Architecture and Input Volume Design for Analyzing Somatosensory ERP Signals Evoked by a Tactile P300-based Brain-Computer Interface," in Proc. the 39th Annual International Confernce of the IEEE Engineering in Medicine and Biology Society (EMBC 2017), IEEE Engineering in Medicine and Biology Society, pp. (scheduled), Jul. 2017.
Conference Papers (Lead; 3)
35
![Page 36: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/36.jpg)
1. T.M. Rutkowski, H. Mori, T. Kodama and H. Shinoda, "Airborne Ultrasonic Tactile Display Brain-computer Interface - A Small Robotic Arm Online Control Study," in Proc. the 10th AEARU Workshop on Computer Science and Web Technologies (CSWT-2015), University of Tsukuba, pp. 7-8, Feb. 2015.
2. K. Shimizu, T. Kodama, P. Jurica, A. Cichocki and T.M. Rutkowski, "Tactile BCI Paradigms for Robots' Control," in Proc. the 6th Conference on Systems Neuroscience and Rehabilitation (SNR 2015), National Rehabilitation Center for Persons with Disabilities, pp. 28, Mar. 2015.
3. T.M. Rutkowski, K. Shimizu,T. Kodama, P. Jurica, A. Cichocki and H. Shinoda, "Controlling a Robot with Tactile Brain-computer Interfaces," in Proc. the 38th Annual Meeting of the Japan Neuroscience Society (Neuroscience 2015), Japan Neuroscience Society, pp. 2P332, July 2015.
4. K. Shimizu , D. Aminaka , T. Kodama, C. Nakaizumi, P. Jurica, A. Cichocki, S. Makino and T.M. Rutkowski, "Brain-robot Interfaces Using Spatial Tactile and Visual BCI Paradigms - Brains Connecting to the Internet of Things Approach," in Proc. the International Conference on Brain Informatics & Health (BIH 2015), Imperial College London, pp.9-10, Sep. 2015.
Conference Papers (Co; 1)
36
![Page 37: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/37.jpg)
Conference Papers (Co; 2)
37
5. K. Shimizu, T. Kodama, S. Makino and T.M. Rutkowski, "Visual Motion Onset Virtual Reality Brain–computer Interface," in Proc. the International Conference on Bio-engineering for Smart Technologies 2016 (BioSMART 2016), IEEE Press, pp. 24-27, Dec. 2016.
![Page 38: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/38.jpg)
38
Many thanks for your attention!
![Page 39: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/39.jpg)
fbBCI demonstration
39https://www.youtube.com/watch?v=sn6OEBBKsPQ
![Page 40: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/40.jpg)
Result Ⅰ - Psychophysical
● Response time differences for each stimulus pattern
40
![Page 41: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/41.jpg)
● How to train the P300-based BCI classifier?○ Each stimulus pattern was given 10 times in random○ Altogether 360 (60×6) times for a classifier training
Experiment Ⅱ - EEG online
41
ω1 : Target
Classifier (2cls)
Target 1
1
2
345
6
1
6
5
4
3
2
ω2 : Non-Target
× 10
× 10
× 10
× 10
× 10
× 10Session: 1/6
![Page 42: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/42.jpg)
Experiment Ⅱ - EEG online
42
ω1 : Target
Classifier (2cls)
Target 2
1
2
345
6
1 × 102 × 10
Session: 2/6
6
5
4
3
2
ω2 : Non-Target× 20
× 20
× 20
× 20
× 10
1 × 10
● How to train the P300-based BCI classifier?○ Each stimulus pattern was given 10 times in random○ Altogether 360 (60×6) times for a classifier training
![Page 43: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/43.jpg)
Experiment Ⅱ - EEG online
43
ω1 : Target
Classifier (2cls)
Target 3
1
2
345
6 ω2 : Non-Target
Session: 3/6
1 × 102 × 10
6
5
4
3
2
× 30
× 30
× 30
× 20
× 20
1 × 20
3 × 10
● How to train the P300-based BCI classifier?○ Each stimulus pattern was given 10 times in random○ Altogether 360 (60×6) times for a classifier training
![Page 44: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/44.jpg)
Experiment Ⅱ - EEG online
44
ω1 : Target
Classifier (2cls)
Target 4
1
2
345
6 ω2 : Non-Target
Session: 4/6
1 × 102 × 10
6
5
4
3
2
× 40
× 40
× 30
× 30
× 30
1 × 30
3 × 104 × 10
● How to train the P300-based BCI classifier?○ Each stimulus pattern was given 10 times in random○ Altogether 360 (60×6) times for a classifier training
![Page 45: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/45.jpg)
Experiment Ⅱ - EEG online
45
ω1 : Target
Classifier (2cls)
Target 5
1
2
345
6 ω2 : Non-Target
Session: 5/6
1 × 102 × 10
6
5
4
3
2
× 50
× 40
× 40
× 40
× 40
1 × 40
3 × 104 × 105 × 10
● How to train the P300-based BCI classifier?○ Each stimulus pattern was given 10 times in random○ Altogether 360 (60×6) times for a classifier training
![Page 46: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/46.jpg)
Experiment Ⅱ - EEG online
46
ω1 : Target
Classifier (2cls)
Target 6
1
2
345
6 ω2 : Non-Target
Session: 6/6
1 × 102 × 10
6
5
4
3
2
× 50
× 50
× 50
× 50
× 50
1 × 50
3 × 104 × 105 × 106 × 10
60 300
● How to train the P300-based BCI classifier?○ Each stimulus pattern was given 10 times in random○ Altogether 360 (60×6) times for a classifier training
![Page 47: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/47.jpg)
Experiment Ⅱ - EEG online
● How to predict user’s intention with a trained classifier?○ Correct example
47
ω1 : Target
Classifier (2cls)
1 × 10
72.6 %
Target 1
Session: 1/6
ω1 : Target
Classifier (2cls)
2 × 10
24.4 %ω1 : Target
Classifier (2cls)
3 × 10
56.3 %ω1 : Target
Classifier (2cls)
4 × 10
44.1 %ω1 : Target
Classifier (2cls)
5 × 10
62.9 %ω1 : Target
Classifier (2cls)
6 × 10
39.8 %
1
2
345
6
![Page 48: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/48.jpg)
Experiment Ⅱ - EEG online
48
ω1 : Target
Classifier (2cls)
1 × 10
35.1 %
Target 6
Session: 6/6
ω1 : Target
Classifier (2cls)
2 × 10
48.1 %ω1 : Target
Classifier (2cls)
3 × 10
69.2 %ω1 : Target
Classifier (2cls)
4 × 10
54.3 %ω1 : Target
Classifier (2cls)
5 × 10
50.9 %ω1 : Target
Classifier (2cls)
6 × 10
64.3 %
1
2
345
6
● How to predict user’s intention with a trained classifier?○ Wrong example
![Page 49: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/49.jpg)
Experiment Ⅱ - EEG online
Target 11/6
5
Target 2
Target 3
3
5
● Calculate stimulus pattern classification accuracy○ How many user sessions could be classified with correct
targets?
Target 4
Target 5
Target 6
2
4
Result
1
Session
2/6
3/6
4/6
5/6
6/6
1 Trial
Classification accuracy rate:
4/6 = 0.667 ⇒ 66.7 %
Correct
Correct
Wrong
Correct
Correct
Wrong
Target Status
![Page 50: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/50.jpg)
50
● Event related potential (ERP) interval○ captures 800 ms long after vibrotactile stimulus onsets○ will be converted to feature vectors with their potentials
Lxi …
Ch○○
p1 pL
ex.) fs = 512 [Hz] nd = 4 tERP = 800 [ms] = 0.8 [sec] L = ceil((512/4)・0.8) = 103
L = ceil(( fs / nd )・tERP),where fs [Hz] , tERP [sec]
Experiment Ⅱ - EEG online
![Page 51: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/51.jpg)
Result Ⅱ - EEG online
51
● P300 peaks were shifted to later latencies from #1 to #6
#1 Left arm
#2 Right arm
#3 Shoulder
#4 Waist
#5 Left leg
#6 Right leg
![Page 52: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/52.jpg)
Result Ⅱ - EEG online
52
● Times series of the Target vs. Non-Target AUC scores
![Page 53: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/53.jpg)
Result Ⅱ - EEG online
53
● Information Transfer Rate (ITR)○ Averaged score: 1.31 bit/minute
![Page 54: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/54.jpg)
Result Ⅱ - EEG online
54
● Grand mean fbBCI classification accuracy: 53.67 %
![Page 55: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/55.jpg)
Exp. Ⅲ - Accuracy Refinement
● Architecture diagram of the off-line ERP classification
55
![Page 56: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/56.jpg)
Exp. Ⅲ - Accuracy Refinement
56
● Down-sampling (nd)○ ERPs were decimated by 2 (256
Hz), 4 (128 Hz), 8 (256 Hz), 16 (32 Hz) or kept intact (512 Hz)
○ To reduce a vector length L
nd = 4 (128 Hz) nd = 16 (32 Hz)
Ch○○ Ch○○
![Page 57: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/57.jpg)
57
● Epoch averaging (ne)○ ERPs were averaged using 2, 5,
10 ERPs or no averaging○ To cancel background noise
ne = 1 ne = 10
Ch○○ Ch○○
Exp. Ⅲ - Accuracy Refinement
![Page 58: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/58.jpg)
● Concatenating all feature vectors
Exp. Ⅲ - Accuracy Refinement
ex.) fs = 128 [Hz] (nd = 4) L = ceil(128・0.8) = 103
58
…
Lx1 …
L…
L…
…
… … ……Vex.) Lconcat = L・8 = 103・8 = 824
Lconcat
…
Ch1 Ch2 Ch8
x2 x8
![Page 59: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/59.jpg)
● Training the classifier
Exp. Ⅲ - Accuracy Refinement
59
X1
X2
Lconcat
Classifier (2cls)
XNTAR
・
・
・
・
・
・
NTAR = 60 / ne NNTAR = 60 / ne
Random chooseas many as Tmax
}
Non-Target Target
X1
X2
XNNTAR
Lconcat
![Page 60: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/60.jpg)
● Evaluation with the trained classifier○ Same nd and ne were applied
Exp. Ⅲ - Accuracy Refinement
60
1L
・
・
NERP = 10 / ne
Target? orNon-Target? Classifier (2cls)
Test data
![Page 61: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/61.jpg)
● Machine learning algorithms○ SWLDA○ Linear SVM
○ Non-linear SVM (Gaussian)
where γ > 0 , c = 1
Exp. Ⅲ - Accuracy Refinement
61
//
![Page 62: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/62.jpg)
Result Ⅲ - Accuracy Refinement
62
![Page 63: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/63.jpg)
Experiment Ⅳ - CNN application
● Transform feature vectors to input volumes
L = 410xi …
p1 p410
fs = 512 [Hz] tERP = 800 [ms] = 0.8 [sec] L = ceil(512・0.8) = 410
1. Feature vector length L was reduced from 410 to 400 (first 10 ERP elements were removed) to create squared matrices for filter training
63
![Page 64: Full Body Spatial Vibrotactile Brain Computer Interface Paradigm](https://reader030.vdocuments.us/reader030/viewer/2022020314/5a6636267f8b9a212b8b4595/html5/thumbnails/64.jpg)
Experiment Ⅳ - CNN application
● One-hidden layer multilayer perceptron○ Input: 7200 > Hidden: 500 > Output: 2 units
64