![Page 1: FMRI CORRELATES OF COMPREHENDING REAL-WORLD EVENTS Tatiana Sitnikova, Ph.D. 1,2,4 Alexis Coty 1,2,4 Daphne Robakis 1,2,4 Phillip J. Holcomb, Ph.D. 5 Gina](https://reader035.vdocuments.us/reader035/viewer/2022062216/56649d4d5503460f94a2c57c/html5/thumbnails/1.jpg)
FMRI CORRELATES OF COMPREHENDING
REAL-WORLD EVENTS
Tatiana Sitnikova, Ph.D.1,2,4
Alexis Coty1,2,4
Daphne Robakis1,2,4
Phillip J. Holcomb, Ph.D.5
Gina R. Kuperberg, M.D., Ph.D.2,3,4
W. Caroline West, Ph.D.1,2,4
1 Dept of Neurology, 2 Martinos Center for Biomedical Imaging, 3 Dept of Psychiatry, Mass Gen Hosp, Boston, MA, USA4 Harvard Med Sch, Boston, MA, USA 5 Dept of Psychology, Tufts Univ, Medford, MA, USA
![Page 2: FMRI CORRELATES OF COMPREHENDING REAL-WORLD EVENTS Tatiana Sitnikova, Ph.D. 1,2,4 Alexis Coty 1,2,4 Daphne Robakis 1,2,4 Phillip J. Holcomb, Ph.D. 5 Gina](https://reader035.vdocuments.us/reader035/viewer/2022062216/56649d4d5503460f94a2c57c/html5/thumbnails/2.jpg)
Introduction
How do we use visual information to comprehend real-world events?
• Static pictures Naturalistic video clips
• Paradigm: Contextual anomalies
![Page 3: FMRI CORRELATES OF COMPREHENDING REAL-WORLD EVENTS Tatiana Sitnikova, Ph.D. 1,2,4 Alexis Coty 1,2,4 Daphne Robakis 1,2,4 Phillip J. Holcomb, Ph.D. 5 Gina](https://reader035.vdocuments.us/reader035/viewer/2022062216/56649d4d5503460f94a2c57c/html5/thumbnails/3.jpg)
anomalous ending
Contextual Anomaly Paradigm with Videos
c o n t e x t
congruous ending
congruous ending
anomalous ending
![Page 4: FMRI CORRELATES OF COMPREHENDING REAL-WORLD EVENTS Tatiana Sitnikova, Ph.D. 1,2,4 Alexis Coty 1,2,4 Daphne Robakis 1,2,4 Phillip J. Holcomb, Ph.D. 5 Gina](https://reader035.vdocuments.us/reader035/viewer/2022062216/56649d4d5503460f94a2c57c/html5/thumbnails/4.jpg)
Introduction: Electrophysiological Findings
-2µV
400 800 1200 1600 ms
N400 effect
LPC effect
congruous final sceneanomalous final scene
Cz
Pz
![Page 5: FMRI CORRELATES OF COMPREHENDING REAL-WORLD EVENTS Tatiana Sitnikova, Ph.D. 1,2,4 Alexis Coty 1,2,4 Daphne Robakis 1,2,4 Phillip J. Holcomb, Ph.D. 5 Gina](https://reader035.vdocuments.us/reader035/viewer/2022062216/56649d4d5503460f94a2c57c/html5/thumbnails/5.jpg)
Stimuli
• 40 congruous and 40 anomalous videos: context 6 s long final scene 2 s long
![Page 6: FMRI CORRELATES OF COMPREHENDING REAL-WORLD EVENTS Tatiana Sitnikova, Ph.D. 1,2,4 Alexis Coty 1,2,4 Daphne Robakis 1,2,4 Phillip J. Holcomb, Ph.D. 5 Gina](https://reader035.vdocuments.us/reader035/viewer/2022062216/56649d4d5503460f94a2c57c/html5/thumbnails/6.jpg)
Stimuli: Congruous Videoclick in the middle of the slide to see the video
![Page 7: FMRI CORRELATES OF COMPREHENDING REAL-WORLD EVENTS Tatiana Sitnikova, Ph.D. 1,2,4 Alexis Coty 1,2,4 Daphne Robakis 1,2,4 Phillip J. Holcomb, Ph.D. 5 Gina](https://reader035.vdocuments.us/reader035/viewer/2022062216/56649d4d5503460f94a2c57c/html5/thumbnails/7.jpg)
Stimuli: Anomalous Videoclick in the middle of the slide to see the video
![Page 8: FMRI CORRELATES OF COMPREHENDING REAL-WORLD EVENTS Tatiana Sitnikova, Ph.D. 1,2,4 Alexis Coty 1,2,4 Daphne Robakis 1,2,4 Phillip J. Holcomb, Ph.D. 5 Gina](https://reader035.vdocuments.us/reader035/viewer/2022062216/56649d4d5503460f94a2c57c/html5/thumbnails/8.jpg)
Methods• 16 participants, mean age 21
• Pseudorandom stimulus presentation including
fixation trials
• Counterbalanced design
• 3T Siemens MR scanner
• Event-related fMRI:
TR 2 s, TE 25 ms, 33 slices, 3 mm skip 1 mm
Talairach and spherical space
![Page 9: FMRI CORRELATES OF COMPREHENDING REAL-WORLD EVENTS Tatiana Sitnikova, Ph.D. 1,2,4 Alexis Coty 1,2,4 Daphne Robakis 1,2,4 Phillip J. Holcomb, Ph.D. 5 Gina](https://reader035.vdocuments.us/reader035/viewer/2022062216/56649d4d5503460f94a2c57c/html5/thumbnails/9.jpg)
Timecourse of Hemodynamic Response
xcx
time (s)0 6 10 20 30
% s
igna
l cha
nge
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
-8 -4 0 4 8 12 16 20 24 28
![Page 10: FMRI CORRELATES OF COMPREHENDING REAL-WORLD EVENTS Tatiana Sitnikova, Ph.D. 1,2,4 Alexis Coty 1,2,4 Daphne Robakis 1,2,4 Phillip J. Holcomb, Ph.D. 5 Gina](https://reader035.vdocuments.us/reader035/viewer/2022062216/56649d4d5503460f94a2c57c/html5/thumbnails/10.jpg)
Results: Summed Activity between 10 – 18 s
in the Left Hemisphere
xcx anomalous vs. congruous
BA 9
BA 44, 45
congruous vs. fixation
p=10-4
p=.03
![Page 11: FMRI CORRELATES OF COMPREHENDING REAL-WORLD EVENTS Tatiana Sitnikova, Ph.D. 1,2,4 Alexis Coty 1,2,4 Daphne Robakis 1,2,4 Phillip J. Holcomb, Ph.D. 5 Gina](https://reader035.vdocuments.us/reader035/viewer/2022062216/56649d4d5503460f94a2c57c/html5/thumbnails/11.jpg)
Results: Anatomical Regions of Interest
xcx
left posterior-lateral temporalleft inferior-frontal gyrus
![Page 12: FMRI CORRELATES OF COMPREHENDING REAL-WORLD EVENTS Tatiana Sitnikova, Ph.D. 1,2,4 Alexis Coty 1,2,4 Daphne Robakis 1,2,4 Phillip J. Holcomb, Ph.D. 5 Gina](https://reader035.vdocuments.us/reader035/viewer/2022062216/56649d4d5503460f94a2c57c/html5/thumbnails/12.jpg)
Results: Anatomical Regions of Interest
xcx
anomalous vs. congruousaverage activity 10 - 18 sleft hemisphere
p=.03 p=10-4
left posterior-lateral temporal
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
-8 -4 0 4 8 12 16 20 24 28
time (s)10 20 30
% s
igna
l cha
nge
solid – congruent; dotted - anomalous
0.9
0.6
0.3
![Page 13: FMRI CORRELATES OF COMPREHENDING REAL-WORLD EVENTS Tatiana Sitnikova, Ph.D. 1,2,4 Alexis Coty 1,2,4 Daphne Robakis 1,2,4 Phillip J. Holcomb, Ph.D. 5 Gina](https://reader035.vdocuments.us/reader035/viewer/2022062216/56649d4d5503460f94a2c57c/html5/thumbnails/13.jpg)
Results: Anatomical Regions of Interest
xcx
anomalous vs. congruousaverage activity 10 - 18 sleft hemisphere
left inferior frontal (BA 44, 45)
p=.03 p=10-4-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
-8 -4 0 4 8 12 16 20 24 28
solid – congruent; dotted - anomalous
time (s)
0.9
0.6
0.3
10 20 30%
sig
nal c
hang
e
![Page 14: FMRI CORRELATES OF COMPREHENDING REAL-WORLD EVENTS Tatiana Sitnikova, Ph.D. 1,2,4 Alexis Coty 1,2,4 Daphne Robakis 1,2,4 Phillip J. Holcomb, Ph.D. 5 Gina](https://reader035.vdocuments.us/reader035/viewer/2022062216/56649d4d5503460f94a2c57c/html5/thumbnails/14.jpg)
left caudate
Results: Anatomical Regions of Interest
xcx
anomalous vs. congruousaverage activity 10 - 18 s
p=.03 p=10-4
- 0 . 2
- 0 . 1
0
0 . 1
0 . 2
0 . 3
0 . 4
0 . 5
0 . 6
0 . 7
0 . 8
0 . 9
-8 -4 0 4 8 12 16 20 24 28
solid – congruent; dotted - anomalous
time (s)
0.9
0.6
0.3
10 20 30%
sig
nal c
hang
e
![Page 15: FMRI CORRELATES OF COMPREHENDING REAL-WORLD EVENTS Tatiana Sitnikova, Ph.D. 1,2,4 Alexis Coty 1,2,4 Daphne Robakis 1,2,4 Phillip J. Holcomb, Ph.D. 5 Gina](https://reader035.vdocuments.us/reader035/viewer/2022062216/56649d4d5503460f94a2c57c/html5/thumbnails/15.jpg)
Discussion: How do fMRI and electrophysiological findings
in language and visual real-world events tie together?
Anomalies in language and videos
Common brain activations
Common electrophysiology
Temporal cortexInferior-frontal cortex
Basal ganglia
N400LPC
![Page 16: FMRI CORRELATES OF COMPREHENDING REAL-WORLD EVENTS Tatiana Sitnikova, Ph.D. 1,2,4 Alexis Coty 1,2,4 Daphne Robakis 1,2,4 Phillip J. Holcomb, Ph.D. 5 Gina](https://reader035.vdocuments.us/reader035/viewer/2022062216/56649d4d5503460f94a2c57c/html5/thumbnails/16.jpg)
Future Directions
• Understanding cognitive functions of these brain regions
• Understanding the flow of information between these regions
Higher-level visual comprehension
Conclusion
A wide neural network
![Page 17: FMRI CORRELATES OF COMPREHENDING REAL-WORLD EVENTS Tatiana Sitnikova, Ph.D. 1,2,4 Alexis Coty 1,2,4 Daphne Robakis 1,2,4 Phillip J. Holcomb, Ph.D. 5 Gina](https://reader035.vdocuments.us/reader035/viewer/2022062216/56649d4d5503460f94a2c57c/html5/thumbnails/17.jpg)
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
-8 -4 0 4 8 12 16 20 24 28
left inferior frontal (BA 44, 45)left caudate
left posterior-lateral temporal
Results: Anatomical Regions of Interest
xcx
Solid – congruent; dotted - anomalous
anomalous vs. congruousaverage activity 10 - 18 sleft hemisphere
time (s)
0.9
0.6
0.3
10 20 30%
sig
nal c
hang
ep=.03 p=10-4