![Page 1: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/1.jpg)
Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks
Alexis BattleGal ChechikDaphne Koller
Department of Computer ScienceStanford University
![Page 2: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/2.jpg)
PBAI Competition
● Provided rich data set– Interesting interactions across time, subjects, and
stimuli
● Challenged us to come up with reliable techniques
● Thanks to the organizers!
![Page 3: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/3.jpg)
Key Points
● Predictive voxels selected from whole brain
● Probabilistic model makes use of additional correlations– Subjects’ ratings across time steps
– Ratings between subjects
● Learn strength of each relationship
![Page 4: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/4.jpg)
Modeling the fMRI Domain
A joint distributionin high dimension
Voxels across time
BOLD signal
Ratings across time
User Ratings
funnytimbodylanguage
![Page 5: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/5.jpg)
Modeling the fMRI Domain
A joint distributionin high dimension
Voxels across time
BOLD signal
Ratings across time
User Ratings
funnytimbodylanguage
Training:Use two movies to learn the relations between voxels and ratings
![Page 6: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/6.jpg)
Modeling the fMRI Domain
A joint distributionin high dimension
Voxels across time
BOLD signal
Ratings across time
User Ratings
funnytimbodylanguage
Training:Use two movies to learn the relations between voxels and ratings
Testing:Use the learned relations to predict ratings from fMRImeasurements
![Page 7: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/7.jpg)
Probabilistic Model
• Each voxel measurement• Each rating to predict
Language
fromVox1 Vox2 Vox3
![Page 8: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/8.jpg)
Probabilistic Model
• Each voxel measurement• Each rating to predict• Rating predicted from voxel measurements
– Linear regression model (Gaussian distribution)
Language
fromVox1 Vox2 Vox3
![Page 9: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/9.jpg)
Probabilistic Model
• Each voxel measurement• Each rating to predict• Rating predicted from voxel measurements
– Linear regression model (Gaussian distribution)
Language
fromVox1 Vox2 Vox3• Selected predictive
voxels from whole brain• Regularize (Ridge,
Lasso) to handle noise
![Page 10: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/10.jpg)
Probabilistic Model
Language
Vox1 Vox2
Language
Vox1 Vox2
…T =1 T =2
![Page 11: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/11.jpg)
Probabilistic Model
● Ratings correlated across time– Language at time 1 makes language at time 2 likely
Language
Vox1 Vox2
Language
Vox1 Vox2
…T =1 T =2
![Page 12: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/12.jpg)
Probabilistic Model
● Ratings correlated across time– Language at time 1 makes language at time 2 likely
Language
Vox1 Vox2
Language
Vox1 Vox2
…T =1 T =2
![Page 13: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/13.jpg)
Probabilistic Model
● Ratings correlated across time– Language at time 1 makes language at time 2 likely
– Weight A – how correlated?
Language
Vox1 Vox2
Language
Vox1 Vox2
…T =1 T =2
A*Lang (1)*Lang(2)
A
![Page 14: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/14.jpg)
Probabilistic Model
Subject 1
Subject 2
Language
Vox1 Vox2
Language
Vox1 Vox2
T =1 T =2
Language
Vox1 Vox2
Language
Vox1 Vox2
…
![Page 15: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/15.jpg)
Probabilistic Model
● Ratings likely to be correlated between subjects
Subject 1
Subject 2
Language
Vox1 Vox2
Language
Vox1 Vox2
T =1 T =2
Language
Vox1 Vox2
Language
Vox1 Vox2
…
![Page 16: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/16.jpg)
Probabilistic Model
● Ratings likely to be correlated between subjects– Weighted correlation, NOT equality
Subject 1
Subject 2
Language
Vox1 Vox2
Language
Vox1 Vox2
T =1 T =2
Language
Vox1 Vox2
Language
Vox1 Vox2
…
B B
![Page 17: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/17.jpg)
Probabilistic Model
Joint model over all time points:
Time
…Sub1
Sub2
Gaussian Markov Random Field – joint Gaussian over all rating nodes conditioned on voxel data
![Page 18: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/18.jpg)
Voxel Parameters
● Regularized linear regression for voxel parameters
Language
Vox1 Vox2 Vox3
![Page 19: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/19.jpg)
Voxel Parameters
● Regularized linear regression for voxel parameters
Language
Vox1 Vox2 Vox3
![Page 20: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/20.jpg)
Voxel Parameters
● Regularized linear regression for voxel parameters
β1 = 0.45
β2 = 0.55
Language
Vox1 Vox2 Vox3
![Page 21: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/21.jpg)
Inter-Rating parameters
L(1)
Vox1 Vox2
L(2)
Vox1 Vox2B
● Other weights also learned from data– Example: cross-subject weights
![Page 22: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/22.jpg)
Inter-Rating parameters
L(1)
Vox1 Vox2
L(2)
Vox1 Vox2B
● Other weights also learned from data– Example: cross-subject weights
Attention Faces
![Page 23: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/23.jpg)
Inter-Rating parameters
L(1)
Vox1 Vox2
L(2)
Vox1 Vox2B
B = 0.3 B = 0.7
● Other weights also learned from data– Example: cross-subject weights
Attention Faces
![Page 24: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/24.jpg)
Prediction Results
● Use full learned model, including all weights
● Predict ratings for a new movie given fMRI data
![Page 25: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/25.jpg)
Prediction Results
● Use full learned model, including all weights
● Predict ratings for a new movie given fMRI data
![Page 26: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/26.jpg)
Prediction Results
● Comparison to models without time or subject interactions
![Page 27: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/27.jpg)
Voxel Selection
● Voxels selected by correlation with rating– Number of voxels determined by cross-validation
![Page 28: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/28.jpg)
Voxel Selection
● Voxels selected by correlation with rating– Number of voxels determined by cross-validation
![Page 29: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/29.jpg)
Selected Voxels
Faces Language
L L
![Page 30: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/30.jpg)
Selected Voxels
Motion Arousal
L L
![Page 31: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/31.jpg)
Voxel Selection
● Voxels selected for Language included some in ‘Face’ regions:
L
![Page 32: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/32.jpg)
Voxel Selection
● Voxels selected for Language included some in ‘Face’ regions:
L
● Language and face stimuli correlated in videos
● Complex, interwoven stimuli affect voxel specificity
![Page 33: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/33.jpg)
Voxel Selection
0.330.38
● Voxel selection extension – “spatial bias”– Prefer grouped voxels
* after competition submission
![Page 34: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/34.jpg)
Voxel Selection
* after competition submission
0.330.38
● Voxel selection extension – “spatial bias”– Prefer grouped voxels
![Page 35: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/35.jpg)
Voxel Selection
* after competition submission
0.330.38
● Voxel selection extension – “spatial bias”– Prefer grouped voxels
– Additional terms in linear regression objective:
● |β1| |β2| D(Vox1, Vox2)
D
|| Vox1 –Vox2||2
![Page 36: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/36.jpg)
Adding Spatial Bias
Faces
L L
![Page 37: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/37.jpg)
Conclusions
● Reliable prediction of subjective ratings from fMRI data
● Time step correlations aid in prediction reliability
● Cross-subject correlations also beneficial
● Individual voxels selected from whole brain– Reliability from regularization
– Some found in expected regions
– Some “cross-over” for correlated prediction tasks