temporal and cross-subject probabilistic models for fmri prediction tasks
DESCRIPTION
Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks. Alexis Battle Gal Chechik Daphne Koller Department of Computer Science Stanford University. PBAI Competition. Provided rich data set Interesting interactions across time, subjects, and stimuli - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/1.jpg)
Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks
Alexis BattleGal ChechikDaphne Koller
Department of Computer ScienceStanford University
![Page 2: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/2.jpg)
PBAI Competition
● Provided rich data set– Interesting interactions across time, subjects, and
stimuli
● Challenged us to come up with reliable techniques
● Thanks to the organizers!
![Page 3: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/3.jpg)
Key Points
● Predictive voxels selected from whole brain
● Probabilistic model makes use of additional correlations– Subjects’ ratings across time steps
– Ratings between subjects
● Learn strength of each relationship
![Page 4: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/4.jpg)
Modeling the fMRI Domain
A joint distributionin high dimension
Voxels across time
BOLD signal
Ratings across time
User Ratings
funnytimbodylanguage
![Page 5: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/5.jpg)
Modeling the fMRI Domain
A joint distributionin high dimension
Voxels across time
BOLD signal
Ratings across time
User Ratings
funnytimbodylanguage
Training:Use two movies to learn the relations between voxels and ratings
![Page 6: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/6.jpg)
Modeling the fMRI Domain
A joint distributionin high dimension
Voxels across time
BOLD signal
Ratings across time
User Ratings
funnytimbodylanguage
Training:Use two movies to learn the relations between voxels and ratings
Testing:Use the learned relations to predict ratings from fMRImeasurements
![Page 7: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/7.jpg)
Probabilistic Model
• Each voxel measurement• Each rating to predict
Language
fromVox1 Vox2 Vox3
![Page 8: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/8.jpg)
Probabilistic Model
• Each voxel measurement• Each rating to predict• Rating predicted from voxel measurements
– Linear regression model (Gaussian distribution)
Language
fromVox1 Vox2 Vox3
![Page 9: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/9.jpg)
Probabilistic Model
• Each voxel measurement• Each rating to predict• Rating predicted from voxel measurements
– Linear regression model (Gaussian distribution)
Language
fromVox1 Vox2 Vox3• Selected predictive
voxels from whole brain• Regularize (Ridge,
Lasso) to handle noise
![Page 10: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/10.jpg)
Probabilistic Model
Language
Vox1 Vox2
Language
Vox1 Vox2
…T =1 T =2
![Page 11: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/11.jpg)
Probabilistic Model
● Ratings correlated across time– Language at time 1 makes language at time 2 likely
Language
Vox1 Vox2
Language
Vox1 Vox2
…T =1 T =2
![Page 12: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/12.jpg)
Probabilistic Model
● Ratings correlated across time– Language at time 1 makes language at time 2 likely
Language
Vox1 Vox2
Language
Vox1 Vox2
…T =1 T =2
![Page 13: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/13.jpg)
Probabilistic Model
● Ratings correlated across time– Language at time 1 makes language at time 2 likely
– Weight A – how correlated?
Language
Vox1 Vox2
Language
Vox1 Vox2
…T =1 T =2
A*Lang (1)*Lang(2)
A
![Page 14: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/14.jpg)
Probabilistic Model
Subject 1
Subject 2
Language
Vox1 Vox2
Language
Vox1 Vox2
T =1 T =2
Language
Vox1 Vox2
Language
Vox1 Vox2
…
![Page 15: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/15.jpg)
Probabilistic Model
● Ratings likely to be correlated between subjects
Subject 1
Subject 2
Language
Vox1 Vox2
Language
Vox1 Vox2
T =1 T =2
Language
Vox1 Vox2
Language
Vox1 Vox2
…
![Page 16: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/16.jpg)
Probabilistic Model
● Ratings likely to be correlated between subjects– Weighted correlation, NOT equality
Subject 1
Subject 2
Language
Vox1 Vox2
Language
Vox1 Vox2
T =1 T =2
Language
Vox1 Vox2
Language
Vox1 Vox2
…
B B
![Page 17: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/17.jpg)
Probabilistic Model
Joint model over all time points:
Time
…Sub1
Sub2
Gaussian Markov Random Field – joint Gaussian over all rating nodes conditioned on voxel data
![Page 18: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/18.jpg)
Voxel Parameters
● Regularized linear regression for voxel parameters
Language
Vox1 Vox2 Vox3
![Page 19: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/19.jpg)
Voxel Parameters
● Regularized linear regression for voxel parameters
Language
Vox1 Vox2 Vox3
![Page 20: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/20.jpg)
Voxel Parameters
● Regularized linear regression for voxel parameters
β1 = 0.45
β2 = 0.55
Language
Vox1 Vox2 Vox3
![Page 21: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/21.jpg)
Inter-Rating parameters
L(1)
Vox1 Vox2
L(2)
Vox1 Vox2B
● Other weights also learned from data– Example: cross-subject weights
![Page 22: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/22.jpg)
Inter-Rating parameters
L(1)
Vox1 Vox2
L(2)
Vox1 Vox2B
● Other weights also learned from data– Example: cross-subject weights
Attention Faces
![Page 23: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/23.jpg)
Inter-Rating parameters
L(1)
Vox1 Vox2
L(2)
Vox1 Vox2B
B = 0.3 B = 0.7
● Other weights also learned from data– Example: cross-subject weights
Attention Faces
![Page 24: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/24.jpg)
Prediction Results
● Use full learned model, including all weights
● Predict ratings for a new movie given fMRI data
![Page 25: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/25.jpg)
Prediction Results
● Use full learned model, including all weights
● Predict ratings for a new movie given fMRI data
![Page 26: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/26.jpg)
Prediction Results
● Comparison to models without time or subject interactions
![Page 27: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/27.jpg)
Voxel Selection
● Voxels selected by correlation with rating– Number of voxels determined by cross-validation
![Page 28: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/28.jpg)
Voxel Selection
● Voxels selected by correlation with rating– Number of voxels determined by cross-validation
![Page 29: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/29.jpg)
Selected Voxels
Faces Language
L L
![Page 30: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/30.jpg)
Selected Voxels
Motion Arousal
L L
![Page 31: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/31.jpg)
Voxel Selection
● Voxels selected for Language included some in ‘Face’ regions:
L
![Page 32: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/32.jpg)
Voxel Selection
● Voxels selected for Language included some in ‘Face’ regions:
L
● Language and face stimuli correlated in videos
● Complex, interwoven stimuli affect voxel specificity
![Page 33: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/33.jpg)
Voxel Selection
0.330.38
● Voxel selection extension – “spatial bias”– Prefer grouped voxels
* after competition submission
![Page 34: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/34.jpg)
Voxel Selection
* after competition submission
0.330.38
● Voxel selection extension – “spatial bias”– Prefer grouped voxels
![Page 35: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/35.jpg)
Voxel Selection
* after competition submission
0.330.38
● Voxel selection extension – “spatial bias”– Prefer grouped voxels
– Additional terms in linear regression objective:
● |β1| |β2| D(Vox1, Vox2)
D
|| Vox1 –Vox2||2
![Page 36: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/36.jpg)
Adding Spatial Bias
Faces
L L
![Page 37: Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks](https://reader036.vdocuments.us/reader036/viewer/2022062518/56814682550346895db3a2d2/html5/thumbnails/37.jpg)
Conclusions
● Reliable prediction of subjective ratings from fMRI data
● Time step correlations aid in prediction reliability
● Cross-subject correlations also beneficial
● Individual voxels selected from whole brain– Reliability from regularization
– Some found in expected regions
– Some “cross-over” for correlated prediction tasks