1

Classifying Instantaneous Cognitive States from fMRI Data

Tom Mitchell, Rebecca Hutchinson, Marcel Just, Stefan Niculescu, Francisco Pereira, Xuerui Wang

Carnegie Mellon University

November, 2003

2

…

3

4

Cognitive state sequence

COGNITIVE TASK

“Virtual sensors” of cognitive state

1. Does fMRI contain enough information?

2. Can we devise learning algorithms to construct such “virtual sensors”?

5

Learning Virtual Sensors

• Learn fMRI(t,t+k) CognitiveState

• Classifiers:– Gaussian Naïve Bayes, SVM, kNN

• Trained per subject, per experiment

• Feature selection/abstraction– Select subset of voxels (by signal, by anatomy)– Select subinterval of time– Average activities over space, time– Normalize voxel activities

6

Study 1: Pictures and Sentences

• Trial: read sentence, view picture, answer whether

sentence describes picture

• Picture presented first in half of trials, sentence first

in other half

• Three possible objects: star, dollar, plus

• Collected by Just et al.

7

It is true that the star is above the plus?

8

9

+

---

*

10

11

Is Subject Viewing Picture or Sentence?

• Learn fMRI(t,t+8) {Picture, Sentence}

• Leave two out cross-validation was used to assess the performance of the classifiers

• SVMs and GNB worked better than kNN

• Some Details: – 12 subjects, 40 pictures, 40 sentences– 1397 - 2864 voxels per subject, 7 ROIs – fMRI snapshot taken every half second

12

Error for Single-Subject Classifiers 

 

•Error computed by averaging over all subjects

•95% confidence intervals per subject are ~ 10% large

• Error of default classifier is 50%

Dataset \ Classifier GNB SVM 1NN 2NN 5NN

Picture vs Sentence 0.16 0.09 0.20 0.18 0.19

13

• Approach: define supervoxels based on anatomically defined regions of interest– Normalize per voxel activity for each subject

• Each value scaled now in [0,1]

– Abstract to seven brain region supervoxels– 16 snapshots for each supervoxel

• Train on n-1 subjects, test on nth– Leave one subject out cross validation

Can We Train Subject-Indep Classifiers?

14

• NO Feature Selection used in this experiment

•95% confidence intervals approximately 5% large

•Error of default classifier is 50%

Error for Cross Subject Classifiers

Dataset \ Classifier GNB SVM 1NN 2NN 5NN

Cross-Subject Pict vs Sent

0.30 0.25 0.36 0.33 0.32

15

Study 2: Word Categories

• Family members

• Occupations

• Tools

• Kitchen items

• Dwellings

• Building parts

• 4 legged animals

• Fish

• Trees

• Flowers

• Fruits

• Vegetables

16

Word Categories Study

• Stimulus:

– 12 blocks of words:

• Category name (2 sec)

• Word (400 msec), Blank screen (1200 msec); answer

• Word (400 msec), Blank screen (1200 msec); answer

• …

– Subject answers whether each word in category

– 20 words per block, nearly all in category

17

Training Classifier for Word Categories

• Learn fMRI(t) Word Category

• Training methods: kNN, GNB

• Leave one example out from each class used to assess performance

• Some Details: – 10 subjects, 20 examples per class– 8470 - 11,136 voxels per subject, 30 ROIs– fMRI snapshot taken every second

18

Study 2: Results

Classifier outputs ranked list of classesEvaluate by the fraction of classes ranked ahead of true

class– 0=perfect, 0.5=random, 1.0 unbelievably poor

 

Dataset \ Classifier

GNB 1NN 3NN 5NN

Words 0.08 0.30 0.20 0.16

19

Study 3: Syntactic Ambiguity

• Is subject reading ambiguous or unambiguous sentence?– “The experienced soldiers warned about the dangers

conducted the midnight raid.”

– “The experienced soldiers spoke about the dangers before the midnight raid.”

• Almost random results if no feature selection used • With feature selection:

– SVM - 77% accuracy

– GNB - 75% accuracy

– 5NN – 72% accuracy

20

• Four feature selection methods:

• Active (n most active available voxels compared to baseline fixation activity, according to a t-test)

• RoiActive (n most active voxels in each ROI)

• RoiActiveAvg (average of the n most active voxels in each ROI)

• Disc (n most discriminating voxels according to a trained classifier)

• Active works best

Feature Selection

21

Dataset

Feature

Selection

GNB SVM 1NN 3NN 5NN

PictureSent

No 0.29 0.32 0.43 0.41 0.37

  Active 0.16 0.09 0.20 0.18 0.19

Words

No 0.10 N/A 0.40 0.40 0.40

  Active 0.08 N/A 0.30 0.20 0.16

SyntAmb

No 0.43 0.38 0.50 0.46 0.47

  Active 0.25 0.23 0.29 0.29 0.28

Feature Selection

22

Summary

• Proved that there is enough information in the fMRI signal to allow decoding of Cognitive States

• Successful training of classifiers for instantaneous cognitive state in three studies

• Cross subject classifiers trained by abstracting to anatomically defined ROIs

• Feature selection and abstraction are essential

23

Research Opportunities

• Learning temporal models– HMM’s, Temporal Bayes Nets

• Learn to discriminate whether a subject has certain mental disease

• Discovering useful data abstractions– ICA, PCA, hidden layers in Neural Nets

• Merging data from multiple sources– fMRI, ERP, reaction times