1

Preliminary Experiments: Learning Virtual Sensors

• Machine learning approach: train classifiers– fMRI(t, t+ ) CognitiveState

• Fixed set of possible states

• Trained per subject, per experiment

• Time interval specified

2

Approach

• Learn fMRI(t,…,t+k) CognitiveState

• Classifiers:– Gaussian Naïve Bayes, SVM, kNN

• Feature selection/abstraction– Select subset of voxels (by signal, by anatomy)

– Select subinterval of time

– Average activities over space, time

– Normalize voxel activities

– …

3

Study 1: Word Categories

• Family members

• Occupations

• Tools

• Kitchen items

• Dwellings

• Building parts

• 4 legged animals• Fish• Trees• Flowers• Fruits• Vegetables

[Francisco Pereira]

4

Word Categories Study

• Ten neurologically normal subjects• Stimulus:

– 12 blocks of words:• Category name (2 sec)• Word (400 msec), Blank screen (1200 msec); answer• Word (400 msec), Blank screen (1200 msec); answer• …

– Subject answers whether each word in category– 32 words per block, nearly all in category– Category blocks interspersed with 5 fixation blocks

5

Training Classifier for Word CategoriesLearn fMRI(t) word-category(t)

– fMRI(t) = 8470 to 11,136 voxels, depending on subject

Feature selection: Select n voxels– Best single-voxel classifiers

– Strongest contrast between fixation and some word category

– Strongest contrast, spread equally over ROI’s

– Randomly

Training method:– train ten single-subect classifiers

– Gaussian Naïve Bayes P(fMRI(t) | word-category)

6

Learned Bayes Models - MeansP(BrainActivity | WordCategory = People)

7

Learned Bayes Models - MeansP(BrainActivity | WordClass)

Animal words People words

Accuracy: 85%

8

Results

Classifier outputs ranked list of classesEvaluate by the fraction of classes ranked ahead of true

class– 0=perfect, 0.5=random, 1.0 unbelievably poor

Try abstracting 12 categories to 6 categories

e.g., combine “Family Members” with “Occupations”

9

Impact of Feature Selection

10

X1=S1+N1 X2=S2+N2

N0

Goal: learn f: X ! Y or P(Y|X)

Given:

1. Training examples <Xi, Yi> where Xi = Si + Ni , signal Si ~ P(S|Y=i), noise Ni ~ Pnoise

2. Observed noise with zero signal N0 ~ Pnoise

“Zero Signal” learning setting.

(fixation)

Class 1 observations

Class 2 observations

11

12

[Haxby et al., 2001]

13

Study 1: Summary

• Able to classify single fMRI image by word category block

• Feature selection important

• Is classifier learning word category or something else related to time?– Accurate across ten subjects– Relevant voxels in similar locations across subjs– Locations compatible with earlier studies– New experimental data will answer definitively

14

Study 2: Pictures and Sentences

• Trial: read sentence, view picture, answer whether sentence describes picture

• Picture presented first in half of trials, sentence first in other half

• Image every 500 msec • 12 normal subjects• Three possible objects:

star, dollar, plus• Collected by Just et al.

[Xuerui Wang and Stefan Niculescu]

15

It is true that the star is above the plus?

16

17

+

---

*

18

19

Is Subject Viewing Picture or Sentence?

• Learn fMRI(t, …, t+15) {Picture, Sentence}– 40 training trials (40 pictures and 40 sentences)– 7 ROIs

• Training methods:– K Nearest Neighbor– Support Vector Machine– Naïve Bayes

20

Is Subject Viewing Picture or Sentence?

• Support Vector Machine worked better on avg.

• Results (leave one out) on picture-then-sentence, sentence-then-picture data

– Random guess = 50% accuracy

– SVM using pair of time slices at 5.0,5.5 sec after stimulus: 91% accuracy

21

Accuracy for Single-Subject Classifiers

22

Can We Train Subject-Indep Classifiers?

23

Training Cross-Subject Classifiers

• Approach: define supervoxels based on anatomically defined regions of interest– Abstract to seven brain region supervoxels

• Train on n-1 subjects, test on nth

24

Accuracy for Cross-Subject GNB Classifier

25

Accuracy for Cross-Subject and Cross-Context Classifier

26

Possible ANN to discover intermediate data abstraction across multiple subjects. Each bank of inputs corresponds to voxel inputs for a particular subject. The trained hidden layer will provide a low-dimensional data abstraction shared by all subjects. We propose to develop new algorithms to train such networks to discover multi-subject classifiers.

Subject 1 Subject 2 Subject 3

Learned cross-subject representation

Output classification