APRIL GARDNER APRIL 2013

Reconstructing visual experiences from brain activity

evoked by natural movies

Background

Understanding neural decoding of processing early visual information – color, shape, location – has been explored in past studies. Reconstructing still images

fMRI is the tool of choice has a built-in time lag; the level of oxygen in the

blood doesn’t change unti about 4 seconds after neuron activity

Neural decoding had to account for this in a multi-step process

What is the problem area?

How does our visual perception work to process what we see in everyday life? Theory: Seeing is like watching a movie – a dynamic

experience.

Hypothesis: Dynamic brain activity of natural image processing can be decoded with fMRI technology.

The problem is important! Visual information is dominant in how we receive information High implications for those that are disabled or dream

Question 6

What is neural decoding?

Question 6

What is neural decoding?

Reconstruction of sensory and other stimuli from information that has already been encoded and represented in the brain.

Can we predict what sensory stimuli the subject is receiving, purely based on action potentials?

Methods

Q3: What is the function of the V1 region?

Author Claims

The first new motion-energy encoding model that is optimized with fMRI The model reveals how motion information is

represented in early visual areas The model provides reconstructions of natural movies

from evoked BOLD signals

Methods

Obtain BOLD signals while watching a series of natural color movies

Fixation task to control eye position Two separate data sets obtained

TRAINING DATA from 7,200s of each movie, presented once

TEST DATA: BOLD signals from 540s of color natural movies, each repeated ten times

Methods

[1] Record brain activity while the subject watches several hours of movie trailers.

[2] Build dictionaries to translate between shapes, edges and motion in the movies and measured brain activity.

[3] Record brain activity to a new set of movie trailers that will be used to test the quality of dictionaries and reconstructions.

[4] Build a new library of ~18,000,000s of video. Select & average the 100 clips whose activity is most similar to the observed brain activity.

Participants

3 human subjects All co-authors

Question 5

What is first order motion perception? responds to moving luminance patterns Detected by early simple “motion sensors”

Results

Success.

Motion-energy encoding model identified specific movie stimulus that evoked anobserved BOLD signal 95% of the time (464 of 486 volumes), within +/- one volume.Far above chance (<1%).

Main Results

Author Conclusions

Successfully developed a computational model of brain activity evoked by dynamic natural movies.

Data: Consistent with Claims?

Each model was unique to the person Intersubject variability - how much does the model

from Subject A vary from that of B or C?Model is perception agnostic.

Does the person see the clips, or actually attend to them? What would the differing result be if they didn’t? In this study, measuring early visual areas Model was not accurate in higher level areas

Does a person who sees hallucinations, register anything similar in V1, V2, or V3?

Is this a Visual Experience? No.

Question 4

What is bottom up, and top down processing?

Bottom up processing of a stimulus in which information

from a physical stimulus (rather than from a general context)

Top down Knowledge and memory

play a roleBugelski and Alampay (1961)

Question 2

Describe the neural correlates of consciousness (NCC).

Question 2

Describe the neural correlates of consciousness (NCC). the minimal set of neuronal events and mechanisms

sufficient for a specific conscious percept. explain the exact relationship between

subjective mental states and brain states, the nature of the relationship between the conscious

mind and the electro-chemical interactions in the body

Block N (1996) How can we find the neural correlate of consciousness?Trends Neurosci 19:456–459.Rock I, Linnet CM, Grant P, Mack A (1992) Perception without attention:results of a new method. Cognit Psychol 24:501–534.

Question 1

What is the materialist theory, as applied to consciousness?

Question 1

What is the materialist theory, as applied to consciousness?

As opposed to the dualist theory – mind is a nonphysical substance

Mental = physical All mental states, properties, processes, and operations

are identical to physical ones Behaviorists maintain that all talk of mental causes stem

from environmental stimuli and behavioral responses

Tononi, Giulio. An information integration theory of consciousness. BMC Neuroscience 2004, 5:42

What’s Next?

Reproduce images of the mind that no one else sees INTERNAL IMAGERY Dreams Hallucinations

Communicate with those who verbally cannot, combating Coma Stroke Neurodegenerative disease