lecture 10. the mirror neuron system model (mns) 1 reading assignment:

1Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1

Michael Arbib: CS564 - Brain Theory and Artificial Intelligence

University of Southern California, Fall 2001

Lecture 10. The Mirror Neuron System Model (MNS) 1

Reading Assignment:Schema Design and Implementation ofthe Grasp-Related Mirror Neuron SystemErhan Oztop and Michael A. Arbib


Visual Control of Grasping in Macaque Monkey

F5 - grasp commands inpremotor cortexGiacomo Rizzolatti

AIP - grasp affordancesin parietal cortexHideo Sakata

A key theme of visuomotor coordination:parietal affordances (AIP) drivefrontal motor schemas (F5)


Mirror Neurons

Rizzolatti, Fadiga, Gallese, and Fogassi, 1995: Premotor cortex and the recognition of motor actions

Mirror neurons form the subset of grasp-related premotor neurons of F5 which discharge when the monkey observes meaningful hand movements made by the experimenter or another

monkey.F5 is endowed with an observation/execution matching system


F5 Motor Neurons

F5 Motor Neurons include all F5 neurons whose firing is related to motor activity. We focus on grasp-related behavior. Other F5 motor neurons are related to oro-facial movements.

F5 Mirror Neurons form the subset of grasp-related F5 motor neurons of F5 which discharge when the monkey observes meaningful hand movements.

F5 Canonical Neurons form the subset of grasp-related F5 motor neurons of F5 which fire when the monkey sees an object with related affordances.


What is the mirror system (for grasping) for?

• Action recognition

• Understanding (assigning meaning to other’s actions)

• Associative memory for actions

Mirror neurons: The cells that selectively discharge when the monkey executes particular actions as well as when the monkey observes an other individual executing the same action.

Mirror neuron system (MNS): The mirror neurons and the brain regions involved in eliciting mirror behavior.

Interpretations:


Computing the Mirror System Response

The FARS Model: Recognize object affordances and determine appropriate grasp.

The Mirror Neuron System (MNS) Model:We must add recognition of

trajectory and hand preshape

to recognition of object affordances

and ensure that all three are congruent.

There are parietal systems other than AIP adapted to this task.


Further Brain Regions Involved

cIPS

Spatial coding for objects, analysis of motion during interaction of objects and self-motion7b (PF):

Rostral part of the posterior

parietal lobule Mainly somatosensory Mirror-like responses

STS: Superior Temporal

Sulcus

Detection of biologically meaningful stimuli (e.g.hand actions) Motion related activity (MT/MST part)

cIPScIPS Axis and surface orientation

cIPS:

caudal intraparietal

sulcus

7a (PG):

caudal part of the posterior

parietal lobule


cIPS cell response

Surface orientation selectivity of a cIPS cell

Sakata et al. 1997cIPS

cIPS


Key Criteria for Mirror Neuron Activation When Observing a Grasp

a) Does the preshape of the hand correspond to the grasp encoded by the mirror neuron?

b) Does this preshape match an affordance of the target object?

c) Do samples of the hand state indicate a trajectory that will bring the hand to grasp the object?

Modeling Challenges:i) To have mirror neurons self-organize to learn to recognize grasps in the monkey’s motor repertoire

ii) To learn to activate mirror neurons from smaller and smaller samples of a trajectory.


Initial Hypothesis on Mirror Neuron Development

The development of the (grasp) mirror neuron system in a healthy infant is driven by the visual stimuli generated by the actions (grasps) performed by the infant himself.

The infant (with maturation of visual acuity) gains the ability to map other individual’s actions into his internal motor representation. [In the MNS model, the hand state provides the key representation for this transfer.]

Then the infant acquires the ability to create (internal) representations for novel actions observed.

Parallel to these achievements, the infant develops an action prediction capability (the recognition of an action given the prefix of the action and the target object)


The Mirror Neuron System (MNS) Model


Implementing the Basic Schemas of the Mirror Neuron System (MNS) Model

using Artificial Neural Networks(Work of Erhan Oztop)


Opposition Spaces and Virtual Fingers

The goal of a successfulpreshape, reach and graspis to match the oppositionaxis defined by the virtualfingers of the hand withthe opposition axis defined by an affordance of theobject

(Iberall and Arbib 1990)


Hand State

Our current representation of hand state defines a 7-dimensional trajectory F(t) with the following components

F(t) = (d(t), v(t), a(t), o1(t), o2(t), o3(t), o4(t)):

d(t): distance to target at time tv(t): tangential velocity of the wrist a(t): Aperture of the virtual fingers involved in grasping at time t

o1(t): Angle between the object axis and the (index finger tip – thumb tip) vector [relevant for pad and palm oppositions]

o2(t): Angle between the object axis and the (index finger knuckle – thumb tip) vector [relevant for side oppositions]

o3(t), o4(t): The two angles defining how close the thumb is to the hand as measured relative to the side of the hand and to the inner surface of the palm.


Curve recognition

The general problem: associate N-dimensional space curves with object affordances

A special case: The recognition of two (or three) dimensional trajectories in physical space Simplest solution: Map temporal information into spatial

domain. Then apply known pattern recognition techniques.

Problem with simplest solution: The speed of the moving point can be a problem! The spatial representation may change drastically with the speed

Scaling can overcome the problem. However the scaling must be such that it preserves the generalization ability of the pattern recognition engine.

Solution: Fit a cubic spline to the sampled values. Then normalize and re-sample from the spline curve.

Result:Very good generalization. Better performance than using the Fourier coefficients to recognize curves.


Curve recognition

Spatial resolution: 30

Network input size: 30

Hidden layer size: 15

Output size: 5

Training : Back-propagation with momentum.and adaptive learning rate

Sampled pointsPoint used for spline interpolationFitted spline

Curve recognition system demonstrated for hand drawn numeral recognition (successful recognition examples for 2, 8 and 3).


STS hand shape recognition

Model Matching

Precision grasp

Hand Configuration Classification

Step 2: The feature vector generated by the first step is used to fit a 3D-kinematics model of the hand by the model matching module. The resulting hand configuration is sent to the classification module.

Color Coded Hand Feature Extraction

Step 1 of hand shape recognition: system processes the color-coded hand image and generates a set of features to be used by the second step


STS hand shape recognition 1:Color Segmentation and Feature Extraction

Preprocessing

Color Expert(Network

weights)

Training phase: A color expert is generated by training a feed-forward network to approximate human perception of color.

Features

Actual processing: The hand image is fed to the augmented segmentation system. The color decision during segmentation is done by consulting color expert.

NN augmented segmentation system


STS hand shape recognition2:3D Hand Model Matching

A realistic drawing of hand bones. The hand is modelled with 14 degrees of freedom as illustrated.

Classification

Grasp Type

Result of feature extraction

Feature Vector

Error minimization

The model matching algorithm minimizes the error between the extracted features and the model hand.


Virtual Hand/Arm and Reach/Grasp Simulator

A power grasp and a side grasp

A precision pinch


Power grasp time series data

+: aperture; *: angle 1; x: angle 2; : 1-axisdisp1; :1-axisdisp2; : speed; : distance.


Core Mirror Circuit

Hand state

Mirror Neurons (F5mirror)

Association (7b) Neurons

Mirror Feedback

Object affordance

Mirror Neuron Output

Motor Program(F5 canonical)

Hand shape recognition & Hand motion detection

Hand-Object spatial relation analysis

Object affordance -hand state association

Object Affordances

Action recognition (Mirror Neurons)

Motor program

Motor execution

Mirror Feedback

Integrate temporal association

Motor program

F5canonical

F5mirror


Connectivity pattern

7b

Mirror Feedback

Object affordance (AIP)

7a

STS

F5mirror

Motor Program (F5canonical)


A single grasp trajectory viewed from three different angles

The wrist trajectory during the grasp is shown by square traces, with the distance between any two consecutive trace marks traveled in equal time intervals.

How the network classifies the action as a power grasp. Empty squares: power grasp output; filled squares: precision grasp; crosses:

side grasp output


Power and precision grasp resolution

(a)

(b)

Power GraspMirror Neuron

Precision PinchMirror Neuron

Note that the modeling yields novel predictions for time courseof activity across a population ofmirror neurons.


Research Plan

Development of the Mirror SystemDevelopment of Grasp Specificity in F5 Motor and Canonical Neurons

Visual Feedback for Grasping: A Possible Precursor of the Mirror Property

Recognition of Novel and Compound Actions and their ContextThe Pliers Experiment: Extending the Visual VocabularyRecognition of Compounds of Known MovementsFrom Action Recognition to Understanding: Context and Expectation

lecture 10. the mirror neuron system model (mns) 1 reading assignment:

Documents