Neurological Modeling & Cooperation:Automatic Acquisition of Triggered

Reactions, a Physiological Approach

Cooperative Control of Distributed Autonomous Vehicles in Adversarial Environments2001 MURI: UCLA, CalTech, Cornell, MIT

Mao/Massaquoi/Dahleh/Feron

May 14, 2001

UCLA

• Impression– Useful, complex group behavior is based on a

combination of relatively simple, perhaps identical Triggered Programmed-Reactions existing within a collection of nominally autonomous agents

• Hypothesis– The physiological basis for general behavioral

TPRs is the same as that for TPRs used for elemental body movement control/postural regulation

Basic Route

Examples

ss

s

d

s

gX

d

–Maintenance of upright standing and herding are functionally equivalent in body reference frame

– Both postural defense, herd containment and dancing via triggered reactions require• Assessment of continuous (though perhaps only piecewise,

intermittent) sensory information

• Selection of stereotyped movements (motion primitives) that are appropriately scaled and timed to project beyond the anticipated motion of the target

– Learning based on goals and reinforcement as dictated by environment and higher control levels

Selection,Timing,Scaling

Assessment,Prediction

Multichannel sensory information

Partially pre-programmedbehavior

Goals, constraints,Reward/failure

results

Observations

Observations (ct'd)

– Presumably, scaling, timing and selection also automatically learn to take into account supportive or obstructive features of environment, e.g.• Traction/motion characteristics of floor

• responsiveness of target

Or• Presence or absence of multiple actuators (e.g. ankles and hips

when falling forward, hips only when falling backward)

• Presence or absence of other herders on one side vs. another

– General sensitivity to environment may be physiological substrate for functionally useful group-aware behavior

Modeling Assumptions– Natural motor control system can be represented

as a hierarchy consisting of a high level, largely conscious, discrete state-machine-type ‘computer’ and a low/intermediate level, largely unconscious, continuous signal processing controller.

– In between are structures enabling the development of flexible, simple, semi-conscious ‘motor programs’ (behaviors) that address/adhere to the goals and constraints provided by the high level computer

– That is, our Interest:• Understand Control, Assessment and Learning at the

interface between higher and intermediate/low functional levels of natural sensorimotor system

Action Production

Action Monitoring

Continuous Action Control, Assessment & Adaptation (subconscious?)

Environment

Discrete Behavior Control, Assessment & Adaptation (conscious/preconscious?)

MURI

Natural Sensorimotor Control

• More specifically,– Natural Sensorimotor control Hierarchy

• High level Goals (conscious)

– e.g. win point vs. conserve energy

• Strategic Planning/Decisions (conscious) – e.g. return to right rear baseline

• Tactical Objectives (preconscious/”overlearned’?)

– e.g. contact ball with racket face having particular orientation and velocity

• Tactical Assessment/Planning/Decisions (preconscious/’overlearned’?/development of “motor program”)

– assess/predict ball trajectory, spin, body location in court

– use forehand, assume particular posture, generate specific trajectory

Natural Sensorimotor Control

MURI

– Natural Control Hierarchy (cont’d)• Action (force, position) generation& on-line

control (subconscious)

• Action (continuous trajectory) improvement (optimization?) with practice (subconscious motor learning)

• Behavior (discrete program, trajectory) improvement (optimization?) with practice (conscious--> preconscious: ‘tactical motor learning’, ‘motor programming’)

• Behavior improvement (optimization?) with practice (conscious: ‘strategic motor learning’, ‘gamesmanship’)

Natural Sensorimotor Control

MURI

• Natural Sensorimotor Control System

Natural Sensorimotor Control

ST

MTR SENS

MTBGInterface

between highand intermediate/lowcontrol levels involvessensorimotor and association cortices (especially frontal)and the Basal Ganglia.These link ‘automatic’ behavior and reward.Cerebellum likely contributes optimization

(frontal)ASSOC

(parietal)ASSOC

Cbl

Neural signals------------------ executive

sensory

consciousnessgradient

Mtr CtxBrainstem orSpinal Cord

Segment

Im AntCblL Ant

Cbl

Putamen& GP

Caudate& GP

Frontal & ParietalAssoc Ctx

BodyForce/Motion

Muscle & tendon,Joints, skin

“highest level”PLANS (strategy)

“middle level” (“high” and “intermediate”)

PROGRAMS (tactics)

“lower level”ACTION

(force, velocity)

“Motor Servo”

Vestib

Visual

M. Cbl Flocc Cbl

Human motor control principal information flow (adapted from V. Brooks, 1986)

Motor Servo(proprioceptive)

Mtr Ctx

Im AntCblL Ant

Cbl

Putamen & GP, SN (Basal Ganglia)

M. Cbl Flocc Cbl

“high level”PROGRAMS

(discrete control) (tactics:trajectories,

cues)

Frontal & ParietalAssoc Ctx

“highest level”PLANS,

ALGORITHMS (free assoc, strategy)

“intermediate level”CONTROL

(continuous control)(stability, tracking, stiffness,

scaling, movement time)

• MURI to specifically study ‘Programming’ of Triggered Reaction Loops

L Post Cerebellum

Caudate & GP (Basal Ganglia)

TPR LoopCircuitry

Frontal & Parietal Peri-Sensorimotor Ctxs

MURI Goals

Proposed MURI project (Year 1)

– Acquisition of triggered motor reactions

Robot armImplementing virtual targetsandenvironment

Video monitorshowing virtual targetsandenvironment

Proposed MURI project questions: with respect to physiological structures known

or suspected to be involved in TPRs

(Year 1)– What are the motion primitives?

– How are they generated, scaled, timed, triggered?

– What and how is continuous sensory information used?

– How is prediction performed … evidence for internal models?

– How is reinforcement/suppression mediated?

– What is the statistical nature of the learning and programming?

Background studies and resources

– Existing models for intermediate and low/level motor control based on cerebellar and sensorimotor cortical physiology

– Robot arm laboratory– Access to human subjects including those with diseases of

the basal ganglia and cerebellum

Beyond Year 1

– Useful, complex group behavior may emerge from relatively simple, perhaps identical Triggered Programmed-Reactions existing within a collection of nominally autonomous agents

– Link to emergent group behavior possible via experimental observations / prior and similar approaches in Air Traffic Control (eg Mao, Feron and Bilimoria, IEEE ITS, 06/01)

Research funded by NASA, ONR