neural wiring optimization paradigm christopher cherniak & zekeria mokhtarzada committee for...

Neural Wiring Optimization ParadigmNeural Wiring Optimization Paradigm

Christopher Cherniak & Zekeria MokhtarzadaChristopher Cherniak & Zekeria Mokhtarzada  Committee for Philosophy & the SciencesCommittee for Philosophy & the Sciences

University of MarylandUniversity of Maryland  

www.glue.umd.edu/~cherniak/ www.glue.umd.edu/~cherniak/ 

Network optimization in brainNetwork optimization in brain"Save wire": connection minimization"Save wire": connection minimization

Steiner treeSteiner treeComponent placement optimization [QAP]Component placement optimization [QAP]

NP-complete problems: How does Nature solve them? NP-complete problems: How does Nature solve them? Physics Physics Optimization Optimization Neural Structure Neural Structure

Two-way street between: philosophy & empirical computational neuroanatomy. Bounded-resource philosophical framework: "We don't have God's brain." Human computational resources are critically constrained; in particular, neural connectivity. Hence, strong pressure to optimize use of brain's limited neuro-wiring.

Generative rule of neural structure: "Save wire." Combinatorial network optimization theory.

1. Neuron Arbor Optimization. "Neuron arbors act like flowing water." Tree-optimization concept: Steiner tree. Locally, for isolated "Y-tree".

(a) Fluid-dynamic model: At junction, trunk costs more than branches. Fits fluid-dynamical model: relation of diameters of branches to trunk minimizes internal wall-drag of laminar flow [ t 3 = b1

3 + b23 ].

(b) Fluid-static model: Then at local junction, for given trunk & branch weights, cost-minimizing branch angle is via vector-mechanical tug-of-war ("triangle of forces law") [ cos θ = (t 2 - b1

2 - b22) / 2b1b2 ].

Best fit is for cost = total volume (vs surface area, length, signal-delay).

Globally, for multi-junction tree. Local optimization large-scale optimization. = combinatorial problem ("NP-hard"). E.g., for 12 terminals, 6 x 108 alternative topologies must be exhaustively searched(!)

Observed trees cost ~5% more than volume-minimal trees, across all topologies: For planar dendrites (rabbit retina ganglion, amacrine; cat ganglion), axons (mouse thalamus). Also nonliving trees, e.g., river drainage networks.

2. Component Placement Optimization [qap]: Given: connections among components. Find: layout of components, on 2-d surface, that

minimizes total cost of interconnections (e.g., wirelength). NP-hard: For n-component system, n! alternative layouts must be exhaustively searched.

"Brain as ultimate microchip" = organizing principle, at multiple hierarchical levels, of nervous system anatomy.

"Why brain is in head" of invertebrates & vertebrates: minimizes total nerve connection costs to & from brain.

Nematode C. elegans nervous system is first ever fully mapped. The actual layout of 11 ganglia is the wirelength-minimizing one [87,803 um] -- out of 40 million possibilities(!)

Similarly, CPO predicts actual layout of 40 functional areas of cat cerebral cortex. To best-in-a-billion optimality level.

To computational limits of detectability [ 40! = 1047 layouts ]. "Best of all possible brains," a predictive success story.

3. Mechanisms of NP-hard neural optimization. Blind trial & error exhaustive search for, e.g., minimum-wiring layout of a 50 component system would require more than age of Universe. Generally, exact solutions are computationally intractable.

So, quick & dirty approximate/probabilistic heuristics.

In particular, optimization "for free, directly from physics". (i) E.g., "instant neuron arbors, just add water":

i.e., arbor optimization via fluid dynamics. (ii) Layout optimization (e.g., for roundworm ganglia) via "mesh of springs" force-

directed placement simulation. Also, simple genetic algorithm performs well for worm ganglia, and for cortex

areas. Functional role of "best of all possible brains" wiring optimization? Perhaps an economical means of complex structure generation that is transmissible thru limited-capacity "genomic bottleneck".

So: Physics -> Optimization -> Neuroanatomy

A neural optimization paradigm is a structuralist position, postulating innate abstract internal structure. (Vs tabula rasa, empty-organism, no structure in hardware.) Continental rationalist; but for brain rather than mind. Non-Genomic Nativism: Hardwired; however, not via genome, but by exploiting basic physical processes -- complex biological structure as self-organizing.

Metaphilosophy: Holistic, vs compartmentalized, model of interrelation of philosophy & science.

www.glue.umd.edu/~cherniak/

Minimal Spanning Tree Steiner Tree

Minimal spanning tree (A) and Steiner tree (B) for 5 nodes on a plane. Steiner tree is shorter, but much more computationally costly to construct (NP-hard). Steiner tree applies to neuron arbors; with cost measure as arbor volume, not tree length.

Arbor OptimizationArbor Optimization

Neuron arbor junction (cat retina ganglion cell dendrite). (a) Branch and trunk diameters conform to t3 = b1

3 + b23, a fluid-dynamic model for minimum internal walldrag of pumped

flow (laminar regime). (b) In turn, angle θ conforms to the "triangle of forces" law, a cosine function of the diameters: cos θ = (t2 - b1

2 - b22) / 2b1b2 . This yields the minimum volume for a

Y-tree junction. "Neuron junctions act like flowing water.”

The simplest network optimization problem: Find shortest distance between 2 points. -- Via vector-mechanics.

Minimizing cost of 1-junction network. Via tug-of-war "Triangle of Forces" cos law of vector mechanics.

Even a simple 5-terminal tree has 15 alternative possible topologies, or connecting patterns. Arbor optimization requires not just (a) best embedding of an arbor, but also (b) exhaustive search of all possible topologies.

Optimization analysis of 5-terminal subtree of rabbit retina ganglion cell dendrite. Dendritic arbors best fit minimum-volume model.

Vol err: 2.6% Surf err: 27.2% Leng err: 60.6%

Opt

imal

topo

ls

Act

ual t

opol

s

Complex biological structure arising "for free, directly from physics". -- "Instant arbors, just add water." In each case, from micron to meter scale, actual structure is within a few percent of minimum-vol configuration shown.

Elm twig. Arbor includes multiple junctions.

Lichtenberg Figure, a planar high-voltage discharge pattern. This nonliving structure also conforms to the min-vol arbor model.

Mouse thalamus axon arbor. (A) Actual multi-junction tree in broken lines, optimal embedding of actual topol (wrt vol) in solid lines; Vol err: 2.2%. (B) "Best of all possible topologies" (wrt vol); Vol err: 2.5%. Only 10 of the 10,395 alternative topologies here have lower total vol costs, when optimally embedded, than the actual topol. And, topol search yields little improvement, compared to embedding.

9-terminal arbor of mouse thalamus axon: Distribution of vol costs of all 135,135 possible topologies, each optimally embedded. Histogram shows usual pattern for natural arbors, living and nonliving -- more costly topologies are more common, cheapest ones are rarest. The most costly optimally embedded "pessimal" layouts have only about 12% greater vol than cheapest one. Hence, for optimization, "topology does not matter".

0

5

10

15

20

25

30

35

40

2325 2395 2465 2535 2605

Volume Cost

Fre

qu

ency

(x1

000)

Component placement optimization: Connection cost-minimization. Total system is 1-d array of components 1 - 3 (each dimensionless, with 0 length-cost). (A) a suboptimal layout (cost: 5), vs (B) globally optimal layout (cost: 4).

A B

(Cost: 5) (Cost: 4)

1 2 3 3 1 2

Component Placement OptimizationComponent Placement Optimization

Head Tail

1.3 mm

C. elegans adult hermaphrodite nematode. Nervous system contains 11 ganglia.

PH

AN

RN

GD

OL

AV

NR

V

VC

a

VC

p

PA

DR

LU

C. Elegans ganglia: their body locations and schematized shapes.

50 um

Cell bodies of all neurons in head of C. elegans. (White et al, 1986)

Neuron process tracts in head of C. elegans.

AINL & AINR, interneurons with cell bodies in Lateral ganglion in head. Every synapse is shown; all are in Ring.

Complete ganglion-level connectivity map for C. elegans nervous system. Each horizontal microline represents one of its 302 neurons. Horiz scaling: ~ 100x. This actual ganglion layout requires the least total connection length of all ~40 million alternative orderings.

Zoom-in:

PH,AN,DO = Ganglia

+ = Cell body

= Sensor

~ = Muscle

= Chemical Synapse

= Electrical Synapse

0

2000

4000

6000

8000

10000

12000

87.0 111.3 135.6 159.9 184.2 208.5 232.8 257.1 281.4 305.7 330.0 354.3

Fre

qu

en

cy

Layout Wirecost (mm)

Distribution of wirecosts (total wirelength) of all possible layouts of ganglia of C. elegans. 10,000 bin histogram of 39,916,800 alternative orderings. Least costly and most costly layouts are rarest. The last-place "pessimal" layout requires ~ 4 times as much total connection fiber as the actual optimal one.

Actual

Generation: 16 8 11 10 7 5 3 12 6 4 2 1 220947.75----------------------------------------- 6 12 7 10 11 3 5 8 4 2 1 168993.75------------------------------------- 6 12 7 10 11 3 5 8 4 2 1 168993.75------------------------------------- 1 11 10 7 5 3 12 6 4 8 2 144033.50-------------------------- 8 11 10 7 5 3 12 6 4 2 1 220947.75----------------------------------------- 8 11 10 7 5 3 12 6 4 1 2 221697.75----------------------------------------- 6 12 7 10 11 3 5 2 4 1 8 124148.75----------------- 1 11 10 7 5 3 12 6 4 8 2 144033.50-------------------------- 8 11 10 7 12 3 5 6 4 2 1 221238.75----------------------------------------- 8 11 10 7 5 3 12 6 4 1 2 221697.75----------------------------------------- 185673.30========================================= Generation: 17 12 6 7 10 11 3 5 2 4 1 8 124964.75----------------- 6 12 7 10 11 3 5 8 4 2 1 168993.75------------------------------------- 6 12 7 10 11 3 5 8 4 2 1 168993.75------------------------------------- 1 11 10 7 5 3 12 6 4 8 2 144033.50-------------------------- 6 12 7 10 11 3 5 8 4 2 1 168993.75------------------------------------- 8 11 10 7 5 3 12 6 4 2 1 220947.75----------------------------------------- 6 12 7 10 11 3 5 2 4 1 8 124148.75----------------- 1 11 10 7 5 3 12 6 4 8 2 144033.50-------------------------- 1 11 10 7 5 3 12 6 4 8 2 144033.50-------------------------- 1 11 10 7 5 3 12 6 4 8 2 144033.50-------------------------- 155317.66=============================== Generation: 18 12 6 7 10 11 3 5 2 4 1 8 124964.75----------------- 4 11 10 7 5 3 12 6 1 8 2 150274.00----------------------------- 6 12 7 10 11 3 5 2 4 1 8 124148.75----------------- 1 11 10 7 5 3 12 6 4 8 2 144033.50-------------------------- 1 11 10 7 5 3 12 4 6 8 2 144277.00-------------------------- 1 11 10 7 5 3 12 6 4 8 2 144033.50-------------------------- 6 12 7 10 11 3 5 2 4 1 8 124148.75----------------- 1 11 10 7 5 3 12 6 4 8 2 144033.50-------------------------- 1 11 10 7 5 3 12 6 4 8 2 144033.50-------------------------- 1 11 10 7 5 3 12 6 4 8 2 144033.50-------------------------- 138798.08======================== Generation: 19 12 6 7 10 11 3 5 2 4 1 8 124964.75----------------- 6 12 7 10 11 3 5 2 4 1 8 124148.75----------------- 6 12 7 10 11 3 5 2 4 1 8 124148.75----------------- 1 6 12 7 10 11 3 5 2 4 8 117829.75-------------- 6 12 7 10 11 3 2 5 4 1 8 125990.75------------------ 12 6 7 10 11 3 5 1 4 2 8 125909.75------------------ 6 12 7 10 11 3 5 2 4 1 8 124148.75----------------- 6 12 7 10 11 3 5 2 4 1 8 124148.75----------------- 1 11 10 7 5 3 12 6 4 8 2 144033.50-------------------------- 1 11 10 7 5 3 12 6 4 8 2 144033.50-------------------------- 127935.70===================

GenAlg, a simple genetic algorithm, rapidly and reliably finds the optimal (minimum wirelength) layout of C. elegans ganglia among 11! alternatives. Here, population size is only 10.

Mea

n W

ire

cost

(u

m)

Generation Number

GenAlg rapidly finds the optimal (minimum wirelength) layout of C. elegans ganglia among 11! alternatives. Here, in only 150 generations.

80000

130000

180000

230000

280000

0 20 40 60 80 100 120 140 160

Tensarama, a force-directed placement algorithm for optimizing layout of C. elegans ganglia. This "mesh of springs" vector-mechanical energy-minimization simulation represents each of the worm's ~ 1,000 connections acting upon the movable ganglia PH, AN, etc. The key feature of Tensarama performance for the actual worm connectivity matrix is its low susceptibility to local minima traps. -- Unlike Tensarama performance for small modifications of the actual connectivity matrix ("butterfly effect"), and unlike FDP algorithms in general for circuit design. Here Tensarama is trapped with a “killer” connectivity matrix that differs from the actual matrix by only one less connection.

Adjacency Rule conformance, vs total wirecost, of 100,000 C. elegans ganglion layouts. ["If components a & b are connected, then a & b are adjacent."] Generally, the Adjacency Rule is not an effective heuristic to good wirecost. However, the small set of layouts best fitting the Adjacency Rule (points at far left) behave markedly differently: they correspond closely to the best wirecost layouts.

Actual

Layout Adjacency Cost

La

yo

ut

Wir

ec

os

t (m

m)

Parcellation of functional areas of macaque cerebral cortex. Component placement optimization analysis of layout of 17 core areas (white) of visual cortex, along with immediately contiguous edge areas (dark gray). Reported interconnections among core areas are indicated by lighter straight lines. Rostral is to right. In a connection cost analysis, this actual layout of the core visual system ranks in the top one-millionth of all alternative layouts.

Core

Edge

Cerebral cortex of cat. Placement of 39 interconnected functional areas of visual, auditory, somatosensory systems. Exhaustive search of samples of alternative layouts suggests this actual layout ranks in top 100 billionth of all possible layouts wrt Adjcost. -- "Best of all possible brains"?

ReferencesReferences

Cherniak, C (1995) Neural component placement. Cherniak, C (1995) Neural component placement. Trends in NeurosciencesTrends in Neurosciences, , 18: 522-527. 18: 522-527.

Cherniak, C., Changizi, M., & Kang, D. (1999) Large-scale optimization of Cherniak, C., Changizi, M., & Kang, D. (1999) Large-scale optimization of neuron arbors. neuron arbors. Physical ReviewPhysical Review E, 59: 6001-6009. E, 59: 6001-6009.

Cherniak, C, Mokhtarzada, Z, Rodriguez, R, & Changizi, K (2004) Global Cherniak, C, Mokhtarzada, Z, Rodriguez, R, & Changizi, K (2004) Global optimization of cerebral cortex layout. optimization of cerebral cortex layout. Proc Nat Acad SciProc Nat Acad Sci, 101: 1081-1086. , 101: 1081-1086.

Cherniak, C (2005) Innateness and brain-wiring optimization. In: A Zilhao, ed, Cherniak, C (2005) Innateness and brain-wiring optimization. In: A Zilhao, ed, Evolution, Rationality and CognitionEvolution, Rationality and Cognition (Routledge) 103-112. (Routledge) 103-112.

www.glue.umd.edu/~cherniak/www.glue.umd.edu/~cherniak/