A theory unifying movement, music and speech

DavidNLeePercep.onMovementAc.onResearchConsor.um

UniversityofEdinburghPaperpresentedattheICTsymposium‘Onthemeaningofmovement’

Paris,28March2019

Essence of life

• Allorganisms,fromcellstogiantsequoias,compriseasetofmoveablepartswhichtheyhavetocoordinatetocouplewithotherorganismsandwiththeinanimateenvironment.

Biological Informa;on

•  Inordertocoordinatemovements,organismsmustincludeanorganizedsystemofflowofbiologicalinforma.on.• Biologicalinforma.onisatemporalpaRernofphysicalpower,mainlyelectro-chemicalpowerflowinginthenervoussystem,orequivalentsystemincells.• Biologicalinforma.onisaphysicalanaloguequan.tythatdoessomethingdirectly,notviaanarbitrarydigitalcode.

Use of biological informa;on

Tocontrolpurposefulac.ons,by• Prescribingac.ons• Perceptuallymonitoringac.ons• Performingac.ons

Nature of biological informa;on

• Mustbeprospec.ve• Mustbeindependentofthelevelofphysicalpower• Mustbespecifictoitssource• Mustfitwithphysicallaws-notablytheInverseSquareLaw

Rho/Tau Body-environment perceptual informa;on

•  Intensityofs.mula.onatareceptorispropor.onalto(intensityofsource)/(squareofdistanceaway).• Meansthatdistanceawayisnotspecified.•  Thus,thereisnodirectinforma.onaboutdistance,velocity,accelera.on,orjerkofthegapbetweensourceandreceptor.•  Theonlyinforma.onthatisdirectlyavailableisRho,therela.ve-rate-of-changeofthegap,X,tothereceptor(orTau,theinverseofRho).•  Rhoofintensityatreceptor=-2xrhoofdistanceaway(fromtheInverseSquareLaw).

(dX /dt)/ X

S;mulus Rho => Perceptual Neural Rho

• Rho(op.calpower)=>Rho(neuralpower)• Rho(acous.cpower)=>Rho(neuralpower)• Rho(chemicalpower)=>Rho(neuralpower)• Rho(mech.power)=>Rho(neuralpower)• Rho(thermalpower)=>Rho(neuralpower)• Rho(electricpower)=>Rho(neuralpower)

Rho/Tau The currency of biological informa;on

Biologicalinforma.onmustbeinthesamecurrency,ofthesameform,bothwithinandacrossorganisms,tomakecouplingofmovementsefficient.Since,asshown,rho/tauistheperceptualinforma.oncurrency,thismustbethegeneralbiologicalinforma.oncurrency.

Controlling movement requires three func;onal Rhos

• Prescrip.ve• Perceptual• Performatory

Prescrip;ve Rhos

Ahypothesisedprescrip.verhoisfoundedontheecologicalinvariantgravity.Itis,whichequalstherhoofthegapbetweenanobjectfallingfromresttotheground,mul.pliedbyan‘oomph’factor,.Thatis

whereTisthetotalmovementdura.onand.met=0toT.

ρ(X )= λρG =2λt /(t2 −T2)

λρG λ

Effect of the oomph factor k (=1/λ) on the velocity of closure of a gap X

0.00

0.50

1.00

1.50

2.00

0 0.2 0.4 0.6 0.8 1

k=0.25k=0.5k=0.75k=1.0k=1.25

no

rma

lize

d v

el. o

f g

ap

X

normalized time

tauX = ktauG

Ac;vi;es found to use RhoG/TauG-coupling

•  Flying(bats,birds,humans)• Gesturing(babies)• Hibng(golf)•  Intercep.ngandcatching• Playingmusicalinstrument

• Reaching• Running•  Singing•  Suckling(babies)•  Swimming(cells)

Func;on of Basal Ganglia in Controlling Ac;on

•  Singleunitandmovementrecordingfromrhesusmonkeysreachingtotargets.

Regionsrecorded:• Gpe:globuspalidusext.• Gpi:globuspalidusint•  STN:sub-thalamicnucleus•  ZI:zonainserta

0

5

10

15

20

25

0

0.2

0.4

0.6

0.8

1

1.2

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

Nraw (MC)N (MC)Nmov (MC)Nraw (PC)N (PC)Nmov (PC)

Nraw (GPe)N (GPe)Nmov (GPe)Nraw (GPi)N (GPi)Nmov (GPi)Nraw (STN)N (STN)Nmov (STN)Nraw (ZI)N (ZI)Nmov (ZI)

neur

al p

ower

in c

ortic

al c

ells

(spi

kes

mtu-1

)neural pow

er in basal ganglia cells (spikes mtu -1)

normalized time (mtu)

movement

neural power x time

Conclusions: Func;ons of Basal Ganglia • GPinvolvedincrea.ngtheprescrip.veneuralpowerforrhoG.•  STNinvolvedinmonitoringtheperceptualneuralpowerabouttheac.on.•  ZIinvolvedincombiningtheprescrip.veandperceptualneuralpowerstocreateaperforma.veneuralpowertothemuscles.

Summary 1 Body-environment-coupling law

NP=powerflowinginneuronsEP=powerflowingfromenvironmenttobody

ρ(NP)= λρ(EP)

Summary 2 Neural Informa;on for Crea;ng Ac;ons

Note:Allarevectors

ρ(NPperceptual )

ρ(NPprescriptive )= ρG

ρ(NPperformatory )

RhoG/TauG-coupling makes possible

coordina;on and coupling of movements

Someexamplesfollow

Ella baby

Ella Fitzgerald and one-day-old baby in duet

-0.3

-0.2

-0.1

0

0.1

0.2

0.72 0.7205 0.721 0.7215 0.722 0.7225 0.723

soun

d pr

essu

re

time (s)

Aup

Tup

Adn

Tdn

kup k

dn

Skew A = A

up/(A

up+A

dn).

similarly for k and T

tauG parameters of a sound wavelet

-0.4

-0.2

0

0.2

0.4

0.309 0.31 0.311 0.312 0.313 0.314 0.315

raw wavwav G4

soun

d pr

essu

re

time (s)

Ella Fitzgerald's riff from "Too close for comfort"

Acous;c mo;on wavelets

Acous;c mo;on parameter profiles

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

0 0.5 1 1.5 2 2.5

Ella duration profiles

upT sm (ms)dnT sm (ms)

dura

tions

upT

and

dnT

of w

avel

et (m

s)

time (s)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0 0.5 1 1.5 2 2.5

Ella amplitude profiles

upA smdnA sm

ampl

itude

s up

A an

d dn

A of

wav

elet

(arb

dim

s)

time (s)

0

0.1

0.2

0.3

0.4

0.5

0.6

0 0.5 1 1.5 2 2.5

Ella oomph profiles

upK smdnK sm

oom

phs

upK

and

dnK

of w

avel

et (n

o di

ms)

time (s)

Measures on acous;c mo;on wavelets & profiles

0

0.2

0.4

0.6

0.8

1Ella audio profiles Props

upProp MN

dnProp MN

upProp CV

dnProp CV

propo

rtion

tauG-

guide

d

profilewav upT dnT upA dnA upK dnK

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6Ella audio profiles Ts

upT MN

dnT MN

upT CV

dnT CV

T (s)

profilewav upT dnT upA dnA upK dnK

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6Ella audio profiles As

upA MN

dnA MN

upA CV

dnA CV

A (ar

b dim

s)

profilewav upT dnT upA dnA upK dnK

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6Ella audio profiles Ks

upK MN

dnK MN

upK CV

dnK CV

K (no

ndim

.)

profilewav upT dnT upA dnA upK dnK

2 pianists playing same piece, different expression

Recita;on vs regular speech

0

0.2

0.4

0.6

0.8

1Burton audio profiles Props

upProp MN

dnProp MN

upProp CV

dnProp CV

propo

rtion

tauG-

guide

d

profilewav upT dnT upA dnA upK dnK

0

0.2

0.4

0.6

0.8

1S2 audio profiles Props

upProp MN

dnProp MN

upProp CV

dnProp CV

propo

rtion

tauG-

guide

d

profilewav upT dnT upA dnA upK dnK

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6Burton audio profiles Ts

upT MN

dnT MN

upT CV

dnT CV

T (s)

profilewav upT dnT upA dnA upK dnK

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6Burton audio profiles As

upA MN

dnA MN

upA CV

dnA CV

A (ar

b dim

s)

profilewav upT dnT upA dnA upK dnK

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6S2 audio profiles As

upA MN

dnA MN

upA CV

dnA CV

A (ar

b dim

s)

profilewav upT dnT upA dnA upK dnK

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6Burton audio profiles Ks

upK MN

dnK MN

upK CV

dnK CV

K (no

ndim

.)

profilewav upT dnT upA dnA upK dnK

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6S2 audio profiles Ks

upK MN

dnK MN

upK CV

dnK CV

K (no

ndim

.)

profilewav upT dnT upA dnA upK dnK

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6S2 audio profiles Ts

upT MN

dnT MN

upT CV

dnT CV

T (s)

profilewav upT dnT upA dnA upK dnK

Joyful vs Sad Handel’s Messiah

0

0.2

0.4

0.6

0.8

1Ferrier Zion audio profiles Props

upProp MN

dnProp MN

upProp CV

dnProp CV

propo

rtion

tauG-

guide

d

profilewav upT dnT upA dnA upK dnK

0

0.5

1

1.5

2

2.5

3

3.5

4Ferrier Zion audio profiles Ts

upT MN

dnT MN

upT CV

dnT CV

T (s)

profilewav upT dnT upA dnA upK dnK

0

0.5

1

1.5

2

2.5

3

3.5

4Ferrier Zion audio profiles As

upA MN

dnA MN

upA CV

dnA CV

A (ar

b dim

s)

profilewav upT dnT upA dnA upK dnK

0

0.5

1

1.5

2

2.5

3

3.5

4Ferrier Zion audio profiles Ks

upK MN

dnK MN

upK CV

dnK CV

K (no

ndim

.)

profilewav upT dnT upA dnA upK dnK

0

0.2

0.4

0.6

0.8

1Ferrier Desp audio profiles Props

upProp MN

dnProp MN

upProp CV

dnProp CV

propo

rtion

tauG-

guide

d

profilewav upT dnT upA dnA upK dnK

0

0.5

1

1.5

2

2.5

3

3.5

4Ferrier Desp audio profiles Ts

upT MN

dnT MN

upT CV

dnT CV

T (s)

profilewav upT dnT upA dnA upK dnK

0

0.5

1

1.5

2

2.5

3

3.5

4Ferrier Desp audio profiles As

upA MN

dnA MN

upA CV

dnA CV

A (ar

b dim

s)

profilewav upT dnT upA dnA upK dnK

0

0.5

1

1.5

2

2.5

3

3.5

4Ferrier Desp audio profiles Ks

upK MN

dnK MN

upK CV

dnK CV

K (no

ndim

.)

profilewav upT dnT upA dnA upK dnK

Sprin;ng (red lines - tauG-coupled movements)

PrincipalcollaboratorsMalcolmFairweatherMadeleineGrealyGert-JanPeppingDennisRewt

BenjamanSchogler

Pergolesi: Vanne, Vale, Dico Addio

Ella Fitzgerald glissando

Rhapsody in Blue opening glissando

Humpback whale sounds

Amjad Ali Khan: Sarod Pitch-glides

TauG-coupling of lip movements playing note on trombone (slowed down 40 times)

Analysis of lip movements

Parkinson’s disease

Paradoxicalmovement:•  Tau-coupledmovementsguidedfromwithoutcanbenormal.•  TauG-coupledmovementsguidedfromwithinarelessskillful.Ques5on:MightcouplingmovementtotauG-coupledsoundhelp?

Upward ‘Whoop’

TheshapesoftauG-coupledupward‘Whoops’(X=f0)

Up and down pitch(f0) ‘Whoop’

‘Whoops’ helped tauG-coupled sway in Parkinson’s

Collaborators

CathyCraigJonathanDelafield-BuRApostolosGeorgopoulosMayaGra.erMadeleineGrealyClaesvonHofstenMikeLandMarcLeeAudreyvanderMeerMateoObregon

GarethPadfieldGert-JanPeppingPaulReddishBenSchoglerJimSimmonsHeng-RuMayTanColwynTrevarthenAliceTurkBillWarrenRuudvanderWeel