memory effects in metaplastic binarized neural networks · 2019. 6. 22. · binarized neural...

MEMORYEFFECTSINMETAPLASTICBINARIZEDNEURALNETWORKS

AxelLaborieux,TifennHirtzlin,LizaHerrera-Diez,DamienQuerliozCentredeNanosciencesetdeNanotechnologies,Univ.Paris-Sud,CNRS,France

BinarizedNeuralNetworks(BNNs)areattractiveforlowpowerhardwareimplementationofartificialintelligence.Inthiswork,westudyhowmetaplasticbinarizedsynapsesenableBNNstobeusedintheframeworkofmulti-headlearning,i.e.sequentiallylearningseveraltasksandinferencerequiresspecifyingthetask.

Binarized Neural Network (BNN) Synapses{+1,-1}

Synapses{+1,-1} Synapses{+

1,-1}

Hubara,Courbariauxetal.NIPS2016

•  BinarizedNeuralNetworksachievestateoftheartresultsinimagerecognition,andrelyonsimplelogicoperations.

•  Abinaryweightisthesignofafloatingvaluewhichisnotaweightasthelossandgradientsarecomputedusingbinaryvaluesonly.

→ ↔

MagneticTunnelJunction

TE

BE

TE

BE

LRS HRS

ResistiveRAMPhaseChangeMemory

Nanodevices Using Magnetism, Spintronics, Ionics Provide Artificial Synapses

Richsynapticbehaviorscanbeemulatedbynanodevices

Real Synapses Are Metaplastic

•  Connectionnistmodelsaresubjecttocatastrophicforgetting,whenlearningsequentiallyseveraltasks.

•  Rememberingprevioustasksandlearningnewtasksseemsincompatiblewithrespecttosynapseplasticity:weneedtopreventsynapsesfromchanginginordertorememberandlearningrequiressynapsestochange.

•  Metaplasticsynapseswithawiderangeofplasticityareawayofsolvingthisparadox. Fusietal.Neuron2005

Weight:-1+1

Consolidation Processes for BNN Synapses •  OptimizationisdonewithAdam(Kingma,LeiBaICLR2015)onthefloatingvalueunderlyingthe

binaryweight.p(t)isthepointonthehypercubeonwhichtheBNNisevaluatedattimestept.

•  Synapticmetaplasticityisintroducedbymodulatingtheadamupdate.

•  Thefloatingvalueofthebinaryweightcanencodeforametaplasticstate.Asynapseisdescribedbyabinaryweightusedforinferenceandthehiddenfloatingvalueusedforlearningandmemorypurpose.

Permuted Tasks Benchmark

Non Permuted Tasks

•  Fixedpermutationsofpixelsprovidealistofnoncorrelatedtasks.

10 20 30 40Epochs for each tasks

50

60

70

80

90

100

Testaccuracies

Metaplastic BNN

1st tsk

2nd tsk

10 20 30 40Epochs for each tasks

50

60

70

80

90

100

Testaccuracies

Regular BNN

1st tsk

2nd tsk

−4 −2 0 2 40.0

0.2

0.4

0.6

0.8

1 .0fmeta(Wfloat) = 1 − tanh(|Wfloat|)2

−4 −2 0 2 40.0

0.2

0.4

0.6

0.8

1 .0

fmeta(Wfloat) = 1 |Wfloat|<1

0 10 20

0

5

10

15

20

25

0 10 20

0

5

10

15

20

25

0 10 20

0

5

10

15

20

25

0 10 20

0

5

10

15

20

25

•  MetaplasticBNNscanlearnandconsolidateknowledgeandstillbeableoflearninganewtask.

−4 −2 0 2 40.0

0.2

0.4

0.6

0.8

1 .0fmeta(Wfloat) = 1 − tanh(|Wfloat|)2

•  Poorlycorrelatedtasksaregraduallyforgottenbyregularnetworksbecauseofongoingplasticity,whereasmetaplasticityenablesthenetworktolearnseveraltaskssequentially.fCIFAR10correspondstoCIFAR10featuresextractedbyResNet18pretrainedonImageNet.

•  Learningcorrelatedtasksismoredifficultassomeoftherelevantpixelsareshared.Theaccuracyofthefirsttaskabruptlydropswhilelearningthesecondtaskwithregularmodels.

•  Startingwithrandomlyconsolidatedsynapsesbytuningthewidthofweightinitializationisanothermetaplasticityingredientwhichimprovesperformance.

0.3 0.7 1.1 1.5 1.9 2.3Weight initialization width

20

30

40

50

60

70

80

90

100

Testaccuracies

Metaplastic BNN

1st tsk

2nd tsk

0.3 0.7 1.1 1.5 1.9 2.3Weight initialization width

20

30

40

50

60

70

80

90

100

Testaccuracies

Regular BNN

1st tsk

2nd tsk

−4 −2 0 2 40.0

0.2

0.4

0.6

0.8

1 .0fmeta(Wfloat) = 1 − tanh(|Wfloat|)2

•  Fornoncorrelatedtasksitissufficienttostartlearningwithonlyplasticsynapsesandconsolidateuponlearning.Butforcorrelatedtasks,startinglearningwithconsolidatedsynapsesprovideplasticsynapsesforthenexttask.

Conclusions •  Neuroscientists(Fusietal.)haveshownthatbiologicallyplausiblesynapses

maybedescribedbymorethanoneparameter(i.eoneweight)andthatcomplexsynapsedynamicsallowsforlongtermmemory.

•  BinarizedNeuralNetworksseemstocontainonly+1and-1synapticweights,yetweightswithfloatingvaluesfarfrom0arelesslikelytoswitchthanweightswithfloatingvaluescloseto0.WecanthusintroduceametaplasticdynamicsandweshowthatitallowsBNNstohavelongtermmemory.

•  Spintronicsnanodevicesarepromisingfordesigningmetaplasticsynapsesdirectlyfromthephysicsofthematerialandwithlowenergycost.

Count

Count