exploring and exploiting intrinsic synergies of...
TRANSCRIPT
Exploringandexploitingintrinsicsynergiesofmultivariatesystems
FernandoE.RosasResearchFellow
CentreofComplexityScience
DepartmentofMathematics
DepartmentofElectricalandElectronicEngineering
ImperialCollegeLondon
!!!!!Imperial)College)London)))Centre)for)Complexity)Science)
!
!
Complexity
Science
Acknowledges:
Thisworkistheresultofthefortunatecollaborationwith:
!2
PedroA.M.MedianoDepartmentofComputingImperialCollegeLondon HenrikJ.Jensen
DepartmentofMathematicsImperialCollegeLondon
MartinUgarteDepartmentofComputerScienceUniversitéLibreBruxelles
MichaelGastparSchoolofComputerandCommunicationSciencesEPFL
BorzooRassouliSchoolofComputerScienceandElectronicEngineeringUniversityofEssex
DenizGündüzDepartmentofE&EEngineeringImperialCollegeLondon
Acknowledges:MarieSlodowska-CurieActions
!3
Objectives:
(a) tounderstandandexploitintrinsicstatisticalsynergiesofmultivariatesystems
(b) applythistodataprivacyandneuraldataanalysis
!4
Today’smenu
1.Self-organisation
2.O-information
3.Dataprivacy
4.Summaryandcurrentwork
!5
1.Self-organisation:whatisapattern
Whatisapattern/structure?
Shannon—>compressibilityofanstatisticalsourceRegularities/interdepedencies(i.e.deviationsfromstatisticalindependence)
Organisationasstatisticalinterdependency
!6
1.Whatisself-organization?
Attractor:destination/resultoftheevolutionLinearrecurrence/evolutionissimplesimple/uninterestingattractorsNon-linearevolutioninteresting/strangeattractors
Challenge:Isitpossibletorelatetheattractorspropertiestoorganizationproperties?
!7
1.Whatisself-organization?
Attractor:destination/resultoftheevolutionLinearrecurrence/evolutionissimplesimple/uninterestingattractorsNon-linearevolutioninteresting/strangeattractors
Idea:high-ordercorrelationsallowtodistinguishqualitativelydifferentarrangements…
redundancy! synergy!
1.Self-organisation:ourapproach
Approachtostudyspontaneouscreationofcorrelations:
—Consideranflatinitialdistribution(independence)
—Computeit’stimeevolution()usingthecorresponding“masterequation”
—Iterateformanysteps,oruntilitreachastationarystate.
—Analysethe“structureofthecorrelations”of.
Whystartfromanuniformdistribution?Becauseanycorrelationfoundisduetotheevolutionandnottheinitialcondition.
!8
µ0
µt
µtst+1 = µtst
(X1tst , . . . , X
Ntst)
1.Self-organisation:dynamicsassculpture…
!9
“TheAtlas”MichelangeloCirca1530-1534
“TheAwakeningSlave”MichelangeloCirca1520-1523
�t
�0t
1.Self-organisation:entropyisnotenough
Iseasytoprovethat,becauseofdeterminism,thejointShannonentropyofthesystemisnon-increasing.
Thisnotsufficienttoguaranteeself-organization!!(example:fixpointattractors)
!10
H(X1t+1, . . . , X
Nt+1) H(X1
t , . . . , XNt )
InformationisbeingdissipatedIttakesmoreinformationtospecifyarandomstate
thanapointinanattractor
1.Self-organisation:entropyisnotenough
Iseasytoprovethat,becauseofdeterminism,thejointShannonentropyofthesystemisnon-increasing.
Thisnotsufficienttoguaranteeself-organization!!(example:fixpointattractors)
!11
H(X1t+1, . . . , X
Nt+1) H(X1
t , . . . , XNt )
InformationisbeingdissipatedIttakesmoreinformationtospecifyarandomstate
thanapointinanattractor
Keyidea:Thedestructionofinformation
cancreatecorrelations!!
H(X) H(Y )
H(X) H(Y )
I(X;Y )
1.Self-organisation:proposedframework
!12
Ourproposedtotalentropydecomposition:
F.Rosas,P.A.M.Mediano,M.Ugarte,H.J.Jensen,“Aninformation-theoreticapproachtoself-organization:emergenceofcomplexinterdependenciesincoupleddynamicalsystems”,Entropy20,no.10(2018):793.
"Residualentropy”
“Bindingentropy”
1.Self-organisation:proposedframework
Decompositionbysharingmodes:
!13
bymeasuringiagentscanguessthestateofn…
1.Self-organisation:proposedframework
Decompositionbysharingmodes:
where
!14
1.Self-organisation:proofofconcept
Rule232
!15
1.Self-organisation:proofofconcept
Rule232
!16
1.Self-organisation:proofofconcept
Rule30
!17
1.Self-organisation:proofofconcept
Rule30
!18
1.Self-organisation:proofofconcept
!19
1.Self-organisation:proofofconcept
!20
0 5 10 15
0.0
0.5
1.0
yL
Rule 232: Redundancy
0 5 10 15
0.0
0.5
1.0
Rule 30: Moderate synergy
0 5 10 15
0.0
0.5
1.0
Number of cells (L)
yL
Rule 90: Strong synergy
0 5 10 15
0.0
0.5
1.0
Number of cells (L)
Rule 106: Mixed profile
Rule106
!21
Today’smenu
1.Self-organisation
2.O-information
3.Dataprivacy
4.Summaryandcurrentwork
2.O-information:fundamentals
1.Interaction-information:
2.Totalcorrelation:
3.Dualtotalcorrelation:
Stateofaffairs:
(+)TCandDTCaremetricsofglobalcorrelationstrength.
(-)TC=0ifandonlyifDTC=0.Besidesthat,theirrelationshipisunclear.
(+)
(+)=TC-DTC.
(-)Themeaningoftheinteraction-informationforn>3isunclear.
!22
ispositiveforredundantsystems,andnegativeforsynergisticones.I(X1;X2;X3) = I(X1;X3) + I(X2;X3)� I(X1X2;X3)
I(X1;X2;X3)
TC(Xn) =nX
j=1
H(Xj)�H(Xn)
DTC(Xn) = H(Xn)�nX
j=1
H(Xj |Xn�j)
I(X1;X2; . . . ;Xn) = �X
�✓{1,...,n}
(�1)|�|H(X�)
2.O-information:fundamentals
1.Interaction-information:
2.Totalcorrelation:
3.Dualtotalcorrelation:
!23
HowcouldI“fix”theinteraction-infofor
anyn??
TCorDTC?thatisthequestion…
Theinformation-theoreticHamlet
TC(Xn) =nX
j=1
H(Xj)�H(Xn)
DTC(Xn) = H(Xn)�nX
j=1
H(Xj |Xn�j)
I(X1;X2; . . . ;Xn) = �X
�✓{1,...,n}
(�1)|�|H(X�)
2.O-information:thetwofacesofinterdependency
Considerasystemwhere,describedbyap.d.f..
!24
nX
j=1
log |Xj |
N (Xn) =X
j
log |Xj |�H(Xn)
H(Xn) = �X
xn
pxn log pxn
Negentropy
Entropy
Xn = (X1, . . . , Xn) Xj 2 Xj pXn(xn)
H(Xn)
N (Xn)
2.O-information:thetwofacesofinterdependency
Considerasystemwhere,describedbyap.d.f..
!25
nX
j=1
log |Xj |
Negentropy
Entropy
X1, X2 2 {0, 1}
pX1X2(0, 0) = pX1X2(1, 1) = 0
pX1X2(0, 1) = pX1X2(1, 0) = 1/2
H(X1X2) = N (X1X2) = 1
(0, 0) (0, 1)
(1, 0) (1, 1)+
+
Example:
H(Xn) = �X
xn
pxn log pxn
N (Xn) =X
j
log |Xj |�H(Xn)
Xn = (X1, . . . , Xn) Xj 2 Xj pXn(xn)
H(Xn)
N (Xn)
2.O-information:thetwofacesofinterdependency
!26
N (X5)N (X4)N (X3)N (X2)N (X1)
TC
N (X6)
nX
j=1
log |Xj |
H(Xn) = �X
xn
pxn log pxn
N (Xn) =nX
j=1
N (Xj) + TC
IndividualconstraintsCollectiveconstraints
Marginalnegentropy: N (Xj) = log |Xj |�H(Xj)
Collectiveconstraints: TC(Xn) = N (Xn)�nX
j=1
N (Xj) =nX
j=1
H(Xj)�H(Xn)
2.O-information:thetwofacesofinterdependency
!27
N (X5)N (X4)N (X3)N (X2)N (X1)
TC
N (X6)
nX
j=1
log |Xj |
H(Xn) = �X
xn
pxn log pxn
N (Xn) =nX
j=1
N (Xj) + TC
(0, 0) (0, 1)
(1, 0) (1, 1)+
+Example: (0, 0) (0, 1)
(1, 0) (1, 1)+ +
N (X1X2) = 1
TC(X1X2) = 1
N (X1X2) = 1
TC(X1X2) = 0
IndividualconstraintsCollectiveconstraints
2.O-information:thetwofacesofinterdependency
!28
R1R2 R3 R4 R5
N (X5)N (X4)N (X3)N (X2)N (X1)
TC
DTC
N (X6)
R6
N (Xn) =nX
j=1
N (Xj) + TC
H(Xn) =nX
j=1
Rj +DTC
nX
j=1
log |Xj |
Privaterandomness
Collectiveconstraints
Sharedrandomness
Individualconstraints
Residualentropies:
Bindingentropy: DTC(Xn) = H(Xn)�nX
j=1
H(Xj |Xn�j)
Rj = H(Xj |Xn�j)
2.O-information:thetwofacesofinterdependency
!29
R1R2 R3 R4 R5
N (X5)N (X4)N (X3)N (X2)N (X1)
TC
DTC
N (X6)
R6
N (Xn) =nX
j=1
N (Xj) + TC
H(Xn) =nX
j=1
Rj +DTC
nX
j=1
log |Xj |
Privaterandomness
Collectiveconstraints
Sharedrandomness
Example: X1 = X2 = X3
TC = 2 > DTC = 1
X3 = X1 �X2
TC = 1 < DTC = 2
redundancy! synergy!
Individualconstraints
2.O-information:thetwofacesofinterdependency
!30
R1R2 R3 R4 R5
N (X5)N (X4)N (X3)N (X2)N (X1)
TC
DTC
N (X6)
R6
N (Xn) =nX
j=1
N (Xj) + TC
H(Xn) =nX
j=1
Rj +DTC
nX
j=1
log |Xj |
Privaterandomness
Collectiveconstraints
Sharedrandomness
Example: X1 = X2 = X3
TC = 2 > DTC = 1
X3 = X1 �X2
TC = 1 < DTC = 2
redundancy! synergy!
Individualconstraints
2.O-information:definition
Occam’sraizor(lexparsimoniae):givepreferencetothesimplestdescription
• Ifitisshortertodescribetheallowedstates.
• Ifitisshortershortertodescribetheconstraints.
!31
Definition
⌦(Xn) = TC(Xn)�DTC(Xn)O-information:
⌦(Xn) > 0
⌦(Xn) < 0
F.Rosas,P.A.M.Mediano,M.Gastpar,andH.J.Jensen."QuantifyingHigh-orderInterdependenciesviaMultivariateExtensionsoftheMutualInformation,”accepted,tobepublishedinPRE,2019
2.O-information:definition
Occam’sraizor(lexparsimoniae):givepreferencetothesimplestdescription
• Ifitisshortertodescribetheallowedstates.
• Ifitisshortershortertodescribetheconstraints.
!32
Definition
⌦(Xn) = TC(Xn)�DTC(Xn)O-information:
⌦(Xn) > 0
⌦(Xn) < 0
Example:
redundancy! synergy!
X1 = X2 = · · · = Xn
TC = n� 1 > DTC = 1 TC = 1 < DTC = n� 1
Xn = X1 �X2 � · · ·�Xn�1
F.Rosas,P.A.M.Mediano,M.Gastpar,andH.J.Jensen."QuantifyingHigh-orderInterdependenciesviaMultivariateExtensionsoftheMutualInformation,”accepted,tobepublishedinPRE,2019
2.O-information:definition
Occam’sraizor(lexparsimoniae):givepreferencetothesimplestdescription
• Ifitisshortertodescribetheallowedstates.
• Ifitisshortershortertodescribetheconstraints.
!33
Definition
⌦(Xn) = TC(Xn)�DTC(Xn)O-information:
⌦(Xn) > 0
⌦(Xn) < 0
Definition
1. Asystemisredundancy-dominatedif
2. Asystemissynergy-dominatedif
⌦(Xn) � 0
⌦(Xn) 0
2.O-information:definition
Occam’sraizor(lexparsimoniae):givepreferencetothesimplestdescription!
• Ifitisshortertodescribetheallowedstates.
• Ifitisshortershortertodescribetheconstraints.
Note:
1.n=2:,becausei.e.sharedrandomnessisequaltopredictability
!34
Definition
⌦(Xn) = TC(Xn)�DTC(Xn)O-information:
⌦(Xn) > 0
⌦(Xn) < 0
⌦(X1X2) = 0 TC(X1X2) = DTC(X1X2) = I(X1;X2)
2.O-information:definition
Occam’sraizor(lexparsimoniae):givepreferencetothesimplestdescription!
• Ifitisshortertodescribetheallowedstates.
• Ifitisshortershortertodescribetheconstraints.
Note:
1.n=2:,becausei.e.sharedrandomnessisequaltopredictability
2.n=3:—>interaction-information!
!35
Definition
⌦(Xn) = TC(Xn)�DTC(Xn)O-information:
⌦(Xn) > 0
⌦(Xn) < 0
⌦(X1X2) = 0 TC(X1X2) = DTC(X1X2) = I(X1;X2)
⌦(X1X2X3) = I(X1;X2)� I(X1;X2|X3) =: I(X1;X2;X3)
2.O-information:(1)itissumoftripleinteraction-informations
• DecompositionsfortheTC,DTCando-information:
!36
O-informationisanaggregationoftriplemulti-informations!!
2.O-information:(1)itissumoftripleinteraction-informations
• DecompositionsfortheTC,DTCando-information:
!37
2.O-information:(1)itissumoftripleinteraction-informations
• DecompositionsfortheTC,DTCando-information:
!38
HXY ZW
HXY +HZW
HXZ +HYW
HXW +HY Z
HY +HZWXHX +HY ZW HZ +HWXY HW +HXY Z
HY +HZ +HXW
HX +HW +HY ZHX +HZ +HYWHX +HY +HZW HY +HW +HXZ HZ +HW +HXY
HX +HY +HZ +HW
HXY |ZW +HZW |XY
HXZ|YW +HYW |XZ
HXW |Y Z +HY Z|XW
HY |ZWX +HZWX|YHX|Y ZW +HY ZW |X HZ|WXY +HWXY |Z HW |XY Z +HXY Z|W
HY |ZWX +HZ|WXY +HXW |Y Z
HX|Y ZW +HW |XY Z +HY Z|XW
HX|Y ZW +HZ|WXY +HYW |XZHX|Y ZW +HY |ZWX +HZW |XY HY |ZWX ++HW |XY Z +HXZ|YW HZ|WXY +HW |XY Z +HXY |ZW
HX|Y ZW +HY |ZWX +HZ|WXY +HW |XY Z
2.O-information:(2)characterizationofextremevalues
• DecompositionsfortheTC,DTCando-information:
• Upperandlowerbounds:
• Characterizationofuniqueextremesoftheo-information:
!39
Maximum: X1 = X2 = · · · = Xn Minimum: Xn =n�1X
j=1
Xj (mod k)
2.O-information:(3)nullvalueofnon-overlappingpairwiseinteractions
• O-informationofindependentsubgroupsisadditive:
• Ifasystem’sisdecomposableindisjointpairwiseinteractionsthen
• Theconverseisnottrue!!(synergiesandredundanciescanceleachother)
• Localo-informationcangiveafine-graineddescriptionofthesystem…
!40
⌦(Xn,Y m) = ⌦(Xn) + ⌦(Y m)pXn,Y m(xn,ym) = pXn(xn)pY m(ym)
⌦(Xn) = 0
!i,j = I(Xi;Xj ;Xn�i�j)
2.O-information:(4)valueimpliesboundsoverdifferentscales
• ThevalueoftheO-informationprovideconstraintsovertheinterdependenciesofsubgroups!
•
!41
2.O-information:(5)redundancyoftrees
• O-informationisnon-negativeingraphicalmodelswiththreestructure!
• ForNaiveBayes,theo-informationisgivenby
• ForMarkovchains,theo-informationis
!42
⌦(Xn) = C(Xn2 ) � 0
1 2 n…3
2.O-information:(6)synergyispervasiveinlargesystems
• Themeanvalueoftheo-informationoverrandomdistributionsgrowsnegative!
!43
2 4 6 8
�4
�2
0
System size n
Gaussian systems
2 4 6 8
�0.8
�0.6
�0.4
�0.2
0
System size n
⌦
Binary systems
AnalyticalMonte CarloApproximation
(Williams & Beer, 2010)(Williams & Beer, 2010)(Williams & Beer, 2010)(Williams & Beer, 2010)
2.O-information:relationshipwithstatisticalmechanics
1.Connectedinformation(Schneidmanetal.2003):
Stateofaffairs:
(+)Interestingconnectionwithstatisticalmechanics.
(+)Intuitiveinterpretation.
(-)Veryhardtocompute…
!44
Con(Xn) =
Pnk=1 H(Xj)�H(Xn
maxent)
TC(Xn)
2.O-information:relationshipwithstatisticalmechanics
1.Connectedinformation(Schneidmanetal.2003):
• Hamiltonianwithhigh-ordertermstendtohavenegativeO-information
!45
Con(Xn) =
Pnk=1 H(Xj)�H(Xn
maxent)
TC(Xn)
2.O-information:analysispipeline
!46
Procedure
1.Computeandasmetricsofglobalcorrelationstrength.
2.Computetofinddominantglobalbehaviour(redundancyorsynergy).
3.Studythelocalo-informationterms,i.e.foralliandj,asameasureoflocalisedbehaviour.
⌦(Xn)
I(Xi;Xj ;Xn�i,�j)
B(Xn) C(Xn)
2.O-information:Casestudyonbaroquemusicscores
DataanalysisovermusicscoresfromtheBaroqueperiod(Python,Music21package)
i)choralesforfourvoicesbyJ.S.Bach(1685–1750)
!47
2.O-information:Casestudyonbaroquemusicscores
DataanalysisovermusicscoresfromtheBaroqueperiod(Python,Music21package)
i)choralesforfourvoicesbyJ.S.Bach(1685–1750)
Dataformat:foursynchronoustimeseries,alphabetof13values(12tones+silence)
Database:300~chorales,43kfour-notechords
!48
2.O-information:Casestudyonbaroquemusicscores
DataanalysisovermusicscoresfromtheBaroqueperiod(Python,Music21package)
i)choralesforfourvoicesbyJ.S.Bach(1685–1750)
(43kfour-notechords)
ii)Op.1,3,4,5and6ofA.Corelli(1653–1713)(80kfour-notechords)
!49
2.O-information:Casestudyonbaroquemusicscores
DataanalysisovermusicscoresfromtheBaroqueperiod(Python,Music21package)
!50
2.O-information:Casestudyonbaroquemusicscores
DataanalysisovermusicscoresfromtheBaroqueperiod(Python,Music21package)
!51
Synergyallowsthecoexistenceoflocalindependencyandglobalcoordination
!52
Today’smenu
1.Self-organisation
2.O-information
3.Dataprivacy
4.Summaryandcurrentwork
3.Synergyanddataprivacy:keyideas
Privacyfunnel:maximizecorrelationwithfeatureofinterest,whilekeepingprivatestuffsecure
!53
3.Synergyanddataprivacy:keyideas
Privacyfunnel:maximizecorrelationwithfeatureofinterest,whilekeepingprivatestuffsecure
However,itisoftenthecasewherethefeatureofinterestisunknown
ButtheDPinequalitymakeitseemunfeasible…
!54
3.Synergyanddataprivacy:keyideas
Idea:makeYindependentofeachcoordinateofX,butcorrelatedwiththewhole!!
!55
3.Synergyanddataprivacy:keyideas
Idea:makeYindependentofeachcoordinateofX,butcorrelatedwiththewhole!!
!56
3.Synergyanddataprivacy:mainresults
!57
3.Synergyanddataprivacy:mainresults
!58
Tolearnmoreaboutthis:1. B.Rassouli,F.E.Rosas,andD.Gunduz."DataDisclosureunderPerfectSamplePrivacy."Submitted
toIEEETransactionsinInformationForensicsandSecurity(TIFS),underreview.arXivpreprintarXiv:1904.01711(2019).
2. B.Rassouli*,F.E.Rosas*,andD.Gunduz,"LatentFeatureDisclosureunderPerfectSamplePrivacy."In2018IEEEInternationalWorkshoponInformationForensicsandSecurity(WIFS),pp.1-7.IEEE,2018.
!59
Today’smenu
1.Self-organisation
2.O-information
3.Dataprivacy
4.Summaryandcurrentwork
Furtherreading:
!60
1. B.Rassouli*,F.E.Rosas*,andD.Gunduz,"LatentFeatureDisclosureunderPerfectSamplePrivacy."In2018IEEEInternationalWorkshoponInformationForensicsandSecurity(WIFS),pp.1-7.IEEE,2018.
2. B.Rassouli,F.E.Rosas,andD.Gunduz."DataDisclosureunderPerfectSamplePrivacy."SubmittedtoIEEETransactionsinInformationForensicsandSecurity(TIFS),underreview.arXivpreprintarXiv:1904.01711(2019).
3. F.Rosas,P.A.M.Mediano,M.Ugarte,H.J.Jensen,“Aninformation-theoreticapproachtoself-organization:emergenceofcomplexinterdependenciesincoupleddynamicalsystems”,Entropy20,no.10(2018):793.
4. F.Rosas,P.A.M.Mediano,M.Gastpar,andH.J.Jensen."QuantifyingHigh-orderInterdependenciesviaMultivariateExtensionsoftheMutualInformation,”accepted,tobepublishedinPRE,2019
PD:currentwork
Drivingquestions:1.Isthebrainasynergisticmulti-agentsystem?2.Isconsciousnessrelatedwithhigh-orderstatistics?3.CanPIDprincipleshelpustounderstandtheeffectofpsychedelicdrugs?
!61
!!!!!Imperial)College)London)))Centre)for)Complexity)Science)
!
!
Complexity
Science
Acknowledges:
Thisworkistheresultofthefortunatecollaborationwith
- PedroMediano(ImperialCollegeLondon)- HenrikJ.Jensen(ImperialCollegeLondon)- MichaelGastpar(EPFL)- MartinUgarte(UniversitélibredeBruxelles)- BorzooRasouli(UniversityofEssex)- DenizGündüz(ImperialCollegeLondon)
SpecialthanksfortheMarieSlodowska-CurieprogramfromH20202andtheEuropeanCommission.
!62