hierarchically modular structure in complex...
TRANSCRIPT
![Page 1: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/1.jpg)
Hierarchically ModularStructure in
Complex Networks
Aaron ClausetSanta Fe Institute
3 November 2008DIMACS / DyDAn
“Network Models of Biological and Social Contagion”
![Page 2: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/2.jpg)
Modular Hierarchies
Grassland species**thank you: Jennifer Dunne
plant!
!
herbivore
!
parasite
![Page 3: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/3.jpg)
Modular Hierarchies
c
![Page 4: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/4.jpg)
Modular Hierarchies
c
![Page 5: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/5.jpg)
The TaskHow can we extract
• this hierarchical (multi-scale) structurefrom complex networks?
!
network
?
c
hierarchy
![Page 6: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/6.jpg)
One Approach
Model-based inference
1. describe how to generate hierarchies (a model)
2. “fit” model to empirical data
3. test “fitted” model
4. extract predictions + insight
5. profit!
![Page 7: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/7.jpg)
A Model of Hierarchy
![Page 8: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/8.jpg)
!
A Model of Hierarchy
probability
assortative modules
pr
D, {pr}
![Page 9: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/9.jpg)
“inhomogeneous” random graph
!!
model
instance
!
Pr(i, j connected) = pr
i
j
i j
= p(lowest common ancestor of i,j)
![Page 10: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/10.jpg)
Model Features
• explicit model = explicit assumptions
• very flexible (many parameters)
• captures structure at all scales
• arbitrary mixtures of assortativity, disassortativity
• learnable directly from data
![Page 11: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/11.jpg)
Learning From Data
a direct approach• likelihood function
( scores quality of model)
• sample the good models
via Markov chain Monte Carlo
• technical details in arXiv : physics/0610051
L = Pr( data | model )
![Page 12: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/12.jpg)
From Graph to Ensemble
![Page 13: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/13.jpg)
From Graph to Ensemble
• Given graph• run MCMC to equilibrium• then, for each sampled , draw a resampled
graph from ensemble
A test: do resampled graphs look like original?
D
G!
G
![Page 14: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/14.jpg)
Grassland species*
plant!
!herbivore
!parasite
*thank you: Jennifer Dunne
![Page 15: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/15.jpg)
100 10110!3
10!2
10!1
100a
Degree, k
Frac
tion
of v
ertic
es w
ith d
egre
e k
Degree Distribution
resampled!
original!
![Page 16: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/16.jpg)
Clustering Coefficient
resampled!
original!
0 0.05 0.1 0.15 0.2 0.25 0.30
0.05
0.1
0.15
0.2
0.25
Frac
tion
of g
raph
s wi
th c
lust
erin
g co
effic
ient
c
Clustering coefficient, c
resampled!
original!
![Page 17: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/17.jpg)
2 4 6 8 1010!3
10!2
10!1
100b
Distance, d
Frac
tion
of v
erte
x!pa
irs a
t dis
tanc
e d
Distance Distribution
resampled!
original!
![Page 18: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/18.jpg)
Missing Links
A test: can model predict missing links?
![Page 19: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/19.jpg)
Predicting is Hard
• remove edges from• how easy to guess a missing link?
n = 75
m = 113
pguess !k
n2" m + k
= O(n!2)
k
pguess = k/(2662 + k)
G
![Page 20: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/20.jpg)
• Given incomplete graph• run MCMC to equilibrium• then, over sampled , compute average
for links • predict links with high values are missing
Test idea via leave-k-out cross-validationperfect accuracy: AUC = 1no better than chance: AUC = 1/2
(i, j) !" G
Predicting Missing Links
D !pr"
G
!pr"
![Page 21: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/21.jpg)
Missing Structure
0 0.2 0.4 0.6 0.8 10.4
0.5
0.6
0.7
0.8
0.9
1
Area
und
er R
OC
curv
e
Fraction of edges observed, k/m
Grassland species network
Pure chanceCommon neighborsJaccard coeff.Degree productShortest pathsHierarchical structure
simple predictors
!
hierarchy
!
pure chance!
AUC
![Page 22: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/22.jpg)
0 0.2 0.4 0.6 0.8 10.4
0.5
0.6
0.7
0.8
0.9
1
AUC
Fraction of edges observed
Terrorist association networka
Pure chanceCommon neighborsJaccard coefficientDegree productShortest pathsHierarchical structure
Other Networks
0 0.2 0.4 0.6 0.8 10.4
0.5
0.6
0.7
0.8
0.9
1
AUC
Fraction of edges observed
T. pallidum metabolic networkb
Pure chanceCommon neighborsJaccard coefficientDegree productShortest pathsHierarchical structure
![Page 23: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/23.jpg)
Summary
• Many real networks are hierarchically modular• Hierarchies can
• model multi-scale structure• generalize a single network• predict missing links
• Model-based inference is very powerful
Acknowledgments:C. Moore, M.E.J. Newman, C.H. Wiggins, and C.R. Shalizi
![Page 24: Hierarchically Modular Structure in Complex Networksdimacs.rutgers.edu/Workshops/Contagion/slides/clauset.pdf · 2008-11-05 · Hierarchically Modular Structure in Complex Networks](https://reader033.vdocuments.us/reader033/viewer/2022042923/5f701d1e09508a685e3c2f93/html5/thumbnails/24.jpg)
Fin