short networks vis - ghostweather.com · release by joint economic committee minority republicans...
TRANSCRIPT
Visualizing Networks
Lynn Cherny@arnicas
4 Nov 2014
The Hairball: A Metaphor for Complexity
h6p://www.nd.edu/~networks/PublicaBon%20Categories/01%20Review%20ArBcles/ScaleFree_ScienBfic%20Ameri%20288,%2060-‐69%20(2003).pdf
h6p://www.linkedin.com/today/post/arBcle/20121016185655-‐10842349-‐the-‐hidden-‐power-‐of-‐networks
Release by Joint Economic Committee minority Republicans led by Boehner (2010), Iliinsky & Steele fig 4-14 4
Release by Joint Economic Committee minority Republicans led by Boehner (2010), Iliinsky & Steele fig 4-14
Neither Exploratory Nor Explanatory:Political.
4
Redesign by Robert Palmer(Iliisnky & Steele fig 4-15) 5
Redesign by Robert Palmer(Iliisnky & Steele fig 4-15)
“By releasing your chart, instead of meaningfully educating the public, you willfully obfuscated an already complicated proposal. There is no simple proposal to
solve this problem. You instead chose to shout ‘12! 16! 37! 9! 24!’ while we were trying to count
something.”
5
WHAT IS A NETWORK?
It’s not a visualizaBon. Think of it as a data structure.
Data relaBonship: enBBes + relaBonships to other objects (node/edge, vertex/link)
Nodes and Edges may have a6ributes, eg. gender, age, weight, tv prefs connecBon date, frequency of contact, type of exchange, direcBonality of relaBonship a6ributes may be calculated from network relaBons too
Lane Harrison: h6p://blog.visual.ly/network-‐visualizaBons/
Lane Harrison: h6p://blog.visual.ly/network-‐visualizaBons/
Lane Harrison: h6p://blog.visual.ly/network-‐visualizaBons/
9h6p://blog.visual.ly/network-‐visualizaBons/
Best!
A User Study on Visualizing Directed Edges in Graphs” Danny Holten and Jarke J. van Wijk, 27th SIGCHI Conference on Human Factors in Computing Systems (Proceedings of CHI 2009),
10
10
10
10
It’s a natural human trait to see visual similarity and proximity as meaningful.
Be very careful about your display choices and layout methods!
Reading a network visualizaBon
There’s obviously something important going on here, structurally....
Reading a network visualizaBon
Look at this outlier case!
There’s obviously something important going on here, structurally....
Reading a network visualizaBon
Look at this outlier case!
There’s obviously something important going on here, structurally....
A ménage à
trois ove
r here
Reading a network visualizaBon
Look at this outlier case!
There’s obviously something important going on here, structurally....
A ménage à
trois ove
r here
MIS?
Using a “random” Gephi layout on the dolphins
Reading a network visualizaBon
Look at this outlier case!
There’s obviously something important going on here, structurally....
A ménage à
trois ove
r here
MIS?
Using a “random” Gephi layout on the dolphins
Random!
12
TOOLS AND CONCEPTS FOR TODAY
CreaBng network layouts...
13
Gephi (and a li6le D3)
J BerBn: Semiology of Graphics
BerBn, J. Semiology of Graphics: Diagrams, Networks, Maps (1967)
Linear
Circular
Irregular
Regular (Tree)
3D
Matrix /
BiparBte
J BerBn: Semiology of Graphics
BerBn, J. Semiology of Graphics: Diagrams, Networks, Maps (1967)
Linear
Circular
Irregular
Regular (Tree)
3D
Matrix /
BiparBte
MATRIX LAYOUTS / REPRESENTATIONS
h6p://barabasilab.neu.edu/networksciencebook/download/network_science_October_2012.pdf
Real social networks are generally quite sparse.
h6p://www.cise.ufl.edu/research/sparse/matrices/Newman/dolphins.html
D3 demo by me h6p://www.ghostweather.com/essays/talks/networkx/adjacency.html
D3 demo by me h6p://www.ghostweather.com/essays/talks/networkx/adjacency.html
20
Readability: Matrix Be6er (except for path finding)
Jean-Daniel Fekete http://lamut.informatik.uni-wuerzburg.de/gd2014/data/uploads/gd2014-fekete-keynote.pdf
ARC / LINEAR LAYOUTS
Philipp Steinweber and Andreas Koller Similar Diversity, 2007
For a D3 example in another domain: h6p://tradearc.laserdeathstehr.com/
Topics in Da Vinci Code (by me)
23
https://www.dropbox.com/s/090vwj12kh3mu96/Screenshot%202014-11-04%2020.45.02.png?dl=0
Lynn Cherny - http://www.ghostweather.com/essays/talks/openvisconf/topic_arc_diagram/TopicArc.html
Topics in Da Vinci Code (by me)
23
https://www.dropbox.com/s/090vwj12kh3mu96/Screenshot%202014-11-04%2020.45.02.png?dl=0
Lynn Cherny - http://www.ghostweather.com/essays/talks/openvisconf/topic_arc_diagram/TopicArc.html
CIRCULAR / CHORD LAYOUTS
h6p://circos.ca/images/
Circos“If it's round, Circos can
probably do it”
TIL (shocking): It’s all perl code?!
h6p://www.ghostweather.com/essays/talks/networkx/chord.html
h6p://www.ghostweather.com/essays/talks/networkx/chord.html
A Nice UI Improvement -‐ DifferenBate Source/DesBnaBon
28Nikola Sander, http://www.global-migration.info/
Hierarchical Edge Bundling demo by @mbostock
D3: h6p://bl.ocks.org/1044242
"Hierarchical Edge Bundles: VisualizaBon of Adjacency RelaBons in Hierarchical Data”, Danny Holten, IEEE TransacBons on VisualizaBon and Computer Graphics (TVCG; Proceedings of Vis/InfoVis 2006), Vol. 12, No. 5, Pages 741 -‐ 748, 2006.
SIMPLE CALCULATIONS ON NETWORKS CAN TELL YOU LOADSOven you need to visualize the structure/role of the graph elements as part of the visualizaBon: So, do some simple math.
Degree (In, Out)
“Degree” is a measure of the edges in (directed), out (directed), or total (in directed or undirected graphs) to a node
“Hub” nodes have high in-‐degree. In scale-‐free networks, we see preferenBal a6achment to the popular kids.
h6p://mlg.ucd.ie/files/summer/tutorial.pdf
The Threat of Hub-‐Loss
Albert-‐László Barabási and Eric Bonabeau, Scale-‐Free Networks, 2003.h6p://www.scienBficamerican.com/arBcle.cfm?id=scale-‐free-‐networks
VisualizaBon Aside: If Some Names are Huge, the Others are Invisible-‐?
33CorrecBng for text size by degree display issue
VisualizaBon Aside: If Some Names are Huge, the Others are Invisible-‐?
33
Gephi Panel
CorrecBng for text size by degree display issue
VisualizaBon Aside: If Some Names are Huge, the Others are Invisible-‐?
33
Gephi Panel
CorrecBng for text size by degree display issue
VisualizaBon Aside: If Some Names are Huge, the Others are Invisible-‐?
33
Gephi Panel
CorrecBng for text size by degree display issue
BetweennessA measure of connectedness between
(sub)components of the graph
“Betweenness centrality thus tends to pick out boundary individuals who play the role of brokers between communiBes.”
h6p://en.wikipedia.org/wiki/Centrality#Betweenness_centralityLusseau and Newman. h6p://www.ncbi.nlm.nih.gov/pmc/arBcles/PMC1810112/pdf/15801609.pdf
35
Judging By Eye Will Probably Be Wrong...
35
? This one?
Judging By Eye Will Probably Be Wrong...
35
? This one?
Sized by Betweenness
Judging By Eye Will Probably Be Wrong...
Eigenvector Centrality
IntuiBon: A node is important if it is connected to other important nodes A node with a small number of influenBal contacts may outrank one with a larger number of mediocre contacts
h6p://mlg.ucd.ie/files/summer/tutorial.pdf h6p://demonstraBons.wolfram.com/NetworkCentralityUsingEigenvectors/
Pagerank
Wikipedia image
Community DetecBon Algorithms
E.g., the Louvain method, in Gephi as “Modularity.” Many layout algorithms help you intuit these structures, but don’t rely on percepBon of layout!
h6p://en.wikipedia.org/wiki/File:Network_Community_Structure.png
You can even do it in your browser now…
39http://bl.ocks.org/john-guerra/ecdde32ab4ad91a1a240
You can even do it in your browser now…
39http://bl.ocks.org/john-guerra/ecdde32ab4ad91a1a240
CitaBon: Lusseau D (2007) Why Are Male Social RelaBonships Complex in the Doub|ul Sound Bo6lenose Dolphin
PopulaBon?. PLoS ONE 2(4): e348. doi:10.1371/journal.pone.0000348
IdenBfying th
e role th
at animals p
lay in th
eir social networks (2
004)
D Lusseau, M
EJ New
man
Proceedings o
f the
Royal Society of Lon
don. Series B
: Biological Scien
ces
IdenBfying th
e role th
at animals p
lay in th
eir social networks (2
004)
D Lusseau, M
EJ New
man
Proceedings o
f the
Royal Society of Lon
don. Series B
: Biological Scien
ces
IdenBfying th
e role th
at animals p
lay in th
eir social networks (2
004)
D Lusseau, M
EJ New
man
Proceedings o
f the
Royal Society of Lon
don. Series B
: Biological Scien
ces
IdenBfying th
e role th
at animals p
lay in th
eir social networks (2
004)
D Lusseau, M
EJ New
man
Proceedings o
f the
Royal Society of Lon
don. Series B
: Biological Scien
ces
IdenBfying th
e role th
at animals p
lay in th
eir social networks (2
004)
D Lusseau, M
EJ New
man
Proceedings o
f the
Royal Society of Lon
don. Series B
: Biological Scien
ces
IdenBfying th
e role th
at animals p
lay in th
eir social networks (2
004)
D Lusseau, M
EJ New
man
Proceedings o
f the
Royal Society of Lon
don. Series B
: Biological Scien
ces
Eduarda Mendes Rodrigues, Natasa Milic-‐Frayling, Marc Smith, Ben Shneiderman, Derek Hansen, Group-‐in-‐a-‐box Layout for MulB-‐faceted Analysis of CommuniBes. IEEE Third InternaBonal Conference on Social CompuBng, October 9-‐11, 2011. Boston, MA
h6p://mbostock.github.com/d3/talk/20111116/force-‐collapsible.html Also see Ger Hobbelt D3: h6p://bl.ocks.org/3616279
h6p://mbostock.github.com/d3/talk/20111116/force-‐collapsible.html Also see Ger Hobbelt D3: h6p://bl.ocks.org/3616279
ALGORITHMIC LAYOUTS
Gephi / D3.js / Other tools
Gephi à Sigma.js / D3
Gephi.org: Open source, runs on Mac, Linux, PC Can be run from a python-‐esque console plugin or UI Can be run “headless” for layouts with Java or Python (I have code for this) Plugins include a Neo4j graph db access, and streaming support
Sigma.js : Will display a gexf gephi layout file with minimal work, using a plugin interpreter for sigma site export Also offers a force-‐directed layout plugin for graphs without x&y coords Does CANVAS drawing, not SVG
Export JSON from Gephi and load into D3 network layout
h6p://www.barabasilab.com/pubs/CCNR-‐ALB_PublicaBons/200705-‐14_PNAS-‐HumanDisease/Suppl/Goh_etal_poster.pdf
Sigma.js version of the Gephi export h6p://exploringdata.github.com/vis/human-‐disease-‐network/
Movie:
Sigma.js version of the Gephi export h6p://exploringdata.github.com/vis/human-‐disease-‐network/
Movie:
Sample Layout Plugins in Gephi
h6ps://gephi.org/tutorials/gephi-‐tutorial-‐layouts.pdf
Gephi Plugin Layout DetailsLayout Complexity Graph Size Author Comment
Circular O(N) 1 to 1M nodes Ma6 Groeninger Used to show distribuBon, ordered layout
Radial Axis O(N) 1 to 1M nodes Ma6 Groeninger Show ordered groups (homophily)
Force Atlas O(N²) 1 to 10K nodes Mathieu Jacomy Slow, but uses edge weight and few biases
Force Atlas 2 O(N*log(N)) 1 to 1M nodes Mathieu Jacomy
(does use weight)
OpenOrd O(N*log(N)) 100 to 1M nodes S. MarBn, W. M. Brown, R. Klavans, and K. Boyack
Focus on clustering (uses edge weight)
Yifan Hu O(N*log(N)) 100 to 100K nodes Yifan Hu (no edge weight)
Fruchterman-‐Rheingold O(N²) 1 to 1K nodes Fruchterman & Rheingold!
ParBcle system, slow (no edge weight)
GeoLayout O(N) 1 to 1M nodes Alexis Jacomy Uses Lat/Long for layout
h6ps://gephi.org/2011/new-‐tutorial-‐layouts-‐in-‐gephi/
Dolphins Again
50
OpenOrd + “No Overlap” ForceAtlas2
Dolphins Again
50
OpenOrd + “No Overlap” ForceAtlas2
Dolphins Again
50
OpenOrd + “No Overlap” ForceAtlas2
51
Weight 2: Force Atlas Weight 4: Force Atlas
Unweighted dolphins, Force Atlas
51
Weight 2: Force Atlas Weight 4: Force Atlas Weight 4: Yifan Hu
Unweighted dolphins, Force Atlas
Nick Diakapolous: h6p://nad.webfacBonal.com/ntap/graphscale/
h6ps://docs.google.com/spreadsheet/ccc?key=0AtvlFoSBUC5kdEZJNVFySG9wSHZka0NsOT
ZDSkt3Nnc#gid=0
Canvas/SVG benchmarks from the d3.js group:
Nick Diakapolous: h6p://nad.webfacBonal.com/ntap/graphscale/
h6ps://docs.google.com/spreadsheet/ccc?key=0AtvlFoSBUC5kdEZJNVFySG9wSHZka0NsOT
ZDSkt3Nnc#gid=0
Canvas/SVG benchmarks from the d3.js group:
53
FINAL DESIGN THOUGHTS… BY HAND? BY ALGORITHM-‐?
Jeff Heer: h6p://hci.stanford.edu/courses/cs448b/f11/lectures/CS448B-‐20111110-‐GraphsAndTrees.pdf
Conspiracy Theorist Mark Lombardi
Learning from Lombardi: h6p://benfry.com/exd09/
Conspiracy Theorist Mark Lombardi
Learning from Lombardi: h6p://benfry.com/exd09/
J. BerBn Again…
BerBn, J. Semiology of Graphics: Diagrams, Networks, Maps (1967)
Age Groups and Movement to/from Paris to Rural Spots
(P) Paris
(Z) Paris Suburbs
(+50) Communes of >50K
(+10) Communes of >10K
(-‐10) Communes of <10K
(R) Rural
Ger Hobbelt in D3: h6p://bl.ocks.org/3637711
Hybrid Method: Use algorithmic layout, and then adjust nodes by hand.
Minimize Info: Less is More
60Color by important data value, sized by degree, no edges
61http://www.ghostweather.com/essays/talks/openvisconf/topic_docs_network/
61http://www.ghostweather.com/essays/talks/openvisconf/topic_docs_network/
Brian Keegan’s “15 Minutes of Fame as a B-‐List GamerGate Celebrity”
62http://www.brianckeegan.com/2014/10/my-15-minutes-of-fame-as-a-b-list-gamergate-celebrity/
Brian Keegan’s “15 Minutes of Fame as a B-‐List GamerGate Celebrity”
62http://www.brianckeegan.com/2014/10/my-15-minutes-of-fame-as-a-b-list-gamergate-celebrity/
Brian Keegan’s “15 Minutes of Fame as a B-‐List GamerGate Celebrity”
62http://www.brianckeegan.com/2014/10/my-15-minutes-of-fame-as-a-b-list-gamergate-celebrity/
Actually:
Wrap it up on design...
Reminders
Do data analysis / reducBon -‐ why would you want to show 1T of network data? Calculate and reveal important facts about node relaBonships. Consider your visual encodings: – visualizaBon, not data vomit. Make it interacBve -‐ details on demand.
Help people find things in your network! (Search?) Show more on hover/click
Reminders
Do data analysis / reducBon -‐ why would you want to show 1T of network data? Calculate and reveal important facts about node relaBonships. Consider your visual encodings: – visualizaBon, not data vomit. Make it interacBve -‐ details on demand.
Help people find things in your network! (Search?) Show more on hover/click
Too much data
=Not always good
data science!
More Reminders!
Different layouts communicate different things to your viewer -‐ choose wisely.
Take care: people will infer things from proximity/similarity even if it was not intended!
Consider if it has to be a “network” display at all: Is it the stats you care about? The important nodes who branch sub-‐communiBes? The ones with the most in/out links? Graph those instead.
65
The Map is Not the Territory…
Forest Pi6s (thanks to Noah Friedkin) h6p://www.analyBctech.com/networks/pi6s.htm
The Map is Not the Territory…
Forest Pi6s (thanks to Noah Friedkin) h6p://www.analyBctech.com/networks/pi6s.htm
68
http://www.ghostweather.com/workshops/nyc_visdata.html
lynn @ ghostweather.com
Dec 11-12, NYC
A Few More ReferencesJeff Heer class slides: h6p://hci.stanford.edu/courses/cs448b/w09/lectures/20090204-‐GraphsAndTrees.pdf A great in-‐progress book on networks: h6p://barabasilab.neu.edu/networksciencebook/ Mark Newman’s new book-‐ NetWorks: An IntroducBon Eyeo FesBval videos from Moritz Stefaner, Manuel Lima, Stefanie Posavec
Journal of Graph Algorithms and ApplicaBons: h6p://jgaa.info/home.html
Jim Vallandingham’s D3 network tutorials: h6p://flowingdata.com/2012/08/02/how-‐to-‐make-‐an-‐interacBve-‐network-‐visualizaBon/, h6p://vallandingham.me/bubble_charts_in_d3.html
Brian Keegan’s post about GamerGate tweets and the dangers of network vis
Robert Kosara’s post: h6p://eagereyes.org/techniques/graphs-‐hairball
Lane Harrison’s post: h6p://blog.visual.ly/network-‐visualizaBons/ MS Lima’s book Visual Complexity Jason Sundram’s tool to drive Gephi layout from command line: h6ps://github.com/jsundram/pygephi A couple arBcles on community structure:
Overlapping Community DetecBon in Networks: State of the Art and ComparaBve Study by Jierui Xie, Stephen Kelley, Boleslaw K. Szymanski
Empirical Comparison of Algorithms for Network Community DetecBon by Leskovec, Lang, Mahoney
My posts on NetworkX with D3 and Twi6er network vis with Gephi