code4lib 2014 miller presentation

Visualizing Library Resources as Networks Ma8 Miller

[email protected] | @thisismmiller NYPL Labs

New York Public Library

Code4Lib 2014, Raleigh

Agenda

Why Networks? Networks in Archives Networks in the Catalog

Why Networks?

•  Libraries and archives consist of independent documents. How do we connect them and why is it important?

Why Networks?

The document, then, is no longer for history an inert material through which it tries to reconsOtute what individuals have done or said, the events of which only the trace remains; history is now trying to define within the documentary materials itself uniOes, totaliOes, series, relaOons. -‐Michel Foucault The Archaeology of Knowledge

Why Networks?

•  Scale – A few hundred to hundreds of thousands

•  See the larger pa8erns – Visualize collecOon strengths

– Find outliers

Why Networks?

•  Serendipity –  Interconnect lots of things and you start geVng interesOng relaOonships.

– An alternaOve way to browse.

Networks in Archives

•  A single collecOon viewed as a network. •  h8p://archives.nypl.org/mss/2993

•  (shout out to Trevor Thornton, @trevorthornton)


•  Explore all of the archive’s collecOon through a network.

•  h8p://archives.nypl.org/terms/


•  Limits of control access terms. – Most collecOons do not have component level access terms.

– CollecOon level terms are not specific to find the interesOng connecOons that we are looking for.

– We need more data!

•  Technical limits of browser based networks.


•  Next steps – NLP the EAD files for more controlled terms. – Leverage exiOng term’s linked data to bolster terms associated with a collecOon.

– Add insOtuOonal data to the graph, who donated what.

– Start building links between collecOons based on shared web visits.

Networks in the Catalog

Each subject heading is a node, size indicates number of occurrences.

Subject heading must be used in at least two resources.

When two subjects occur in the same resource a edge is created between them. More co-‐occurrences creates a closer connecOon between them.


Lincoln, Abraham, -‐-‐ 1809-‐1865. PoliOcal leadership -‐-‐ United States -‐-‐ History. Genius -‐-‐ Case studies. United States -‐-‐ History -‐-‐ Civil War, 1861-‐1865. Lincoln, Abraham, -‐-‐ 1809-‐1865 -‐-‐ Friends and associates. Presidents -‐-‐ United States -‐-‐ Biography. United States -‐-‐ PoliOcs and government -‐-‐ 1861-‐1865.


•  8.5+ million MARC Records •  430,000+ subjects •  11+ million connecOons

h8p://bit.ly/nyplnetwork


•  How to: – Compile MARC records into node/edge relaOonships.

– A tool to render the network based on exisOng network analysis algorithms.

–  Take that informaOon and make a visual representaOon of the network.

– Present that visual representaOon in a web base format.


•  Compile MARC records into node/edge relaOonships. – pyMARC to process subject headings into a Gephi XML document.


•  A tool to render the network based on exisOng network analysis algorithms. – Using Gephi, but not the GUI.

•  Gephi Toolkit – a library of the core rendering funcOons of Gephi. •  h8ps://gephi.org/toolkit/ •  Using the Force Atlas 2 layout algorithm.

•  Video.


•  Take that informaOon and make a visual representaOon of the network. – Used Python bindings of GraphicsMagick to draw one very large PNG.

– Cut into Oles with libvips. – Keep track of the coordinates of all nodes.


•  Present that visual representaOon in a web base format. – Used OpenSeaDragon image Ole viewer to navigate.

– Used ElasOcSearch to translate client clicks into node posiOons and provide search funcOon.


•  Next VisualizaOon Steps: – Show edges – Use subfield subject in more specific ways. – Provide more interacOvity, sub-‐networks, shared connecOons, etc.

•  Next Catalog Steps: – Mapping MARC to find cataloging pracOces – AddiOonal discovery experiments – Proof of concept discovery layer

Thanks! •  Ma8 Miller – [email protected] – @thisismmiller

•  Tools + Code –  h8p://archives.nypl.org/terms –  h8p://bit.ly/nyplnetwork –  h8ps://github.com/thisisma8miller/catalog-‐network

Icon Credits: Network designed by Hyeji Michelle Jun from the Noun Project Book designed by Chris Thoburn from the Noun Project Box designed by Michael Rowe from the Noun Project Book designed by Pedro Lalli from the Noun Project Enlarge designed by Cornelius Danger from the Noun Project Present designed by Naomi Atkinson from the Noun Project

code4lib 2014 miller presentation

Documents