DEVELOPMENTS AND TRENDS IN THE LMS AND DISCOVERY ARENAS

Marshall BreedingDirector for Innovative Technology and ResearchVanderbilt University LibraryFounder and Publisher, Library Technology Guideshttp://www.librarytechnology.org/http://twitter.com/mbreedingProgram on National

Infrastructure

26 August 2010Stockholm

Seminar Goal

The aim of the seminar is to create an understanding of the infrastructural challenges and to contribute to a plan of action for the future.

Library Directors and System managers will discuss different solutions of availability and management of e- resources in order to make strategic choices for the development of the infrastructure at a national level.

Presentation Themes

Trends and recent developments in the library system market,

resource discovery services and resource management as indexing/knowledge bases

Creation and management of data wells for metadata

Ongoing discussion regarding options for building data wells in-house, open source or partnering with commercial actors.

Summary

• development and trends in the library system market, regarding resource discovery services and resource management as indexing/knowledge bases. If I should emphasize something special, it is the question of data wells for metadata. We have been investigating the data well question in a report (plesase see below, Summary in English) and there is a discussion about building data wells in-house, open source or with commercial actors. We have also invited three commercial actors to the seminar. Not an easy question!Related is also the topic of the national catalogue LIBRIS as a local OPAC for the libraries. How can Libris work as, not only the national catalogue, but also as a local OPAC? The third topic is the future for ExLibris, Metalib/SFX in Sweden. We´re happy with SFX, but not with Metalib/federated search, how to continue? But the main focus at the seminar will be resource management/data well, although Libris and Metalib/SFX questions need to be included in the discussions.   

Basic Discovery Concepts

Crowded Landscape of Information Providers on the Web

Lots of non-library Web destinations deliver content to library patrons Google Search / Google Scholar Amazon.com Wikipedia Ask.com

User expectations

Evolution of library collection discovery tools

Bound handwritten catalogs Card Catalogs Library online catalogs – OPACs Next-Gen Catalogs / Discovery interfaces Web-scale discovery services

Bound Catalog

Card Catalog

Online Card Catalog

Web-based online catalog

Next-generation Catalog

Next-generation Catalog

Modernized Interface

Single search box Query tools

Did you mean Type-ahead

Relevance ranked results Faceted navigation Enhanced visual displays

Cover art Summaries, reviews,

Recommendation services

Web site as menu of search options

Disjointed approach to information and service delivery

Silos Prevail Books: Library OPAC (ILS module) Articles: Aggregated content products, e-journal

collections OpenURL linking services E-journal finding aids (Often managed by link

resolver) Local digital collections

ETDs, photos, rich media collections Metasearch engines

All searched separately

Lack of unified Web presence User’s don’t understand the distinctions

we make Catalog? Articles and Databases? Digital Library? Search our Site? Search interfaces based on content formats

or management applications Non-library Web sites are much more

unified

A simple vision

A single point of entry to all the content and services offered by the library

…but with precision, nuanced sophistication, and multiple dimensions

Search:

Web-scale discovery

Online Catalog vs. Discovery Layer

Online Catalog Interface

conventions from an earlier Web era

Scope: Tied to the ILS and its content domain

Discovery Layer Modern interface

elements Scope: aims to

address broad range of components that constitute library collections

Discovery Products

http://www.librarytechnology.org/

discovery.pl

Decoupled from ILS

Social discovery

Tags, user-supplied ratings and reviews Leverage social networking interactions to

assist readers in identifying interesting materials: BiblioCommons

Leverage use data for a recommendation service of scholarly content based on link resolver data: Ex Libris bX service

Deep indexing Metadata can no longer serve as the only basis for

discovery Increasing opportunities to search the full contents

Google Library Print, Google Publisher, Open Content Alliance, government publications, etc.

High-quality metadata will improve search precision Commercial search providers already offer “search

inside the book” and searching across the full text of large book collections

Important transition to full-text book search beginning in library projects HathiTrust indexing 6 million volumes Must become a routine component of library discovery

Deep search highly improved by high-quality metadata

Discovery product Trend

Initial products focused on technology AquaBrowser, Endeca, Primo, Encore, VUfind Mostly locally-installed software

Current phase focused on integrated access to both local content and remote articles to deliver Web-scale discovery. Examples: Summon (Serials Solutions) WorldCat Local (OCLC) EBSCO Discovery Service (EBSCO) Primo Central Encore Synergy

Beyond Federated search

Federated Search / Metasearch use real-time queries against multiple information targets

No centralized index – presentation of dynamic results

Shallow results -- only a few results initially fetched from each target

Difficult to calculate relevancy Performance challenges

Beyond local discovery interfaces

Pre-populated indexes Web-scale

Exploits the full depth and breadth of library collections

Beyond the bounds of the local library’s collection

Targets the universe of objective, vetted library content

Pre-populated discovery services

New-generation interface Harvested local content

ILS metadata Institutional repositories, ETDs, Digital Collection

platforms Vendor-supplied indexes of library content

E-journals, databases, e-books Full-text and metadata corresponding to e-content

subscriptions Book collections beyond local library collections

Includes full-text indexing to the fullest extent possible

Online Catalog

Search:

Search Results

ILS Data

Federated Search

Search: Digital

Collections

ProQuest

EBSCOhost

…MLA

Bibliography

ABC-CLIO

Search Results

Real-time query and responses

ILS Data

Discovery Interface

Search: Digital

Collections

ProQuest

EBSCOhost

…MLA

Bibliography

ABC-CLIO

Search Results

Real-time query and responses

ILS Data

Local Index

Meta

Search

En

gin

e

Web-scale Search

Search: Digital

Collections

ProQuest

EBSCOhost

…MLA

Bibliography

ABC-CLIO

Search Results

Pre-built harvesting and indexing

Con

solid

ate

d In

dex

ILS Data

Web-scale Search + Federated Search

Search: Digital

Collections

ProQuest

…MLA

Bibliography

ABC-CLIO

Search Results

Pre-built harvesting and indexing

Con

solid

ate

d

Index

ILS Data

FedSearch Non-

harvestable

Resources

Discovery Delivery

Discovered content delivered through original repositories

Publisher agreements generally preclude exposing content for direct access

Should necessarily circumvent core role of publisher

Benefits

Libraries: increased access to high-cost electronic content

Users: Easer access to research resources

Publishers: Increased impact of content products

IT perspective: advance harvesting makes more efficient use of resources than simultaneous real-time queries

Toward a Large-scale National Discovery environment

Obstacles and Challenges

Scaleable technology platform Acceptable relevancy-based retrieval for

large heterogeneous collections Acquisition of data and metadata for

aggregated index

Opportunities

Climate more favorable to harvesting e-content for indexing

Highly scaleable, open source tools for discovery infrastructure Lucene SOLR

Many ongoing synergistic projects as possible collaborative partners

Potential Commercial Partners Three commercial organizations will

participate in the seminar: Ex Libris Serials Solutions EBSCO

Each has negotiated access to commercial content products

Paved the way for library driven projects

Other similar projects

Summa

State and University Library of Denmark Locally built integrated search

Catalogs + articles Failed to receive EU funding due to lack

of guarantees to receive article data from publishers

Now Partnering with Serials Solution to use article index from Summon via API

Trove

National Library of Australia Previously called Single Business Discovery

Project Brings together many previously separate

discovery systems Built in-house at NLA Prototype released May 2009 Includes some full-text as well as metadata Technology: Java, Lucene, SOLR, MySQL Details:

http://www.nla.gov.au/pub/gateways/issues/101/story01.html

What about OCLC?

WorldCat: ever expanding repository of metadata Books mostly, increasing article metadata Focused on expanding WorldCat for broad

discovery ArticleFirst 23 million records April 2009 agreement with EBSCO for article

metadata (withdrawn?). Quantity of article metadata apparently not on

track to attain the same level of comprehensiveness as seen in Summon, EDS, Primo Central

Developing the Data Well / Aggregated index

Aggregation of metadata and content Normalization – map metadata to make

indexing, facets, and presentation meaningful

De-duplication of records within and between content sources

FRBR – Collapsible groupings according to FRBR concepts: work – expression -- manifestation – item

Content sources populating the Aggregated Index

Article metadata and full text Index views according to profile Coordinated with local OpenURL knowledge bases

Digital Collections LMS Metadata

Books, Microfilm, periodical titles, DVD, etc Blending of vendor provided metadata and

locally managed unique content At the cusp of being able to represent library

collections comprehensively

Acquiring content for Aggregated Index

Agreements with publishers and providers of article content to libraries

Open access content Any OAI target Local digital collections Relevant library catalog data

OK with OCLC record use policies when aggregated at a national level?

Data Well Construction

Technical Assembling technologies of adequate scale and capacity Indexing, Search and retrieval Normalizing

Business / Political Agreements with commercial publisher to provide

metadata or content Increasing expectation from libraries to allow harvesting

for discovery (Similar to COUNTER compliance, OpenURL support)

Improved performance at delivering library end users to publisher content

Relationship with OpenURL Knowledgebase

The aggregation of article-level citations and content relates to journal title-level profile and availability data in the OpenURL knowledgebase

Important source of profiling needed to deliver appropriate views of the index for different libraries.

A labor-intensive project

Business process Develop relationships with providers and publishers Construct contracts and licenses

Technical Create import process for each source:

Normalization, Mapping, de-duplication, FRBR groupings Initial load + constant incremental updates

Creation of highly scalable indexing and retrieval platform Must scale up to 1 billion articles Develop algorithms and tunings for appropriate relevancy

rankings Interface design

Building Expectations for Article Discovery

Libraries should require agreements for harvesting as part of content licensing process

Library licenses have led to broad support for: COUNTER SUSHI OpenURL Linking

Beyond Metadata

Increasing expectation for full-text indexing

Capacity present in e-journals for many years

Full-text book indexing more problematic Much full text not available Complex to index

Heterogeneous index

Books – mere millions Articles – many hundreds of millions Digital objects – many hundreds of

millions

How to deal with non-harvestable resources

Metasearch? Resource recommendation service Database spotlighting

Positioning of Discovery vs native Interfaces

Current generation of discovery interfaces lack important features Service delivery (items borrowed, renewals,

fee payments, etc) Browse and other advanced search or

retrieval features Many libraries use native Web-based

catalog to supplement Native interfaces of major information

products appeal to discipline specialists

Content + Services

Must go beyond discovery to fulfillment Further integration of user services

features into discovery interface Increased resource sharing capabilities

LIBRIS

National Union Catalog > Local catalog? Local LMS?

LMS deployments in Sweden -- Academic

LMS deployments in Sweden -- Public

The next new front for Library Discovery

Mobile

Relevant Technology Trends

Service-oriented architecture Key technology for interoperability

among diverse software applications New applications built with SOA

throughout Legacy applications with a services layer

Aggregating data and metadata Open source Commercial partnerships

Mobile access to library content and services

New opportunity to retain and attract library users

Mobile web and apps Working toward a unified Mobile library

presence Unify disjointed mobile silos the same

ambitions as we have for our the Web

Questions and Discussion