lecture 14 interoperability for information discovery

1 herbert van de sompel

Lecture 14 Interoperability for information discovery

CS 502 Computing Methods for Digital Libraries

Cornell University – Computer ScienceHerbert Van de [email protected]

Acknowledgements: Carl Lagoze


Why interoperability?

The distributed information environment creates enormous challenges regarding the provision of coherent services. Addressing these challenges requires some form of interoperability.


OPAC FTXT

FTXTe-printA&I

A&I

distributed

herbert van de sompel

Information resources: distributed


OPAC FTXT

FTXTe-printA&I

A&I

range of authorities, technologies


Information resources: different authorities, technologies


OPAC FTXT

FTXTe-printA&I

A&I

¡¡ challenges re integrated access !!


Information resources: challenges re coherent services


Interoperability helps!

• UNICODE• XML• XML Schema, XML Namespaces• HTTP• Metadata formats (MARC, Dublin Core, …)

• This lecture: interoperability for resource discovery (searching)• Next week: interoperability for reference linking


• reaching interoperability is an organizational challenge (not only technical)

• the challenge is to create incentives for independent digital libraries to adopt specifications

Interoperability: technical & organizational


Functionality versus cost of acceptance

Functionality

Cost of acceptance

Metadata HarvestingOAI

SDLIP

Z39.50

This picture does not show the eventual benefit of a protocol in the creation of coherent services


A&I

federated searching

image

FTXT

OPAC

e-print


Z39.50

http://www.loc.gov/z3950/agency/


• Permits one computer (z3950 client) to search and retrieve information on another computer serving a database (z3950 target)

• Important both technically and for its wide use in library systems, A&I databases

• Most development has concentrated on bibliographic data

• Most implementations emphasize searches that use a bibliographic set of attributes to search databases of MARC records

Aims of z39.50


• Developed for X.25 networks (connection orientation), conversion to run over TCP fitted later

• Original concept in days when repeating a search was expensive computation (about 1980)

• NISO standard

Technical history of z39.50


Abstract view of database searching.

• Server stores a set of databases with searchable indexes

• Interactions are based on a session

• The client opens a connection with the server, carries out a sequence of interactions and then closes the connection.

• During the course of the session, both the server and the client remember the state of their interaction.

z39.50 - principles


• The server carries out the search and builds a results set

• Server saves the results set.

• Subsequent message from the client can reference the result set.

• Thus the client can modify a large set by increasingly precise requests, or can request a presentation of any record in the set, without searching entire database.

z39.50 - state


init -- client connects to the server and exchanges initial information, e.g., preferred message size

explain -- client inquires of the server what databases are available for searching, the fields that are available, the syntax and formats supported, etc.

search -- client presents a query to a database; choices of syntax for specifying searches

• Boolean queries widely implemented

z39.50 - services

manipulation of results sets -- e.g., sort or delete

present -- requests the server to send specified records from the results set to the client in a specified format

scan – browse indexes


In the database named Books find all records for which the access point title contains the value evangeline and the access point author contains the value longfellow.

Z39.50 defines a rich variety of search access points (use attributes) that can be extended by implementers

z39.50 – sample query

http://lcweb.loc.gov/z3950/agency/defns/bib1.html


z39.50 – issues

No general acceptance (for instance not supported by web browsers => httpd/z3950 gateways)

Merging results for distributed search

Semantic interoperability ~ z39.50 profiles


Simple Digital Library Interoperability Protocol

http://www-diglib.stanford.edu/~testbed/doc2/SDLIP/


• Compromise between a full-scale, all encompassing approach such as Z39.50 and the “anything goes” approach typical for ad-hoc search interface design on web

• Developed jointly by Stanford, Berkeley, and UC Santa Barbara

• Heavily influenced by DASL from IETF

SDLIP


SDLIP – search middleware


SDLIP – managing complexity via separating interfaces


• Search Interface – defines simple query language, protocol can then include other languages

• Result Interface – parking meter metaphor (server decides on length of session)

• Source Metadata Interface – allows to query a library proxy about its capabilities (subcollections, attributes that can be searched, …)

SDLIP - interfaces


Open Archives Initiative Metadata Harvesting Protocol

http://www.openarchives.org


• Low-barrier framework for repository interoperability• Minimal burden for parties providing databases• Plug-in concept to allow community and service

specialization (placeholders – about, descriptor ; parallel metadata formats; …)

OAMH protocol


metadata

A&I

image

OPAC

e-print

FTXT

harvester

FTXT

OAMH - metadata harvesting


metadata

A&I

image

FTXT

e-print

AuthorTitleAbstractIdentifer

OPAC

OAMH – federated services


service provider data provider

Requests

Replies

repos i tory

harves ter

6

OAMH – protocol


• low-barrier interoperability

• data-provider & service-provider model

• metadata harvesting model OAMH protocol

Dublin Core

HTTP basedReply • XML Schema

• Self contained• shared metadata format and parallel, community-

specific metadata formats

OAMH – core concepts

• authentication etc. : on purpose outside of the

protocol



DatestampIdentifierSet

Records

repos i tory

harves ter

OAMH – protocol tools


Supporting protocol requests:• Identify• ListMetadataFormats• ListSets

Harvesting protocol requests:• ListRecords• ListIdentifiers• GetRecord

repos i tory


harves ter

OAMH – protocol tools


ListMetadataFormats

ListMetadataFormats / Time / Request REPEAT

• Format prefix• Format XML schema

/REPEAT

repos i tory


harves ter

OAMH – a supporting protocol request


* from=a * until=b * set=klmListRecords * metadataPrefix=dc

ListRecords / Time / Request REPEAT

• Identifier• Datestamp

• Metadata/REPEAT

repos i tory


harves ter

OAMH – a harvesting protocol request


• federated services [S&R, SDI, alerting, linking, ...]• database synchronization• harvesting the deep Web• ...

OAMH – applications?


OAMH – issues

• sync between data provider and service provider (cf. Web page harvesters)

• everyone harvesting from everyone? (cf. brokers)

• CAN attractive cross-community services be built?

• gaining critical mass


• There is (and will never be) one right solution (technical vs. cost vs. complexity vs. ??) (cf metadata)

• Distributed technical solutions have organizational ramifications

• A revival of the simplicity approach: specifications that require little effort by the digital libraries involved, but that can eventually lead to appealing services (when combined with brute force computing and intelligent tools)

Some thoughts

lecture 14 interoperability for information discovery

Documents