DIGITAL LIBRARIESDIGITAL LIBRARIES

Feasibility, Features, Functionalities and the Future

Dr. M.G. SreekumarDr. M.G. SreekumarUNESCO Coordinator, Greenstone Support, South AsiaUNESCO Coordinator, Greenstone Support, South AsiaUNESCO Coordinator, Greenstone Support, South AsiaUNESCO Coordinator, Greenstone Support, South Asia

Indian Institute of Management Kozhikode (IIMK)Indian Institute of Management Kozhikode (IIMK)

Agenda• Organizational Knowledge – Enterprise Content

Management • Digital Library – Concepts, Principles and

Technologies, Architecture…• Web and its variant forms• Open (Source) Digital Libraries

M t d t C t F ti d St d d• Metadata – Concepts, Functions and Standards• DL : Functional Components, Workflows & Procedures

DL B ild St t i• DL : Build up Strategies• Hardware / Storage / Space

Software Selection• Software Selection• DL Architecture - Major Tasks - DL Hardships

Digital LibrariesDigital LibrariesDigital LibrariesDigital LibrariesDigital Libraries are becoming important and they are gaining Digital Libraries are becoming important and they are gaining g g p y g gg g p y g gincreasing social attention, academic and research interestincreasing social attention, academic and research interestDemand for improved information and knowledge Demand for improved information and knowledge management solutionsmanagement solutions -- universities, enterprises anduniversities, enterprises andmanagement solutions management solutions universities, enterprises and universities, enterprises and institutionsinstitutionsNeed for Integrated access to disparate information resourcesNeed for Integrated access to disparate information resourcesKey challenge Key challenge -- how to create online information how to create online information environments facilitating internal content publishing and environments facilitating internal content publishing and single point access to internal/external information sources single point access to internal/external information sources Latest DL technologies Latest DL technologies VsVs Traditional libraries and Traditional libraries and knowledge managementknowledge managementFortunately we have a large number of operational digitalFortunately we have a large number of operational digitalFortunately we have a large number of operational digital Fortunately we have a large number of operational digital libraries and serviceslibraries and services

World of Digital Information :World of Digital Information :FFFeaturesFeatures

Great Potential and DynamicGreat Potential and DynamicGreat Potential and DynamicGreat Potential and DynamicEasy to access, disseminate, store, retrieve, archive, Easy to access, disseminate, store, retrieve, archive, copy, transmit ...copy, transmit ...py,py,Ubiquity of the Net / WebUbiquity of the Net / WebInformationInformation -- Any time / Anywhere / AnyoneAny time / Anywhere / AnyoneInformation Information Any time / Anywhere / AnyoneAny time / Anywhere / AnyoneAccess by a wide spectrum of UsersAccess by a wide spectrum of UsersEasiness of accessEasiness of access -- Plug & PlayPlug & PlayEasiness of access Easiness of access -- Plug & PlayPlug & PlayCurrency of the material / informationCurrency of the material / informationIncrease in valueIncrease in valueIncrease in value Increase in value

Unique Features of the Net/WebUnique Features of the Net/WebUnique Features of the Net/WebUnique Features of the Net/Web

ReachReach unprecedentedunprecedentedReach Reach -- unprecedentedunprecedentedRichness Richness -- unquestionedunquestionedFeedback Feedback -- excellent excellent Content HolderContent HolderContent PublisherContent PublisherContent CommunicatorContent CommunicatorContent CommunicatorContent CommunicatorAsynchronousAsynchronousD th f Di t / TiD th f Di t / TiDeath of Distance / TimeDeath of Distance / Time

The Current EnvironmentThe Current EnvironmentThe Current EnvironmentThe Current EnvironmentFascinatingFascinating timestimes inin thethe historyhistory ofof libraries,libraries,FascinatingFascinating timestimes inin thethe historyhistory ofof libraries,libraries,informationinformation systemssystems andand electronicelectronic publishingpublishing

PossibilitiesPossibilities ofof b ildingb ilding largelarge scalescale ser icesser icesPossibilitiesPossibilities ofof buildingbuilding largelarge--scalescale servicesservices

MaterialsMaterials areare storedstored onon computerscomputers

NetworkNetwork connectsconnects thethe computerscomputers toto personalpersonal computerscomputersonon thethe users'users' desksdesksonon thethe usersusers desksdesks

InIn aa completecomplete digitaldigital library,library, nothingnothing needneed everever reachreachpaperpaper

Internet / IntranetInternet / Intranet

Multimedia Library Info System

Multimedia Library Info System

Gateway-out Data capture

USER @ anywhere (access to information from anywhere)

Challenges of the DayChallenges of the Dayg yg yProliferation of ContentCollection Building – Acquisition, Subscriptions, Licensing…Diverse Datastreams - Content Categories, Publication TypesMultimedia, Multiformats, Multilingual Copyright, Intellectual Property, Fair Use…Technology Complexities, Infrastructure IssuesPublishers’ Stringent Policies / MonopoliesIntegration of legacy systems and the new genre

Penetration of E Content in LibrariesE-Content in Libraries

PUBLICATION TYPES DOCUMENT FORMATS

• E-Books, E-Journals… • ASCII, RTF, HTML, SGML, • Aggregated Scholarly E-

Journal Databases

• Databases CBT/ WBT

Postscript, PDF, Proprietary, Native Application Formats• Databases, CBT/ WBT

• Portals, Vortals…

• Value added services

pp cat o o ats

• Images, Graphics

• AudioValue added services

• Preprints, Eprints, E-Documents….

• Video

• XHTML, ASP, PHP, XML ...

What’s a DL ? What’s a DL ? "Digital libraries are organized collections of digital information. They "Digital libraries are organized collections of digital information. They combine the structuring and gathering of information, which libraries and combine the structuring and gathering of information, which libraries and

hi h l d ith th di it l t ti th t thi h l d ith th di it l t ti th t tarchives have always done, with the digital representation that computers archives have always done, with the digital representation that computers have made possible." (have made possible." (Michael LeskMichael Lesk) ) “Is a managed collection of information, with associated services, where “Is a managed collection of information, with associated services, where the information is stored in digital formats and accessible over a network.the information is stored in digital formats and accessible over a network.the information is stored in digital formats and accessible over a network. the information is stored in digital formats and accessible over a network. A crucial part of this definition is that the information is managed. A A crucial part of this definition is that the information is managed. A stream of data sent to earth from a satellite is not a library. The same data, stream of data sent to earth from a satellite is not a library. The same data, when organized systematically, becomes a digital library collection." when organized systematically, becomes a digital library collection." ((William ArmsWilliam Arms))((William ArmsWilliam Arms) ) DigitalDigital librarylibrary isis "a"a focusedfocused collectioncollection ofof digitaldigital objects,objects, includingincluding text,text,video,video, andand audio,audio, alongalong withwith methodsmethods forfor accessaccess andand retrieval,retrieval, andand forforselection,selection, organization,organization, andand maintenancemaintenance ofof thethe collectioncollection..""((II WittWitt dd D idD id B i b idB i b id ))((IanIan WittenWitten andand DavidDavid BainbridgeBainbridge)).."Digital libraries are different [from traditional library automation] in that "Digital libraries are different [from traditional library automation] in that they are designed to support the creation, maintenance, management, they are designed to support the creation, maintenance, management, access to, and preservation of digital content. access to, and preservation of digital content. (Bernie Hurley,(Bernie Hurley, the Director the Director , p g, p g ( y,( y,for Library Technologies at U.C.Berkeley. Quoted in for Library Technologies at U.C.Berkeley. Quoted in Digital library technology Digital library technology trendstrends. Sun Microsystems. August 2002) . Sun Microsystems. August 2002)

What is a “digital library”?

Collection of digital objects (text, video, audio) along with methods for access and retrieval [ ]with methods for access and retrieval, [user]and for selection, organization, and maintenance [lib]

Traditional user/librarian distinction is blurredComputers make information activeKit h f k l d tiKitchens for knowledge preparationWWW ≠ DL!—organization, selectivityNice Web site ≠ DL!—import new documents easily

Ian Witten

DL FeaturesDL FeaturesDL FeaturesDL Features

Digital libraries encompass a whole range of information Digital libraries encompass a whole range of information services related work such asservices related work such as

Organization of digital informationOrganization of digital informationInformation retrievalInformation retrievalUser interfaceUser interfaceArchiving and preservationArchiving and preservationServices and social issuesServices and social issuesEvaluation and applications to particular areasEvaluation and applications to particular areas

Digital Libraries as ‘Collections’Digital Libraries as ‘Collections’

Digital Libraries as ‘Institutions’ Digital Libraries as ‘Institutions’ gg

DigitalDigital librarieslibraries areare organizationsorganizations thatthat provideprovide thetheDigitalDigital librarieslibraries areare organizationsorganizations thatthat provideprovide thetheresources,resources, includingincluding thethe specializedspecialized staff,staff, towardstowardsbuildingbuilding andand operatingoperating DLsDLs

DigitalDigital librarieslibraries asas aa dynamic,dynamic, growinggrowing organismorganism

DigitalDigital librarieslibraries evolveevolve andand becomebecome thethe predominantpredominantmodemode ofof accessaccess toto knowledgeknowledge andand learning,learning,institutionalizationinstitutionalization ofof digitaldigital librarieslibraries appearsappears toto bebe ananincreasingincreasing possibilitypossibilityincreasingincreasing possibilitypossibility

Benefits of DLsBenefits of DLs

DLsDLs areare structuredstructured storagestorage environmentsenvironments ofof digitaldigitaldatadata withwith aa consistentconsistent formatformat forfor indexindex andandcontentcontent abstractionabstraction..DLsDLs dodo enableenable thethe seamlessseamless integrationintegration ofof thethescholarlyscholarly electronicelectronic information,information, helphelp inin creatingcreatingandand maintainingmaintaining locallocal digitaldigital contentcontent andandandand maintainingmaintaining locallocal digitaldigital content,content, andandstrengthenstrengthen thethe mechanismsmechanisms andand thethe capacitycapacity ofof thethelibrary’slibrary’s informationinformation systemssystems andand servicesservices..library slibrary s informationinformation systemssystems andand servicesservices..DLsDLs increaseincrease thethe portability,portability, efficiencyefficiency ofof access,access,flexibility,flexibility, availabilityavailability andand preservationpreservation ofof digitaldigitaly,y, yy pp ggobjectsobjects..

Benefits of DLsBenefits of DLsOutreach Outreach -- Library goes to the user Library goes to the user

Seamless Access Seamless Access -- Searching and browsing Searching and browsing

Borderless Dissemination Borderless Dissemination

Instantaneous and Current Instantaneous and Current

Always (24X7) available Always (24X7) available

L t tiL t tiLong term preservationLong term preservation……

LimitationsLimitations of DLsof DLsTechnological obsolescence Technological obsolescence

HardwareHardwareHardwareHardwareSoftwareSoftware

Q it T d d h F il tQ it T d d h F il tQuite Tender and hence Fragile tooQuite Tender and hence Fragile tooSecurity Issues Security Issues –– Being rigorously addressedBeing rigorously addressed

i hl i i di hl i i d llllHighly sensitive to Commands Highly sensitive to Commands –– Even a small Even a small ignorance or carelessness could be very fatal at timesignorance or carelessness could be very fatal at timesResources Cost ManpowerResources Cost ManpowerResources, Cost, ManpowerResources, Cost, ManpowerBandwidthBandwidthRights ManagementRights ManagementRights ManagementRights Management……

Functional ComponentsFunctional Componentspp

Creation of DLsCreation of DLs

Digital ObjectsDigital ObjectsDigital ObjectsDigital Objects

Digital ObjectsDigital ObjectsDigital ObjectsDigital ObjectsDigital objects of analogue/physical equivalents:Digital objects of analogue/physical equivalents: pictures, pictures, g j g p y qg j g p y q ppvideo clips, music, publications, maps, artifacts (e.g. museum video clips, music, publications, maps, artifacts (e.g. museum objects), living beings (plants, animals, people), animation's, objects), living beings (plants, animals, people), animation's, slide shows, print publications, etc. In case of some of these slide shows, print publications, etc. In case of some of these , p p ,, p p ,entities (for example, artifacts like buildings and museum entities (for example, artifacts like buildings and museum objects and living beings) digital objects may only carry objects and living beings) digital objects may only carry relevant metadata information and possibly some form of relevant metadata information and possibly some form of p yp ymultimedia representation of the entity (e.g. photographs). multimedia representation of the entity (e.g. photographs).

Digital objects that do not have physical counterparts and Digital objects that do not have physical counterparts and th t d d i ll d i lth t d d i ll d i l titi l t il t ithose created dynamically and in realthose created dynamically and in real--time:time: electronic electronic publications, software, spread sheets, databases, data gathered publications, software, spread sheets, databases, data gathered from remote sensors, software agents, and live capture of from remote sensors, software agents, and live capture of di it l i f h i d iddi it l i f h i d iddigital versions of speech, music and video. digital versions of speech, music and video.

LibrariesLibraries -- ShiftsShiftsLibraries Libraries ShiftsShifts

Traditional / AutomatedTraditional / AutomatedOrganization is physicalOrganization is physicalOrganization is physical Organization is physical Shelving of documents Shelving of documents -- Based on Subject ClnBased on Subject ClnKey Key -- Index / Catalogues / Cards / Digital Catalogs Index / Catalogues / Cards / Digital Catalogs CardsCards -- Real/VirtualReal/Virtual -- Author Title DescriptionsAuthor Title DescriptionsCards Cards -- Real/Virtual Real/Virtual -- Author, Title, DescriptionsAuthor, Title, Descriptions

DigitalDigitalOrganization in terms of digital files /objectsOrganization in terms of digital files /objectsg g jg g jContains material digitized formContains material digitized formContains digital materialContains digital materialArchitecture Architecture Key Key -- MetadataMetadata

Shift in ApproachesShift in ApproachesppppTraditional Automated Dig. Library

Limited/ RigidEfficient/ Flexible

Improved

AACR2AACR2 MetadataISO 2709CCFMARC

CCCCC / LCCSDDC / UDC

MetadataDCMI -- W3CEAD, TEI, DTDMETS,MODS,

ThesauriThesauri/LCSH Z39.50MARC21OAI-PMH

What are digital libraries for?Knowledge/content management

Manage and access internal information assetsScholarly communication, education, research

E j l i t b k d t t l iE-journals, e-prints, e-books, data sets, e-learningAccess to cultural collections

Cultural heritage historical & special collectionsCultural, heritage, historical & special collections, museums, biodiversity

E-governanceImproved access to government policies, plans, procedures, rules and regulations

Archiving and preservationArchiving and preservationMany more …

Features of Digital Libraries…g

• Dynamic Electronic Information Systems• Seamless Aggregation and Integration of Scholarly

ContentCreate / Maintain Local Content• Create / Maintain Local Content

• Strengthens - mechanisms and capacity -Information Systems / Servicesy /

• Increase Portability• Efficiency of Access• Flexibility• Availability

L t ti• Long term preservation

UNESCO

What Distinguishes a DL?What Distinguishes a DL?

Site Neutrality (3 in 1 Access-Anytime,Site Neutrality (3 in 1 Access Anytime,Anywhere by Anyone Access)Open AccessOpen AccessGreater variety and granularity of informationSh i f i f ti ‘Sh i ’Sharing of information ‘Sharium’Up-to-date nessAlways available (365*7*24)New forms of rendering (New Genre)

Digital Libraries: An Overview

Lib iDigital Libraries

C i N ki C C ll i S i C iComputing Networking Content Collections Services Community

Special Requirements p q

• InfrastructureInfrastructure• Acceptability• Access Restrictions• Readability• Standardization• Authentication• Preservation

Copyright• Copyright• User Interface

Need for Content Integration / O i tiOrganization

A i S l A t th C t t • Assuring Seamless Access to the Content • Need for a single Info. Gateway / Access Point • Multi - Formats Media Platforms (Content / Data • Multi - Formats, Media, Platforms (Content / Data

in different formats)• Data encoding (role of markup languages)• Role of Metadata (role of Standards)• Structured Metadata (role of XML)

d f bili• Need for Interoperability• Interface / Delivery / Presentation• Exorbitant cost of proprietary DL S/W• Exorbitant cost of proprietary DL S/W

Digital Library TechnologiesDigital Library Technologies

• Open architectures (Open DLs)

• Componentized vs Monolithic systems

• Interoperability (role of Z39.50, OAI etc.)p y

• Unified interface for heterogeneous libraries

• Metadata mapping across different libraries pp g

• OAI-compliant data and service providers

• Multilingual digital libraries g g

• Scalable digital library architectures

• Publication tools

• Searching tools

Space Requirements: For 100,000 Space Requirements: For 100,000 Articles (Text) having 5 pages eachArticles (Text) having 5 pages each

Space Requirements: For 100,000 Space Requirements: For 100,000 Images (640X480 in 256 colours)Images (640X480 in 256 colours)

Space Requirements: For 100,000 Audio Space Requirements: For 100,000 Audio di ( lf S d 8 i 11di ( lf S d 8 i 11Recordings (Half Sound, 8 Bit 11 KHzRecordings (Half Sound, 8 Bit 11 KHz-- Mono Mono

and 16 Bit 44 KHz Stereo, 10 Mins each)and 16 Bit 44 KHz Stereo, 10 Mins each)

Space Requirements: For 100,000 Video Space Requirements: For 100,000 Video Clips (320X200 and 256 colours at 15 fps)Clips (320X200 and 256 colours at 15 fps)

Bandwidth RequirementsBandwidth Requirements

Desirable Features of DL SoftwareDesirable Features of DL SoftwareDesirable Features of DL SoftwareDesirable Features of DL Software

St tSt tStructuresStructuresAccessibleAccessibleSearchableSearchableExtensibleExtensibleExtensibleExtensibleMassiveMassiveH tH tHeterogeneousHeterogeneousPersistentPersistent

Desirable Features of DL …Desirable Features of DL …Low cost, including all hardware and software Low cost, including all hardware and software componentscomponentsTechnically simple to install and manageTechnically simple to install and manageRobustRobustScalableScalableOpen and interoperableOpen and interoperableModularModularUser FriendlyUser FriendlyM ltiM lti (i l di b th hi d i t )(i l di b th hi d i t )MultiMulti--user (including both searching and maintenance) user (including both searching and maintenance) Multimedia digital object enabledMultimedia digital object enabledPlatform independent (including both client and serverPlatform independent (including both client and serverPlatform independent (including both client and server Platform independent (including both client and server components) interoperablecomponents) interoperable

DL’s operation should be DL’s operation should be examined under…examined under…

Architectural design Architectural design –– Modular and OpenModular and Open

Backend Database Backend Database –– scalable, robust, data formatsscalable, robust, data formats

Network capabilities Network capabilities –– webweb--based and seamless based and seamless operations, persistent Ids, security and authenticationoperations, persistent Ids, security and authentication

Metadata and Interoperability Metadata and Interoperability –– compatible with compatible with world standards such as Dublin Core and OAIworld standards such as Dublin Core and OAI--PMHPMH

Technical IssuesTechnical IssuesTechnical IssuesTechnical Issues

Open source software Vs Commercial OSOpen source software Vs Commercial OSHardware and peripheral requirementsHardware and peripheral requirementsp p qp p qNetwork ComponentsNetwork ComponentsStandards Standards –– data formats, metadata, network, access, data formats, metadata, network, access, interoperability, encodinginteroperability, encoding

Software Selection• Goals and Requirement Specification

P i V O S• Proprietary Vs Open Source

• Fit the existing Information System

A d t f t i ti• Accommodate future migration

• Embrace all possible/predominant formats

Support standard DL technologies/platforms• Support standard DL technologies/platforms

• Easy installation, population, maintenance

• Comprehensive Documentation• Comprehensive Documentation

• Software Development Team

• Active User Groups E Mail Lists (Users / • Active User Groups, E-Mail Lists (Users / Developers)

DL Software: AlternativesDL Software: Alternatives

What are your expectations?What are your expectations?Develop local web-based application?Commercial DL solution?Adopt open source software?p p

GreenstoneEprintsDSpaceFedora…

DLs: Workflows and ProcessesDLs: Workflows and Processes

Content selection PreservationContent selectionContent acquisitionContent publishing

PreservationAccess managementUsage monitoring and

Metadata preparationContent loading

Content indexing & storage

evaluationNetworking and interoperationMaintenanceContent indexing & storage

Content access & deliveryMaintenance

DL Software: Key requirements• Document types (book, journal article, lecture …)• Document formats (text, PDF, Word, PS, …)• Content acquisition (online and offline)• Content acquisition (online and offline)

– Metadata description, content tagging– Content uploading

• Indexing and retrieval– Structured/ full text indexing

Automatic metadata extraction– Automatic metadata extraction• Storage

– Data compression– Efficient storage for metadata– Efficient location of metadata and documents

• Access and deliveryAccess and delivery– Structured search, browse, hierarchical browsing– CD-ROM distribution

DL Software: More requirementsDL Software: More requirements

• Scaling up for large collections• Scaling up – for large collections• Multilingual support• Access management and security• Access management and security• Usage monitoring and reporting• Standards complianceStandards compliance

– XML, Dublin Core, Unicode• Interoperationp

– OAI, Z39.50 compliance, MARC, CDS/ISIS, …

MetadataMetadataGeneral DefinitionGeneral Definition

MetadataMetadata

Metadata in its broadest sense is Metadata in its broadest sense is data about datadata about dataDocumentation about documents and objectsDocumentation about documents and objectsDocumentation about documents and objectsDocumentation about documents and objectsDescribing (Tagging) the contents of the objectDescribing (Tagging) the contents of the objectFor Information Discovery from the Resource BaseFor Information Discovery from the Resource Baseyy

Internet context Internet context

D tD t d ibi th tt ib t f l t id ibi th tt ib t f l t iData Data describing the attributes of an electronic resourcedescribing the attributes of an electronic resource on on the netthe netDublin Core (DCMI)Dublin Core (DCMI) –– WWW Consortium StandardWWW Consortium StandardDublin Core (DCMI)Dublin Core (DCMI) WWW Consortium StandardWWW Consortium StandardXML XML -- The toolThe tool

Dublin Core Metadata ElementsMetadata Definition

Title The name given to the resource by the creator or publisher Creator The person responsible for the intellectual content of the

resource

The Basics:22 Elements

Metadata Definition

resource Subject The Topic of the resourceDescription A textual description of the content of the source Publisher The Entity responsible for making the resource available Contributor A person or organization (other than the Creator) who is

Content

Responsibility

Contributor A person or organization (other than the Creator) who is responsible for making significant contributions to the intellectual content of the resource

Date A date associated with the creation or availability of the resource

Manifestation

Type The nature or genre of the content of the resource Format The physical or digital manifestation of the resource Identifier An unambiguous reference that uniquely identifies the

resource within a given context S A f t d f hi h th tSource A reference to a second resource from which the present

resource is derived Language The language of the intellectual content of the resource Relation A reference to a related resource, and the nature of its

relationshiprelationship Coverage Spatial locations and temporal durations characteristic of

the content of the resourceRights Information about rights held in the resource

DLDL -- HardshipsHardshipsDL DL HardshipsHardships

Copyright IssuesCopyright IssuesTechnology ComplexitiesTechnology ComplexitiesTechnology ComplexitiesTechnology ComplexitiesInfrastructure IssuesInfrastructure IssuesPublications/Formats Publications/Formats –– Diverse DatastreamsDiverse DatastreamsDigital Objects/Formats Digital Objects/Formats -- Multiple Multiple Publishers’ Policies Publishers’ Policies –– Stringent, InconsistentStringent, Inconsistent

Approaches to Building DLApproaches to Building DLApproaches to Building DLApproaches to Building DL

Digitization Digitization –– retroretro--conversion of nonconversion of non--digital digital t di it lt di it lresources to digitalresources to digital

Di i ll bDi i ll b i l ii l i iiDigitally born resources Digitally born resources –– involves interinvolves inter--conversion conversion to standard formats and storageto standard formats and storage

Major TasksMajor TasksMajor TasksMajor TasksContent identification (internal / external)Content identification (internal / external)Content CreationContent CreationContent Collation/SignpostsContent Collation/SignpostsOrganisationOrganisationUpdationUpdationRetrieval / Dissemination Retrieval / Dissemination User TrainingUser TrainingArchivingArchivinggg

DIGITAL LIBRARY ARCHITECTURE

OS

NetworkARCHITECTURE

METS/MODS

Z39.50 /OAI-PMH

DL Software

Data/Objects

METS/MODS

EAD TEIj

DCMI