tefko saracevic 1 preservation of digital records concerns, approaches, efforts tefko saracevic,...

Tefko Saracevic Tefko Saracevic 11

Preservation of digital Preservation of digital records records

Concerns, approaches, Concerns, approaches, efforts efforts

Tefko Saracevic, Ph.D.Tefko Saracevic, Ph.D.

22

ToCToC Introductory musings: definitions & Introductory musings: definitions &

problem statementproblem statement Library involvement over timeLibrary involvement over time Preservation in digital environmentsPreservation in digital environments Technological problems, solutionsTechnological problems, solutions Preservation projectsPreservation projects Preservation standards Preservation standards Concluding musings: issues, questionsConcluding musings: issues, questions

Tefko Saracevic Tefko Saracevic


Preservation – general definitionsPreservation – general definitions

To preserve: To preserve: ““To keep alive, keep from perishing (arch.); to keep in existence, keep from decay, make lasting (a material thing, a name, a memory)” (OED, 2nd ed.)


ProblemProblem

Unlike physical Unlike physical materials, which can materials, which can remain in their current remain in their current state for decades, state for decades, even centurieseven centuries

While we are still able to While we are still able to read our written read our written heritage from several heritage from several thousand years agothousand years ago

Content stored in digital Content stored in digital formats is easily formats is easily altered or even lostaltered or even lost

The digital information The digital information created merely a created merely a decade ago is in decade ago is in serious danger of serious danger of being lostbeing lost

· · · — — — · · · · · · — — — · · ·· · · — — — · · · · · · — — — · · · · · · — — — · · · · · · — — — · · · · · · — — — · · ·· · · — — — · · · · · · — — — · · ·· · · — — — · · ·


Solutions Solutions

PreservationPreservation but digital preservation is only half the battle, but digital preservation is only half the battle,

PermanencePermanence is needed for and closely linked with is needed for and closely linked with

preservation of digital resourcespreservation of digital resources


Historically in libraries:Historically in libraries:long time involvementlong time involvement

PreservationPreservation: : maintaining or maintaining or restoring access to restoring access to artifacts, documents artifacts, documents and records through and records through the study, diagnosis, the study, diagnosis, treatment and treatment and prevention of decay prevention of decay and damageand damage

Conservation: Conservation: the treatment and the treatment and repair of individual repair of individual items to slow decay items to slow decay or restore them to a or restore them to a usable state. usable state. “Conservation” is “Conservation” is occasionally used occasionally used interchangeably with interchangeably with “preservation”“preservation”


Paper degradation problemsPaper degradation problems

Paper embrittlement from acid decay; Paper embrittlement from acid decay; brittle books, newsprint brittle books, newsprint (“slow fire”)(“slow fire”) mass deacidification efforts;mass deacidification efforts; promotion of acid-free paperpromotion of acid-free paper many library projects globallymany library projects globally degradation affects any media, not just paperdegradation affects any media, not just paper

Infestation (mold, fungi, bacteria):Infestation (mold, fungi, bacteria): The use of gamma rays in book conservation

Book conservation


Another solution: reformattingAnother solution: reformatting

Use of other media to store documentsUse of other media to store documents Microfilm became popular & widespreadMicrofilm became popular & widespread

many advantages – easy storagemany advantages – easy storage many disadvantagesmany disadvantages

no such thing as cuddly microfilm readerno such thing as cuddly microfilm reader searching not possiblesearching not possible

Followed by other media: various tapes, Followed by other media: various tapes, cartridges, optical disks, CD-ROMs …cartridges, optical disks, CD-ROMs …

Finally, digital mediaFinally, digital media


One of traditional concerns: One of traditional concerns: preparednesspreparedness

Natural disasters & libraries – many experiencesNatural disasters & libraries – many experiences The The Flood of the River Arno in Florence, Italy, ,

(1966) damaged or destroyed great many rare (1966) damaged or destroyed great many rare books & books & art led to establishing restoration laboratories in many led to establishing restoration laboratories in many

placesplaces

Recognizing importance of having a disaster Recognizing importance of having a disaster preparedness & preservation planspreparedness & preservation plans

e.g U Delaware Library e.g U Delaware Library Disaster Response plan ALA ALA Disaster Preparedness and Recovery


Libraries not aloneLibraries not alone

Sharing preservation concerns withSharing preservation concerns with archivesarchives historical institutionshistorical institutions museumsmuseums antiquarian practices antiquarian practices archeologyarcheology

National Center for Preservation Technology & Training


Preservation in digital environmentsPreservation in digital environments

Management of digital information over Management of digital information over timetime

Constant Constant efforteffort & & expendituresexpenditures to handle to handle rapid technological and organizational rapid technological and organizational advancesadvances main stumbling block for preserving digital main stumbling block for preserving digital

information beyond a couple of years information beyond a couple of years

Terminology not fixed – here is a list of definitionsfrom Digital Preservation Coalition


Digital preservation goalsDigital preservation goals

Ensuring the Ensuring the continued access to continued access to information and all information and all kinds of records, kinds of records, scientific and cultural scientific and cultural heritage existing in heritage existing in digital formats digital formats

Long-term, error-free Long-term, error-free storage of digital storage of digital information, with information, with means for retrieval means for retrieval and interpretation, for and interpretation, for all the time span that all the time span that the information is the information is required for required for

From preservation to permanenceFrom preservation to permanence


The second half of the battle:The second half of the battle: permanence permanence

1.1. Identifier validity:Identifier validity: the extent to which the extent to which the given name or identifier will always the given name or identifier will always provide access to same resource provide access to same resource

2.2. Resource availability:Resource availability: the extent to the extent to which given resource is guaranteed to which given resource is guaranteed to remain available in electronic formremain available in electronic form

3.3. Content invariability:Content invariability: the extent to the extent to which the content of resource could which the content of resource could changechange

from from presentation Digital Archives at NLM


Technological problems: Technological problems: obsolescence obsolescence

Format obsolescenceFormat obsolescence when the software required when the software required

to read the content or data to read the content or data is no longer available or is is no longer available or is unable to understand the unable to understand the format of the dataformat of the data

RequiresRequires copying of content onto copying of content onto

newer formatnewer format converting content from converting content from

one format to another one format to another avoiding loss of fidelity avoiding loss of fidelity

Technology obsolescence Technology obsolescence when the hardware when the hardware

required to read the data is required to read the data is no longer available or new no longer available or new hardware or media hardware or media emergesemerges

RequiresRequires transfer from one transfer from one

technology or media to technology or media to another another

e.g. from one kind of tapes, e.g. from one kind of tapes, disks to anotherdisks to another

from microfilm to digitalfrom microfilm to digital


Technological obsolescence Technological obsolescence solutionssolutions

Transferring of Transferring of content or data to content or data to newer systemsnewer systems

Conversion from one Conversion from one format to another or format to another or one operating system one operating system to another or one to another or one programming programming language to another language to another

EmulationEmulation - content - content is both preserved and is both preserved and presented to readers presented to readers in the original format in the original format

MigrationMigration - content is - content is presented in a current presented in a current format; it may be format; it may be preserved in a preserved in a succession of current succession of current formats formats


Emulation problems:Emulation problems:BBC Domesday project - 1986BBC Domesday project - 1986

Attempt to re-do the survey of England done in 1085

Unfortunately, published on 12-inch laser disc in a format that died quickly in the marketplace

Leeds Univ. & U. Michigan now trying to emulate original hardware

From Michael Lesk


Digital preservation projectsDigital preservation projects Digital Preservation Coalit

ion (DPC) (UK)(DPC) (UK)

““to secure the preservation to secure the preservation of digital resources in the of digital resources in the UK and to work with others UK and to work with others internationally to secure internationally to secure our global digital memory our global digital memory and knowledge base.”and knowledge base.”

Sound Direction (U (U Indiana)Indiana)

““digital preservation & digital preservation & access for global audio access for global audio heritage “heritage “

Porticolaunched by launched by JSTOR

““preserve scholarly literature preserve scholarly literature published in electronic form”published in electronic form”

many publishers & libraries many publishers & libraries participatingparticipating

so far ~ 20 mill. unitsso far ~ 20 mill. units

MetaArchive cooperative

over over 50 institutions50 institutions““The Greatest Threat The Greatest Threat to digital assets is not to digital assets is not

fire, flood or theft. It’s the assumption that fire, flood or theft. It’s the assumption that cultural memory organizations have taken cultural memory organizations have taken the requisite steps to preserve them.”the requisite steps to preserve them.”


General international resourceGeneral international resource

Preserving Access to Digital Information (PADI) (Australia)(PADI) (Australia) ““gateway to international digital preservation gateway to international digital preservation

resources”resources” aims “to provide mechanisms that will help to aims “to provide mechanisms that will help to

ensure that information in digital form is ensure that information in digital form is managed with appropriate consideration for managed with appropriate consideration for preservation and future access”preservation and future access”

wide coverage:wide coverage: policies, topics, projects, legal deposits …policies, topics, projects, legal deposits …


Technology & services support:Technology & services support:Duplication and sharingDuplication and sharing

Protection against loss via multiple copies

Individual backups are traditional, but do not protect against organizational disappearance

if not regularly exercised might not be dependable

LOCKSS (Lots of Copies (Lots of Copies Keep Stuff Safe) at Keep Stuff Safe) at StanfordStanford

““provides tools and support provides tools and support so libraries can easily and so libraries can easily and cost-effectively preserve cost-effectively preserve today’s web-published today’s web-published materials for tomorrow’s materials for tomorrow’s readers” readers”

free, open source softwarefree, open source software and a and a YouTube video

Basic approach – from a quoteBasic approach – from a quote

““...let us save what remains: not by vaults ...let us save what remains: not by vaults and locks which fence them from the and locks which fence them from the public eye and use in consigning them to public eye and use in consigning them to the waste of time, but by such a the waste of time, but by such a multiplication of copies, as shall place multiplication of copies, as shall place them beyond the reach of accident.”them beyond the reach of accident.”

— Thomas Jefferson, February 18, 1791 — Thomas Jefferson, February 18, 1791



"Darwin's tortoise" dies, age 176” ” (June 26, 2006).(June 26, 2006). The LOCKSS logo is a tortoise; tortoises live a veryThe LOCKSS logo is a tortoise; tortoises live a very

long timelong time

LOCKSS Alliance is a library membership organization

they have LOCKSS Boxes libraries around the globe

participate

Large number of publishers participate also

most open access

LOCKSS software turns a PC into a digital preservation appliance (a LOCKSS Box)

collects newly published content

compares it with other LOCKSS Boxes

acts as a web proxy or cache

provides a web-based administrative interface


LOCKSS: time and consensusLOCKSS: time and consensus(Michael Lesk)(Michael Lesk)

The LOCKSS project is particularly interesting for two reasons: A. running slowlyB. relying on a

combination of consensus and reputation

A. By making it fast to find one copy of something, but slow to find all copies, it becomes difficult for a vandal to find and destroy all copies of a file

B. By relying on a weighted polling system in which a site can gain weight only by agreeing with many prior decisions, it is difficult even for an insider to insist on installing bad versions of files.


Authentication of digital Authentication of digital resourcesresources

Authenticity: Authenticity: The digital The digital material is what it material is what it purports to bepurports to be

refers to the refers to the trustworthiness of the trustworthiness of the electronic record as a electronic record as a recordrecord

Confidence in the Confidence in the authenticity of digital authenticity of digital materials over time is materials over time is particularly crucial owing particularly crucial owing to the ease with which to the ease with which alterations can be made. alterations can be made.

In the case of "born In the case of "born digital" and digitized digital" and digitized materials:materials:

the fact that whatever is the fact that whatever is being cited is the same as being cited is the same as it was when it was first it was when it was first created unless the created unless the accompanying metadata accompanying metadata indicates any changesindicates any changes

A number of mechanisms A number of mechanisms used to establish the used to establish the authenticity of digital authenticity of digital materials materials


ArchivingArchiving

OCLC's Digital Archive software & services forsoftware & services for

Web archiving: Item-by-item

Batch archiving: For collections

available to users in multiple ways - through OCLC’s FirstSearch, Connexion, library own OPAC or a Web portal

Dutch National Library agreement with publishers

digital archive for scientific digital archive for scientific research – some dozen research – some dozen publishers deposit journalspublishers deposit journals

““which will be made available which will be made available in perpetuity to the in perpetuity to the research community research community including authors, including authors, researchers, historians and researchers, historians and librarians “librarians “


Really BIG projectReally BIG project

National Digital Information Infrastructure and Preservation Program (NDIIPP) (NDIIPP)

““... implementing a national strategy to collect, ... implementing a national strategy to collect, preserve and make available significant digital preserve and make available significant digital content, especially information that is created in content, especially information that is created in digital form only, for current and future generations. digital form only, for current and future generations. ””

Collaborative approach – building a national Collaborative approach – building a national network – a number of institutions involvednetwork – a number of institutions involved

Congress: $100 mill. + a lot from private Congress: $100 mill. + a lot from private sourcessources Description in Description in Wikipedia


NDIIPP …NDIIPP …

Digital files selected for preservation:Digital files selected for preservation: Geospatial data Geospatial data Web sites Web sites Television Television Social science datasets Social science datasets E-Journals E-Journals Historical materials Historical materials Provides suggestions for Provides suggestions for Personal archiving – your – your

own recordsown records


NDIIPP …NDIIPP …

Some of the NDIIPP-NSF Digital Some of the NDIIPP-NSF Digital Preservation research projectsPreservation research projects U California Libraries: U California Libraries: Tools for Web archiving

Digital Preservation Repository - a - a YouTube video Preserving Digital Public Television (PBS) National Geospatial Digital Archive (NGDA) North Carolina Geospatial Data Archiving Project

(NCGDAP) Sustainability of digital formats (LoC)


and then there isand then there is

CybercemeteryCybercemetery Maintained by U North Texas LibrariesMaintained by U North Texas Libraries ““archive of government websitesarchive of government websites that have that have

ceased operation (usually websites of defunct ceased operation (usually websites of defunct government agencies and commissions that government agencies and commissions that have issued a final report).”have issued a final report).”


And in Europe: EU projectsAnd in Europe: EU projects Open Planets Open Planets

Foundation Foundation ““A community hub for A community hub for

digital preservationdigital preservation ““to provide practical to provide practical

solutions and expertise solutions and expertise in digital preservation”in digital preservation”

about 20 members about 20 members internationallyinternationally

open sourcesopen sources

CASPAR CASPAR - Cultural, Artistic and Scientific knowledge for Preservation, Access & Retrieval (EU project)

“... research, implement, and ... research, implement, and disseminate innovative solutions disseminate innovative solutions for digital preservation for digital preservation ”

a community of membersa community of members DigitalPreservationEurope DigitalPreservationEurope

(DPE)(DPE) ““to improve coordination, to improve coordination,

cooperation and consistency in cooperation and consistency in current activities to secure current activities to secure effective preservation of digital effective preservation of digital materials” materials”


Developing standards for Developing standards for approaches to preservationapproaches to preservation

Standards needed to deal withStandards needed to deal with impacts of changing technologies, including support impacts of changing technologies, including support

for new media & data formatsfor new media & data formats changing user communities changing user communities

Open Archival Information System (OAIS)Open Archival Information System (OAIS)Reference ModelReference Model conceptualization of a system that addresses digital conceptualization of a system that addresses digital

preservation - provides a general frameworkpreservation - provides a general framework model describes components and services required to model describes components and services required to

develop and maintain archives develop and maintain archives

Learn more about preservationLearn more about preservation

Cornel University Tutorial – moved to MITCornel University Tutorial – moved to MITDigital Preservation Management – Digital Preservation Management – Implementing Short-term Strategies for Implementing Short-term Strategies for Long-term ProblemsLong-term Problems

Well done & exhaustive treatment with Well done & exhaustive treatment with quizzes, explanations, and questions.quizzes, explanations, and questions.



Summary of preservation requirementsSummary of preservation requirements(from Cornell/MIT tutorial)(from Cornell/MIT tutorial)


Summary: preservation requiresSummary: preservation requires Organizational Infrastructure:Organizational Infrastructure:

WhatWhat are the requirements and parameters for the are the requirements and parameters for the organization's digital preservation program? organization's digital preservation program?

Technological InfrastructureTechnological Infrastructure HowHow will the organization meet defined digital will the organization meet defined digital

preservation requirements? preservation requirements?

Resources Framework - $$$$Resources Framework - $$$$ WhatWhat resources will it take to develop and maintain resources will it take to develop and maintain

the organization’s digital preservation program?the organization’s digital preservation program?


Concluding issues – Lesk’s questions Concluding issues – Lesk’s questions

Should there be compulsory clear-text deposit of electronic resources?

How should digital preservation be funded?

Should we select or just keep everything? Whose responsibility?Whose responsibility??

What if the publisher will not let subscribers do archiving and copying?

What if the publisher provides temporary access only to encrypted files, and then goes bankrupt?


Concluding warningsConcluding warnings

Current approaches to digital preservation are still limited

They are labor intensive And very costly And require institutional commitment &

organization But: sustainable digital libraries depend

upon the availability of preservation tools, services & efforts


We We need need many many

Rosetta Rosetta stonesstones

tefko saracevic 1 preservation of digital records concerns, approaches, efforts tefko saracevic,...

Documents

efforts tefko saracevic

digital formats

lostthe digital information

slow decay

prevention of decay

acid decay brittle books

traditional concerns

tocintroductory musings