Digital Preservation:A Brief Introduction

Wen-Chin LanAssistant Professor

Department of Library & Information Science, National Taiwan University

10/19/2005

Prelude

“There is no access without preservation. … Action has to be taken at the outset to ensure viable and sustainable access to digital content. …In essence, preservation decisions need to be integrated with the creation process that will determine the way in which the material is created, described and stored.” (Brindley, 2000)

“Like it or not, the digital domain will have to include a preservation component.” (Hunter, 2000, p.2)

A Landmark Report

Garrett, J., & Waters, D. (Eds.) (1996). Preserving digital information: Report of the Task Force on Archiving of Digital Information. Washington, DC: Commission on Preservation and Access. Available online: http://www.rlg.org/legacy/ftpd/pub/archtf/final-report.pdf

Much recent awareness of the digital preservation issue was facilitated by this report.

Preservation and

Digital Preservation

What Is Preservation?

“Preservation itself is primarily concerned with the survival of information in a usable form for as long as it is required. Preservation … is not just concerned with the conservation or restoration of physical artifacts, but includes all of the strategic and organization considerations that relate to the survival of information over time.” (Day, 2003, p.168-169)

“Preserving anything beyond data means that we also have to preserve other things: the communication of the information, the context of the document, the integrity of the record, and the value of the archives.” (Hunter, 2000, p.4)

What is Preservation? (cont.)

Preservation encompasses a wide variety of interrelated activities, such as policies, standards, procedures, processes, etc., designed to prolong the usable life of human artifacts.

Preservation won’t happen naturally. In essence, preservation is costly and takes a lot of

time and effort. Preservation is a continuous obligation.

What Is Digital Preservation?

“Digital preservation is the ability to keep digital documents and files available for time periods that can transcend technological advances without concern for alteration or loss of readability” (The Association for Information and Image Management)

“Digital preservation refers to the series of managed activities necessary to ensure continued access to and preservation of digital materials” (RLG/OCLC report)

(Both definitions were cited in Chapman, 2001)

What Is Digital Preservation? (cont.)

Digital preservation refers to the series of managed activities necessary to ensure continued access to digital materials for as long as necessary. Digital preservation…refers to all of the actions required to maintain access to digital materials beyond the limits of media failure or technological change.

Long-term preservation – Continued access to digital materials, or at least to the information contained in them, indefinitely.

Medium-term preservation – Continued access to digital materials beyond changes in technology for a defined period of time but not indefinitely.

Short-term preservation – Access to digital materials either for a defined period of time while use is predicted but which does not extend beyond the foreseeable future and/or until it becomes inaccessible because of changes in technology.(A definition adapted from the Preservation Management of Digital Materials: A Handbook, Available: http://www.dpconline.org/graphics/intro/definitions.html)

What Is Digital Preservation? (cont.)

Preserving digital content entails far more than making backup copies and storing them in disparate location. Digital preservation is … to extend the usable life of machine-readable files and protect them from media failure, physical loss, and hardware and software obsolescence, these activities include:

Ensuring the long-term maintenance of a bitstream (the zeros and ones):

Backing up files and keeping a copy at an offsite location Running checks to track the deterioration of storage media, fil

es or bitstreams Providing continued accessibility of the contents:

Viability – making sure that information is intact and readable from the storage media

Renderability – making sure that information is viewable by humans and able to be processed by computers

Understandability – making sure that information is able to be interpreted by humans. (Hixson, 2004)

What Makes Digital Preservation Different?

No benign neglect – Digital storage is an all-or-nothing proposition. Information loss will happen without an action. Compared with traditional preservation, it requires perpetual maintenance .

The focus is not to preserve the artifact itself. Instead, digital preservation is more concerned with the preservation of the information content and its integrity as well as the means to access and use it.

Digital preservation consists of, at least, three different aspects: medium preservation, technology preservation, and intellectual preservation. (Graham, 1997)

Some Confusion in Terms

Digital reformatting (digitizing for preservation) - Using digital technology to preserve the information content of print and non-print materials. The product is a digital reproduction for the original.

Digitizing for access – Converting artifactual information into digital form so that it may be viewed by a larger community and from a distance.

Preserving digital-only information – long-term archiving of information that exists in digital form, such as born digital data, e-journals, etc.

(Lazinger, 2001, p.xx)

Can Digitization Be Preservation?

“We digitized it, so it is preserved.” – NOT REALLY!

Comparatively, digitized materials seem inherently ephemeral (i.e., they have relatively shorter period of life span or “shelf-life”)

Since digitization has become the mainstream of publishing and communication, if we don’t make effort to preserve these digitized objects, they may be lost for good. Indeed, “the origins of the digital era are probably already lost!” (Lyman & Kahle, 1999)

Why Is Digital Preservation

An Issue?

The Advent of Digital Information Introduces New Challenges

“The root of the digital preservation problem is technological, but any proposed solution also needs to take account of organizational and economic issues. … Almost all kinds of digital information need to be interpreted by machines before they can become intelligible to humans.” (Day, 2003, p.179)

Three Different Aspects of Digital Preservation

According to Graham (1997), digital preservation problems are associated with three distinct aspects:

Medium preservation – the preservation of the physical media on which the bits and bytes of electronic information reside.

Technology preservation – refreshing of technologies from old to new as they become available.

Intellectual preservation – addressing the integrity and authenticity of the information as originally recorded.

Medium Longevity Issues

Fragile physical media – relatively short lifetimes in comparison with paper

Silent deterioration – Digital storage is an all-or-nothing proposition.

Subject to rapid media degradation and “bit rot” Prone to damage from careless handling The need to copy data regularly

Medium preservation is of course an issue, but as Peter Graham (1998) pointed out, “the medium becomes of less intrinsic interest and preserving the intellectual content becomes more complex.” (p. 81) For scholarly e-journals, the preservation of information content is the main issue.

Technology Preservation Concerns

Technological obsolescence – technology rapidly changing and evolving

Hardware and software dependence – obsolete technology may cause the dependent digital materials not accessible and useable

Digital information has been made available in a variety of different formats

Intellectual Preservation Concerns

The malleability and volatility of digital information – the authenticity of digital materials is not guaranteed.

Inadvertent destruction of data – digital data can be damaged accidentally, e.g., data loss during transfer, accidents in updating, inadvertent mistakes in manipulation, etc.

Unauthorized tampering with data – digital information is highly vulnerable to destruction through unauthorized tampering or deliberately modifying.

Intellectual Preservation (cont.)

The integrity of digital information - Information integrity means those features of an information object that distinguish it as a whole and singular work. In the digital world, there are five features that determine information integrity

Content Fixity Reference Provenance Context

(Garrett & Waters, 1996, pp.11-19)

Legal & Ethical Challenges

Digital preservation often occurs while materials are still under copyright – can we reproduce, reformat, or migrate these materials?

Many digital materials are obtained through license or subscription. These materials are outside custody of institutions with mandate to preserve – how can one preserve something that he/she doesn’t really own?

Can we preserve Web sites hosted by others?

Economic challenges

Large sums of money are invested in digitizing materials, and these digital assets need to be managed and preserved.

Digital preservation is costly. Economic sustainability – Digital preservation is not

an one-off, short-term operation. It’s a long-term obligation and continuous effort.

Organizational Issues

Most of the challenges associated with digital preservation are organization – not technical, e.g.,

Creators of digital information ≠Owners of server space

Supporting infrastructure Trusted and empowered organizations or repositori

es Certification process

(Marcum, 1997, p.358; Lazinger, 2001, pp.11-14)

Electronic Data in Forms That Are Hard to Preserve (1) “An electronic serial can be a dynamic document i

ncluding embedded multimedia or active links to related publications or data. Electronic serials can also be regularly updated to take account of comments made by scholars in reviews or in other publications.” (Day, 2003, p.178)

Some digital materials are designed to prevent any copying and some digital materials are updated regularly or frequently. (Lazinger, 2001, pp.9-11)

In the case of continually updated documents, should all versions of the same document be preserved, or only the final version?

Electronic Data in Forms That Are Hard to Preserve (2) The transitory nature of e-mail – “In facilitating access to i

nformation and in reducing the time for information to pass from one place to another, e-mail has made information transitory and non-essential; in doing so, it contributes to the loss of our cultural memory” (Gould & Varlamoff, 2000, p.3)

“This new publication models will present a severe challenge for preservation…How long should any given data be expected to reside in automatically accessible form on the Internet? …Can this data be preserved in such a way that they can be retrieved in the future without also using the specialized software tools that have been developed to process them?”(Day, 2003, p.179)

Digital Preservation Strategies“It is now becoming clear that the successful long-term preservation of digital information will depend upon relevant organization identifying and implementing suitable preservation strategies.”

(Day, 2003, p.180)

What Digital Materials Should Be Preserved? (1)“The first act of preservation of any material … is identifying or selecting which material merits the effort of preservation. … With digital information, the time frame within which selection choices must be made is shortened due to the problems of media instability and technological obsolescence. … Digital resources require decisions not only about what items to keep but also about which elements of the resource should be preserved. …This may involve saving not only the resource, but also contextual information such as information on the medium and metadata about the contents.” (Lazinger, 2000, p.17)

What Digital Materials Should Be Preserved? (2)

“Not everything can be preserved, and that a selection process is necessary and unavoidable. The enormous amount of digital information which exists, and the ease with which it can be created or changed makes selection criteria even more essential, but in a way even more difficult. What should those selection criteria be? Can we be sure that what is selected for preservation now will be what is required in the future?” (Gould & Varlamoff, 2000, p.4)

What Content Should Be Preserved? An Example

For E-journal , what content should be preserved? While articles are the intellectual core of E-journals, they also contain many other kinds of information, such as:

Editorial board Rights and usage terms Copyright statement Journal description Advertisements Reprint information Editorials Events lists Errata Conference announcements Various sorts of digital files related to individual articles (data sets, images, tables,

videos, models, etc.)

Which of these need to be archived and preserved for the future? This should be carefully considered by the institutions with mandate to preserve. (Flecker, 2002, p.19)

Who Should Be Responsible for Digital Preservation? (1) Creators Publishers Distributors System administrators Libraries Archives Users

(Lazinger, 2000, pp.49-53)

Who Should Be Responsible for Digital Preservation? (2)

“There is some agreement that it is unfair of libraries to expect publishers to begin to take on the role of archiving when they have never done so before, but similarly publishers cannot expect libraries to preserve material which they do not own and do not have long term access to. There is good reason to expect licensing agreements between publishers and libraries to change in due course to take account of this dilemma.” (Gould & Varlamoff, 2000, p.7)

Where is the win-win solution?

How Can Digital Information Be Preserved?Roughly speaking, there are four main technical options but none of these can provide a perfect solution:

Medium refreshing – copying digital files from one storage medium to another medium

Data migration – transfer of digital materials from one hardware/software configuration to another

Technology preservation – preserve an information object together with all of the hardware and software needed to interpret it

Software emulation – software designed to perform the functions of obsolete hardware and other software

(Day, 2003, p. 180; Lazinger, 2000, pp.76-77)

Media Renewal / Refreshing

A common response to the medium deterioration problem is known as media renewal or media refreshing. There is a need to clarify the following terms:

Copying – transferring data from old storage media to new storage media with the same format specification.

Reformatting – there is a change to the underlying bit stream, but there is no change in the intellectual content of the records.

Converting – data export or import from one software environment to another.

Migrating – usually involves moving data from proprietary legacy systems that lack software functionality to open systems. (Hunter, 2000, pp.57-58)

Media refreshing may solve the problem of medium deterioration, but it won’t be able to solve the problem of technology obsolescence.

Data Migration

Data migration involves the transfer of the entire technology environment, usually from one generation of computer technology to a subsequent one, not just the physical storage medium

Data migration is a more tried and tested preservation strategy.

Data migration is labor-intensive and it’s difficult to predict what it will involve.

Developing Migration Strategies

Less stable to more stable media Highly software-dependent formats to less softwar

e-intensive formats Multiplicity of formats to smaller number of com

mon formats Development of backward compatibility paths Standard development and imposition Development of process centers for migration and

reformatting(Lazinger, 2000, pp.79-83)

Technology Preservation

This approach intends to retain the needed hardware and software.

This approach “may have an important role for the recovery of data from obsolete storage media and platforms, but it is unlikely to become a viable long-term strategy.” (Day, 2003, p.180)

This approach my end up with “a museum of ageing and incompatible computer hardware.” (Feeney, 1999, p.42)

Software Emulation - “Mimicking” obsolescent hardware in software This approach is based on “the premise that the

best way to preserve the functionality and ‘look and feel’ of digital resources would be to preserve the original software and then to run this on emulators that would mimic the behavior of the obsolete hardware and operating systems.”

This approach requires “the encapsulation of a digital data object together with the application software used to create or interpret it and a description of the required hardware environment that could be used as a specification for an emulator.” (Day, 2003, p. 181)

Software Emulation (cont.)

The emulator involves encapsulating three kinds of information with each digital document: The document and its software environment, includin

g document itself, application software, and operating system

The specification of an emulator for the document’s original hardware platform

Explanatory material, labeling information, annotations, metadata about the document and its history, and documentation for the software and emulated hardware included in the encapsulation.

(Lazinger, 2000, pp.84-85)

Ensuring Long-Term Access to E-Journals

“Commercial publishers tend to license the use of information to customers, thus ensuring that contract law rather than copyright law governs the use of the information. … Because more licenses are time limited. …libraries’ concerns about ongoing access can be addressed in the agreements negotiated between libraries and publishers.” (Day, 2003, p.185)

Examples: NESLi2 (National E-Journals Initiative)

Coordinates e-journal licensing for UK higher education community Model License: requires publishers to incorporate a clause for

ensuring long-term access http://www.nesli2.ac.uk/

Ensuring Long-Term Access to E-Journals (cont.) It is expected that library consortia can act on behalf of

their members to negotiate with E-journals providers to retain perpetual access to licensed content as well as uninterrupted access

Many libraries seem to move towards “Purchase model” for e-content

The International Coalition of Library Consortia (ICOLC) suggests publishers to provide permanent access to licensed content and agrees to and supports consortia in developing local archiving solutions

More and more publishers agree to provide content to fulfill library archiving needs

Economic Infrastructure

Digital preservation relies heavily upon continual investment and long-term administrative supporting.

Economic infrastructure for digital preservation includes: Identify preservation responsibilities and appropriately alloc

ate them Provide sufficient incentives to encourage people to carry ou

t these responsibilities Leverage infrastructure in support of digital preservation, eli

minate redundancies and use scarce resources effectively More information:

http://www.oclc.org/research/projects/digipres/incentives-dp.pdf http://www.clir.org/pubs/reports/pub126/pub126.pdf

Open Archival Information System (OAIS) Reference Model OAIS Reference Model is a conceptual framework for an archival system

dedicated to preserving and maintaining access to digital information over long term.

The reference model has been well-received by a diverse community of institutions. Many current digital archiving projects are based on OAIS, such as CEDARS, PANDORA, and NEDLIB.

The reference model defines roles for three players in archiving: creators (producers), archive operators (management), and consumers (designated users). These three players and the archive itself consist of the OAIS environment.

This model also specify functional components of complete digital archiving system.

More information:http://ssdoo.gsfc.nasa.gov/nost/wwwclassic/documents/pdf/CCSDS-650.0-B-1.pdf

http://www.dpconline.org/docs/lavoie_OAIS.pdf

OAIS Reference Model (cont.)

Source: Lavoie (2004)

OAIS Reference Model (cont.)Source: http://www.oclc.org/research/projects/pmwg/pm_framework.pdf

OAIS Functional Model (cont.)

Source: Lavoie (2004)

Preservation Metadata

Preservation metadata is information that supports and documents the long-term preservation of digital materials. It addresses the following attributes of an archived digital object:

provenance - documenting the custodial history of the object authenticity - validating that the digital object is in fact what it

purports to be, and has not been altered in an undocumented way preservation activity - documenting the actions taken to preserve

the digital object, and any consequences of these actions that impact its look, feel, or functionality

technical environment - describing the technical requirements, such as hardware and software, needed to render and use the digital objects

rights management - recording any binding intellectual property rights that may limit the repository’s ability to preserve and disseminate the digital object over time

(Lavoie & Gartner, 2005, p.2)

Preservation Metadata (cont.)

Preservation metadata helps make an archived digital object self-documenting over time, even as the intellectual, economic, legal, and technical environments surrounding the object are in a constant state of change. (Lavoie & Gartner, 2005, p.2)

Examples: PREMIS (Preservation Metadata: Implementation Strategies)

http://www.oclc.org/research/projects/pmwg/ CDEARS Guide to Preservation Metadata

http://www.leeds.ac.uk/cedars/guideto/metadata/guidetometadata.pdf National Library of Australia. Preservation Metadata for Digital

Collections http://www.nla.gov.au/preserve/pmeta.html

Trusted Digital Repositories

A trusted digital repository is one whose mission is to provide reliable, long-term access to managed digital resources to its designated community, now and in the future. (Trusted Digital Repositories: Attributes and Responsibilities, p.i)More information:

Trusted Digital Repositories: Attributes and Responsibilities. An RLG-OCLC report (May 2002) http://www.rlg.org/en/pdfs/repositories.pdf

Minimum Criteria for an Archival Repository of Digital Scholarly Journals (May 15, 2000) http://www.diglib.org/preserve/criteria.htm

Examples of Current Initiatives and Projects

Organizations / Cooperatives

Digital Library Federation (DLF) http://www.diglib.org/dlfhomepage.htm

Council on Library and Information Resources (CLIR) http://www.clir.org/pubs/reports/strategies.html

Electronic Media Group (EMG) http://aic.stanford.edu/sg/emg/ The Joint Information Systems Committee (JISC)

http://www.jisc.ac.uk/ UKOLNhttp://www.ukoln.ac.uk/ Electronic Resource Preservation and Access Network (ERPANET)

http://www.erpanet.org/ Digital Preservation Coalition

http://www.dpconline.org/graphics/index.html

Initiatives, Programs & Projects

The International Research on Permanent Authentic Records in Electronic Systems (InterPARES) http://www.interpares.org/

Library of Congress. National Digital Information Infrastructure and Preservation Program (NDIIPP) http://www.digitalpreservation.gov/

U. S. National Archives. Electronic Records Archive (ERA) http://www.archives.gov/era/

Canadian Initiative on Digital Libraries:http://www.collectionscanada.ca/cidl/pres-preserv-e.html

PADI (Preserving Access to Digital Information):http://www.nla.gov.au/padi/

PANDAS: PANDORA Digital Archiving System http://pandora.nla.gov.au/pandas.html

The Joint Information Systems Committee (JISC). Digital Preservation and Record Management. http://www.jisc.ac.uk/index.cfm?name=programme_preservation

Initiatives, Programs & Projects (cont.)

Archiving Electronic Journals. Research Funded by the Andrew W. Mellon Foundation. http://www.diglib.org/preserve/ejp.htm

CAMiLEON Project (Creative Archiving at Michigan & Leeds: Emulating the Old on the New) http://www.si.umich.edu/CAMILEON/

LOCKSS http://lockss.stanford.edu/ Florida Center for Library Automation (FCLA). DAITSS

(Dare Archive In The Sunshine State) http://www.fcla.edu/digitalArchive/pdfs/DAITSS.pdf

Internet Archive http://www.archive.org DiVA: Digital Scientific Archive http://www.div

a-portal.org/index.xsql?lang=en The Nordic Web Archive (NWA)

http://nwa.nb.no

Tools and Services

JHOVE: JSTOR/Harvard Object Validation Environment http://hul.harvard.edu/jhove/

U.K. National Archive. PRONOM: The File Format Registry http://www.nationalarchives.gov.uk/pronom/

University of Pennsylvania. Typed Object Model (TOM) http://tom.library.upenn.edu/

San Diego Supercomputer Center (SDSC). Storage Resource Broker (SRB) http://www.sdsc.edu/srb/

Global Digital Format Registry (GDFR) http://hul.harvard.edu/gdfr/

References

Brindley, L. (2000). Preservation 2000: Keynote speech, in Preservation 2000: An international conference on the preservation and long term accessibility of digital materials. Available: http://www.rlg.org/en/page.php?Page_ID=242

Chapman, S. (2001). What is digital preservation? Paper presented at the Digital Preservation Resources Symposium 2001. Retrieved September 5, 2005, from http://www.oclc.org/education/conferences/presentations/2001/ preservation/chapman.htm

Conway, P. (1996). Preservation in the digital world. Washington, DC: Commission on Preservation and Access.

Cornell University Libraries. Digital preservation management tutorial. Retrieved September 3, 2005, from http://www.library.cornell.edu/iris/tutorial/dpm/index.html

Council on Library and Information Resources, Digital Library Federation, and Coalition for Networked Information (2000). Minimum criteria for an archival repository of digital scholarly journals (version1.2). Retreived September 7, 2005, from http://www.diglib.org/preserve/criteria.htm

Croft, J. A. (2003). The preservation evolution: A review of preservation literature, 1999-2001. Library Resources & Technical Services, 47(2): 59-70.

Day, M. (2003). Digital preservation and long-term access to the content of electronic serials. In W. Jones (Ed.), E-serials: Publishers, libraries, users, and standards (2nd ed.) (pp. 167-195). Binghamton, NY: Haworth.

References (cont.)

Feeney, M. (Ed.) (1999). Digital culture: Maximizing the nation’s investment. London: National Preservation Office.

Flecker, D. (2001). Preserving scholarly e-journals. D-Lib Magazine, 7(9). Retrieved August 13, 2005, from http://www.dlib.org/dlib/september01/flecker/09flecker.html

Flecker, D. (2002). Preserving digital periodicals. In Building a national strategy for digital preservation: Issues in digital media archiving (pp. 10-23). Washington, DC: Council on Library and Information Resources.

Garrett, J., & Waters, D. (Eds.) (1996). Preserving digital information: Report of the Task Force on Archiving of Digital Information. Washington, DC: Commission on Preservation and Access. Retrieved July 7, 2005, from http://www.rlg.org/legacy/ftpd/pub/archtf/final-report.pdf

Gould, S., & Varlamoff, M-T. (2000). The preservation of digitized collections: Recent progress and persistent challenges world-wide. In R.W. Manning & V. Kremp (comp.), A reader in preservation and conservation (pp.1-12). Munchen: Saur.

Graham, P. S. (1997). Building the digital research library: Preservation and access at the heart of scholarship. Retrieved August 15, 2005, from http://ukoln.bath.ac.uk/services/papers/follett/graham/paper.html

Graham, P. S. (1998). Long-term intellectual preservation. Collection Management, 22(3/4): 81-82.

Hixson, C. (2004). Digital preservation at the UO Libraries: A snapshot. Retrieved September 15, 2005, from http://libweb.uoregon.edu/diglib/Dig_Pres_Snapshot2.pdf

Hunter, G. S. (2000). Preserving digital information: A how-to-do-it manual. New York: Neal-Schuman Publishers.

References (cont.)

Lavoie, B. F. (Ed.). (2001). Preservation metadata for digital objects: A review of the state of the art. White paper by the OCLC/RLG working group on preservation metadata. OCLC Research publication. Retrieved August 30, 2005, from http://www.oclc.org/research/projects/pmwg/presmeta_wp.pdf

Lavoie, B. F. (2004). The open archival information system reference model: Introductory guide. A joint report of the Digital Preservation Coalition (DPC) and OCLC, published electronically as a DPC Technology Report. Retrieved September 2, 2005, from http://www.dpconline.org/docs/lavoie_OAIS.pdf

Lavoie, B. F., & Dempsey, L. (2004). Thirteen ways of looking at ... digital preservation. D-Lib Magazine, 10(7/8). Retrieved August 13, 2005, from http://www.dlib.org/dlib/july04/lavoie/07lavoie.html

Lavoie, B. F., & Gartner, R. (2005). Preservation Metadata. A joint report of OCLC, Oxford Library Services, and the Digital Preservation Coalition (DPC), published electronically as a DPC Technology Watch Report (No. 05-01). Retrieved September 13, 2005m, from http://www.dpconline.org/docs/reports/dpctw05-01.pdf.

Lazinger, S. S. (2001). Digital preservation and metadata: History, theory, practice. Englewood, Colo.: Libraries Unlimited.

Lyman, P., & Kahle, B. (1000). Archiving digital cultureal artifacts. D-Lib Magazine, 4(7/8). Retrieved August 13, 2005, from http://www.dlib.org/dlib/july98/07lyman.html

Marcum, D. (1997). A moral and legal obligation: Preservation in the digital age. International Information and Library Review, 29(3/4): 357-365.

Thank you!