Download - The New DRS: Plan for Metadata Migration
The New DRS: Plan for Metadata Migration
Harvard Library & Library Technology ServicesFebruary 26, 2014
AgendaWelcome and introduction …... Franziska Frey, Tracey Robinson
Migration challenges .…………... Randy Stern
Creating the plan ………………..... Kate Bowers
Walkthrough of plan …………….. Andrea Goethals
Imaging Services:
minimizing disruption……….… Bill Comstock
Wrap-up & next steps ………...… Kate Bowers, Andrea Goethals
Q & A ……………………………….…... All
WELCOME & INTRODUCTIONFranziska Frey, Tracey Robinson
The DRS Advisory Group…
…provides oversight and guidance during the rollout phase of the DRS2 project and ensures that the user community of active DRS depositors and content owners contributes to decisions about the rollout.
Amy BensonKate BowersBill ComstockFranziska Frey (chair)
Andrea GoethalsWendy GogelTracey Robinson Randy Stern
Why a New DRS?
• Upgrade to best-in-breed technologies• Adopt digital preservation best practices
and standards• Preserve metadata better• Improve collection management• Support preservation planning & activities• Improve access to content & metadata• Support more formats & genres
Preservation Capability Before and After the DRS2 Project
Level One Level Two Level Three Level Four
Storage & Geographic Location
File Fixity and Data Integrity
Information Security
Metadata
File Formats
= already compliant = will be compliant after the DRS2 project
Based on the NDSA Levels of Digital Preservation
Evolution of the DRS
2000 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 20122001
Current DRS in
production
New DRS in productionDRS enhancements
New DRS infrastructuredevelopment
2013 2014 2015
New DRS metadata migration
& user adoption
New DRS - Completed
2009 2010 2011 2012
convened DRS
Advisory Group
software in production
2013 2014 2015
users trained,phase 1
hardware in production
migrated content to new hardware
InfrastructureDevelopment
Metadata Migration
& User Adoption
Fedora assessment
DuraCloud pilot test
early release beta 1 beta 2
beta 3
first object deposited to the new
DRS
New DRS - Upcoming
2009 2010 2011 2012 2013 2014 2015
InfrastructureDevelopment
Metadata Migration
& User Adoption
metadata migration tools
created
metadata migrated
users moved
MIGRATION CHALLENGESRandy Stern
Why “Metadata” Migration?
Why not“content” migration?
Pre-migration
DRS Content
Current DRS
Database
Post-migration
DRS Content
Current DRS
Database New DRS Database
New DRS Index
New DRS Object Descriptors
New DRS Data Model
• Not a simple metadata conversion• A new DRS object is a logical intellectual
entity that unifies multiple DRS files– Still image objects - archival and production
masters, and deliverables including thumbnails – Audio objects - archival and production
masters and deliverables– PDS objects - page image and text files
Object Descriptors
• METS files generated for each object– Standards-based internal schemas (PREMIS,
MODS, MIX, etc.)
• Metadata gathered from multiple sources– Current DRS database– Every content file– HOLLIS records
Technical Challenges
• Many formats– Images, audio, text, digitized books, web
sites, documents, biomedical image stacks, opaque files
• Unique migration rules per format – technical metadata, roles, relationships
• Large (>5000 file) PDS documents• 45+ million DRS files
Technical Challenges
• At 1 sec/file, 45 million files would take 520 days!
• We are designing the migration software tools for parallel processing
• We are configuring multiple servers to run the migration
CREATING THE PLANKate Bowers
Formulating a Migration Strategy
• Analysis of:– DRS content
• Technical (relationships, etc. for building objects)• Volume and type by repository
– metadata for mapping– user activity in the DRS– survey of highest volume, active users– training and testing registration lists
Migration Strategy Factors
• Combines needs of users with technical requirements
• User sequencing will be based on:– Current deposit & administrative activity – Level of preparation (training and participation
in beta testing)
WALKTHROUGH OF PLANAndrea Goethals
Migrating Content in 5 Stages
Migrate 1st : Tier 1 contentMigrate 2nd: Tier 2 contentMigrate 3rd: Tier 3 contentMigrate 4th: Tier 4 contentMigrate 5th: Tier 5 content
Migrating Content in 5 Stages
Migrate 1st : Tier 1 contentMigrate 2nd: Tier 2 contentMigrate 3rd: Tier 3 contentMigrate 4th: Tier 4 contentMigrate 5th: Tier 5 content
simpler objects
more complex objects
Migrating Content in 5 Stages
Migrate 1st : Tier 1 contentMigrate 2nd: Tier 2 contentMigrate 3rd: Tier 3 contentMigrate 4th: Tier 4 contentMigrate 5th: Tier 5 content
dependenciesbetween tiers
dependencieswithin tiers
Migrating Content in 5 StagesTier Content
1 Text (Methodology, ESRI World File), Document, Color Profile, Target Image
2 PDS Document, Still Image
3 Audio, Text (SMIL)
4 Web Harvest, Opaque Container
5 Biomedical Image; Google Document Container 1, 2, 3
Migrating Content in 5 StagesTier Content
1 Text (Methodology, ESRI World File), Document, Color Profile, Target Image
2 PDS Document, Still Image
3 Audio, Text (SMIL)
4 Web Harvest, Opaque Container
5 Biomedical Image; Google Document Container 1, 2, 3
Migrating Content in 5 StagesTier Content
1 Text (Methodology, ESRI World File), Document, Color Profile, Target Image
2 PDS Document, Still Image
3 Audio, Text (SMIL)
4 Web Harvest, Opaque Container
5 Biomedical Image; Google Document Container 1, 2, 3
Tiers 1, 3, 4, 5: Migrate across all DRS owner codes at one timeTier 2: Migrate one DRS owner code at a time
Tier 2: Sequence by DRS Owner Code
• Migrate just your unit’s PDS document and still image content
• Minimize the amount of time the content you manage the most is in 2 different systems
DRS Owner Codes to be Migrated
DIV.LIBR
FHCL.HOUGH
FHCL.MUSI
GSD.LIBR
RAD.ARCH
RAD.SCHL
FHCL.JUD
FHCL.FAL
FHCL.MAPS
FMUS.MCZ
HLS.LIBR
HUL.ARCH
HUAM.MUSE
HBS.BAKR
VIT.BERE
HUL.PRES
FMUS.GRAY
HPPM.PIRC DOAK.RESLIB
DOAK.MUS
FCOR.REISCH
FMUS.ARN
HLNC.LIBRARB.AAHOD
DOAK.LIBR
FCOR.FORST
FCOR.WOLBACH
FMUS.FARL
FMUS.HUH
FMUS.ORC
FMUS.PEAB
HMS.COUNT
HPRE.WARD
HUAM.SARDISHUL.GGL
HUL.OIS
FCOR.CARP
FCOR.HCO FCOR.URI
FHCL.CAB
FHCL.COLL
FHCL.DAVIS
FHCL.ENV
FHCL.FUNG
FHCL.GOV
FHCL.LITTFHCL.MED
FHCL.SLV
FHCL.TOZ FHCL.YENCH
FMUS.SEM
FMUS.WARE
GSE.GUTMN
KSG.LIBR
Timing
• Current estimates: – Building & testing migration tools: Now– Begin Tier 1 content: Spring 2014– Begin Tier 2 content: Summer 2014
• Units will be contacted about their Tier 2 migration schedule
After Your Tier 2 Migration
• You, and anyone depositing on your behalf, will begin depositing only to the new DRS
• All of your management tasks will be done only in the new DRS
IMAGING SERVICES: MINIMIZING DISRUPTION
Bill Comstock
Minimizing Disruption
• Testing by Imaging Services• Uninterrupted services• Migration sequencing• Participating as a “pioneer”
Testing by Imaging Services
Alpha and beta testing:• Depositing processes• DRS content maintenance tools
– Searching and assembling content for download
– Editing PDS objects
Uninterrupted Services
Providing services before and after your migration • Content needs to be deposited• Content needs to be searched • Content needs to be assembled
• may need to be edited• may need to be downloaded
Migration Sequencing
• We will synchronize deposits with your migration– start depositing for you in the new DRS
after your Tier 2 content is migrated
Imaging Services as Pioneers
As pioneers, we:• Learn to use the new tools• Refine the new depositing workflows• Identify bugs• Suggest improvements• Create a group of local experts that can support those
that follow
We’ll wear the scars so that you can stay pretty!
WRAP-UP AND NEXT STEPSKate Bowers and Andrea Goethals
Nine Pioneers
• Limited number of first depositors to new DRS• Factors
– New DRS-ready content from new systems • EAS (Electronic Archiving Service), ACORN
(Weissman Preservation Center conservation treatments), DASH (for ETD)
– Prepared and trained staff– No content to migrate
• HUA example: opaque objects
First Deposit in the New DRS
大藏經 Da Zang Jing - Buddhist sutra, Qing dynasty (1644 -1911), China, Tibetan language
Email List
http://hul.harvard.edu/ois/systems/drs/drs2.html
Coming Attractions
• Open meetings– Technical aspects brown bag (March)– Digital preservation & DRS intro (Summer)
• Training and instruction– Refresher training– New training– Onsite assistance
Q & AThanks!