american archives horror story
TRANSCRIPT
ARCHIVE
LTO FAILURE AND DATA LOSS
Who we are: WGBH MLA
Who We Are: AAPB
...and more than 120 public radio and television stations and archives nationwide
Digitization recently completed
WGBH’s 7,010 tapes that were sent for digitization
Returned on 17 LTO-6 tapes
• 5,000 hours of digitized and born digital media
• Up to 59,000 files
• Not to exceed 5.24 terabytes after transcoding has occurred
The Born Digital Deliverable
• Lack of staff resources at stations• Absence of existing metadata• Unique identifiers ≠ actual names of
files• Limitations of our metadata
management system • Bicycling hard drives• Access quality vs. preservation
quality• 5.24 terabytes became 300+
terabytes
We had some challenges
• Send multiple batches totaling 13,500 video and audio files
• Pull 300TB of files over our network and place on 76 3TB hard drives
– Stored on LTO-4 robotic machine in IT
– Checksums for most files did not exist
– Many files up to 100GB each
The Plan at WGBH
THE PROBLEM
Out of a set of 2069 files pulled for Batch 3 part 1, 1195 proved to have failed on reaching Crawford
693 failed initial analysis
394 failed QC
108 failed transcode
= 57% failure rate
The next batch had 1310 failures out of 2826 files
THE PROCESS
start with csv file containing final name of file at receiving end, full path to file on source end, ID value of offline storage tape
shell script:
- sorts files by # of storage tape
- logs into DAM using ssh
- transfers file using scp through Artesia from LTO 4 tape (stored as tarball) onto 3 TB hard drive
later versions used tar rather than scp
THE PROCESS (REVISED)post-transfer, compare the megabyte block counts of source and destination products
(no checksum – took too much time to perform on such large files while under time pressure)
failed items automatically removed from drive
transfer script re-run until all files download successfully
if files fail repeatedly, assume they have failed on LTO; backup tape called from Iron Mountain and attempted to be staged from there
THE PROGRESS
Many files that initially failed eventually transferred successfully, either from the initial tape or from a backup, after multiple attempts
Others were never successfully transferred
Out of a planned 10,648 files in the batch, 2173 were never successfully downloaded – a 20% failure rate
BREAKING DOWN THE FAILURES
ffmpeg –i ${filename}mediainfo –f ${filename}
“moov atom not found”
QC FAILURE
Playable files with evidence of corruption defined by Crawford as “issues that would make the file unusable,” for example:
a green screen with no audio
a video that plays for two seconds before the screen going black or grey
pixels shift out of place in zigzag pattern
audio is digital noise only
THE PROGNOSIS
Sample data: 5000 files with checksums generated at creation
1012 of those files could not be transferred from LTO, after multiple attempts
However, MD5s on LTO show the files are unchanged
So the files are good – but can’t be reached?
THE POSSIBILITIES
Files were bad before they went onto LTO –production environment provides little opportunity for QC
Files are good, but inaccessible on LTO because of problems with the way the data is stored on the tape or the interaction of the different technologies used to get it out c
THE PROBLEMS NOW
Administrative distance between institutional IT and archival needs makes it difficult to get clear answers about the technology we’re using
Staff turnover means information about original systems/data transfer processes are lost
Local LTO systems incompatible with older tapes, making direct testing currently impossible
NEXT STEPS
Acquire Linux machine for direct testing of LTO 4 tapes
Test different transfer protocols
More investigation into the SL8500 SAMFS/QFS
Look for patterns in inaccessible files (file size, date uploaded, system architecture on storage tape)
