migration and preservation of main-frame tape data

Migration and PreservationMigration and Preservationof Mainof Main--frame Tape Dataframe Tape Data

Muller Media Conversions was founded in 1978 to provide documentediting and conversion services to financial printers, banks and law

firms. The company has always developed most of its own software.(We “eat our own cooking” as they say.)

Some of our most interesting projects over the yearshave been for the U.S. National Archives.

That work has made us particularly aware of the requirements andopportunities in handling “Persistent Data”.

Muller Media Conversions www.mullermedia.com

Please excuse our taking just a moment to talk about our background in datapreservation. One day, we saw this article in our local newspaper. It ultimatelylead to our longstanding relationship with the U.S. National Archives…

The NationThe Nation’’s Records Are a Messs Records Are a MessWASHINGTON-- A slice of America's history has become asunreadable as Egyptian hieroglyphics before the discovery ofthe Rosetta stone. Vast untold volumes of historic, scientificand business data are in danger of dissolving into ameaningless jumble of letters, numbers and computersymbols. Much information from the last 30 years is strandedon computer tape from primitive or discarded systems-unintelligible or soon to be so…

...ASSOCIATED PRESS JANUARY 3, 1991

www.mullermedia.com

On an incompatiblemedium ...

In an indecipherable code or format ...

Trapped in an obsolete filing system ...

On tapes in dangerof decay ...

Information Assets in DangerInformation Assets in Danger


NARA uses tape conversion system topreserve records

The National Archives and Records Administration hasstepped up its preservation of electronic records withthe installation of a $510,000 records maintenancesystem. Responsible for preserving the government’selectronic mail and other data stored in digital formats,NARA’s Center for Electronic Records in 1992contracted Muller Media Conversions Inc. of NewYork, to develop and install…

…GOVERNMENT COMPUTER NEWS, AUGUST, 1994

IAC/IRM honors federal I/T leadersFynette Eaton of the National Archives and RecordsAdministration's Center for Electronic Recordsreceives a GSA Technology Excellence Award fordeveloping the Archival Preservation System to storerecords and capture data about their physical andtechnical attributes.…

…GOVERNMENT COMPUTER NEWS, JUNE, 1996

• Award-winning

• Contract extended continually

• Ongoing enhancements

• MMC owns the software.

MMC and the National ArchivesMMC and the National Archives

www.mullermedia.com

The Archival Preservation System is designed to perform a wide variety of functionswith robust, bullet-proof reliability. In daily operation at NARA and MMC. *DPS is the

designation used for the non-government version of the package.

Copying Functions

• Supports virtually any type of tape drive.

• Copy and compare tape-to-tape, tape-to-disk,disk-to-tape.

• Block-for-block (duplication) or "smart copy”(output can have different label conventions,record formats, block sizes than the input).

• Standard ANSI and IBM tape formats,unlabeled files and several proprietaryformats such as VMS Backup.

• ASCII-to/from-EBCDIC conversion plus acomprehensive scripting language for field &record parsing. (packed, integer, float, etc.)

• ”Tape-Image" capability which enables tapepreservation and duplication or smart-copyingof tapes with only one tape drive.

• Tape-Image files can be preserved on CD,DVD or M/O disks (or even on other higher-capacity tapes). Later, original tape can bereconstructed byte-for-byte.

Data Recovery

• Great flexibility/control in error handling.

• Merge data from two or more tapes to a singleoutput tape or file. (by record or by block)

• Automatic repositioning after interruption for tapecleaning/conditioning.

APS/DPS* CapabilitiesAPS/DPS* Capabilities

Record Keeping/Reporting

Central database for detailed records of:

• Tape processing (e.g.- which system, drives,operators, hard/soft errors, file names, sizes, etc.)

• Media tracking with bar-code reader support.

• Media performance by manufacturer/lot.

• A wide variety of production reports.

Analysis/Review

• Block/record dumps in hex or octal with charinterpretation in EBCDIC, ASCII-7 or ASCII-8.

• Tape map (directory) brief and verbose listings.

…and lots more


Storm Cloud or Sweet Spot?Storm Cloud or Sweet Spot?

Stormy

• The “vapor trail” includes endangeredmedia containing data of lasting value.

• The ability to handle such media/fileformats is becoming increasingly rare.

• Regulators and compliance officers insistthat it be readable, useful.

Sweet

• A good ratio between capacity of old mediaand inexpensive, durable new media.

• With MMC’s software, experience and skills,this legacy of important data can be verifiedand preserved just as they do it at theUS National Archives.

Time & Computer Media (very rough scale)

Techno-Rocket

Your “Vapor Trail”of Older Information

Time


Over the past 28 years MMC has developed theOver the past 28 years MMC has developed theskills and tools to transform information for askills and tools to transform information for a

variety of purposes.variety of purposes.

•• Preservation. (Transfer to inexpensive, long-

lived media for statutory or other reasons--

with extensive verification and record-keeping.)

•• Litigation/Investigation. (Rendering electronic

evidence to a format suitable for legal review.)

•• Conversion. (Translation of file content for use

on another platform or software package.)

Some projects involve aspects of two or more of these. Please see

our website for examples of our work in these areas.


Possibly saved for many years.

(3a) Using DPS™, many types of filescan be processed on PC, no tapeinvolved.

(2) Preserve on hard drives or DVDs.An inexpensive external drive canhold the contents of thousands ofreels or cartridges.

Tapes to be processed

(3b) If needed, any tape can bere-created with bit-for-bitaccuracy. (On same ordifferent medium.)

Basic* OverviewBasic* OverviewTape Conversion/PreservationTape Conversion/Preservation

__________* Other features & variations* Other features & variations

are presented later.are presented later.

(1) DPS system captures datawith 100% accuracy


Photo of a DPSPhoto of a DPS™™ WorkstationWorkstation


DPS creates a PC disk file containing byte-for-byte, block-for-block, filemark-for-filemark representation ofthe tape. This is called a Tape Image. The tape ID is automatically picked up and used to uniquely namethe tape image file. (Example: 019246.tap.) Optionally, the tape image is then automatically compared tothe original tape to ensure 100% accuracy—though most tape drives have reliable error checking makingthis step unnecessary. A report file detailing success/failure and counts of blocks, bytes and filemarks isalso captured and named to match the tape image. (Example: 019246.txt.) The “tap” and “txt” files are thencompressed to a single file.

Compressibility of content can vary widely depending on the nature of the material, tape length, block-sizesand so forth. A modern external hard drive can hold thousands of compressed 349x tape images.

The Tape Capture ProcessThe Tape Capture Process

compare

copy

019246.tap

015953.tap

A01293.tap


… The originalhard drive maybe used, or thetape-imagesmay have beencopied to anynetwork ordirect-attachedstorage youmay choose.

Using an ordinary PC, a SCSI-attached tape drive and MMCsoftware, any captured tape can bere-created with bit-for-bit accuracy.(On same or different medium.)

Restoration/Conversion of Preserved DataRestoration/Conversion of Preserved DataLevel 1Level 1 –– ReRe--Create a TapeCreate a Tape

This approach is used for material that’s wanted back on the mainframe forprocessing, or because of proprietary format can be handled in no other way.See Levels Two and Three for other possibilities.


Restoration/Conversion of Preserved DataRestoration/Conversion of Preserved DataLevel 2Level 2 –– Extract Individual FilesExtract Individual Files

019246\0001_REND.TFB.G0013V00019246\0002_REND.PCF.G0016V00019246\0003_……

015953\0001_30.ITEMFILE.BACKUP

A01293\0001_LAYOUTA01293\0002_PFINVUPL

*File names in standard IBM labels are limited to 17 characters. The name may have been longer on themainframe’s disk and truncated when written to tape. This can lead to ambiguous names. Knowing the tape “volser”value and the sequential position of the file on the tape enables the mainframe (and us) to identify the file uniquely.

… the DPS program extracts files from any standard ANSI or IBM tape (or tape-image). Files are placed

within folders named according to the tape VOLSER. The complete tape filename is retained, and

optionally pre-pended with a sequence number*. A log of this process captures information about record

types and counts for each file. Files that contain only EBCDIC text can be translated to ASCII in this

step, with optional cr/lf appended to each record. See “Conversion/Transfer” slide for further steps.


In many instances, extracted files can be transferred toa mainframe via FTP or drive-sharing.

Using our GenScript/RunScript™record-parsing utilities, many typesof files can be translated on PC, nomainframe resources required.

B. For data that must be restored toB. For data that must be restored tothe mainframe.the mainframe.

FTP

A. For files in a translatable format, whereA. For files in a translatable format, wheretarget system is PC or Open System.target system is PC or Open System.019246\0001_REND.TFB.G0013V00

019246\0002_REND.PCF.G0016V00019246\0003_……



Mainframe

Restoration/Conversion of Preserved DataRestoration/Conversion of Preserved DataLevel 3Level 3 –– Further Processing via PC or MainframeFurther Processing via PC or Mainframe


For files with complex layouts containing packed fields, binary integers, floating point, etc., one canuse GenScript to view, and “hack” these files, creating a script to translate each unique file layout.

Subsequently, the RunScript program is used to translate the files to ASCII. Output choices include“flat”, “quotes-and commas”, “tab-delimited”, and “xml-style”. RunScript is quite fast.

NOTE: Given the possibility that the provided record layout may have been incorrect, and the smallincremental cost, it’s often wise to save the “original file” format along with “converted file” format, thuspreserving the possibility of re-conversion.

019246\0001_REND.TFB.G0013V00019246\0002_REND.PCF.G0016V00019246\0003_……



GenScriptGenScript

RunScriptRunScript

Scriptfiles

019246\0001_REND.TFB.G0013V00.txt019246\0002_REND.PCF.G0016V00.txt019246\0003_……

015953\0001_30.ITEMFILE.BACKUP.txt

A01293\0001_LAYOUT.txtA01293\0002_PFINVUPL.txt

Record Parsing and ConversionRecord Parsing and Conversion


WhatWhat’’s Your Preference?s Your Preference?

Chris Muller, presidentChris Muller, presidentMuller Media ConversionsMuller Media Conversions

Preserving the value of your information legacy isPreserving the value of your information legacy isour goal.our goal.

Every situation is somewhat different, and weEvery situation is somewhat different, and weappreciate the opportunity to discuss the needs andappreciate the opportunity to discuss the needs and

technical parameters of your project.technical parameters of your project.


migration and preservation of main-frame tape data

Documents