Cornell's Innovative Batch Processing Techniques Put Into User-Friendly Practice

Boaz Nadav-ManesDirector, Acquisitions and Automated Technical Services

Philosophy Selector 

Cornell University LibraryLibrary Technical Services

 

Background: Cornell Technical Services and IT• "Technical Services" vs. "Automated Technical Services“ (04/05

onwards).• History of innovative co-development.• Reporting lines (not much "red tape").• Staff reductions through attrition, early retirements and some

layoffs.• Transition to electronic.• 2CUL (combining Technical Services with Columbia’s – 2015)

TS Staff and Automation

• Not everything in TS can be automated.

• Ask: what is big enough and/or what will happen a lot?

• TS Staff are enthusiastic about automatic tools if they are the ones to contribute in developing them and they are the ones to operate them.

• Institutional history and “traditional” experience count! As you move forward emphasize more your core values of service and train (not necessarily hire) for higher level positions.

• You don’t need to be a code expert as long as you know how to logically describe and write a scope for the solution.

• Scalability and Flexibility are keys for good solutions.

• Work collaboratively – invite others to contribute and to lead projects.

• Most of the automatic solutions are not "ground breaking" from the start - they relieve some marginal FTE counts that add up later on.

Batch Your Mind!

Have a users’ group - teach each other = BPEG (Batch Processing Exploratory Group)

Introduction to Web Service API's using PHP and HTML

General Training for all staff on how to use and design MS-Access reports to streamline TS processes:

• Automatic claims (is there a note in PO?)• Open purchase orders• How much you spent with whom? • Who has done what and when?

In the ecosystems business... Work on solutions that have value beyond TS:

Patron Driven Acquisitions in electronic and in print

http://poof.library.cornell.edu/content/about http://www.oclc.org/worldcat-selection.en.html

http://2cul.org/ http://www.slideshare.net/boaz22/ala-2011-pda-preconference

Maximize simple building blocks (metadata) 1. SSH-Secure Shell using Pre-written Scripts:

Commands perform tasks quickly, but offer limited flexibility for record quality and script customization.

2. LS-Tools Web InterfaceGreater flexibility for accommodating new vendor plans and modifications of records.

3. Combinations of above, adding Perl scripts, OCLC Batch-searching, Z39.50 protocol; and/or MARC-Edit Tool

More complex operation, but results in better quality of loaded records.

Friendly Interfaces allow non-experts to participate:

Locally designed “LS-Tools” Web Interface

Consists of 3 main steps:

1. General Import of record files to LS-Tools2. Modification of files to suit our needs (adding appropriate 948 fields, deleting

unneeded fields, etc.)3. Bulk Import of newly modified records, to be loaded into Voyager in SSH.

• Benefits over basic SSH commands: not programmer dependent for new vendor’s files; greater control with a “do-it-yourself” approach.

Functional Approach to Tasks

Assessment Incorporated: Variety of Users

Heaviest User (non specialist!)

Power User

Structure and Permissions

Case Study: ISBN extract from YBP approvals

If/then Logic Runs In The Background (invisible and flexible)

Checks and Balances:

**********************************************************************Cornell University Library Batch ProcessingYankee Approval Metadata run on Mon Oct 7 17:36:04 EDT 2013 Who and Where: apache on lstools01.library.cornell.edu Home directory: /libweb/sites/lstools.library.cornell.edu/htdocsWorking directory: /lstools/batch_data/YankAppr.280.1736********************************************************************** Netid: btn3 Voyager Site: Test Output Log File: /lstools/users/btn3/Jobs/vendor.yankee.1381181763.logE-mail notification: ph44@cornell.edu btn3@cornell.edu********************************************************************** ACTION: Misc FTP BULK: YANKEE CATNAME: Yankee EMAIL: ctsbulk-l FTPGET: yankee INFILE: 13300927.mrc OUTF: /lstools/users/btn3/Marc/13300927.mrc PATH: POFLAG: no SCRIPT: yankee.pl TITLE: Yankee Approvals VENDOR: yankee VNAME: Yankee_Appr**********************************************************************STEP 0.1: 183 initial load recordsSTEP 1.2: 183 input marc records with ISBNs.STEP 1.2: 183 unique ISBNs.STEP 2.1: 183 records writtenFINAL: 1 vendor specific file prepared for load into Voyager**********************************************************************SUMMARY FOR PROJECT MANAGEMENT**********************************************************************Packaged for bulkimport using YANKEE: 183 records

filename: /lstools/users/btn3/Marc/13300927.mrc**********************************************************************Job completed on 2069-12-31 at 19:00:00RC=0 

Usage Statistics

Use pre-existing tools:

Using MarcEdit for a more ‘surgical’ approachhttp://people.oregonstate.edu/~reeset/marcedit/html/about.html

Opens MARC files into temporary “MRK” files that can be edited (then compiled back into MARC)

Mix and match tools and approaches:

Using ISBN Extract with Batch Searching in OCLC

After retrieving a MARC file from vednor’s FTP, ISBN-Extract command creates a “.txt” file of ISBNs in a list, and sends it to a designated folder

ISBN File is batch-loaded into OCLC to be searched & exported to Local files.

Cultivate outside the box thinking:

Using PERL Scripts when MARC files are unavailable

Script is designed to navigate to Website, enter password, and extract lists of LCCNs from a set of framed pages.

Extracting LCCN lists as “.txt” for all un-received LC Jakarta shipments…

SHARE (FOLDERS)

LCCN Text files are sent directly to relevant vendor folder, pre sorted by country (IO, MY, PH, BR, TH).

Allow experts to review exceptions in rotation.

Communal deposit of itterative procedures.

smile

Contact information: btn3@cornell.edu