you can do it! start using perl to handle your voyager needs
DESCRIPTION
Getting started with Perl with a view towards using it with the Voyager LMSPresented at ELUNA 2008TRANSCRIPT
You Can Do It!Start Using Perl to
Handle Your Voyager Needs.
Some Perl nomenclature
PERL – Practical Extraction and Report PERL – Practical Extraction and Report LanguageLanguage
(camel by O’Reilly)
Some Perl nomenclature
PERL – Practical Extraction and Report PERL – Practical Extraction and Report LanguageLanguage
PERL – Pathologically Eclectic Rubbish Lister PERL – Pathologically Eclectic Rubbish Lister (not really)(not really)
(camel by O’Reilly)
Some Perl nomenclature
PERL – Practical Extraction and Report PERL – Practical Extraction and Report LanguageLanguage
PERL – Pathologically Eclectic Rubbish Lister PERL – Pathologically Eclectic Rubbish Lister (not really)(not really)
TMTOWTDI – There’s More Than One Way To TMTOWTDI – There’s More Than One Way To Do ItDo It
(camel by O’Reilly)
Some Perl attributes
it’s a scripted language, not compiled -it’s a scripted language, not compiled -
faster, easier developmentfaster, easier development
runs plenty fast for most thingsruns plenty fast for most things
Some Perl attributes
it’s a scripted language, not compiled -it’s a scripted language, not compiled -
faster, easier developmentfaster, easier development
runs plenty fast for most thingsruns plenty fast for most things
Loose variable typing -Loose variable typing -
both good and bad,both good and bad,
but mostly goodbut mostly good
Your first program
#!/usr/local/bin/perl
print "Hello, World\n";
“Protecting” your program (Unix)
By default, your program is not executable.
chmod 744 your_program
You can execute it as owner of the file, anyone else can only read it.
Variables
$name
can be text or number: a character,a whole page of text,or any kind of number
context determines type
can go “both” ways
Variables, array of
@employee
Array of $employee variables
$employee[0]
$employee[1]
etc.
Variables, hash of
$lib{‘thisone’} = “2 days”;
$lib{‘thatone’} = “5 days”;
Thus can use
$grace_period = $lib{$libname}
when $libname is thatone,
$grace_period is 5 days
Variables, list of
($var1, $var2, $var3) =function_that_does_something;
This function returns a list of elements.
A list is always inside parentheses ().
Variables, assigning a value to
$var = value or expression
$array[n] = something;
@array = (); # empty array
%hash = (); # empty hash
Can be done almost anywhere, anytime.
use strict;
Requires that you declare all variables like this:
my $var;
my $var = something;
my @array = ();
Also makes Perl check your code.
Best Practices!
Variable scope, and good practices
use strict;
my $var;
my $var = something;
my @array = ();
A variable declared like this is visible throughout your program.
Best Practices!
Variable scope, and good practices
Variable scope, and good practices
use strict;
my $var;
my $var = something;
my @array = ();
A “my” declaration within code grouped within { and } is visible only in that section of code; it does not exist elsewhere.
Best Practices!
Scope: where in a program a variable exists.
$_ default parameter for many functions
$. current record/line number in current file
$/ input record separator (usually the newline character)
$, print() function output separator (normally anempty string)
$0 name of the Perl script being executed
$^T time, in seconds, when the script begins running
$^X full pathname of the Perl interpreter running the current script
some Special Variables
@ARGV array which contains the list of the commandline arguments
@Inc array which contains the list of directorieswhere Perl can look for scripts to execute(for use DBI and other modules)
%ENV hash variable which contains entries for yourcurrent environment variables
some Special Variables
STDIN read from the standard input file handle(normally the keyboard)
STDOUT send output to the standard output file handle(normally the display)
STDERR send error output to the standard error filehandle (normally the display)
DATA file handle referring to any data following __END__
and dozens more…
some Special Variables
String manipulation & other stuffGiven
$stuff = “this is me”;
These are not equivalent:
“print $stuff”
‘print $stuff’
`print $stuff`
Given
$stuff = “this is me”;
These are not equivalent:
“print $stuff” is “print this is me”
‘print $stuff’
`print $stuff`
String manipulation & other stuff
Given
$stuff = “this is me”;
These are not equivalent:
“print $stuff” is “print this is me”
‘print $stuff’ is ‘print $stuff’
`print $stuff`
String manipulation & other stuff
Given $stuff = “this is me”;
`print $stuff` would have the operating system try to execute the command <print this is me>
String manipulation & other stuff
This form should be used as$something = `O.S. command`
Example: $listing = ‘ls *.pl`;
The output of this ls command is placed, as possibly a large string, into the variable $listing. This syntax allows powerful processing capabilities within a program.
String manipulation & other stuff
printf, sprintf
printf(“%s lines here”, $counter)
if $counter is 42, we get
42 lines here
for the output
printf(“%c lines here”, $counter)
if $counter is 42, we get
* lines here
for the output, since 42 is the ASCII value for “*”, and we’re printing a character
printf, sprintf
Some additional string formatting…
%s – output length is length($var)
%10s – output length is absolutely 10 (right justified)
%10.20s – output length is min 10, max 20
%-10.10s – output length is absolutely 10 (left justified)
Any padding is with space characters.
printf, sprintf
Some additional number formatting…
%d – output length is length($var)
%10d – output length is absolutely 10 (leading space padded)
%-10d – left justified, absolutely 10 (trailing space padded)
%-10.10d – right justified, absolutely 10 (leading zero padded)
printf, sprintf
Still more number formatting…
%f – output length is length($var)
%10.10f – guarantees 10 positions to the right of the decimal (zero padded)
printf, sprintf
printf whatever outputs to the screen
printf, sprintf
printf whatever outputs to the screen
printf file whatever outputs to that file
Ex: printf file (“this is %s fun\n”, $much);
(print functions just like the above, as to output destination.)
printf, sprintf
printf whatever outputs to the screen
printf file whatever outputs to that file
Ex: printf file (“this is %s fun\n”, $much);
(print functions just like the above, as to output destination.)
sprintf is just like any printf, except that its output always goes to a string variable.
Ex: $var = sprintf(“this is %s fun\n”, $much);
printf, sprintf
substr get a portion of a string
index get the location of a string in a string
length get the length of a string
ord, char convert a character to its ASCII valueand vice versa
Some other common functions
$var = ƒ(x);
uc, lc convert a string entirely to upper orlower case
ucfirst, convert the first character of a string tolcfirst upper or lower case
Some other common functions
$var = ƒ(x);
split convert a string into pieces based on asupplied character
join convert a list of strings into one string, joinedby a supplied character
Some other common functions
$var = ƒ(x);
@person contains a large number of people
foreach $individual (@person){ print “this is person $individual\n”;}
no subscript required!cleaner code
Loop stuffforeach, with an array
@person contains a large number of people
$idnum = 0;
while ($idnum < @person){ print “this is person $person[$idnum]\n”; $idnum++;}
not as clean as using foreach,but sometimes this makes more sense
Loop stuffwhile, with an array
@person contains a large number of people
for ($idnum=scalar(@person); $idnum--; $idnum>=0){ print “this is person $person[$idnum]\n”;}
conventional for loop
Loop stufffor, with an array
(backwards traversal)
@person contains a large number of people
for ($idnum=scalar(@person); $idnum--; $idnum>=0){ next if ($person[$idnum] eq “Harry”) print “this is person $person[$idnum]\n”;}
skip anybody named Harry
Loop stuff,more control
@person contains a large number of people
for ($idnum=scalar(@person); $idnum--; $idnum>=0){ last if ($person[$idnum] eq “Penelope”) print “this is person $person[$idnum]\n”;}next_program_line;
once we get to Penelope, leave the loop, andresume execution at next_program_line
Loop stuff,more control
@person = ();…while (“reading a file”) # this line is not real code!{ $name = substr($file_line, 0, 30); push @person, $name;}
populate an array simply,no hassle with an index variable
One last bit of array stuff…
File input and output (I/O)
“slurping” a file
File test operators
Here are a few:
-d tests if the file is a directory
-e tests if the file exists
-s returns the size of the file in bytes
-x tests if the file can be executed
Example: $filesize = -s $file
Date and Time in Perl, basic
### "create" today's date
my ($sec, $min, $hour, $day, $month, $year, $wday, $yday, $isdst) = localtime;
This gets the date and time information from the system.
### "create" today's date
my ($sec, $min, $hour, $day, $month, $year, $wday, $yday, $isdst) = localtime;
my $today = sprintf ("%4.4d.%2.2d.%2.2d", $year+1900, $month+1, $day);
This puts today’s date in “Voyager” format, 2006.04.26
Date and Time in Perl, basic
Date and Time in PerlThe program, datemath.pl, is part of your handout. The screenshot below shows its output.
Regular expressions, matching
m/PATTERN/gi
If the m for matching is not there, it is assumed.
The g modifier means to find globally, all occurrences.
The i modifier means matching case insensitive.
Modifiers are optional; others are available.
Regular expressions, substituting
s/PATTERN/REPLACEWITH/gi
The s says that substitution is the intent.
The g modifier means to substitute globally, all occurrences.
The i modifier means matching case insensitive.
Modifiers are optional; others are available.
Regular expressions, transliterating
tr/SEARCHFOR/REPLACEWITH/cd
The tr says that transliteration is the intent.
The c modifier means transliterate whatever is not in SEARCHFOR.
The d modifier means to delete found but unreplaced characters.
Modifiers are optional; others are available.
Regular expressions
# if the pattern matches
if ($var =~ /regular expression/)
{
make_something_happen;
}
Regular expressions
# if the pattern does NOT match
if ($var !~ /regular expression/)
{
make_something_happen;
}
Regular expressions
# contents of $var will be changed
# 1st occurrence of this changes to that
$var =~ s/this/that/;
# all occurrences of this are changed to that
$var =~ s/this/that/g;
Regular expressions
# contents of $var will be changed
# converts all lower case letters to# upper case letters
$var =~ tr/a-z/A-Z/;
Regular expressions
Some simple stuff to get started…
m/thisx*/ * find zero or more ‘x’ right after ‘this’
m/thisx+/ + find one or more ‘x’ right after ‘this’
m/thisx?/ ? find zero or one ‘x’ right after ‘this’
Some simple stuff to get started…
m/thisx*/ * find zero or more ‘x’ right after ‘this’
m/thisx+/ + find one or more ‘x’ right after ‘this’
m/thisx?/ ? find zero or one ‘x’ right after ‘this’
m/[0-9]{5}/ find exactly five consecutive digits
m/[0-9]{5,}/ find at least five consecutive digits
m/[0-9]{5,7}/ find from five to seven consecutive digits
Regular expressions
Some more simple stuff…
m/^this/ find ‘this’ only at the beginning of the string
m/this$/ find ‘this’ only at the end of the string
Regular expressions
Some more simple stuff…
m/^this/ find ‘this’ only at the beginning of the string
m/this$/ find ‘this’ only at the end of the string
Some specific characters:
\n newline (line feed)
\r carriage return
\t tab
\f form feed
\0 null
Regular expressions
Some more simple stuff…
m/^this/ find ‘this’ only at the beginning of the string
m/this$/ find ‘this’ only at the end of the string
Some specific characters: Some generic characters:
\n newline (line feed) \d any digit
\r carriage return \D any non-digit character
\t tab \s any whitespace character
\f form feed \S any non-whitespace character
\0 null
Regular expressions
Look in the Perl book (see Resources) for an explanation on how to use regular expressions. You can look around elsewhere, at Perl sites, and in other books, for more information and examples.
Looking at explained examples can be very helpful in learning how to use regular expressions.
(I’ve enclosed some I’ve found useful; see Resources.)
Regular expressions
Very powerful mechanism.
Often hard to understand at first glance.
Can be rather obtuse and frustrating!
If one way doesn’t work, keep at it. Most likely there is a way that works!
Regular expressions
DBI stuff
What is it and why might I want it?
DBI is the DataBase Interface module for Perl. You will also need the specific DBD (DataBase Driver) module for Oracle.
This enables Perl to perform queries against your Voyager database.
Both of these should already be on your Voyager box.
DBI stuff, how to
You need four things to connect to Voyager:
machine name your.machine.here.edu
username your_username
password your_password
SID VGER (or LIBR)
$dbh is the handle for the database
$sth is the handle for the query
Create a query…then execute it.
NOTE: SQL from Access will most likely NOT work here!
DBI stuff, how to
Get the data coming from your query.
DBI stuff, how to
Get the data coming from your query.
You’ll need a Perl variable for each column returned in the query.
Commonly a list of variables is used; you could also use an array.
DBI stuff, how to
Get the data coming from your query.
You’ll need a Perl variable for each column returned in the query.
Commonly a list of variables is used; you could also use an array.
Typically, you get your data in a while loop, but you could have
$var = $sth->fetchrow_array;
when you know you’re getting a single value.
DBI stuff, how to
When you’re done with a query, you should finish it. This becomes important when you have multiple queries in succession.
You can have multiple queries open at the same time. In that case, make the statement handles unique…$sth2, or $sth_patron.
Finally, you can close your database connection.
DBI stuff, how to
CPAN
Comprehensive Perl Archive Network
http://cpan.org
You name it and somebody has probably written a Perl module for it, and you’ll find it here.
There are also good Perl links here; look for the Perl Bookmarks link.
CPANInstalling modules
You need to be root for systemwide installation on Unix systems.
On Windows machines, you’ll probably need to be administrator.
You can install them “just for yourself” with a bit of tweaking, and without needing root access.
If you’re not a techie, you’ll probably want to find someone who is, to install modules.
Installing modules from CPAN is beyond the scope of this presentation.
Perl on your PC
You can get Perl for your PC from ActiveState.
They typically have two versions available; I recommend the newer one. Get the MSI version.
Installation is easy and painless, but it may take some time to complete.
A lot of modules are included with this distribution; many additional modules are available. Module installation is made easy via the Perl Package Manager (PPM).
To use ppm in ActiveState Perl, open a command prompt window and enter ppm.
Help is available by simply typing help.
Some useful commands in ppm are:
query * show what’s already installed
search pkg look for package pkg at ActiveState’s repository
install pkg retrieve and install package pkg on your machine
Perl on your PC
If you can’t find the module you’re looking for at ActiveState, you should be able to find it at CPAN, and will have to install it manually.
Perl on your PC
Voyager examples
Based on my experience (your mileage may vary), there are two main types of applications, for Voyager:
Based on my experience (your mileage may vary), there are two main types of applications, for Voyager:
reports, or data retrievals, from the database
Voyager examples
Based on my experience (your mileage may vary), there are two main types of applications, for Voyager:
reports, or data retrievals, from the database
data manipulation, mainly of files to be imported
Voyager examples
Voyager example,a simple report
This report finds patrons with multiple email addresses
Tells the system where to find Perl
Voyager example,a simple report
Will be querying the Voyager database
Voyager example,a simple report
Set up output file name
Voyager example,a simple report
Carefully open the output file for use
Voyager example,a simple report
Keep password data in ONE file. Why?
one point of maintenance (less work when the password changes)
reduces opportunities for error
anyone can see the source code without seeing the password data
Voyager example,a simple report
Get some information for each patron
Voyager example,a simple report
Get the patron identifying data in a loop, and…
Voyager example,a simple report
Get the patron identifying data in a loop, and set up the query to get the email address(es) for this patron
Voyager example,a simple report
In an “inner” loop, get email address data for this patron
Voyager example,a simple report
In an “inner” loop, get email address data for this patron.
Preformat the fields for future output.
Voyager example,a simple report
In an “inner” loop, get email address data for this patron.
Preformat the fields for future output.
Populate the address array with each address for this patron.
(note that this array starts out empty for each patron, see previous slide)
Voyager example,a simple report
If this patron has more than one email address, then we are interested
Voyager example,a simple report
Remove trailing spaces from the name parts, then concatenate the parts together
Voyager example,a simple report
Now output the multiple email addresses for this patron
Voyager example,a simple report
A sample of the output
Voyager example,a simple report
Voyager example,some data manipulation
This program processes incoming authority records:
remove records whose 010 |a fields begin with "sj“
remaining records are stripped of the 9xx fields
Specify the file to be processed as a command line parameter.
If no parameter is supplied, display a short paragraph that shows how to use this program, then exit.
Voyager example,some data manipulation
Set up the |a subfield “delimiter”.
This will be used later in the 010 field.
Voyager example,some data manipulation
We could have used $ARGV[0] as the filename variable, but using $marcin makes the program more readable
Voyager example,some data manipulation
An example of “slurping”, reading the file into an array without resorting to a loop
Voyager example,some data manipulation
This an example of early code sticking around too long. It should be rewritten:
Insert this line before accessing the file:$/ = chr(0x1d); # use the MARC end-of-record terminator
Then get the data this way:@marcrecords = <marcin>;
The above code can be eliminated by these simple changes.
Voyager example,some data manipulation
Voyager example,some data manipulation
This an example of early code sticking around too long. It should be rewritten:
Insert this line before accessing the file:$/ = chr(0x1d); # use the MARC end-of-record terminator
Then get the data this way:@marcrecords = <marcin>;
The above code can be eliminated by these simple changes.
The end result is that we have an array of the MARC records from the input file
Voyager example,some data manipulation
Determine the base address for data in this record, and get ready to read the directory
Voyager example,some data manipulation
Get each field’s particulars, figure out where its data is, and read the data
Voyager example,some data manipulation
We look for field 010, subfield a
Voyager example,some data manipulation
If subfield a is found, does its data start with “sj”? If so, we do not want this record.
Voyager example,some data manipulation
Looks like this record is a keeper.
If this is a 9xx field, i.e., the tag id starts with ‘9’, keep track of these fields in an array until we’ve looked at all the fields.
Voyager example,some data manipulation
When done reading the record that’s a keeper, we need to delete the 9xx fields, and output the record.
Voyager example,some data manipulation
If the record is not a keeper, put it in the deleted file
Voyager example,some data manipulation
Resources
Learning Perl Perl in a Nutshell
Programming Perl Perl Cookbook
I use these two a lot
All books are from O’Reilly.
Advanced Perl Programming
Perl Best Practices
These will start to be useful once you have some Perl experience.
All books are from O’Reilly.
Perl Hacks
Intermediate Perl
Resources
Active State Perlhttp://activestate.com/Products/Download/Download.plex?id=ActivePerl
CPANhttp://cpan.org
a great link to links http://www.thepeoplestoolbox.com/programmers/perl
Resources
The files listed below are available at
http://homepages.wmich.edu/~zimmer/files/eugm2007
youcandoitPerl.ppt this presentation
findmanyemail.plfind patrons with multiple email addresses(available by request)
noauthsj.pldelete record if 010 |a starts with “sj”, and strip 9XX fields from remaining records
datemath.plsome program code for math with dates
snippet.grepvarious regular expressions I’ve found useful
Resources
Thanks for listening.
Questions?
269.387.3885
Picture © 2005 by Roy Zimmer
Thanks for listening.
Questions?
269.387.3885
Picture © 2006 by Roy Zimmer