Unix as a Web Development Platform

Norman White

Stern School of Business

Why is UNIX so popular?

• Platform Support– Unix runs on virtually an hardware platform

– Designed as a portable Operating System

• Toolset(s)– Unix has many powerful development tools

• Shell Programming Languages (sh,csh,ksh,tcsh)

• C language

• “filters” like awk, grep, sed, find, pr

• “pipe” process output to glue filters together

– Open Source Community

Networking

• Unix has ALWAYS supported TCP/IP

• Very powerful internet capabilities– telnet, ftp, NFS, …– most unix apps support networking– Even the smallest Unix/linux system can

support full-scale Internet apps.– Great development environment– REMOTE ACCESS via Telnet

“Free” UNIX software

• UNIX (.e. Freebsd, Linux)

• Languages (GNU Compilers, Java etc..)

• Web development environments– PhP– PERL (ModPerl)– Apache– FastCGI– NCSA web server

Other Reasons ….

• UNIX is the development environment for almost all computer science departments

• If you have a computer science degree > 1980 you know Unix…

• =====> Many, many UNIX programmers

• Supported by all major vendors– SUN. IBM, HP, SGI, DELL, Unisys etc.

UNIX OS features

• Process oriented environment• Hierarchical file system integrated with

virtual memory system (what does that mean?)

• Small kernel size allows easy “porting” to new environments (like CRUSOE)

• But full set of capabilities allow it to be used on very large systems

Recent Additions - Linux/Beowulf

• Linux growth has made it even easier to learn Unix

• Linux modifications allow it to scale to super computer size

• Beowulf Project - NASA Ames– Modify Linux so it can run on hundreds or

even thousands of cooperating systems

Beowulf

• Configuration– 1 Master computer

• Used for job scheduling, data output etc..

– Many identical Beowulf machines all connected by a separate private, high speed network switch (sometimes multiple networks)

– Can be used for almost any type of processing that can be done in parallel (computation, web serving, database etc.).

UNIX problems

• Command language(s) powerful, but cryptic and terse– Designed to minimize typing– Designed by Ph.D. computer scientists at Bell

labs who believe that everyone understands regular expressions

• Steep learning curve, not for novices...

Huh???

• "There are two major products to come out of Berkeley: LSD and UNIX. We don't believe this to be a coincidence."

• -- Unknown

• What’s the matter with this quote????

UNIX as a web platform

• UNIX quickly became dominant web platform because of its scalability and large set of tools available

• Many development choices available

Some UNIX Web environments

• Apache Web server - most popular web server– Server side toolkits

• CGI (PERL, C, PYTHON …)

• “FASTCGI”– link application directly into Apache so that it is always

running

• MODPERL - FASTCGI version of PERL

• PHP – Server side, imbedded HTML scripting language

• Java servlets, Java Server Pages etc..

Other Web Servers

• Netscape/Sun/Iplanet server(Stern)

• IBM Websphere

• NCSA

• ….

• ALL support the same toolsets

Other UNIX Tools - X

• Xwindows environment– Client server graphical programming

environment– Originally developed to allow scientists to

display results graphically even if the application (Bubble chamber) was running on another computer

– Quickly became a standard on many platforms, but especially UNIX

Xwindows User Interfaces

• Graphical input was supported in early versions of X• This allowed the development of Windows based

command environments for UNIX, as well as many applications (SAS, SPSS etc.)

• This now allowed UNIX to support remote command and windows based interfaces

• Great environment for systems programming support (especially if you have hundreds of web servers)

XwindowsWho is client?

Conclusion

• UNIX is a powerful and robust environment

• Managers need to know it’s capabilities

• Here are some examples...

General UNIX commands

• cd Changes working directory • cp Copies a file • history Lists history of previous commands• ln Creates a link • ls Lists contents of directory • mkdir Creates a new directory • mv Moves or renames files • pwd Prints working directory name • rm Removes files and links • rmdir Removes a directory

• !! (Bang operator) Repeats previous command

UNIX File Manipulation

• apropos Locates commands by keyword lookup• awk Pattern scanning and processing language • cat Concatenates a file• cmp Compares two files • diff Displays differences between pairs of text files • find Finds files • grep, egrep, fgrep Searches a file for a pattern • head Gives first few lines of file • man Finds and displays manual information • more, page, less Filters used to display files on a screen • sed Edits a stream • tail Displays the last part of a file • troff Text formatting and typesetting

Process commands

• bg Places a stopped process in the background

• fg Places a stopped process in the foreground

• jobs Gives a list of current processes created from this

• shell

• kill Sends a signal to the specified process

• ps Gives process status

• & Puts a command in the background when appended to the command

Miscellaneous Commands

• chmod Changes file permissions

• clear Clears terminal screen

• csh Invokes a shell (command interpreter) with C-like

• syntax

• date Displays the date

• du Summarizes disk usage

• hostname Sets or displays name of the current host system

• passwd Changes login password

• script Makes transcript of a terminal sessions

• sh Invokes the Bourne shell

• umask Sets/gets default file permission mask

• which Locates a program file, including aliases and paths

But that’s not all folks…

• Pipes – UNIX supports routing the output of one

command or program (stdout or standard output) to the input of another (stdin or standard input)

– This allows the rapid construction of simple? systems using combinations of existing commands (like our homework)

– This concept was carried forward into DOS

UNIX Pipes and Redirection

• Pipe symbol is | – so proga | progb

• runs program a and program b simultaneously, piping the output (stdout) of proga in to the input (stdin) of progb

– so ps -eaf | grep nwhite will list all of my processes running on unix

– ps -eaf | grep nwhite |tee lpr -Pm9is2 • will also print a copy in the IS department printer• (what do you think “tee” does?)

Redirection

• Symbols <, >, >> allow you to “redirect” the output or input of a program (including shell procedures)

• so proga< inputfile– Starts proga up and “feeds” inputfile to it (instead of

user input)

• proga <inputfile >outputfile– uses inputfile as input and outputfile as output

• >> appends to file

No, we are not done yet!!!

• Unfortunately, we still need to know about shell programming…

Unix Shells

• Shell is the name of any Unix command that provides a command oriented interface to the user.

• Lot’s of shells– Why?

• 25% of all MS students think they want to write their own shell..

Common shells

• (sh) Bourne - the original

• (csh) C shell - modelled after “c” programming language syntax

• (tcsh) - C shell on steroids

• (ksh) - Korn shell - very powerful

• (bsh) - BASH

• You get the picture..

Common Elements

• All support the automation of shell commands, but use (slightly ) different syntaxes

• All are very powerful, since they all support pipes and filters and

• All can be used to build sophisticated systems

• Novice users will see little difference

But how do I tell Unix what shell to use?

• First line of file can be used to tell Unix what program to run to process this command

• so…

• #! /bin/sh– will run the sh command (bourne shell)

• #! /usr/local/bin/tcsh– Will run the t shell ...

Bourne Shell - Environment variables

• CDPATH The search path for the cd command• HOME The subdirectory that becomes current upon login and as a default for cd• LANG Determines the locale to use when LC_ALL does not specify one• LC_ALL Determines the locale to be used to override any previously set values• LC_COLLATE Defines the collating sequence to use when sorting• LC_CTYPE Determines the locale for the interpretation of sequence of bytes• LC_MESSAGES Determines the language in which messages should be written• LIBPATH The search path for shared libraries• LOGNAME Your login name, marked read-only by the /etc/profile script• MAIL The path name of the file used by the mail system to detect the arrival of new mail• MAILCHECK The number of seconds that the shell lets elapse before checking for new mail• MAILPATH A list of programs separated by colons where the shell will notify you of new mail• MAILMSG The mail notification message• PATH The search path for commands separated by colons• PS1 The string to be used as the primary system prompt• PS2 The value of the secondary prompt (when the shell finds a new-line character)• IFS Characters that are to be used as internal field separators• SHACCT The shell writes an accounting record in the file for each shell script executed.• SHELL The path name of the shell (should be exported by the $HOME/.profile script• TIMEOUT The number of minutes the shell remains inactive before it exits.

Setting Environment Variables

• MYVAR=blahblah

• export MYVAR {make it global}

• ….

• Later … cd $MYVAR

• or

• echo my name is $MYVAR

Quotation marks

• Using Quotation Marks• There are three different kind of quotation marks that you

can place around variables, ' (single quotes), " (double quotes), and ` (backquotes). Each performs a completely different function.

Double Quotes• Double Quotes: are the weakest of the three ways of quoting. Everything between

double quotes is still interpreted by the shell. For example, let's consider the following piece of code:

• name=yourself• foo="hello there $name" • The shell will interpret $name and $foo will have hello there yourself stored in it. If

you wanted to, you can• precede a special character with a backslash to prevent the shell from interpreting

metacharacters. So if you did • name=yourself• foo="hello there \$name" • the shell will not interpret $name and you will have hello there $name stored in $foo. • The most common usage of double quotes is to store text with whitespaces in it. For

example • foo="This is a variable with a bunch of whitespace"

Single Quotes

• Single Quotes: If you had done the same thing with single quotes, namely

• name=yourself• foo='hello there $name' • the text between the quotes will *not* be

interpreted by the shell and you'll end up with hello there $name stored in $foo. Single quotes are often used to hide special characters (called metacharacters) such as *, ?, etc. from the shell.

Back Quotes

• BackQuotes: Let's consider the following line in a Bourne shell script

• foo=`ls -l | fgrep Jul`

• The backquotes do something completely different here. The text between the backquotes (ls -l | fgrep Jul) is executed and the results of the execution are stored in the variable $foo.

• Backquotes are most often used when you want to capture the output of a command in a variable.

Great on-line course

• http://www.uwsg.iu.edu/edcert/session3/shell/

• Nice coverage of Bourne Shell

Command Variables

• Variables $1 $2 $3 … $* and $#• are set from the parameters on the command

line• so• myprog foo1 foo2 foo3• $1=foo1• $2=foo2• $3=foo3

Builtin Variables

• A number of variables are automatically set in the shell. They usually contain state information of one kind or the other:

• $# The number of arguments supplied

• $* All arguments passed to the shell on the command line

• $@ Similar to $*

• $- Options supplied to the shell

• $? The value returned by the last executed command

• $$ The PID of this shell

• $! The PID of the last command started with &

Shell Programming - case

• case string1 in

• str1) commands;;

• str2) commands;;

• *) commands;;

• esac

• string1 is compared to str1 and str2. If there is a match, the commands up to ;; are executed. If there is no match, the commands after the asterisk are executed.

For statement

• for var1 [ in list ]• do• commands• done • The for loop executes once for each item in the list.

The list can be a variable containing several words separated by spaces or it can be a list of values typed directly into the statement. Each time through the loop, the variable var1 is assigned the current item in the list until the last one is reached. [ in list ] is optional. If it is not present, the for loop is executed once for each item in the variable var1.

• Example– For I in 1 2 3 4 5; do echo $I; done

if Statement

• if [ expression ];• then• Commands• elif [ expression2 ];• then• Commands• else• Commands• fi• The expression following if is computed and, if it returns true,

the commands following the first then are executed. If expression returns false, expression2 is computed and follows the same logic. If neither expression nor expression2 return true, the commands following else are executed. Both elif and else are optional.

While statement

• while [ expression ]• do• commands• done• A while command repeatedly executes

the expression and, if the last command in the expression is true, executes the commands in the do list; otherwise the loop terminates.

Test Statement

• The test command is available in Bourne shell to test various things. The usage is • test expression • or• [ expression ] • Here is a list of tests that are allowed: • Integer operators • int1 -eq int2 True if int1 is equal to int2 • int1 -ge int2 True if int1 is greater than or equal to int2 • int1 -gt int2 True if int1 is greater than int2 • int1 -le int2 True if int1 is less than or equal to int2 • int1 -lt int2 True if int1 is less than int2 • int1 -ne int2 True if int1 is not equal to int2 •

Test - String/ File Operators

• String operators • str1 = str2 True if str1 is identical to str2 • str1 != str2 True if str1 is not identical to str2 • str True if str is not null • -n str True if the length of str is greater than zero • -z str True if the length of str is equal to zero • File operators • -d filename True if directory filename exists • -f filename True if file filename exists • -r filename True if filename is readable by the process • -s filename True if filename has a nonzero size • -w filename True if filename is a is writeable by the process • -x filename True if filename is executable •

Test - logical operators

• Logical operators

• ! expr True if expr is not true

• expr1 -a expr2 True if both expr1 and expr2 are true

• expr1 -o expr2 True if either expr1 or expr2 is true

• Example:

• if [ -f /bin/sh ]; then

• echo ”yes, we have the Bourne shell"

• fi

Filename matching

• * match any string, including the null string

• ? match any single character

• [ccc] match any of the characters in ccc

• [a-e1-5] equivalent to [abcde12345]

• "... " match ... exactly; quotes protect metacharacters

• \c match c literally

• / matched only by an explicit / in the expression; in case, matched like any other character

• . as the first character of a filename, is matched only by an explicit . in the expression

• a|b (in case expressions only) match either a or b

Putting it all together

• #!/usr/bin/sh

• #

• # Sample Bourne Shell script to list a directory on the

• # Stern Web Server

• #

• # Norman White, November 2000

• #

• # Only works for Stern grads

• # Only lists the contents of their websys subdirectory

• # Assumes the input is on the command line

• #

• # in the form

• http://www.stern.nyu.edu/~nwhite/lsgrad.cgi?fchar=a&userid=aab211

• # or is passed from the web server as a "GET" action in a cgi program

• #

• # part 1 - Generate intro html to put in front of output

• echo "Content-type: text/html"

• echo

• echo "<HTML><HEAD>"

• echo "<TITLE>Websys Directory Listing</TITLE>"

• echo "</HEAD> <BODY>"

• echo "<pre>"

• # remove next few lines for working script

• echo The arguments to the script are: $*

• # Environment Variable QUERY_STRING contains form fields

• echo QUERY_STRING contains $QUERY_STRING

• # uncomment below for help in debugging

• #set

• # Tricky stuff, use sed to get rid of the & between fields

• # so you have a line like:

• # set fchar=a userid=aab211

• # this sets local shell variables for each of the

• # form field values

• # so they can be used to generate the directory path

• # See http://www.stern.nyu.edu/~nwhite/websys/lsgrad.html for an example

• form to

• # invoke this script

• # note, this script MUST be named .cgi in order to be run

• # on the Stern web server

• # also, you need to do a chmod o+rx on it so it is executable

• # The "eval" line creates a shell variable for each form field

• # by issuing a

• # field1=value1 field2=value2 etc

• # The values of these variables can now be used later in the script

• # by putting a $ in front of them

• # "eval line echos the input line and pipes it to a sed program that changes

• # & to blank, hence fchar=a&userid=aab211 becomes

• # fchar=a userid=aab211

• eval ` ( echo $QUERY_STRING | sed -e "s/\&/ /g") `

• # Note need the { } around userid because of the trailing :

• echo Doing directory listing for user ${userid}:\<p\>

• ls -m /export/grad/$fchar/$userid/public_html/websys/

• echo "</pre>"

• echo "</BODY></HTML>"

So how do we do optional parts?

• Clickable list – Hint how do you get a clickable list of files…

• Grad vs undergrad - how can you tell?

Shell ProgrammingJust one kind of “scripting” .

We will see many others, but they are all similar in concept..

PHP, ASP, PERL etc much easier to use for server side processing.