SOFTWARE TOOLS In addition to the basic UNIX commands, we discuss some software tools that you will find useful. Absolute Basics You must know these commands: Command Description cd change current working directory pwd print current working directory mkdir creates a directory rmdir removes directory, must be empty rm removes a file. cp copies files/dirs mv moves files/dirs chmod change file permissions – discussed in class Basics These were discussed in the assigned exercises last week. Command Description ls lists files – more discussion to come in class cat displays a file more like cat, but displays a file 1 screen at a time ps lists processes kill kills an active process who displays currently logged on users finger displays more details about a user su switch accounts du disk usage head, tail display 1st/last couple of lines wc counts the number of lines,words, characters in a file lpr, lpq, lprm printing related commands clear clears the screen Advanced Tools These tools are very useful and will be discussed in some detail below: Command Description grep,egrep search file(s) for strings sort sorts the contents of files uniq eliminates duplicate lines find search for files and much more tar archive files/dirs diff show change(s) made to a file by comparing to new version compress, uncompress, gzip, gunzip

compress/decompress files

sed stream editor – transforms text as it streams through

grep & regular expressions – search for strings grep stands for “global regular expression print”. It’s a very useful tool you can use to search for occurrences of strings in files. Before you can use grep, you need to understand regular expressions. Regular Expressions A regular expression is a sequence of ordinary characters and special characters. Special characters include the following: back slash (\), caret (^), asterisk (*), dollar ($), square brackets ([]), and dot (.). A simple string The simplest of regular expressions is just a string. If you give grep a string with no special characters, it’ll match whenever it finds the string, even if other characters surround it. So, grep con socket*.cpp will search all files beginning with socket and ending with .cpp for the string con. con would match connect, continue, connection, lexicon ...etc. werewolf:~% grep 'con' socket*.cpp socketclient.cpp: if( connect(sockfd,(struct sockaddr *)&server,SIZE)==-1) socketclient.cpp: printf("connect call fails\n"); socketserv.cpp: printf("waiting for connection...\n"); socketserv.cpp: continue; socketserv.cpp: //parent should close the new socket connection. werewolf:~% Special characters dot (.) matches any single character. use dot (.) to match any single character. For example, w.r would match the strings lowercase, worship or warning. restrict ranges with [] If you want to restrict the range of matches, use the square brackets: w[ea]r matches only lowercase and warning, but not worship. It’s a good idea to pass the regular expression to grep surrounded by a pair of single quotes, so the shell doesn’t do any substitutions on your regular expression. So use grep 'w[ea]r' myfile If the 1st character in the square bracket is the caret (^), then the match occurs on characters NOT in the list. You can specify a range of characters with -. grep 'w[a-z]r' myfile matches any lower case character between w and r. match a specific number of a single character use a pair of escaped braces to specify the minimum and maximum number of a single character: n\{2,4\} matches nn, nnn, or nnnn. If you specify only 1 number, then the exact number of occurrences will be made: werewolf:~% grep 'n\{2\}' socket*.cpp socketclient.cpp: if( connect(sockfd,(struct sockaddr *)&server,SIZE)==-1) socketclient.cpp: printf("connect call fails\n"); socketserv.cpp: printf("waiting for connection...\n"); socketserv.cpp: //parent should close the new socket connection. werewolf:~% * matches zero or more of the preceding expression grep 'p*' myfile displays the entire file, since every line has 0 or more p. grep 'ppp*' myfile displays all lines that have at least 2 ‘p’.

^ anchors in front, $ anchors at the end grep '^[Tt]he' myfile displays all the lines beginning with ‘the’ or ‘The’ grep '\.$' myfile displays all lines that end with a period. Actually, its probably more accurate to say that ^ matches the imaginary character before the 1st character in the line, and $ matches the imaginary character at the end of the line. egrep egrep provides a few more ways you can match stuff. Most notably, + is like *, but matches 1 or more occurrences, and ? matches 0 or 1 occurrences of anything. egrep also allow you to search for either of two regular expressions using the | operator. egrep 'dd+|socket' socket*.cpp displays all lines that have at least 2 ds or the word socket in it. Sort Consider the following raw results from a swim meet: Name Age Team Prelim Final Stanford, Jeffrey 25 HIMA 47.07 46.32 Liggett, Michael 27 DYNA 47.25 48.12 Baker, Chase 29 GMUP 56.28 57.79 Kittredge, Brad 25 TOC 45.05 46.22 Richner, Thomas 27 UCLA 50.00 48.79 McCormick, Aaron 27 RMM 49.00 49.30 Thorum, Thomas 29 UTAH 50.00 49.45 Paul, Darcy 26 NEM 52.00 50.17 Welting, Evan 27 DYNA 50.50 51.04 Wanie, Lee 28 TOC 46.00 46.39 Linderman, Ross 25 PNA 52.55 51.17 Wen, Patrick 29 UCLA 57.50 55.93 Frohlich, Jon 29 UTAH 49.10 49.20 Parnes, Jason 29 SDSM 63.00 59.11 Perunovich, Steven 27 HIMA 59.50 52.93 You might want to sort the file according to certain criteria. Suppose the results are stored in a file called swimresults. sort lines Invoke sort with no arguments just sorts the lines: $ sort swimresults Baker, Chase 29 GMUP 56.28 57.79 Frohlich, Jon 29 UTAH 49.10 49.20 Kittredge, Brad 25 TOC 45.05 46.22 Liggett, Michael 27 DYNA 47.25 48.12 Linderman, Ross 25 PNA 52.55 51.17 McCormick, Aaron 27 RMM 49.00 49.30 Parnes, Jason 29 SDSM 63.00 59.11 Paul, Darcy 26 NEM 52.00 50.17 Perunovich, Steven 27 HIMA 59.50 52.93 Richner, Thomas 27 UCLA 50.00 48.79

Stanford, Jeffrey 25 HIMA 47.07 46.32 Thorum, Thomas 29 UTAH 50.00 49.45 Wanie, Lee 28 TOC 46.00 46.39 Welting, Evan 27 DYNA 50.50 51.04 Wen, Patrick 29 UCLA 57.50 55.93 $ Since the last names were listed first, sorting the lines is equivalent to sorting by last names. Note the sort is performed by comparing ASCII values. As a result, upper case letters are “less” than lower case letters and space is “less” than a character. Skip fields with + What would I do if I wanted to sort on the age of the participants instead of the last name? This can be accomplished by telling sort to skip the first 2 fields (last name, first name): $sort +2 swimresults Wanie, Lee 28 TOC 46.00 46.39 Paul, Darcy 26 NEM 52.00 50.17 Baker, Chase 29 GMUP 56.28 57.79 Wen, Patrick 29 UCLA 57.50 55.93 Welting, Evan 27 DYNA 50.50 51.04 Parnes, Jason 29 SDSM 63.00 59.11 Frohlich, Jon 29 UTAH 49.10 49.20 Thorum, Thomas 29 UTAH 50.00 49.45 Linderman, Ross 25 PNA 52.55 51.17 Kittredge, Brad 25 TOC 45.05 46.22 Richner, Thomas 27 UCLA 50.00 48.79 Liggett, Michael 27 DYNA 47.25 48.12 McCormick, Aaron 27 RMM 49.00 49.30 Stanford, Jeffrey 25 HIMA 47.07 46.32 Perunovich, Steven 27 HIMA 59.50 52.93 ...hmm, that didn’t seem to work too well. The problem is that sort treats the 1st white space as the field separator. The rest of the field, including all the rest of the white spaces are factored into the sort. Because the age field included an uneven number of leading spaces, those spaces where sorted too, leading to the problem we see. Skip leading spaces with –b Try this again with the –b option. This tells sort to ignore leading spaces. % sort -b +2 swimresults Stanford, Jeffrey 25 HIMA 47.07 46.32 Linderman, Ross 25 PNA 52.55 51.17 Kittredge, Brad 25 TOC 45.05 46.22 Paul, Darcy 26 NEM 52.00 50.17 Liggett, Michael 27 DYNA 47.25 48.12 Welting, Evan 27 DYNA 50.50 51.04 Perunovich, Steven 27 HIMA 59.50 52.93 McCormick, Aaron 27 RMM 49.00 49.30 Richner, Thomas 27 UCLA 50.00 48.79 Wanie, Lee 28 TOC 46.00 46.39 Baker, Chase 29 GMUP 56.28 57.79 Parnes, Jason 29 SDSM 63.00 59.11 Wen, Patrick 29 UCLA 57.50 55.93 Frohlich, Jon 29 UTAH 49.10 49.20 Thorum, Thomas 29 UTAH 50.00 49.45 That’s more like it.

Use stop field number with – By default, sort includes the rest of the fields in the sort. So if you specify sort -b +2 swimresults, the rest of the line, such as 25 HIMA 47.07 46.32 is used for the sort. You can tell sort not to include rest of the fields, for example, to sort on the 3rd field only. This can be done with the stop field operator –. The – operator tells sort to ignore everything after the field indicated. So sort -b +2 -3 swimresults says: skip the first 2 fields, ignore all leading blanks, and ignore everything after the 3rd field. Contrast the difference between sort -b +2 -3 swimresults and sort -b +2 swimresults : werewolf:~% sort -b +2 -3 swimresults Kittredge, Brad 25 TOC 45.05 46.22 Linderman, Ross 25 PNA 52.55 51.17 Stanford, Jeffrey 25 HIMA 47.07 46.32 Paul, Darcy 26 NEM 52.00 50.17 Liggett, Michael 27 DYNA 47.25 48.12 McCormick, Aaron 27 RMM 49.00 49.30 Perunovich, Steven 27 HIMA 59.50 52.93 Richner, Thomas 27 UCLA 50.00 48.79 Welting, Evan 27 DYNA 50.50 51.04 Wanie, Lee 28 TOC 46.00 46.39 Baker, Chase 29 GMUP 56.28 57.79 Frohlich, Jon 29 UTAH 49.10 49.20 Parnes, Jason 29 SDSM 63.00 59.11 Thorum, Thomas 29 UTAH 50.00 49.45 Wen, Patrick 29 UCLA 57.50 55.93 werewolf:~% sort -b +2 swimresults Stanford, Jeffrey 25 HIMA 47.07 46.32 Linderman, Ross 25 PNA 52.55 51.17 Kittredge, Brad 25 TOC 45.05 46.22 Paul, Darcy 26 NEM 52.00 50.17 Liggett, Michael 27 DYNA 47.25 48.12 Welting, Evan 27 DYNA 50.50 51.04 Perunovich, Steven 27 HIMA 59.50 52.93 McCormick, Aaron 27 RMM 49.00 49.30 Richner, Thomas 27 UCLA 50.00 48.79 Wanie, Lee 28 TOC 46.00 46.39 Baker, Chase 29 GMUP 56.28 57.79 Parnes, Jason 29 SDSM 63.00 59.11 Wen, Patrick 29 UCLA 57.50 55.93 Frohlich, Jon 29 UTAH 49.10 49.20 Thorum, Thomas 29 UTAH 50.00 49.45 Secondary sorts The stop field operator is useful for performing secondary sorts. Suppose I wanted to list the fastest people according to age groups: werewolf:~% sort -b +2 -3 +5 swimresults Kittredge, Brad 25 TOC 45.05 46.22 Stanford, Jeffrey 25 HIMA 47.07 46.32 Linderman, Ross 25 PNA 52.55 51.17 Paul, Darcy 26 NEM 52.00 50.17 Liggett, Michael 27 DYNA 47.25 48.12 Richner, Thomas 27 UCLA 50.00 48.79 McCormick, Aaron 27 RMM 49.00 49.30 Welting, Evan 27 DYNA 50.50 51.04 Perunovich, Steven 27 HIMA 59.50 52.93 Wanie, Lee 28 TOC 46.00 46.39 Frohlich, Jon 29 UTAH 49.10 49.20 Thorum, Thomas 29 UTAH 50.00 49.45 Wen, Patrick 29 UCLA 57.50 55.93 Baker, Chase 29 GMUP 56.28 57.79 Parnes, Jason 29 SDSM 63.00 59.11

This tells sort to do the following: skip the first 2 fields, ignore all leading blanks, and ignore everything after the 3rd field. For those items considered equivalent on the first sort, further sort it according to the 6th field. uniq uniq gets rid of duplicate lines. Consider: % cat nonuniqresults Baker, Chase 29 GMUP 56.28 57.79 Frohlich, Jon 29 UTAH 49.10 49.20 Kittredge, Brad 25 TOC 45.05 46.22 Liggett, Michael 27 DYNA 47.25 48.12 Linderman, Ross 25 PNA 52.55 51.17 Linderman, Ross 25 PNA 52.55 51.17 McCormick, Aaron 27 RMM 49.00 49.30 Linderman is a duplicate item. We can use uniq to remove it: % uniq nonuniqresults Baker, Chase 29 GMUP 56.28 57.79 Frohlich, Jon 29 UTAH 49.10 49.20 Kittredge, Brad 25 TOC 45.05 46.22 Liggett, Michael 27 DYNA 47.25 48.12 Linderman, Ross 25 PNA 52.55 51.17 McCormick, Aaron 27 RMM 49.00 49.30 As you can see the duplicate item is gone. Be careful, uniq only removes identical lines that are adjacent to one another! find – searches for files find takes as its first parameter a path to start the search from. If you want find to search everything, then give it a first parameter of /. Each parameter after this first one falls into one of three categories: criteria, action, or qualifier. criteria these are things like the name of the file, user id of the file, last access time, size ...etc. If the file matches each criteria specified, then the action specified is executed. $find /usr/include/ -name "fcntl*" –print /usr/include/asm/fcntl.h /usr/include/linux/fcntl.h /usr/include/bits/fcntl.h /usr/include/fcntl.h /usr/include/sys/fcntl.h the above searches the /usr/include/ and all subdirectories of it for all files starting with fcntl. Another example: werewolf:~% find /usr/bin -size +2000 -print /usr/bin/openssl /usr/bin/gs /usr/bin/openjade /usr/bin/gimp-1.2 /usr/bin/Xvnc /usr/bin/linux /usr/bin/pdfetex /usr/bin/vim /usr/bin/ddd /usr/bin/doxygen

/usr/bin/gdb /usr/bin/splint werewolf:~% The above searches the /usr/bin directory for files for files bigger than 2000 disk blocks. Action If I were to issue the following command: $find /usr/include/ -name "fcntl*" $ The output would be nothing, even though as we’ve seen, there are files under /usr/include/ that matches “fcntl*”. The problem here is that no action was specified. For each file matched to the criteria, you must specify an action in order for find to do anything useful. -print prints the matched file. This is probably the most useful action -exec cmd {} \; executes cmd on each matched file, signified by {}. werewolf:~% find ./temp -name "sockets*" -print -exec rm -f {} \; ./temp/socketserv ./temp/socketserv.cpp Here we search for and remove all files that start with “sockets” in the temp directory. Qualifiers – these modify how find performs its search. You can force find to search only on the current file system or limit the depth of search. Read the man pages for more details. Diff The diff utility compares two files and shows the change(s) made. To be more specific, it shows the changes one would have to make to file 1 in order to make it identical to file 2. There are 3 types of changes: additions (a), deletions (d), and changes (c). Add The format for add is: firstStart a secondStart, secondStop This says that the chunk of text between secondStart and secondStop are to be inserted right after firstStart in file 1. delete The format for delete is: firstStart, firstStop d lineCount This says that the lines in between firstStart and firstStop are to be deleted in file 1. change The format for change is: firstStart, firstStop c secondStart, secondStop This says that the chunk of text between firstStart, firstStop are to be replaced by that between secondStart and secondStop Example: Consider the following:

file1: (doesn’t actually contain line numbers 1 Baker, Chase 29 GMUP 56.28 57.79 2 Frohlich, Jon 29 UTAH 49.10 49.20 3 Kittredge, Brad 25 TOC 45.05 46.22 4 Liggett, Michael 27 DYNA 47.25 48.12 5 Linderman, Ross 25 PNA 52.55 51.17 6 Linderman, Ross 25 PNA 52.55 51.17 7 McCormick, Aaron 27 RMM 49.00 49.30 8 Parnes, Jason 29 SDSM 63.00 59.11 9 Paul, Darcy 26 NEM 52.00 50.17 10 Perunovich, Steven 27 HIMA 59.50 52.93 11 Richner, Thomas 27 UCLA 50.00 48.79 12 Stanford, Jeffrey 25 HIMA 47.07 46.32 13 Thorum, Thomas 29 UTAH 50.00 49.45 14 Wanie, Lee 28 TOC 46.00 46.39 15 Welting, Evan 27 DYNA 50.50 51.04 16 Wen, Patrick 29 UCLA 57.50 55.93 file2: (doesn’t contain line numbers) 1 Baker, Chase 29 GMUP 56.28 57.79 2 Frohlich, Jon 29 UTAH 49.10 49.20 3 Linderman, Ross 25 PNA 52.55 51.17 4 Linderman, Ross 25 PNA 52.55 51.17 5 McCormick, Aaron 27 RMM 49.00 49.30 6 Parnes, Jason 29 SDSM 63.00 59.11 7 Paul, Darcy 26 NEM 52.00 50.17 8 Perunovich, Steven 27 HIMA 59.50 52.00 9 Richner, Thomas 27 UCLA 50.00 48.00 10 Stanford, Jeffrey 25 HIMA 47.07 46.32 11 Thorum, Thomas 29 UTAH 50.00 49.45 12 Wanie, Lee 28 TOC 46.00 46.39 13 some additions in the 14 middle 15 Welting, Evan 27 DYNA 50.50 51.04 16 Wen, Patrick 29 UCLA 57.50 55.93 sh-2.05a$ diff file1 file2 3,4d2 < Kittredge, Brad 25 TOC 45.05 46.22 < Liggett, Michael 27 DYNA 47.25 48.12 10,11c8,9 < Perunovich, Steven 27 HIMA 59.50 52.93 < Richner, Thomas 27 UCLA 50.00 48.79 --- > Perunovich, Steven 27 HIMA 59.50 52.00 > Richner, Thomas 27 UCLA 50.00 48.00 14a13,14 > some additions in the > middle 16c16 < Wen, Patrick 29 UCLA 57.50 55.93 --- > Wen, Patrick 29 UCLA 57.50 55.93 The last change is not obvious. Although it looks the same, there are some more spaces in the second file. To make diff ignore spaces, use the –w option. The –i flag makes diff case insensitive. Here is a view of a more modern diff program:

Archiving tar tar was originally designed for archiving files to tape. It’s just as good at producing archive files on disk, however. creating a tar file: tar -cvf tarfile fileList Options: -c means to create archive, -v means verbose, and -f forces tar to write to tarfile, instead of /dev/rmt0, which is the default. extracting a tar file: tar –xvf tarfile fileList Options: -x means to extract. If fileList is not specified, then tar extracts everything. The –t option makes tar display a tableof contents. sh-2.05a$ tar -cvf mytarfile.tar *.cpp findcode.cpp hello.cpp socketclient.cpp socketserv.cpp sh-2.05a$ tar -tvf mytarfile.tar -rw------- kenxu/instrs 10446 2003-05-18 02:59:36 findcode.cpp -rw------- kenxu/instrs 59 2003-05-12 17:59:46 hello.cpp -rw------- kenxu/instrs 850 2003-04-18 15:36:04 socketclient.cpp -rw------- kenxu/instrs 1265 2003-04-18 15:42:42 socketserv.cpp sh-2.05a$ tar -xvf mytarfile.tar hello.cpp hello.cpp sh-2.05a$ tar -xvf mytarfile.tar findcode.cpp hello.cpp socketclient.cpp

socketserv.cpp sh-2.05a$ The archive file is typically compressed, using either compress (.Z), or gzip (.gz). You can decompress these files using uncompress (for .Z) or gunzip (for .gz) sed sed stands for stream editor. Like the name suggests, it is an editor in the sense that it does what most editors do: add new text, replace entire lines of text with other lines, delete lines, or replace certain parts of a line with other text. sed does have one important distinction though – it is designed to work with an input stream – this makes it an ideal fit as a part of the Unix command pipeline. Think of sed as a filter; as input streams through, sed looks for matches either in terms of line numbers or a regular expression. When a match is found, some action specified by a command is performed. text substitution This is probably the most common use for sed. The syntax is: s/expr/str/ - replaces the 1st occurrence of the regular expression expr with str. Consider the following: sh-2.05a$ cat file3 Baker, Chase 29 GMUP 56.28 57.79 Frohlich, Jon 29 UTAH 49.10 49.20 Kittredge, Brad 25 TOC 45.05 46.22 Liggett, Michael 27 DYNA 47.25 48.12 Linderman, Ross 25 PNA 52.55 51.17 sh-2.05a$ sed -e 's/ /-/' file3 Baker,-Chase 29 GMUP 56.28 57.79 Frohlich,-Jon 29 UTAH 49.10 49.20 Kittredge,-Brad 25 TOC 45.05 46.22 Liggett,-Michael 27 DYNA 47.25 48.12 Linderman,-Ross 25 PNA 52.55 51.17 Here I asked the blank character be replaced with a -. Notice just the first space has been replaced with -. We can fix this by adding the g option at the end, which tells sed to replace all occurrences in a line: sh-2.05a$ sed -e 's/ /-/g' file3 Baker,-Chase--------------29-GMUP-------56.28------57.79-- Frohlich,-Jon-------------29-UTAH-------49.10------49.20---- Kittredge,-Brad-----------25-TOC--------45.05------46.22--- Liggett,-Michael----------27-DYNA-------47.25------48.12---- Linderman,-Ross-----------25-PNA--------52.55------51.17-- sh-2.05a$ The –e option tells sed that what follows is to be interpreted as a script. Restricting the range Suppose I want to do the above only for the 2nd, 3rd, and 4th line. This can be accomplished by restricting the range of sed: sh-2.05a$ sed -e '2,4s/ /-/g' file3 Baker, Chase 29 GMUP 56.28 57.79 Frohlich,-Jon-------------29-UTAH-------49.10------49.20---- Kittredge,-Brad-----------25-TOC--------45.05------46.22--- Liggett,-Michael----------27-DYNA-------47.25------48.12---- Linderman, Ross 25 PNA 52.55 51.17 sh-2.05a$

The 2,4 tells sed to restrict the range to lines 2 through 4. The same thing can also be done using pattern matches: sh-2.05a$ sed -e '/Frohlich/,/Liggett/s/ /-/g' file3 Baker, Chase 29 GMUP 56.28 57.79 Frohlich,-Jon-------------29-UTAH-------49.10------49.20---- Kittredge,-Brad-----------25-TOC--------45.05------46.22--- Liggett,-Michael----------27-DYNA-------47.25------48.12---- Linderman, Ross 25 PNA 52.55 51.17 sh-2.05a$ deleting text The general syntax is: addressRange d where addressRange is like that just discussed above. An example: sh-2.05a$ sed -e '/Frohlich/,/Liggett/d' file3 Baker, Chase 29 GMUP 56.28 57.79 Linderman, Ross 25 PNA 52.55 51.17 sh-2.05a$ Notice the middle 3 lines were deleted. replacing lines The syntax for line replacement is: addressRange c\ text It is best to dump this into a file and use the –f option with sed to get it to read the script from the file. This is because sed is very picky about the format above: it has to be addressRange followed by c, followed by \, followed by newline, follwed by text. Anything else produces errors. An example: sh-2.05a$ cat sedtest 2,3c\ Lines 2 and 3 have been replaced by this single line sh-2.05a$ sed -f sedtest file3 Baker, Chase 29 GMUP 56.28 57.79 Lines 2 and 3 have been replaced by this single line Liggett, Michael 27 DYNA 47.25 48.12 Linderman, Ross 25 PNA 52.55 51.17 sh-2.05a$ There are a few other common operations, most notably: Append text: adress a\ text Insert file address r name where name is the name of the file to be inserted. tee • Like a T junction

• • • •

1 input flows in, multiple output comes out Always outputs to screen also outputs to each file mentioned on commandline use –a to append to files, instead of overwrite

touch • • •

update the last access time, modification time etc. creates empty file if not there use –c to suppress file creation

Absolute BasicsBasicsAdvanced ToolsRegular ExpressionsA simple stringSpecial charactersSortName Age Team Prelim Final

Skip fields with +Skip leading spaces with –bUse stop field number with –Secondary sortsActionDiff

Add The format for add is:Archiving

Restricting the rangeInsert file