Download - Unix Files, IO Plumbing and Filters
![Page 1: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/1.jpg)
Unix Files, IO Plumbing and Filters
The file system and pathnamesFiles with more than one linkShell wildcardsCharacters special to the shellPipes and IO redirectionCommands used with redirectionSome useful filters: awk, grep, sed etc.
![Page 2: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/2.jpg)
The file system and pathnames
The filesystem is a set of conventions and the specific data design for organising files and folders, or directories on a hard disk, USB drive, CD or other media. E.G. NTFS, FAT (Windows), EXT3 (Linux)
Files on a filesystem include plain files, directory files (or folders), links (Unix/Linux only) and device files.
All modern OSs map files on a filesystem as an inverted tree.
![Page 3: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/3.jpg)
Inverted tree with root dir as /
/ (root directory) | ------------------------------------------------------ | | | | | dir1 file1 file2 dir2 home | | | ---------------- ------------------- ---------- | | | | | | file3 dir3 file4 file5 fred bill | | ------------------------------- ---------- | | | | | file6 file7 file8 cupboard wardrobe | ---------------------- | | | socks vests shirts
Absolute pathnames start with a / and relative pathnames don't.
![Page 4: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/4.jpg)
Some pathname examples
![Page 5: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/5.jpg)
Differences between Windows and Unix
Pathname delimiters: Unix forward / and Windows back \
Windows has drive letters e.g. a:Unix trees mount filesystems onto empty directories.
So, a:\reports\report.txt on Windows becomes /mnt/floppy/reports/report.txt on Unix
![Page 6: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/6.jpg)
Links
Multiple paths to the same file saves space and keeps objects updated
Hard link: ln /sales/personnel/simon /bonuses/recipients/simon
Soft link:ln -s /usr/wizard/wands/magic/software/binaries /wiz
Hard links only possible within same physical device, soft links needed to link between mounted devices/partitions.
![Page 7: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/7.jpg)
Shell Wildcards
Known as "regular expressions", REs or regexes.
Used in shell commands to match or "glob" multiple files.
Similar language used in Grep, Awk, Perl, Java, Python etc. to find/replace expressions in strings, e.g. tags in HTML
![Page 8: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/8.jpg)
Shell Wildcards
![Page 9: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/9.jpg)
Shell Wildcards can be combined, giving greater flexibility
![Page 10: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/10.jpg)
REs and special characters. at the beginning must match explicitly, and / must always
match explicitly.
Avoid putting *,?$^()[]\/&<>|"'! and spaces in filenames ?
If you can't the backslash special character \ will usually escape the special character following it
rm \$fred deletes the file called $fred
Other special characters:
* . at start of name hides a system file * ~ at start of path means home directory
![Page 11: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/11.jpg)
Pipes and redirection
![Page 12: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/12.jpg)
Redirections can be combined e.g:
grep xxx < f1 | awk '{ print $2 }' 2> f2 \ | tee f3 | wc -l > f4
\ is used to escape the newline so a single command line can be typed using more than one line on the terminal.
![Page 13: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/13.jpg)
Commands often used with redirection
![Page 14: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/14.jpg)
Some useful filters
A filter is a program which reads from stdin and writes to stdout. Unix has many useful programs which filter data based on contents of text lines, or row positions or filtering on columns. We can also sort a file, obtain unique keys or chop the top or tail of it.
These filters will make our pipelines into a general purpose and easy to use text-processing toolkit, once a few examples have been mastered.
![Page 15: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/15.jpg)
awk Replaced as a scripting language by Perl, but still
useful as a simple column filter.
Example 1:
cat file1 | awk '{ print $5, $2 }'
outputs columns 5 then 2 of file1 which has to have spaces or tabs between columns. Use -F<char> to delimit input using <char> e.g. -F: for colon seperated columns. You might prefer cut for this job.
![Page 16: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/16.jpg)
awk example 2Put an awk program of any complexity into a separate file.
Use the -f flag to give filename on shell command. This awkprog file has 3 lines:
BEGIN { total=0 } {if ( $3 == "usr19999" ) total=total+$4 } END { print total/1024 }
Run awk program with command:
ls -l /scratch | awk -f awkprog
Counts the number of 1K blocks used by files belonging to usr19999 in the /scratch directory.
![Page 17: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/17.jpg)
cut
For awk's usual purpose as a column filter, you might prefer to learn cut(1) as an alternative. The following command extracts columns 1, 3, 6 and 7 from /etc/passwd, with colon (:) as input delimiter, and space as output delimiter.
cat /etc/passwd | cut -d: --output-delimiter=" " -f1,3,6,7
![Page 18: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/18.jpg)
Input and output of cut command
input sample from /etc/passwd
bert:x:1003:1003:Bert Trusted User,,,:/home/bert:/bin/bashavahi-autoipd:x:111:111:Avahi autoip daemon,,,:/var/lib/avahi-autoipd:/bin/falseavahi:x:116:121:Avahi mDNS daemon,,,:/var/run/avahi-daemon:/bin/falsesnort:x:117:125:Snort IDS:/var/log/snort:/bin/falsepulse:x:119:126:PulseAudio daemon,,,:/var/run/pulse:/bin/false
Output sample
bert 1003 /home/bert /bin/bashavahi-autoipd 111 /var/lib/avahi-autoipd /bin/falseavahi 116 /var/run/avahi-daemon /bin/falsesnort 117 /var/log/snort /bin/falsepulse 119 /var/run/pulse /bin/false
![Page 19: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/19.jpg)
grep
grep extracts lines containing a particular pattern from a file, using regexes. Similar but not
identical to shell REs.
fgrep is faster and does not interpret REs
egrep interprets more REs than grep
![Page 20: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/20.jpg)
grep examples
![Page 21: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/21.jpg)
Sed is used for automated "stream" editing. Another language best used for one liners.
![Page 22: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/22.jpg)
Another sed example
cat f1 | sed '1,$s/(//g' | sed '1,$s/)//g' > f2
copies all lines from f1 to f2, removing all ( and ) brackets. sed is used twice in the same pipeline and "/" is the search-replace delimiter.
![Page 23: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/23.jpg)
sort
Has many options and parameters for ascending or descending, string or numeric, field and column sort key selection etc. See sort(1) for further details. Example:
cat /etc/passwd | sort -t: +0n -1 | more
Gives paged output of passwd database, sorted by userid.
![Page 24: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/24.jpg)
uniq
Having sorted our input data we may find repeated lines. The following example only outputs unique lines once:
cat file.txt | sort | uniq
The next example precedes all unique output lines with a count showing how many times each line occurred:
cat file.txt | sort | uniq -c
![Page 25: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/25.jpg)
head and tail
Head copies the first part and tail copies the last part of stdin file, see head(1) and tail(1) manual pages.
Example:
head -5 index
displays first 5 lines of index file
![Page 26: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/26.jpg)
tr and colrm filters
tr – is used to translate characterse.g. convert a text from lower to upper case:
richard@saturn:~/misc$ cat keller_security | tr a-z A-Z"SECURITY IS MOSTLY A SUPERSTITION. IT DOES NOT EXIST IN
NATURE... LIFE IS EITHER A DARING ADVENTURE OR NOTHING."-- HELEN KELLER
colrm – similar to cut but filters based only on character positions, and doesn't understand field delimiters.
richard@saturn:~/misc$ cat keller_security | colrm 36"Security is mostly a superstition.either a daring adventure or nothin-- Helen Keller
![Page 27: Unix Files, IO Plumbing and Filters](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815103550346895dbf20c7/html5/thumbnails/27.jpg)
Conclusions
Redirections of input, output and error files combined with the use of some filter programs allows for us to manipulate the text data used for system configurations. We will be able to extract needles of security information from the system log data haystacks.
Combining a few powerful one liners into short scripts will allow for fully customised administration and security monitoring of open-source servers, routers and firewalls.