introduction to linux - nih hpc systems
TRANSCRIPT
Introduction to Linux
Bash and Basic GNU/Linux and Unix Concepts
2/15/171
This class will….
2/15/17
} Get your toes wet. Unix and Linux are gargantuan topics that only come into focus with experience.
} Provide some basic concept information for users familiar with MacOS or Windows.
} Get you familiar with Linux commands.
} Get you started in understanding command line interfaces.
2
Class outline
2/15/173
} History} Kernel and shells} The bash shell} Files and directories} File ownership and permissions} Essential Linux commands with exercises} Processes} File transfer
History
Late 60’s through 1980’s} Unix is the result of Bell Labs
research (Dennis Ritchie, Brian Kerningham, Ken Thompson, et al). Originally written in assembly language.
} Unics (Unix) was named in contrast to MIT’s Multics operating system.
} Berkeley Software Distribution (BSD), or Berkeley Unix derived from Bell Labs’ work due to government monopoly agreements.
} Unix led to the BSD family of operating systems in the 1990’s.
2/15/174
History
Richard Stallman, in 1983} Started the GNU (GNU’s Not Unix!)
project} Open-sourced versions of standard
suite of Unix utilities found in BSD} GNU is also a software license} Utilities used in Linux, BSD-derived and
proprietary Unix operating systems} All commands in this lesson are from
GNU
2/15/175
What is “Open Source?”
2/15/17
} “Open source” refers to a family of software licenses where the source code is available to the public with little or no copyright restrictions
} BSD-style licenses make source code available to be distributed without restriction.
} GNU GPL (General Public License) style licenses require source code availability. Projects that include GPL code mustmake any alterations to that code available under the GPL.
6
HistoryLinus Torvalds, in 1991} Released the first version of his Linux
kernel. } Started as a study in processor
architectures while at the University of Helsinki, Finland, and to this day still has the authority on what gets included in the Linux kernel
} In 1992 adopted the GNU license and rapidly gathered developers
} Combined the GNU suite of utilities with a new operating system kernel (GNU/Linux)
2/15/177
History
} By the late 1990’s/early 2000’s GNU/Linux starts to gather main-stream adoption, especially in research and academic circles due to structural similarities with Unix and BSD
} Gains large market share of commercial servers
} Becomes usable for desktop adoption
} Present on gadgets (e.g. Android)
2/15/178
BSD: 1990s - Current
While Linux is great, the BSD family is also a solid choice for scientific use depending on application.
} FreeBSD} OpenBSD} NetBSD} Others…
2/15/179
Popular Linux Distributions} Red Hat Enterprise Linux} Fedora} CentOS} Debian} Ubuntu} Suse Linux} Linux Mint
2/15/1710
History in a chart:
2/15/1711
Linux in Science (why?)
} Popular due to shared functional legacy with Unix systems associated with research (Irix, SunOS/Solaris, etc.)
} Source code availability and semi-liberal licensing made it easy for researchers to adjust the kernel as needed.
} Community backing and “perfect- storm” of enthusiasm for the project led to critical mass of development (in contract to the BSD family)
} Licensing and well known Unix-style APIs make it easy for vendors of HPC equipment to write drivers for their hardware.
} Wide range of tools available for users (compilers, scientific libraries, debuggers, etc).
} Performance, functionality and portability2/15/1712
Concepts: Kernel
} Operating system “kernel” is the core software used to “talk” to computer hardware
} Is a core and modular system of drivers used to create a standardized environment for interfacing with hardware
} Kernel operates in its own memory or “kernel-space”
} Responsible for allocating memory and time to system and user processes as well as interacting with files.
2/15/1713
Your Shell
} On log-in, the system runs a shell} A shell is
} environment within which you will interface with the kernel via commands
} command line interpreter (similar to cmd on Windows)
} It determines the syntax for complex command-line operations and shell scripting
} The shell you’re using is called “bash,” the successor to the venerable “Bourne Shell” called “sh”
} BASH: “Bourne Again SHell”
2/15/1714
Various Shells
2/15/17
} sh – the original UNIX shell} bash – written as a replacement/extension
of sh} csh –shell for those who prefer the syntax
similar to the C programming language } tcsh – enhanced version of C shell} ksh – Korn shell developed in the early
1980’s, backward compatible with sh, but with some features of csh
} zsh – extended version of sh, developed in 1990
} dash – Debian Almquist shell, replaces sh
15
Linux accounts
2/15/17
} To access a Linux system, you need to have an account} A Linux account includes the following:- username and password- User ID (uid) and Group ID (gid)- a home directory, which is where you are placed by
default when you log in- a default shell
16
Using SSH to log in:
But First! Introducing OpenSSH:} SSH is the “Secure SHell”} All traffic over SSH is encrypted} Developed as a secure alternative to
RSH and Telnet} SSH supports a file-transfer subsystem} SSH can act as an encryption layer for
arbitrary network connections
2/15/1717
Logging in} MacOS:
} Finder -> Applications -> Utilities -> Terminal} Type this: ssh [email protected]
} At the prompt, enter your password
} Windows:} Launch PuTTY. Under “Host Name (or IP address), type:
} …and click “Open”} At the prompt, enter your password
2/15/1718
More on shells
2/15/17
} What shell am I in? "echo $SHELL" displays your default shell's full pathname: /bin/bash"echo $0" displays your current shell: bash
} $SHELL and $0 are shell expressions} more about this and shell variables later
} List of available shells on the system can be displayed by typing "chsh --list-shells"
} Use chsh command to change your default shell. WARNING! on Helix and Biowulf never change it to a shell that ends in LOCKED – you will lock yourself out of your account!
19
Shell preferences
2/15/17
} When you login, startup scripts are run to setup your environment
} For bash, you can customize your environment by adding or modifying environment variables and aliases in the .bashrc file in your home directory.
} Examples:alias list='ls'export PATH=$PATH:/scratch/myusernameexport EDITOR=/usr/bin/vimexport PS1="[\u@\h \w \# ]"set -o noclobber
20
Summary of Linux commands
awk allows manipulation of text
bg place suspended job into background
cat view contents of a file
cd change directory
chmod change permissions on a file/directory
cp copy a file
cut extract a field of data from text output
echo output text to the terminal or to a file
emacs text editor
fg bring suspended job to foreground
file display file type
find search for files
grep search a file or command output for a pattern
head view beginning of file
history display list of most recent commands
less scroll forward or back through a file
ln create a link to a file
ls list files in a directory
man view information about a command
mkdir make directory
more scroll through file a page at a time
mv change the name of a file (move)
nano/pico text editors
printenv display shell variables
ps show current process information
pwd print current working directory
rm delete or remove a file
rmdir delete or remove a directory
sed stream editor
sleep pause
sort perform a sort of text
stat display file status info
tail view end of the file
touch create an empty file or update timestamps
tr character substitution tool
uniq remove identical, adjacent lines
vi/vim text editor
wc print number of lines, words or characters
which shows full path of a command
whoami displays username
2/15/1721
Our cast! (of characters)
Character Name/Location\ Backslash (above the enter key)/ Slash (left of right shift key)` Back quote (left of the number 1, above the tab key)| Pipe (shift-\)[ and ] Brackets (left of the backslash){ and } Braces or “curly” brackets (shift-[ and shift-])< and > Angle brackets (left of the right shift key)~ Tilde (shift-`)! @ # $% ^ & *( )
Bang/exclamation mark, at sign, hash, ($) dollarpercent, caret, ampersand, asterisk/starleft and right parenthesis
2/15/1722
Linux Command Basics
2/15/17
} Linux commands are case-sensitivels is not the same as LS} Linux commands may have options that come after the
command that start with a “–” and followed by a letter or “- -” and a word:
$ ls –r$ ls --reverse} Linux commands may allow for arguments:
$ ls /scratch} You can run more than one command on the same line by
separating the commands with a semicolon (;)$ date; ls
} Most Linux commands have a manual page or help to describe how they can be used….more about this later!
23
Linux Command Example
2/15/17
$ ls –r /home/$USER
24
prompt
command
option
argument
Exercise #1: First commands[username@helix ~]$} Type "whoami", press Enter
} your user id} Type "pwd" press Enter
} your working directory
What do the following show?} echo $HOME } echo $USER} echo $PWD
} HOME, USER and PWD are three more examples of shell variables (like SHELL). These are evaluated with the '$' prefix
2/15/1725
Concepts: Files and ProcessesIn Linux, (almost) everything is either a file or a process. In other words, processes interact file files} Regular files: text, data, documents, …} Directories: special files that contain a
bunch stuff to organize other files} Devices: disks, video and audio
hardware, processors, memory, I/O (USB) ports} look in /dev : special files that facilitate
interactions with devices (hardware)} Processes: things that get executed by
a processor (CPU, core) and are "running" or "sleeping"} identified by a number in /proc
2/15/1726
More on Files
2/15/17
} Each file (and directory) has a name} The filename can contain letters, numbers and
special characters} Every file has a unique path to its location
} e.g.: /home/student2/read-write.txt} A filename MUST be unique within a
directory…though files with the same filename can exist in different directories
} Filenames are also case sensitive} "myfile" and "Myfile" are distinct filenames} In practice, avoid using similar names
} Filenames can be lengthy
27
More on Directories
2/15/17
} A directory is a special type of file that can hold other files
} The “working directory” is the directory with which your shell is currently associated…where you currently are! When you first login, you will normally be in your home directory, /home/username
} Use the pwd command to print working directory
} Special directory notations:} . refers to the current working directory} .. refers to the parent directory
} the parent directory of /home/username would be /home
28
Concepts: The File System
} Linux file systems are arranged in a tree structure. There is a single root node called '/'} Note to Windows users: there are no drive letters here.
} Mapping disks to file systems is transparent to the user. Any disk can have multiple file systems and any file system can be "mounted" in any directory. Normal user activity does not care about the hardware layout of disk storage devices
2/15/1729
Directory Structure/ root/bin bare essential commands/boot OS Kernels/dev hardware device drivers/etc system files, configuration/home user home directories/lib Libraries needed by the system/opt 3rd party applications/proc Running process IDs and other
hardware stuff/sbin administrative commands/tmp temporary space/usr operating system applications/var system written potentially volatile files
2/15/1730
File system layout
Common directory layout on a Linux system
Note that /var is frequently a mount-point to a separate file system. This is often true of /home, /tmp and /usr as well.
On Helix/Biowulf, /home is on a network file system as are data directories
2/15/1731
cd and ls commands} The “cd” command is used to change directory location. Without an
argument, “cd” takes you to your home directory} The “ls” command is used to list the files in a directory. Like many Linux
commands, it can take a number of “flags” as options to change the behavior of the command
$ cd /home/$USER$ ls$ cd /etc$ pwd$ ls –l$ cd ..$ pwd$ ls -l$ cd$ pwd
2/15/1732
Exercise #2: ls, cd & man commands} Type "cd" to get to your home directory} Try "ls –l"} Try "ls –a"} Try "ls –la"} How are the above outputs different?} Try "ls –l /data/classes/linux"The above shows how providing an argument to the ‘ls’ command displays the contents of a directory without first changing to the directory} Type "cd /scratch" to change to the /scratch directory} Try "ls –lt"} How is the output ordered?} Try "ls –help"
} "man" displays a user manualfor a command} Type "man ls"} Scroll with arrow keys} "q" to quit
2/15/1733
Users and Groups} Users are associated with a unique
user identification (UID) number that the system uses internally
} Users can be real people} Users can be system entities} Users can be herded via groups} Groups also are associated with a
unique group identification (GID) number by the system
} Groups allow multiple users to access/share the same files
2/15/1734
Ownership & Permissions
2/15/17
} Linux systems are multi-user environments that allow users to create files, run programs and share data.
} Files and directories have two types of ownership – the user and group. A Linux group consists of one or more users.
} Files and directories have three types of access permissions:read permission (r)write permission (w)execute permission (x)
} Every files and directory has permissions for three levels or entities of permissions:
a) user or ownerb) group (one or more users)c) others or world
35
Permissions tripletsEach triplet indicates the access permissions for that level – in the example below, the user/owner has read, write & execute permission, other group members only have read and execute permissions and all others have no access permissions.
-rwxr-x---.
2/15/1736
Long List Output Explained (a little)} From left to right:Unix permissionsHard linksOwnerGroup ownershipFile size in bytesModification dateName of fileSpecial Directories:. is the current working directory.. is the "parent" directory, one level "up"
$ ls -ladrwxrwx--- 104 patkus staff 110592 Jan 6 13:02 .drwxr-xr-x 2510 root root 196608 Jan 6 12:58 ..-rw-r--r-- 1 patkus patkus 1051 May 8 2012 ad-week-rwxr--r-- 1 patkus staff 239 May 11 2007 alias.pl-rw-r----- 1 patkus staff 1185 Jun 22 1998 bp.txt-rwxr-xr-x 1 root root 27320 Mar 29 2012 getpass.awk-rw-rw-r-- 1 susanc staff 20529 Aug 7 2009 httpd.conf-rwxr--r-- 1 root staff 136236 Sep 10 2013 memcondrwxr-x--- 2 patkus staff 4096 Jun 24 2010 miscdrwx------ 3 patkus staff 4096 Jun 24 2008 test-rwx------ 1 patkus staff 493 Feb 10 2009 unlock-rw-r----- 1 patkus staff 38 Oct 20 2010 world.c-rwxr-x--- 1 patkus staff 6703 Jan 8 2013 world.exe-rwxrwx--- 1 patkus staff 2350 May 22 2009 year.pllrwxrwxrwx 1 patkus staff 7 Jan 6 15:30 year2 -> year.pl
2/15/1737
Permissions described:
File Type: Permissions
- regular filed directoryl symlinkb block devicec character devicep named pipes socket
r readw writex execute
Special values:x executables or t executable and
setuid/setgid/stickyS or T setuid/setgid or
sticky, but not executable.
2/15/1738
Changing Permissions and Ownership
2/15/17
} Use "chmod" to change the file permissions:chmod [ugoa][+/-][rwx] filename
where u=owner, g=group, o=others or world and a=all three
For example, to provide group read access to a file:
$ chmod g+r myfile
Or to remove file access to another than the owner or group members (in other words, others):$ chmod o-rwx myfile
} The "chown" command is used to change file ownership and the "chgrp" command can change group ownership of a file. As a regular user, you can not change the ownership of a file, but you can change the group ownership if you are a member of the group to which you are changing the group ownership
} Use the –R option on any of the above to recursively make changes on a directory of files
39
Exercise #3 pre-exercise
2/15/17
} We’ll talk about some of these commands shortly, but you each need to make a copy of files needed for the rest of class.
} First go to your home directory$ cd /home/$USER} Make a directory called "LinuxClass" in your HOME and go into that
directory:$ mkdir LinuxClass$ cd LinuxClass} Copy the files from the exercise file to your directory:$ cp –r /data/classes/linux/* .$ ls -l} Create a shell variable to your class directory:$ export class=$PWD$ cd $class
40
2/15/1741
Exercise #3: File Permissions} Read Permissions } Execute PermissionsThe cat command displays contents of a file
$ cd /home/$USER/LinuxClass
$ cat read-write.txt
Change the read permission$ chmod u-r read-write.txt
$ cat read-write.txt
What happened? Now restore the read permission
$ chmod u+r read-write.txt
$ cat read-write.txt
$ cd /home/$USER/LinuxClassRun the myhostname file to see the system name$ ./myhostname
Change the file permissions:
$ chmod u-x myhostname$ ./myhostname$ chmod u+x myhostname$ ./myhostname
Change permissions on the directory dir-perms:
$ chmod u-x dir-perms$ ls dir-perms$ ls –l dir-perms
What happened and why?
Shell Variables
Show all currently assigned variables
Useful predefined and important variables
$ printenv
HOSTNAME=helix.nih.govTERM=xtermSHELL=/bin/bashHISTSIZE=500SSH_CLIENT=165.112.93.227 49886 22QTDIR=/usr/lib64/qt-3.3QTINC=/usr/lib64/qt-3.3/includeSSH_TTY=/dev/pts/286HISTFILESIZE=500USER=patkus…
HOSTNAME System hostname
USER Your Username
SHELL Your shell
HOME Home directory
PWD Current directory
PATH Command paths
2/15/1742
Shell Variable Scope
Variables are available only to your immediate shell environment by default.
Export makes variables available to subprocesses(like /usr/bin/printenv)
$ MYVAR="This is my var"$ echo $MYVAR$ printenv | grep MYVAR
Is MYVAR listed?
$ MYVAR="exported"$ printenv | grep MYVAR$ export MYVAR$ printenv | grep MYVAR
Or
$ export MYVAR2=“exported2”$ printenv | grep MYVAR
2/15/1743
Your PATH
Execution pathIn BASH, execution of a program happens when you enter the program name. Your PATH variable keeps you from having to enter the full path to the program
Setting your PATH
$ echo $PATH
$ which date$ which whoami$ which perl
$ PATH=/home/$USER:$PATH$ export PATH
Or
$ export PATH=/home/$USER:$PATH
$ echo $PATH
2/15/1744
Wildcards
2/15/17
* (asterisk) Match zero or more characters
$ ls bear*bears bears7 bears_chicago$ ls *bear*bears bears7 bears_chicago polarbears`
? (question mark) Match exactly one character
$ ls bears?bears7
45
Special Keys
2/15/17
Tab allows for command auto-completion
Arrow keys allow you to:
a) horizontal arrows: edit command without deleting and retyping everything
b) vertical arrows: browse your history of previously typed commands
Back quote
Single quote
Tab
Arrow keys
ESC
Ctrl
46
Ctrl-C and ESC
2/15/17
} If you get into a situation where you can’t seem to get back to a command line prompt, try pressing the Ctrl-C combination or the Esc key – often one of those will abort whatever you are currently doing.
} The Ctrl character is often denoted by the caret symbol:
47
^C press Ctrl and c(no shift)
Let’s go home!
2/15/17
} ~ is a special character (in bash) that is short-hand for your home directory (like $HOME)
} Several ways to get to your home directory:
} Ok to use ~ and $HOME as arguments with other commands:
$ cd ~$ cd $HOME$ cd /home/username$ cd /home/$USER$ cd
$ ls ~/tmp$ ls $HOME/LinuxClass48
Absolute and Relative paths
} The starting '/' in the directory argument explicitly spells out a pathname – specifying an absolute or full path
} No leading '/' means you are specifying a path that is relative to the current working directory.
$ cd /home/username
# Absolute path:$ cd /home/username/tmp
# Relative path:$ cd tmp
# Using ~These do the same:$ cd ~/tmp$ cd /home/username/tmp
2/15/1749
What is that file?
file aproposApropos will search documentation for keywords. If you don’t know what man page to look at, use apropos to search for potentially related material
Tells us what type of file it is:
e.g: text, executable, PDF, jpg, gzip, tar, directory, etc.
$ file read-write.txt
$ file world.exe
$ file examples
$ apropos editor
$ apropos "text editor"
2/15/1750
Cat and Echo
Use cat to display file contents to the terminal:
Use echo to output arbitrary text to the terminal:
$ cat bears$ cat bears7$ cat bears bears7
$ echo 'Hello World!'$ echo without single quotes$ echo 'output' > Myoutput
2/15/17
"cat" is short for concatenate. The "cat" command takes one or more files and concatenates their contents to standard output.
51
Output Redirection to Files
Redirect output:
Append files:
$ cat bears > Newbears$ cat Newbears$ cat bears bears7 > newbears$ cat newbears
$ echo 'Hi there!' > greeting$ cat greeting
$ echo 'Hi yourself!' >> greeting$ cat newbears >> greeting$ cat greeting
2/15/1752
Exercise #4: cat and echo
cat a file to view contents
Using echo
2/15/17
$ pwd$ cd /home/$USER/LinuxClass$ cat lions$ cat tigers$ cat bears$ cat lions tigers > animals$ cat animals$ cat bears >> animals$ cat animals
$ echo my name is Chris$ echo “my name is $USER” > myname$ cat myname$ echo Hello $USER >> myname$ cat myname
53
Creating and deleting files and directories
Using touch and mkdir Using rm to remove files and directories
To create an empty file, use the touch command:$ touch emptyfile
You can also create a file using an editor such as pico, nano, vi or emacs:$ pico penguinfile
To create a directory:$ mkdir Mydirectory$ mkdir 2012$ mkdir –p 2014/Jan/stats
To remove a file:$ rm emptyfile$ touch myFile$ chmod u-rwx myFile$ rm myFile$ rm –f myFile
To remove a directory:$ rmdir Mydirectory$ rm –r 2012rmdir only works if the directory is empty!
Dangerous:$ rm –rf *
2/15/1754
Exercise #5: Creating and deleting files
Creating a file and directory
Deleting a file and directory
2/15/17
$ cd /home/$USER/LinuxClass$ echo 'I love genomic research!' > science$ touch science_project$ mkdir scienceclass$ ls –ld science*
$ rm science*$ ls –ld science*What happened?$ rmdir scienceclass
55
Displaying/Editing Files
more, lesshead, tail
Text editors:
$ more colleges.txt
$ less colleges.txt
$ head colleges.txtprints out the first 10 lines by default. Can use the –n argument to specify the number of lines
$ tail –20 colleges.txtprints out the last 20 lines
Good simple editors:} pico (pine composer)} nano (pico clone)
More powerful and complex editors:} vim (vi-improved)} emacs (Editor MACroS)
2/15/1756
pico – a simple editor
2/15/17
The essentials:} Just start typing – can use arrow keys to position, backspace or
delete key to delete characters to the left} Keystrokes for basic commands at bottom of the screen} ^G – help screen (^C to exit help)} ^O – save the file} ^W – search for a string} ^X – exit pico$ pico --help
$ pico filename
57
Exercise #6: Edit a file using pico
2/15/17
$ cd /home/$USER/LinuxClass$ pico bashrc1) Using the RIGHT arrow key, position the cursor at the end of the first line,2) Use the Backspace key to remove the bracketed text from the first line and
then simply start typing $HOME after the colon3) Press ^W (to search), type PICOPATH and hit Enter – this should place you on
the last line4) Hit ^E to get to the end of the line5) Use the Backspace key to remove everything after the '=' sign and type
'/usr/bin/pico'6) Use the up & the right arrow keys to get to the @ on the 2nd line7) Backspace to remove <USERNAME> and type your username8) Use the down arrow key to get to the 3rd line9) Hit ^K to cut the 3rd line10) Hit the Up arrow to get to the 1st line & ^A to get to the start of the line11) Hit ^U to paste the text – the 3rd line should now be the first12) Hit ^X to exit – type Yes to save the file when prompted and hit Enter when
prompted for the name$ cat bashrc
58
Moving /Copying files
Move (mv) Copy (cp)
2/15/17
Move a file or rename a file
$ touch football$ ls –l football$ mv football volleyball$ ls –l *ball$ mkdir sports$ mv sports sportsNEW$ ls –ld sports*$ mv volleyball sportsNEW$ ls –la sportsNEW
Copy a file or directory with the cp command
$ echo 'Goal!!' > soccer$ cp soccer soccerball$ ls –la soccer*$ cp –p soccer soccerball$ ls –la soccer*$ cp –p soccer sportsNEW
Archival copy:$ cp –a 2012tax 2012save$ cp –pr 2012tax 2012save
59
Exercise #7: Moving/Copying Files
Move (mv) Copy (cp)
$ cd /home/$USER/LinuxClass$ touch baseball$ mv baseball hockey$ ls –la baseball$ ls –la hockey
Create a directory path using mkdir with the –p option:$ mkdir –p 2012/tax/forms$ mv 2012 2013$ ls –Rl 2012$ ls –Rl 2013
$ cp hockey icehockey$ mkdir hockeypuck$ mv icehockey hockeypuck$ cp –pr hockeypuck hockeystick$ ls –la hockey*$ ls –la *hockey$ cp 2013 2012What did you see?$ cp –r 2013 2012Archival copy:$ cp –a 2013 2014
2/15/1760
grep – pattern matching search of a file
grep – global/ regular expression/ print
2/15/17
$ grep cat nonsense.txt$ grep dog nonsense.txt$ grep –i dog nonsense.txt$ grep –v dog nonsense.*$ grep oc nonsense.txt$ grep ^oc nonsense.txt$ grep oc$ nonsense.txt
grep is a powerful tool. Use it !
$ grep --help$ man grep
61
find – where are my files?
2/15/17
$ find [path_to_search] [expression/options]$ find . –name "*.txt"$ find /home/$USER/LinuxClass –iname "capital*"$ find /home/$USER/LinuxClass –type f -mmin 20$ find /home/$USER/LinuxClass –type f -mmin -20$ find /home/$USER/LinuxClass –type f -mmin +20$ find /home/$USER/LinuxClass –type f –mtime 1$ find /home/$USER/LinuxClass –name "*.bak" –delete$ find . –name "*.txt" –exec ls \-la {} \;$ man find
find – search for files (subject to various criteria)
62
Exercise #8: Using find
2/15/17
} Search for files from the class directory as the root
} Use find} locate the file named 'colors'} locate the files that contain the word 'bear' where the
match is case insensitive (hint: -iname option)} locate the files that were modified LESS than 90 minutes
ago(hint: -mmin option)
} How many files did you find for each?
$ cd ~/LinuxClass$ pwd
63
wc – count lines, words and bytes in a file
wc (word count)
$ wc nonsense.txt19 95 505 nonsense.txt
Output shows the number of lines, words and characters in the file
Can use argument to only get one of the three values:
$ wc –l nonsense.txt$ wc –w nonsense.txt$ wc –m nonsense.txt
$ wc -help
2/15/1764
uniq – show or remove duplicate lines
2/15/17
} uniqshow either unique or duplicate consecutive lines in a file or output. Default behavior is to merge adjacent matching lines into one, but can be used to print just the matching lines or provide a count of matching lines…most effective with the sort command
$ uniq bears # supress consecutively repeated lines$ uniq –d bears # show only duplicate lines$ uniq –c bears # show a count of each unique line
65
Sorting
Sort command
sort and output to the terminal
$ cat baseball.txt$ sort baseball.txt$ sort –r baseball.txt$ sort –b -k2 baseball.txt$ sort –bn -k2 baseball.txt$ sort –bnr –k2 baseball.txt$ sort –help
2/15/1766
Pipes (redirect to other processes)Connect the output of one process to the input of another process.
'|'
remember, '>' redirects to a file
$ cat college1 | sort | uniq$ cat college2 | sort | uniq$ cat college1 college2 | sort | uniq -c
Write to a file at the end:
$ cat college1 college2 | sort | uniq |grep ^B > Colleges
2/15/1767
Exercise #9: sort, grep and redirection
2/15/17
} cd ~/LinuxClass} Look at the contents of two files, grocery1 andgrocery2 (use cat command)
} Concatenate the two files with cat command and then use the sort and uniq commands to get a list of sorted, unique items for the grocery list
} Now redirect the output to a file named grocery3} Use the wc command to determine how many
unique items are on the list (in the grocery3 file).} Use grep and wc to determine how many items in
the grocery3 list start with the letter 'c'
68
Exercise #9 continued
2/15/17
$ cat grocery1 $ cat grocery2$ cat grocery1 grocery2 | sort | uniq$ cat grocery1 grocery2 | sort | uniq | wc –l
You should have 32 items
$cat grocery1 grocery2 | sort | uniq > grocery3$ grep ^c grocery3
7 items start with the letter c
69
Other useful commands
2/15/17
} history – displays a history of commands} alias – list aliases or create a new oneExample:$ alias hi="history 20"
} cut – print out selected fieldsExample:$ cat famousdogs | cut –f1,4 –d:
70
Input, Output and Error
2/15/17
} Commands (processes) read from input and write to output. } Standard input is your keyboard by default} Standard output is your terminal screen by default} Standard error is also your terminal screen by default
} this is where error messages appear
71
$ echo foobar > mystuff$ echo foobar > /mystuffbash: /mystuff: Permission denied
Input, Output and Error (cont.)
2/15/17
} STDIN, STDOUT and STDERR have numbers associated with each:
STDIN 0 Standard inputSTDOUT 1 Standard outputSTDERR 2 Standard error
$ chmod ugo-r /scratch/Class/exercises/dogfile2
} Redirect STDERR to a file:$ grep dog $HOME/LinuxClass/exercises/dogfile* 2> errors$ grep dog $HOME/LinuxClass/exercises/dogfile* 2> /dev/null
} Redirect STDOUT and STDERR (both of them) to a file:$ grep dog $HOME/LinuxClass/exercises/dogfile* > out_errs 2>&1
or$ grep dog $HOME/LinuxClass/exercises/dogfile* &> out_errs
72
Combining a chain of commands
Read from a file and do some stuffDo some more stuffWrite to a different file
$ sort -r < foodfile | grep -i ^s > s-foods
} sort reads from a file; its output is piped to grep which then write to a file
} Most programs will read from standard input if no file is specified in arguments
2/15/1773
awk – text manipulation (if time allows)
2/15/17
} awk treats a line of input as numbered fields (words) separated by white space. Each field is referenced by its number like this: $1, $2, etc..
} It is a feature rich program, this is only the simplest, but most often used application
} Use only the fields you need
$ cat hare_tortoiseThe hare beat the tortoise handily.
} We tell awk to print the words in a different order:$ cat hare_tortoise | awk ‘{print $1,$5,$3,$4,$2,$6}’The tortoise beat the hare handily.
74
sed – stream editor
2/15/17
} sed allows you to modify text based on a pattern match
} The '/g' at the end indicates that the change is global.without it, only the first instance (per line) would be changed
$ cat hare_tortoiseThe hare beat my tortoise handily.
} replace 'beat' with 'defeated':$ cat hare_tortoise | sed 's/beat/defeated/g'The hare defeated my tortoise handily.
} We can use both awk & sed on the same command line:$ cat hare_tortoise | awk ‘{print $1,$5,$3,$4,$2,$6}’ | \sed 's/beat/defeated/g'
} The tortoise defeated my hare handily.
75
tr – translation: character substitution
$ echo 'I love Linux!' | tr "a-z" "A-Z"
I LOVE LINUX!
2/15/17
} tr lets you replace all instances of one character with a different one– often used to change case of letters
76
More Linux Command Basics…Quotes
2/15/17
} bash treats single, double and back quotes in commands differently} Contents within a pair of single quotes are used verbatim
} Contents within back quotes are treated as a command; the result is a string formed from the output of the command.
} can be assigned to a variable:
} Contents within double quotes allows shell variable evaluation
$ echo '$USER'$USER
$ echo "The home directory of $USER is $HOME"The home directory of user is /home/user
$ NOW=`date`; echo $NOWThu Nov 21 14:38:13 EDT 201477
ProcessesShow processes
Show your processes:$ ps$ ps –f # full format
$ sleep 5 # do nothing for five seconds
$ sleep 25 & # do nothing for 25 seconds in the background$ ps -f$ ps –f --forest
Show all processes:$ ps –e$ ps –ef --forest$ man ps
2/15/1778
More on Processes
2/15/17
} ^Z suspends a running job/process} bg resume a suspended job in the background. you get a
prompt and can type new command while the bg'd job runs} fg resume a suspended job in the foreground. continues to run
normally} ^C interrupts or kills the currently running process
} Warning: Backgrounded processes will die when you log out of your session unless you use something like nohup or screen.
Background and Foreground processes
} A command/job can be run in the background by adding ‘&’ to end of the command:
$ sleep 50 &[1]+ Done sleep 50
79
More on Processes
Suspend and Resumeprocesses
Killing a process
^Z suspends an active job
$ sleep 300^Z (process is suspended)
$ bg$ ps –f$ fg
$ sleep 300^Z$ bg$ psPID TTY TIME CMD6686 pts/0 00:00:03 bash8298 pts/0 00:00:00 sleep8299 pts/0 00:00:00 ps
find the PID of the process you want to kill
$ kill 8298
2/15/1780
Processes: terminate (kill) them
"kill" only requests that the program exit. Use a signal 9 to force it to exit$ sleep 300
^Z$ psPID TTY TIME CMD6686 pts/0 00:00:03 bash8298 pts/0 00:00:00 sleep8299 pts/0 00:00:00 ps
find the PID of the process to be terminated$ kill -9 8298
} The kill command is slightly misnamed, what it actually does is send a signal to a process
} Most signals are interpreted by the application receiving the signal, so behavior is consistent only by convention
} Signal 9 is the exception: it will terminate the process with extreme prejudice
2/15/1781
Processes: Make them Nicer
"nice" a process. Reduce the priority of your process to reduce its overall impact on the system. Valid values for n are between 0 (highest priority) and 19 (lowest priority).
$ nice –n 10 cat /dev/urandom > /dev/null &$ top –u `whoami`
…
$ renice –n 15 –p [pid of “cat” command]
Now kill the process:
$ ps –ef|grep urandom
$ kill -9 [pid of urandom process]
2/15/1782
Exercise #10: terminate process with "kill"
2/15/17
} Start a 'sleep' process that will run in the background for 300 seconds:
$ sleep 300
} Type ^Z to suspend the sleep process
} Type bg resume the sleep in the background
} Check that the process is running by using the ps command and note the pid, process identification number
} Using the kill command with the pid of the sleepprocess from the previous step, terminate the sleep process
} How can you check that the sleep process is gone?
83
uptime
2/15/17
} In order (left to right):} Current time} Length of time since last boot} Number of users currently logged on} The average system load for the past 1, 5 and 15 minutes
} load = number of running proceses} if load < number of cores, then the system is not yet stressed} Helix has 128 cores
} uptime -- summary of the system status$ uptime14:39:46 up 14 days, 7:00, 305 usersload average: 39.18, 40.68, 38.68
84
Who is doing what…using top
top makes continuous output about running processes$ top
top - 16:19:54 up 28 days, 9:07, 255 users, load average: 32.18, 32.79, 33.22
Tasks: 4749 total, 8 running, 4733 sleeping, 7 stopped, 1 zombie
Cpu(s): 9.6%us, 5.8%sy, 6.0%ni, 78.2%id, 0.2%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 1058656848k total, 955041356k used, 103615492k free, 79064k buffers
Swap: 67108856k total, 547376k used, 66561480k free, 89619996k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
120202 johanesb 39 19 235m 180m 1432 R 96.4 0.0 170:21.86 merlin
252158 liqingli 39 19 58496 26m 756 S 95.0 0.0 17141:15 moe
170176 bozser 33 13 407m 117m 2588 S 60.1 0.0 62:30.33 ascp
218983 jrussler 20 0 18532 4704 872 R 22.3 0.0 0:00.38 top
127988 elliottm 39 19 223m 3544 1064 S 16.8 0.0 782:02.42 sshd
198816 wenxiao 20 0 4280 792 416 D 14.0 0.0 24:50.19 gzip
2/15/17
press 'q' to quit85
Looking at file system (disk) Space 1.To see local file system space:$ df –lFilesystem 1K-blocks Used Available Use% Mounted on/dev/mapper/vg_helix-root
51403396 8502228 40289968 18% /tmpfs 529355640 2216 529353424 1% /dev/shm/dev/sda2 495844 180879 289365 39% /boot/dev/sda1 204580 33228 171352 17% /boot/efi/dev/mapper/vg_helix-tmp
51606140 973788 48010912 2% /tmp/dev/mapper/vg_helix-var
32253856 19349996 11265460 64% /var
2/15/1786
Looking at file system (disk) Space 2.To see numbers in human readable format:$ df –lhFilesystem Size Used Avail Use% Mounted on/dev/mapper/vg_helix-lv_root
50G 8.9G 38G 19% /tmpfs 505G 5.6M 505G 1% /dev/shm/dev/sda2 485M 142M 318M 31% /boot/dev/sda1 200M 256K 200M 1% /boot/efi/dev/mapper/vg_helix-lv_tmp
50G 613M 47G 2% /tmpAll filesystems, including network file systems:$ df –h
2/15/1787
Directory size
Estimate disk usage (du)
$ cd /home/$USER
Estimate a file size:$ du LinuxClass/pbs_user_guide.pdf
Summary:$ du -s LinuxClass
Summary in human-readable format:$ du -sh LinuxClass
Default output $ du
2/15/1788
Checking Quotas on Helix/Biowulf
checkquota} The checkquota command will query all network
storage devices to find the applicable quota(s) for you} This command is specific to Helix and Biowulf and is
not available to Linux in general since it relies on information that is site-specific to this infrastructure.
$ checkquotaMount Used Quota Percent Files/data: 94.2 GB 200.0 GB 47.12% 70424/home: 5.2 GB 8.0 GB 64.71% 133607mailbox: 347.9 MB 1.0 GB 33.98%
2/15/1789
File Transfer
SCP, SFTP and clients Clients
} SCP and SFTP are file transfer protocols that run over SSH, the same protocol that you used to log in
} They are very secure and encrypt both the log-in and content of any transfer
Linux/MacOS:} scp secure copy} sftp secure FTP} fuze-ssh (Linux only)Windows:} WinSCP} Filezilla} Swish
2/15/1790
Using OpenSSH (Linux/macOS)
scp: securely copy files$ scp [email protected]:/tmp/file ~
Recursive copy (whole directory)
$ scp –r [email protected]:/tmp/dir ~
Preserve time stamps $ scp –p [email protected]:/tmp/file ~
From local host to remote.
$ scp ~/file [email protected]:/tmp/
As usual
$ scp --help$ man scp
2/15/1791
Using OpenSSH (Linux/macOS)
sftp: secure file transfer protocol
$ sftp [email protected]> cd /tmpsftp> get fileFetching /tmp/file to file/tmp/file 100% 2048KB 2.0MB/s 00:00sftp> put file newfileUploading file to /tmp/filefile 100% 2048KB 2.0MB/s 00:00sftp> exit
$ man sftp
2/15/1792
Exercise #11: Using scp
2/15/17
} Open a new terminal window on your local machine (not Helix)
} Use scp to copy the file read-write.txt from Helix to your local system:
$ scp [email protected]:/data/classes/linux/read-write.txt .
} Advanced : copy a whole directory:$ scp -pr [email protected]:/data/classes/linux/examples .
93
Logging out
$ exit
2/15/1794
Shell Variables (skip)
Variable assignment From a file
Arbitrary assignment
$ MYWORLD=“Hello World”$ echo $MYWORLD
With program output$ RIGHTNOW=`date`$ echo $RIGHTNOWWed Jan 30 14:12:28 EDT 2014
$ FILE=`cat nonsense.txt`$ echo $FILE
$ echo $FILE | tr “ “ “\n” \|sort | uniq
2/15/1795
Review
2/15/17
} History – Linux Torvalds, 1991} Why Linux? Performance, functionality and portability} Bash shell & shell variables} Files and directories – permissions & ownership} Linux file system} Paths to files/directories} Basic Linux commands to create & access files & directories} pico editor} sort, grep & find} pipe & file redirection} processes} transferring data to and from a Linux system} cron
96
Resources} Linux Documentation Project: http://tldp.org/
} Introduction to Linux - A Hands on Guide} Bash Guide for Beginners
} Helix Web Site: http://helix.nih.gov
2/15/1797
Cron (skip)Cron: run a job whenever you want} crontab -l
} crontab –e
--
15 3 * * * ~/script.sh >> ~/output 2> ~/error.log—
Runs "script.sh" at 3:15AM every day of every week of every month of every year.
* * * * *} First number is the minute at which to run (0-59)} Second is the hour (0-23)} Third is the day of the month (1-31)} Fourth is the month (1-12)} Fifth is the day of the week (0-6), 0 is Sunday
2/15/1798
File Transfer via Helixdrive (skip)
Network drive (Windows) Network Drive (MacOS)
Helixdrive is available to users with Helix accounts:
} Open “Computer” from the start menu
} Click “Map Network Drive”
} Folder: helixdrive.nih.gov/username
} Click Finish
} If prompted, enter Helix username and password
} Menu Bar -> Go -> Connect to Server
} Server Address: smb://helixdrive.nih.gov/username
} Click “Connect”} Check “Registered User”} If prompted, enter Helix
username and password
2/15/1799
date & cal (skip)
2/15/17100
} date – prints the current date and time$ dateMon Apr 20 14:09:01 EDT 2015
$ date +“%D”04/20/15
} cal – print the calendar for the current month or entire year
$ cal 2015
Loops
“For” loops will traverse space-delimited data
Loop over the output of the seq command:
$ FILE=`cat foodfile`
$ for n in $FILE
> do
> echo $n
> done
$ for n in `cat foodfile`; do
> echo $n
> done
$ seq 1 10
$ t=0
$ for n in `seq 1 10`; do
> t=`expr $t + $n`
> echo $t
> done
$ echo $t
55
$ t=1
$ for n in `seq 1 10`; do
> t=`expr $t \* $n`
> done
2/15/17101
Loops, part 2Let’s combine contents of many named files into one$ cd /home/$USER/LinuxClass/loops
$ cat nih1
.
.
$ cat nih9
We can combine the contents of the 9 nih named files into one using a loop:
$ for i in `seq 1 9`
> do cat nih$i >> NIH-ALL
> done
$ cat NIH-ALL
2/15/17102
What is truth?
Equality: if, then, else, fi Existence
$ BAD=“good”$ if [ “$BAD” = “good” ]> then> echo “true”> fi
$ if [ “$BAD” = “no good” ]> then> echo “true”> else> echo “false”> fi
$ if [ -f $HOME/showvar ]
> then
> echo true
> fi
$ if [ -d $HOME ]
> then
> echo true
> fi
2/15/17103
Symbolic links (skip)Allows you to reference same file with different name or path –a symbolic link is a another file type.
$ ln –s /home/$USER/LinuxClass/examples/tmp/colors color-pairs $ ls –la color-pairs$ cat /home/$USER/LinuxClass/examples/tmp/colors$ cat color-pairs
2/15/17
$ ln –s <existing_file> <file_link>$ cat Capitals$ ln –s Capitals Capslink$ ls –l Cap*$ cat Capslink
104
Example:
WinSCP
• Start WinSCP
• Click “New”
• Enter the host name (i.e.: crash.cit.nih.gov)
• Fill in user name
• Leave password blank
• Click Login
• If this is the first time you’ve connected to this host, you’ll have to accept the host’s key
2/15/17105
WinSCP Interface
• Left window is your local workstation, right window is the remote host
• Drag and drop files
• Navigate like a traditional explorer interface
2/15/17106