bash guide summary
TRANSCRIPT
Summary of BASH Guide
• This is a PERSONAL summary of Greg's BASH Guide.
• The official front page URL of original guide is http://mywiki.wooledge.org/.
BASH is,
• an acronym for "Bourne Again Shell". • command interperters. • a layer between system function calls and user.
BASH is NOT,
• your operating system. • your window manager. • your terminal.(but it often runs inside terminal) • control your mouse and keyboard. • configure your system. • activate your screensaver. • open your files when you double-click them. • involved in launching applications.
BASH is,
• ONLY an interface for you to execute statements, using BASH syntax.
• a (possibly interactive) command interpreter, acting as a layer between the user and the system.
BASH mode
• Interactive: A mode of operation where a prompt asks you for one command at a time.
• Script: A file that contains a sequence of commands.
Prompt types
• $ : BASH shell (POSIX, Korn shell, BASH) • % : C shell (csh, tcsh) • # : Running as the system's superuser account
Commands and Arguments
$ # This is comment.
$ # touch: $ # A COMMAND that changes the Last Modified time. $ # If the filename does not exist, $ # it simply creates that file. $ touch a b c $ ls
a b c
$ # There MUST be spaces between command and arguments. $ # The amount of whitespace between arguments $ # does not matter. $ echo a b c
a b c
Strings
• In BASH programming, almost everything is a string.
• When you type a command, the command's name is a string.
• Each argument is a string. • Variable name are strings. • The contents of variables are strings. • A filename is a string, and most files contains strings.
Types of Commands
• Aliases: A way of shortening commands. (only used in interactive shells, not in scripts)
$ ls -laF $ alias ll='ls -laF' $ ll
• Functions: Somewhat like aliases, but can be used in scripts.
$ sum () { echo "$1 $2"; } $ sum foo bar
foo bar
Types of Commands
• Builtins: Basic commands built into BASH
$ cd .. $ echo "echo is a bulitin"
• Keywords: Quite like builtins, but special parsing rules apply to them
$ [[ a < b ]]
Types of Commands
• Executables (Application): Also called an external command or application. Invoked by using a pathname.
$ ./myprogram
How to figure out the type?
• Command Types: aliases, functions, builtins, keywords, executables
$ type rm
rm is /bin/rm
$ type cd
cd is a function
Scripts
• A script is basically a sequence of commands.
• How to make a script: 1. Create a new file. 2. Put this in it at the top: #!/usr/bin/env bash
Shebang
• The top-most header of file. • Also called a sha-bang, hashbang, pound-bang.
• The way this header works: 1. When the kernel executes a non-binary, it looks at the first line of the file. 2. If the line begins with #!, the kernel uses the line to determine the interpreter that the code should be passed to.
#!What??
• #!<interpreter> <argument> This tries to run <interpreter> <argument> to read and run the rest of file.
• #!/bin/bash tells the kernel to run the program '/bin/bash', and give this script as an argument.
• #!/usr/bin/env bash tells the kernel to run the program, '/usr/bin/env', and give it two arguments, 'bash', and this script.
• Using '/usr/bin/env' is more potable. http://en.wikipedia.org/wiki/Env
sh is NOT bash
• Do not fooled by examples on the Internet that uses /bin/sh as interperter.
• sh is NOT bash.
• Refrain from giving scripts .sh extension. It serves no purpose, and it's completely misleading. (since it's going to be a bash scripts, not a sh scripts)
Creating a script
[myscript]
#!/usr/bin/env bash
echo "Hello, BASH"
[prompt] $ # Execute script using bash command. $ bash myscript
$ # Or, give your script executable permissions, $ # so you can execute the script as an application. $ chmod +x myscript $ ./myscript
Special Characters
• [whitespace] Whitespace (spaces, tabs and newlines). BASH uses whitespace to determine each word. The first word of each command is a command name. Any additional words become arguments to that command.
• $ Expansion character. This character is used in most substitutions, including parameter expansion.
• # Comment character. Any word beginning with # begins a comment.
Special Characters
• Quotes protect the text inside from being split into multiple words or arguments.
• 'text' (Single quotes) Prevent the all special characters.
• "text" (Double quotes) This permit substitutions to occur. Except for the $, prevent the special characters.
$ echo "$HOME"
/Users/ohgyun $ echo '$HOME'
$HOME
Special Characters
• \ Escape Character.
• ~ Shortcut for home directory.
• > or < Redirection characters. There character are used modify (redirect) the input and/or output of a command.
• | Pipelines allow you to send to output of one command as the input to another command.
Special Characters
• [[ expression ]] Test Expression. This evaluates the conditional expression.
• { command; } Command grouping. The commands inside the braces treated as though they were only one command.
• `command`, $(command) Command substitution. This executes the inner command first, and then replace the whole `...` or $(...0 with that command's standard ouptut.
Special Characters
• (command) Subshell Execution. This executes the command in a new bash shell.
• ((expression)) Arithemtic Command. Inside the parentheses, operator such as +,-,*,/ are seen as mathmatical operators.
• $((expression)) Arithematic Substitution. This expression is replaced with the result of its arithmetic evaluation.
Special Characters
• (command) Subshell Execution. This executes the command in a new bash shell. A subshell is a child process, and changes in subshell causes no effect on the current shell.
$ pwd
/Users/ohgyun $ (cd ..) $ pwd
/Users/ohgyun
Special Characters
• ((expression)) Arithemtic Command. Inside the parentheses, operator such as +,-,*,/ are seen as mathmatical operators.
• $((expression)) Arithematic Substitution. This expression is replaced with the result of its arithmetic evaluation.
$ echo $((5 + 5))
10 $ (( 5 > 0 )) && echo "True"
True
Parameters
• Parameters are a sort of named space in memory. • Parameters come in two flavors. - Special parameters: read-only, pre-set by BASH. - Variables: parameters you can create and update.
$ echo $$ # Special parameter that retreive the PID
70724
$ # Variable assignment $ # NO space between the = sign $ foo=bar $ echo $foo
bar
Parameters is "$XX"?
• Parameters do NOT start with a $-sign.
• The $-sign means 'Parameter expansion', and replaces the parameter to its content.
$ foo=bar $ # Parameter Name: foo $ # Usage: "$foo"
Parameter Expansion
• Parameter expansion is the substitution of a parameter by its value.
$ foo=bar $ echo "Foo is $foo" # Parameter expands to its value.
Foo is bar
$ # Remember? $ # Single quote does not expand parameters. $ echo 'Foo is $foo'
Foo is $foo
"Double Quote it!"
• Parameter expansion causes the $parameter to be replaced by its contents. Note the following!
$ song="My song.mp3" $ rm $song
rm: My: No such file or directory
rm: song.mp3: No such file or directory
• Because Bash replaced $song by its content, being My song.mp3; then it performed word splitting; and only THEN executed the command. It was as if you had typed this:
$ rm My song.mp3
"Double Quote it!"
• How do we fix this? Put double quotes around every parameter expansion!
$ rm "$song"
Special Parameters
• Parameters that aren't variables. • Parameter Name: 0 / Usage: "$0" Contains the name, or the path, of the script.
• Parameter Name: ? / Usage: $? Expands to the exit code of the most recently completed foreground command.
• Parameter Name: $ / Usage: $$ Expands to the PID of the current shell. (PID = process ID number)
• Parameter Name: ! / Usage: $! Expands to the PID of the command most recently executed in the background.
Special Parameters
• Parameter Name: 1 2 etc. / Usage: "$1" Positional Parameters contain the arguments that were passed to the current script or function.
• Parameter Name: * / Usage: "$*" Expands to words of all the positional parameters. It expands to a single string containing them all.
• Parameter Name: @ / Usage: "$@" Expands to words of all the positional parameters. It expands to a list of them all as individual words.
• Parameter Name: _ / Usage: "$_" Expands to the last argument of the last command that was executed.
Reserved variables
• The variables that the shell provides:
• BASH_VERSION: The version of Bash • HOSTNAME: The hostname of your computer. • PWD: Current working directory • PPID: Contains the PID of the parent process. • RANDOM: A random number between 0 and 32767. • UID: The ID number of the current user. • HOME: The current user's home directory. • PATH: A colon-seperated list of paths that will be searched to find a command.
• PS1: The format of your shell prompt. • TMPDIR: The directory for temporary files.
Variable Types
• Although Bash is not a typed language, it does have a few different types of variables.
• Array: declare -a variable The variable is an array of strings.
• Integer: declare -i variable The variable holds an integer. It automatically triggers Arithmetic Evaluation.
• Read Only: declare -r variable The variable can no longer be modifed or unset.
• Export: declare -x variable The variable will be inherited by any child process.
Variable Types
$ a=5; a+=2; echo $a; unset a
52 $ declare -i a=5; a+=2; echo $a; unset a
7 $ a=5+2; echo $a; unset a
5+2 $ declare -i a=5+2; echo $a; unset a
7 $ declare -r a=5 $ a=10
bash: a: readonly variable
${Parameter expansion}
• ${parameter} You can use curly braces to mark the beginning and the end of the parameter name.
$ foo=bar $ echo "'$foos', '${foo}s'"
'', bars
• ${#parameter} The length in characters of the value.
$ foo=abcde $ echo ${#foo}
5
${Parameter expansion}
• ${parameter:-word} Use default value, if 'parameter' is unset or null.
$ echo "'${xxx:-bar}'"
'bar' $ echo "'$xxx'"
''
• ${parameter:=word} Assign default value, if 'parameter' is unset or null.
$ echo "'${xxx:=bar}'"
'bar' $ echo "'$xxx'"
'bar'
${Parameter expansion}
• ${parameter:+word} Use alternate value, if 'parameter'is NOT unset or null.
$ echo "'${xxx:+bar}'"
'' $ xxx=foo $ echo "'${xxx:+bar}'"
'bar' $ echo "'$xxx'"
'foo'
${Parameter expansion}
• ${parameter:offset:length} Substring expansion. Expands to up to 'length' characters of 'parameter' starting at the 'offset'(0-indexed)
$ foo=abcde $ echo ${foo:1:3}
bcd $ echo ${foo:1}
bcde $ # If offset is negative (use parentheses!), $ # count backword from the end of 'parameter' $ echo ${foo:(-3):2}
cd
${Parameter expansion}
• Substring removal: Remove the value matching the 'pattern'.
• ${parameter#pattern} From the beginning, the shortest match is deleted.
• ${parameter##pattern} • From the beginning, the longest match is deleted.
• ${parameter%pattern} From the end, the shorted match is deleted.
• ${parameter%%pattern} From the end, the longest math is deleted.
${Parameter expansion}
$ foo="a b c d e"
$ # Pattern: '* ' $ echo ${foo#* } # greedy
b c d e $ echo ${foo##* }
e $ # Pattern: ' *' $ echo ${foo% *}
a b c d • echo ${foo%% *}
a
${Parameter expansion}
• ${parameter/pattern/string} The first match of 'pattern' replaced by 'string'.
• ${parameter//pat/string} Every match of 'pattern' is replaced.
$ foo="a a a" $ echo ${foo/a/x}
x a a $ echo ${foo//a/x}
x x x
Patterns
• Pattern matching serves two roles in the shell: 1. Globs: Selecting filenames within a directory. 2. Regular Expression: Pattern matching for strings, not for filename.
Globs
• Globs can be used to match filenames.
• Meta characters: * : Mathces any string, including the null string. ? : Mathces any single character. [...] : Mathces any one of the enclosed characters.
• Globs must match a WHOLE string.
• Use globs instead of ls to enumerate files
Globs
$ ls
a ab abc bc bcd $ echo *
a ab abc bc bcd $ echo b*
bc bcd $ echo ?
a $ echo ?b
ab $ echo *[c-z]
abc bc bcd
Extended Globs
• Bash also supports a feature called Extended Globs. • This feature is turend off by default. You can turn it on using shopt command. (shopt = shell options)
• Patterns: ?(pattern-list) : Matches zero or one occurence *(pattern-list) : zero or more +(pattern-list) : one or more @(pattern-list) : one !(pattern-list) : anything except (pattern-list is a list of items sperated by '|')
Extended Globs
$ shopt -s extglob # turn on extended globs
$ ls
aaab.jpg aab.jpg ab.jpg ab.txt $ echo a?(a)b*
aab.jpg ab.jpg ab.txt $ echo a*(a)b*
aaab.jpg aab.jpg ab.jpg ab.txt $ echo a+(a)b*
aaab.jpg aab.jpg $ echo a@(a)b*
aab.jpg $ echo !(*jpg)
ab.txt
Regular Expressions
• Regular Expression can only be used for pattern matching, not for filename matching.
• Bash supports the =~ operator to the [[ keyword. When the string matches the pattern, [[ returns with an exit code of 0 ("true"). Exit code 1 ("false") is returned if not matched.
Regular Expressions
$ regex='a(b)(c)' # put your regex in variable
$ [[ 'abc' =~ $regex ]] \ # use WITHOUT quotes $ && echo 'matched'
matched
$ # Capturing group assigned to BASH_REMATCH variable $ echo $BASH_REMATCH
abc $ echo ${BASH_REMATCH[1]} / ${BASH_REMATCH[2]}
b / c
Brace Expansion
• Brace expansion is used to generate lists of words.
• Globs only expand to actual filenames, but brace expansion will extpand to any possible permutaion of their contents.
Brace Expansion
$ echo {a,b,c} # NO spaces between words.
a b c
$ echo a{b,c}d
abd acd
$ echo {1..5} # Creating ranges
1 2 3 4 5
$ # Generate all possible combinations $ echo {A..C}{1..3}
A1 A2 A3 B1 B2 B3 C1 C2 C3
Exit status
• Whenever a command ends it notifies its parent of its exit status. This is represented by a number ranging from 0 to 255. This code is a hint as to the success of the command's execution
• Convention dictates that we use 0 to denote success, and any other number to denote failure of some sort.
$ # Make sure that your scripts always return a $ # non-zero exit code if something wrong in execution. $ rm file || { echo 'Something wrong!' >&2; exit 1; }
$ echo $? # Exit code of the last foreground process
Control Operators (&& and ||)
• && and || are used to link commands together. They check the exit code of the previous command to determine whether or not execute the next command. This concept is called conditional execution.
$ mkdir d && cd d
$ rm foo.txt || echo 'I couldn't remove the file'
Grouping statements
• You can group commands using culry brace.
$ # Use a semicolon to separate the commands $ { echo "A"; echo "B"; echo "C"; }
A
B
C
$ # Using multi-line $ cd not_exist || { $ echo "Directory does not exist' >&2 $ exit 1 $ }
Conditional Blocks
• if COMMAND; then COMMAND elif COMMAND; then COMMAND else COMMAND fi
$ if [[ $name = "foo" ]]; then $ echo "Hello, $name" $ elif [[ $name = "bar" ]]; then $ echo "Bonjour, $name" $ else $ echo "Who are you? $name" $ fi
Test commands
• [ or test is a normal command that reads its arguments and does some checks with them.
• [[ is much like [, but i offers far more versatility.
• Whenever you're making a BASH script, you should always use [[ rather than [.
Test commands
$ [[ -e FILE ]] # True if file exists $ [[ -f FILE ]] # True if file is a regular file $ [[ -d FILE ]] # True if file is a directory $ [[ -h FILE ]] # True if file is a symbolic link $ [[ -w FILE ]] # True if a writable by you $ [[ -r FILE ]] # True if redable by you $ [[ FILE -ot FILE ]] # True if the first file is $ # older than second $ [[ FILE -nt FILE ]] # newer than second $ [[ INT -eq INT ]] # True if both integers are equal $ [[ INT -lt INT ]] # less than $ [[ INT -gt INT ]] # greater than $ [[ INT -le INT ]] # less than or equal $ [[ INT -ge INT ]] # greater than or equal
Test commands
$ [[ STRING = STRING ]] # string comparison $ [[ STRING != STRING ]] $ [[ STRING = PATTERN ]] # glob pattern matching $ [[ STRING =~ REGEX ]] # regex pattern matching $ [[ EXPR -a EXPR ]] # logical AND $ [[ EXPR && EXPR ]] $ [[ EXPR -o EXPR ]] # logical OR $ [[ EXPR || EXPR ]] $ [[ ! EXPR ]] # logical NOT
Conditional Loops
• while COMMAND; do COMMAND
• done
• until COMMAND; do COMMAND done
• for VARIABLE in WORDS; do COMMAND
• done
• for (( EXPR; EXPR; EXPR )); do COMMAND done
While Loops
$ # inifinite Loop $ while true; do $ echo "inifinite loop" $ done
$ # Check your email every five minutes $ while sleep 300; do $ kmain --check $ done
$ # Wait for a host to come back online. $ until ping -c 1 -W 1 "$host"; do $ echo "$host is still unavailable" $ done
For Loops
$ # For loop with arithmetic expression $ for (( i=0; i<3; i++ )); do $ echo $i $ done
$ # For loop with brace expansion $ for i in {1..10}; do $ echo $i $ done
$ # For loop with glob pattern $ for file in *.mp3; do $ rm "$file" # wrap with quotes! $ done
Choices
• case VARIABLE in CASE) COMMAND ;; CASE) COMMAND ;; *) COMMAND ;; esac
$ case $LANG in $ en*) echo 'Hello!' ;; $ fr*) echo 'Salut!' ;; $ de*) echo 'Guten Tag!' ;; $ *) echo 'I cannot speak your language.' ;; $ esac
Arrays
• An array is a numbered list of strings
$ # =() syntax: $ names=("A" "B" "C" "D")
$ # You can also specify explicit indexes $ names=([0]="A" [1]="B" [20]="C")
$ # You can also get/set a item with index $ echo ${names[0]} $ names[3]="E"
$ # If you want to fill an array with filenames, $ # you can use globs in there. $ photos=(~/"my photos"/*.jpg)
Arrays
$ # Get filenames of current directory $ files=(*)
$ # Using arrays in for loop. $ # ${myarray[@]} will expand all elements $ for file in "${myfiles[@]}"; do $ cp "$file" /backup/ $ done
$ # you can merge items with delimiter $ # using IFS with "${arrayname[*]}" $ names=("A" "B" "C" "D") $ ( IFS=,; echo "Names: ${names[*]}" )
Names: A,B,C,D
Input and Output
• Input refers to any information that your program receives or reads.
• Bash script can come from: - Command-line arguments (positional parameters) - Environment variables, inherited from parents - Files - Anything else a File Descriptor can point to
Input and Output
• Output refres to any information that your program produces or writes.
• Output from a Bash script: - Files - Anything else a File Descriptor can point to - Command-line arguments to some other program - Environment variable passe to some other program
Command-line Arguments
• There are some special parameters available to every script which contain these arguments. These are called Positional Parameters.
• Positional Parameters: "$1", "$2", ... , "${10}", ... "$@" : The entire set of position parameters
The Environment
• Every progmra inherits information, resources, privileges and restrictions from its parent process. One of thoes resources is a set of variables called Environment Variables.
$ echo $PATH # All-capital letters
$ # If you want to put information into the evironment $ # for your CHILD processes to inherits. $ # Use the export command: $ export MYVAR=something
File Descriptos
• File Descriptors (in short:FDs) are the way programs refer to files, or to other resources that work like files (such as pipes, devices, sockets, or terminals).
• FDs are kind of pointers to sources of data, or places data can be written.
• When something reads from writers to that FD, the data is read from or written to that FD's resources.
• By default, every new process starts with open FDs: - Standard Input (stdin): File Descriptor 0 - Standard Output (stdou): File Descriptor 1 - Standard Error (stderr): File Descriptor 2
File Descriptos
• In an interactive shell, or in a script running on a termnial: FD 0 : the characters type on keyboard FD 1 : normal informations to terminal FD 2 : error informations to terminal
Redirection
• Redirection is used to change the data source or destination of a program's FDs. That way, you can send output to a file insted of the terminal, or have an application read from a file instead of from the keyboard.
$ echo "Hello" > file $ cat file
Hello command > file : Send stdout of command to file command 1> file : Same with above command < file : Use contents of file when command reads from stdin. command 0< file : Same with above
Redirection
$ # Redirect stderr to errors file $ rm not_exist 2> errors
$ # Append new data to end of file. $ rm not_exist 2>> errors
$ # Ignore error message. $ # /dev/null is a virtual divice that is always empty. $ rm not_exist 2> /dev/null
$ # >& syntax: $ # Duplicate FD1 and put this duplicate in FD2 $ grep keyword file not_exist_file > log 2>&1
Heredocs And Herestrings
• Heredocs are useful if you're trying to embed short blocks of multi-line data inside your script.
$ grep proud <<END # END = terminator string $ I am a proud sentence. $ END
• The most common use of Heredocs is dumping documentation to the user:
$ usage() { $ cat <<EOF # Terminator string can be any word $ usage: foobar [-xyz] [file ...] $ EOF $ }
Heredocs And Herestrings
• Herestrings are shorter, less instrusive and overall more convenient that Heredoc.
$ grep proud <<<"I am a proud sentence"
Pipes
• Pipe connects the stdout of one process to the stdin of another. (I/O Redirecting)
$ echo "I am proud sentence" | grep proud
• The pipe operator creates a subshell environment for each command.
$ foo=bar $ # Pipe creates a subshell and modifications in $ # subshell are lost. $ echo 'Foo changed' | read foo $ echo $foo
bar
Arithmetic Evaluation
$ (( a = (5+2)*3 )) $ if (($a == 21 )); then echo 'Blackjack!'; fi
Blackjack!
$ # Finding error log in input file $ flag=0 $ while read line; do $ if [[ $line = *err* ]]; then flag=1; fi $ done < inputfile $ if ((flag)); then echo "Find error!"; fi
Functions
$ sum() { $ echo "$1 + $2 = $(($1 + $2))" $ } $ sum 2 5
2 + 5 = 7
$ count() { $ local i # local variable $ for ((i=1;i<=$1;i++)); do echo $i; done $ } $ count 2
1
2