introduction to unix/linux - hpc | usc · pdf fileintroduction to unix/linux ken-ichi nomura,...

Introduction to Unix/Linux

Ken-ichi Nomura, Ph.D.

Center for High-Performance Computing and Communications

Outline

1. HPCC Overview

2.  Basic Commands and Utilities

3.  Text Editor

4.  Directory, Software and Account Issues

HPCC clusters: -  HP SL230 (128GB memory), HP SL250

(64 GB memory, dual NVIDIA K20m GPUs), Dual Xeon 8-core 2.4 GHz, on 56.6-Gbit FDR Infiniband network

-  2,225-node, 4-core, 6-core, and 12-core dual-processor cluster contains Dell, Oracle Sun, HP, and IBM compute nodes on a 10-Gbit Myrinet network

-  4 large-memory nodes with 1 TB of RAM and 4x10-core Intel Xeon processors.

-  2 head nodes, hpc-login1.usc.edu and hpc-login2.usc.edu

HPCC Overview

Infiniband   Myrinet  

hpc-­‐login1  hpc-­‐login2  

HPCC  

Internet  

HPCC website: http://hpcc.usc.edu/

HPCC Overview

-  About -  Research -  User Support -  New User

Guide -  Frequently

Asked Questions

-  Contact Us

HPCC Website: HPCC Policies

5

http://hpcc.usc.edu/support/accounts/hpcc-policies/

HPCC Website: Account Application

6

http://hpcc.usc.edu/support/accounts/applying-for-a-hpcc-account/

Machine Room

Outline

1. HPCC Overview

2.  Basic Commands and Utilities

3.  Text Editor

4.  Directory, Software and Account Issues

What is Linux?

Command Line Interface

Graphical User Interface

Operating System (OS)

ls List file and/or directory names pwd Print working (current) directory mkdir/rmdir Create/remove directory cd Change directory chmod Change file or directory permission cp/mv/rm Copy/move/remove file or directory cat/more/less Display file contents chmod/chown change permission/ownership man Display online manual

Basic Commands

[~]$ ls docs hpc-23 Makefile R util workshop [~]$ ls –a … [~]$ pwd /home/rcf-12/knomura [~]$ mkdir tmp [~]$ cd tmp/ [tmp]$ cp ~/.bash_profile dot_bash_profile [tmp]$ cat dot_bash_profile [tmp]$ ls -l dot_bash_profile [tmp]$ chmod u-r dot_bash_profile [tmp]$ cat dot_bash_profile cat: dot_bash_profile: Permission denied [tmp]$ chmod u+r dot_bash_profile

Basic Commands: Example

[tmp]$ cp dot_bash_profile DOT_bash_profile [tmp]$ ls dot_bash_profile DOT_bash_profile [tmp]$ ls -l *_bash_profile -rw-r--r-- 1 knomura m-cacs 208 Oct 3 11:43 dot_bash_profile -rw-r--r-- 1 knomura m-cacs 208 Oct 3 11:47 DOT_bash_profile [tmp]$ mv dot_bash_profile Dot_Bash_Profile [tmp]$ ls –la … [~]$ cd .. [~]$ ls [~]$ pwd [~]$ rm -rv tmp/ removed `tmp/DOT_bash_profile' removed `tmp/Dot_Bash_Profile' removed directory: `tmp'

Basic Commands: Example

Shell (bash and tcsh) •  Shell is a command processor by which users tell what to do to

system. •  Command-line Interface (CLI) or Character user interface (CUI) •  To check which shell you are using, type echo $0 •  Auto logout after 20 minutes of inactivity

Bash Config Files •  Configuration files to setup user environment, for

examples command prompt, path, alias so on. Sometimes called “dot file”

•  .bash_profile and .bashrc are stored in each user’s home directory

•  When bash is invoked as a login shell, it first reads /etc/profile, if that file exists, then looks for .bash_profile and .profile

•  When bash is invoked as a non-login shell, only read .profile

Csh & Tcsh Config Files

•  .login & .cshrc are in each user’s home directory

•  When tcsh is invoked as a login shell, it reads first .tcshrc or, if .tcshrc is not found, .cshrc, then .history, then .login, and finally .cshdirs

•  When tcsh is invoked as a non-login shell, only read /etc/csh.cshrc and .cshrc

-  File and directory have ownership and permission -  Three types of permission, readable, writeable and

executable -  Each permission is given to three groups, owner, group and

others [~]$ ls -l .bash_profile -rw-r--r-- 1 knomura m-cacs 208 Sep 26 14:11 .bash_profile

r readable, w writable, x executable u user (owner), g group, o others, a all

Permission & Ownership

chmod Change file/directory permission chown Change file/directory owner

Example) chmod a+w file Add W permission to all users chmod o-x file Remove E permission from others chmod 750 file Add R/W/E permission to myself, R/E

permission to group no permission to others chown usr:grp file Change ownership of a file/directory

r(4) readable, w(2) writable, x(1) executable u(I) user (owner), g(II) group, o(III) others, a all

Permission & Ownership

Task 1: Change Permission and Run clock.sh

#!/bin/bash while true ; do echo -n " " date +"%r" sleep 1 done

1. Add executable permission to clock.sh

2. Type ./clock.sh 3. Terminate the script by Ctrl+c [~]$ ./clock.sh 02:21:02 PM 02:21:03 PM 02:21:04 PM

…

Useful Commands env display environmental variables echo/export display/set an environmental variable grep print lines matching a pattern head/tail show first/last several lines sort sort text ps/top display currently running jobs kill/killall terminate a process (not for PBS job) tar archive/unarchive files bc calculator

[~]$ env [~]$ echo $PATH [~]$ export PATH=$PATH:~/bin [~]$ env | grep PATH [~]$ env > env.txt [~]$ cat env.txt [~]$ head (or tail) env.txt [~]$ grep -n knomura env.txt [~]$ sort env.txt | grep -n knomura > env2.txt [~]$ ps -u knomura [~]$ ps -ef | grep knomura [~]$ top [~]$ kill 123456

Useful Commands: Example Archive multiple files/directories into one file, then compress it. [~]$ tar cvfz out.tar.gz file1 file2 dir1 Unzip and extract from an archived file. [~]$ tar xvfz output.tar.gz Display archived file contents [~]$ tar tvfz output.tar.gz

Special Characters/Commands ~ home directory . current directory .. parent directory * wild card > redirect output >> redirect output (append) <<, <<< here document/string | pipe `command` back quotes, replace `…` with

command output

tab Complete name of command, file or directory

clear/ctrl+l Clear terminal ctrl+a Go to the beginning of line ctrl+e Go to the end of line history Display command history éê Display command history ! Rerun a command in history

Special Characters/Commands

Outline

1. HPCC Overview

2.  Basic Commands and Utilities

3.  Text Editor

4.  Directory, Software and Account Issues

Text Editor: GNU nano

http://www.nano-editor.org/

Arrow-keys Move cursor Enter Change line CTRL+a Move to the beginning of line CTRL+e Move to the end of line CTRL+v Move forward one page CTRL+y Move backward one page

Text Editor: GNU nano

CTRL+o Save file CTRL+w Search text CTRL+d Delete a character CTRL+k Remove a line CTRL+u Paste buffer CTRL+x Save data and exit

Text Editor: GNU nano

Task 2: bash scripting 1 #!/bin/bash 2 3 # function definition 4 function pi { 5 6 # max number of RNG. use this for normalization. 7 RANDMAX=32767 8 9 # reset the number dot within a circle 10 in=0 11 for ((n=0; n<${1}; n++)); do 12 x=${RANDOM} 13 y=${RANDOM} 14 z=`echo "(${x}^2 + ${y}^2)/(${RANDMAX}*${RANDMAX})" | bc` 15 16 [ $z -lt 1 ] && let in+=1; 17 done 18 }

pi.sh

Task 2: pi.sh 20 # read total number of trial, number of procs 21 echo -n "type total number of trial: " 22 read total 23 24 # call the PI estimator 25 pi ${total} 26 27 echo "total trial=${total} PI = `echo ${in}/${total}*4 | bc -l`"

1. Fix pi.sh with text editor 2. Try 10, 50, 100, 1000 for the total number of trail

to see how the result improves

Outline

1. HPCC Overview

2.  Basic Commands and Utilities

3.  Text Editor

4.  Directory, Software and Account Issues

System Directories

/bin, /sbin executable files (commands) /etc system config files /home users home directories /lib, /lib64, /usr/lib64 library files /proc process information /tmp temporary space /usr/usc software repository

Directory and Quota

• Home directory /home/rcf-xx/username Each user has 1 GB of disk quota and 100,000 files of file quota •  Project directory /home/rcf-proj/projectid Quota on project directory varies depending on each project If you need more space, request from project page. https://www-rcf.usc.edu/rcfdocs/hpcc/allocations/

Monitoring Your Quota: myquota

----------------------------------------- Disk Quota for knomura ID 55302 Used Soft Hard Files 8146 100000 101000 Bytes 839.56M 4.30G 4.30G ----------------------------------------- Disk Quota for /home/rcf-proj/hpcc ID 419 Used Soft Hard Files 400792 1000000 1100000 Bytes 236.94G 400.00G 402.00G -----------------------------------------

myquota shows the quota on your home & project directories.

•  Program may crash due to the quota, being unable to write files in either home directory or project directory

•  Pay attention to the file quota (max number of files) also.

Account, Balance & core-hour •  To be able to submit your jobs to the HPCC cluster,

your account needs some balance (core-hour) •  Whenever your job finishes (successfully or

unsuccessfully), your account is charged by the number of processors used times the wallclock time the job spent

Example) If you request 2 nodes with 4 processors per nodes for 2 hours (-l nodes=2:ppn=4,walltime=2:00:00), the total charge is 2x4x2 = 16 core-hour.

Monitoring Your Balance: mybalance mybalance shows the current balance of your account [~]$ mybalance -h Balance Name ----------- ------ 11106267.27 lc_an 20000.00 lc_kn1

-  Job will not start if your balance is not enough. -  Submit a request from HPCC website for additional

core-hour. -  Make sure the ppn value matches with the number of

processors actually used. (Don’t use ppn=12 for single-core job!)

Software Repository: /usr/usc Many applications are available. Compilers: gnu, intel, pgi Numerical Libraries: mpich, openmpi, cuda, fftw, petsc Molecular Simulation: NAMD, gromacs, amber Quantum Chemistry: gaussian, schrodinger Numerical Environment: matlab, R, python

hpc-login1.usc.edu for 32-bit applications hpc-login2.usc.edu for 64-bit applications

Software Repository: /usr/usc How to use the installed software on HPCC: - Find application and version you want to use - Source setup script. There are two setup scripts, one for

bash the other for tcsh

Example) Matlab R2013a  

[~]$ ls /usr/usc/matlab/ 2009a 2009b 2010b 2011a 2013a 2013b default Scripts [~]$ source /usr/usc/matlab/2013a/setup.sh [~]$ matlab -nodisplay

< M A T L A B (R) > Copyright 1984-2013 The MathWorks, Inc.

R2013a (8.1.0.604) 64-bit (glnxa64) February 15, 2013

>>

System Software vs Repo Software •  Some software and libraries (e.g. gcc, python, fftw) come

with OS

•  Although command name is the same, the system software and repo software are often different (versions, libraries, developers). Make sure that you use what you want to use

•  which command shows the absolute path of a command

[~]$ which python /usr/bin/python [~]$ source /usr/usc/python/enthought/default/setup.sh [~]$ which python /usr/usc/python/enthought/default/bin/python