minnesota supercomputing institute · © 2013 regents of the university of minnesota. all rights...

© 2013 Regents of the University of Minnesota. All rights reserved.

Minnesota SupercomputingInstitute


Introduction to Job Submission and Scheduling

Andrew Gustafson


Interacting with MSI Systems


Connecting to MSISSH is the most reliable connection method

Linux and Mac users can use the terminal command:ssh login.msi.umn.edu

Windows users will need to use an ssh capable program, like PuTTY or Cygwin.

SSH connections must first connect to login.msi.umn.edu. From there you can connect to other systems.

For graphical connections use NICE:www.nice.msi.umn.edu


MSI Computing Environment

MSI systems are primarily Linux compute clusters running CentOS.

Software is managed via a module system.

Jobs are scheduled via a queueing system.

Home directories are unified across systems.


Machine Architecture: Cluster

Source: http://en.wikipedia.org/wiki/Cluster_%28computing%29


○ Mesabi■ About 17,700 total cores, on Intel Haswell processors.■ 24 cores and 62gb per node in the large primary queues.■ Special queues with large memory (up to 1TB), and GPUs.■ Allows node sharing: good for both small and large jobs.■ mesabi.msi.umn.edu

○ Itasca■ About 9,000 total cores, on Intel Nehalem processors.■ 8 cores and 22gb per node in the large primary queue.■ Special queues with larger memory and 16 cores per node.■ itasca.msi.umn.edu

○ Lab Server■ About 500 total cores, on older hardware.■ For interactive, or small single node jobs.■ 8 cores and 15gb per node in the primary queue.■ lab.msi.umn.edu

Clusters at MSI


Clusters at MSI

Mesabi LabItasca

Login

First connect to login.msi.umn.edu,then connect to a cluster.

Must be on-campus, or using the VPN.https://it.umn.edu/virtual-private-network-vpn


Home Directories

Home directories are unified across all Linux systems.

Each group has a disk quota which can be viewed with the command: groupquota

Panasas ActivStor 14: 3.01PB storage, capable of 30 GB/sec read/write, and 270,000 IOPS


Loading SoftwareSoftware modules are used to alter environmental variables, in order to make software available. MSI has hundreds of software modules.

Description Command Example

See all available modules: module avail module avail

Load a module: module load module load matlab/2015a

Unload a module: module unload module unload matlab/2015a

Unload all modules: module purge module purge

See what a module does: module show module show matlab/2015a

List currently loaded modules: module list module list

Module Commands:


Job SchedulingOn MSI systems, calculations are performed within “jobs”. A job is a planned calculation that will run for a specified time length on a specified set of hardware.

There are two types of job:1. Non-interactive (vast majority)2. Interactive

The job scheduler front-end is called the Portable Batch System (PBS).

Jobs start in your home directory with no modules loaded.


Job ScriptsTo submit a non-interactive job, first make a

PBS job script.

Example:#!/bin/bash -l#PBS -l walltime=8:00:00,nodes=3:ppn=8,pmem=1000mb#PBS -m abe#PBS -M [email protected]

cd ~/program_directorymodule load intelmodule load ompi/intelmpirun -np 24 program_name < inputfile > outputfile


Job SubmissionTo submit a job script use the command:

qsub -q queuename scriptname

A list of queues available on different systems can be found here:https://www.msi.umn.edu/queues

Submit jobs to a queue which is appropriate for the resources needed.

Resources to consider when choosing a queue:● Walltime● Total cores and cores per node● Memory● Special hardware (GPUs, etc)


Job SubmissionTo view queued jobs use the commands:

qstat -u usernameshowq -w user=username

For detailed information:checkjob -v jobnumber

To cancel a submitted job use the command:qdel jobnumber


Interactive Jobs

Nodes may be requested for interactive use using the command:

qsub -I -X -l walltime=1:00:00,nodes=1:ppn=8,mem=2gb

The job waits in the queue like all jobs, and when it begins the terminal returns control.


Service Units (SUs)

Jobs on the high performance computing (HPC) systems consume Service Units (SUs), which roughly correspond to processor time.

Each research group is given a service unit allocation at the beginning of the year. To view the number of service units remaining use the command: acctinfo

If a group is using service units faster than the "fairshare target", then the group's jobs will have lower queue priority.


Simple Parallelization: BackgroundingMost easily done with single node jobs.

#!/bin/bash -l#PBS -l walltime=8:00:00,nodes=1:ppn=8,pmem=1000mb#PBS -m abe#PBS -M [email protected]

cd ~/job_directorymodule load example/1.0./program1.exe < input1 > output1 &./program2.exe < input2 > output2 &./program3.exe < input3 > output3 &./program4.exe < input4 > output4 &./program5.exe < input5 > output5 &./program6.exe < input6 > output6 &./program7.exe < input7 > output7 &./program8.exe < input8 > output8 &wait


Simple Parallelization: Job ArraysWorks best on Mesabi.

Template Job Script, template.pbs:#!/bin/bash -l#PBS -l walltime=8:00:00,nodes=1:ppn=8,pmem=1000mb#PBS -m abe#PBS -M [email protected]

cd ~/job_directorymodule load example/1.0./program.exe < input$PBS_ARRAYID > output$PBS_ARRAYID

Submit an array of 10 jobs:qsub -t 1-10 template.pbs


Minnesota Supercomputing Institute

The University of Minnesota is an equal opportunity educator and employer. This PowerPoint is available in alternative formats upon request. Direct requests to Minnesota Supercomputing Institute, 599 Walter library, 117 Pleasant St. SE,

Minneapolis, Minnesota, 55455, 612-624-0528.

Web: www.msi.umn.edu

Email: [email protected]

Telephone: (612) 626-0802

minnesota supercomputing institute · © 2013 regents of the university of minnesota. all rights...

Documents