High Performance Computing Systems for IU Researchers – An Introduction

IUB Wells Library 10-Sep-2012

Jenett Tillotson <jtillots@iu.edu>George Turner turnerg@iu.edu

High Performance – Systems <hps-admin@iu.edu>

Getting Started

• BigRed: http://kb.iu.edu/data/avjx.html

• Quarry: http://kb.iu.edu/data/avkx.html

Getting an account

• http://itaccounts.iu.edu/• Manage my IU computing accounts• create more accounts• choose "BigRed" or "Quarry”• Takes about 15 minutes• You will receive a welcome email

Logging in

• ssh bigred.teragrid.iu.edu• ssh quarry.teragrid.iu.edu• MOTD

Nodes

• BigRed: b1-b1050, s1c1b1-s19c3b14• Quarry-classic: b001-b140, q001-q140• Quarry-pg: p1-p230, pg1-pg230

Head nodes

• BigRed: b513 - b516• Quarry-classic: b001 - b004• Quarry-pg; p230

Wild West nodes

• BigRed: b509-b512

• Quarry: q005-q008

Compute nodes

• BigRed: b1-b508, b561-b1050

• Quarry: q009-q140, pg1-pg229

File systems

• Home directory• DataCapacitor (Lustre)

File systems : Home directory

• /N/u/{username}/BigRed• /N/u/{username}/Quarry• ${HOME}• 10GB quota• quota -v• Slow, limited, backed up

File systems : DataCapacitor (lustre)

• http://kb.iu.edu/data/avvh.html• IU users: /N/dc/scratch/{username}• Fast, unlimited, not backed up• Permanent project file space available

Softenv

• Modifies environment• ${PATH} and ${MANPATH}• softlist• soft add• .soft• resoft

Resource ManagerKeeps tracks of resources: nodes, jobs, queues

• BigRed – LoadLeveler• llsubmit• llq• llcancel• llclass

• Quarry – TORQUE• qsub• qstat –a –u $USER• qdel• qstat -Q

Queues : BigRed• LONG: 32 nodes/job, 64 nodes/user, 14 days wall clock time

• NORMAL: 256 nodes/job, 512 nodes/user, 2 days wall clock time

• SERIAL: 1 proc/job, 512 proc/user, 2 days wall clock time

• DEBUG: 4 nodes/jobs, 4 nodes/user, 15 minutes wall clock time, 1 idle job

• Limit of 768 jobs in all the queues per user

• Limit of 16 idle jobs per user (except for the DEBUG queue)

Queues : Quarry• long: 42 nodes/job, 14 days wall clock time, 50 jobs/user

• normal: 6 nodes/job, 7 days wall clock time, 500 jobs/user

• serial: 1 node/job, 12 hours wall clock time, 500 jobs/user

• debug: 4 nodes/job, 15 minutes wall clock time,2 jobs/user

• himem: 28 nodes/job, 14 days wall clock time, 50 jobs/user

• batch: default queue

• Limit of 16 idle jobs per user

Job Scripts

• http://kb.iu.edu/data/axpz.html

BigRed#!/bin/bash -l # @ step_name = step1 # @ initialdir = /N/u/jtillots/BigRed/myoutput# @ output = step1.out # @ error = step1.err # @ notification = always # @ notify_user = jtillots@indiana.edu # @ class = DEBUG# @ wall_clock_limit = 15:00 # @ account_no = NONE # @ queue /bin/datesleep 10 /bin/date

Quarry

#!/bin/bash -l#PBS -N step1#PBS -j oe#PBS -k o#PBS -m abe#PBS -M jtillots@indiana.edu#PBS -q debug#PBS -l nodes=1,walltime=15:00/bin/datesleep 10/bin/date

Job ids

• BigRed: s10c2b5.{jobid}.0

• Quarry: {jobid}.qm2

Scheduler : MoabDecides which jobs get run on what nodes at what time.• showq• by queue: -w class={queuename}• running jobs: -r• idle jobs: -i• blocked jobs: -b

• checknode• checkjob• showstart• Single-user mode

Job Priority

• mdiag -p• Fair share• XFactor• QOS• Backfill• showbf