thing and how do i use it? what is this...

what is this openmind thing and how do i use it?

7/31/14

Everyone go to: http://github.mit.edu/MGHPCC/OpenMind/wiki/

1. Storage on mindhive and openmind2. Software on openmind3. Submitting jobs on openmind4. Leftovers

Getting started with openmind

Mindhive storage

nklab11TB (98% full)

nklab247TB (39% full)

old_data11TB (95% full)

Mindhive storage

nklab11TB (98% full)

nklab247TB (39% full)

old_data11TB (95% full)

(Check where things are actually located with “pwd -LP”)

Mindhive storage mounted on openmind

nklab11TB (98% full)

nklab247TB (39% full)

old_data11TB (95% full)

You can still access the drives with /mindhive/nklab and /mindhive/nklab2All links to things stored on old_data are brokenold_data may be mounted at a later date, just not right now

Openmind storage

Home Directory: /home/<<yourusername>>This has a 1GB maximum, but it is backed up. Use it for scripts if you want

User Directory: /om/user/<<yourusername>>No maximum for now, but it is not backed up. If you store things here mentally prepare yourself for it to disappear at any moment.Good to use as temp storage (or use /om/scratch) while you are running something on openmind, and then move the files to nklab2. In theory this will save some read/write time.

Software on Openmind

Software on OpenmindSoftware is through modulefiles! (just like on mindhive)

See what modules are available with: module avail

Load modules with add/load:module add mit/matlab/2013a

module add openmind/freesurfer/5.3.0

Software on Openmind: what existsMatlab and freesurfer (and fsl) are there to get started.

From the openmind wiki:New software, requests for softwareSoftware that are generally useful to everybody and available through the OS can be requested by submitting an issue. For project specific or specialized software, please consider using modulefiles to install versioned software or use Vagrant virtual machines to maintain your own software stack.If you want to help create module files submit a ticket on the wiki to let scott know -- there is a group with different permissions into the module directories

Submitting Jobs on Openmind

Submitting Jobs on Openmind

SLURM queueing system● sbatch -- used to submit your jobs● squeue -- check the queue● scancel <<JOBID>> -- cancel your job● srun -- runs an interactive job

http://www.nccs.nasa.gov/primer/slurm/slurm.html#quickstarthttp://www.ceci-hpc.be/slurm_tutorial.html

Submitting Jobs on Openmind

sbatch● Like qsub, you can write a script with resources specified

or specify everything from the command line● Have a little more control than qsub

Submitting Jobs on Openmindsbatch● Like qsub, you can write a script with resources specified or

specify everything from the command line● Have a little more control than qsub

examples:sbatch --gres=gpu:1 --mem=16000 -n 4 my_gpu_script arg1 arg2

(specifies 1 gpu, 16GB RAM, and 4 nodes)sbatch submitMatlab

(runs the submitMatlab script on the next slide)

Example slurm script to run a matlab script#!/bin/bash

#SBATCH --job-name=matlab

#SBATCH --output=MatlabJob-%j.out

#SBATCH --error=MatlabJob-%j.err

#SBATCH --mem=32000

#SBATCH --nodes=1

matlab -nosplash -singleCompThread -r "myMatlabScript, exit"

● Run recon-all with -N1 -c1 and it should take ~8 hours if the system is not overloaded (1 node, 1 CPU per task)

● You can run pretty much as many as you want , (Satra ran ~2300 recons recently)

● Are you in a huge hurry? Run with -N1 -c5 and specify -openmp 5 for recon-all (but don’t run 2300 like this because it is inconsiderate)

Submitting Jobs on Openmind: Recons

An example recon script#!/bin/bash

#SBATCH --job-name=myrecon

#SBATCH --output=myrecon%j.out

#SBATCH --error=myrecon%j.err

#SBATCH --cpus-per-task=1

#SBATCH --nodes=1

SUBJECTS_DIR=/home/jfeather/testFSDir/

recon-all -subject VWFA_01c -i /mindhive/nklab2/zeynep/VWFA/DICOMS/VWFA_01/TrioTim-35115-20140331-115625-687000/*-9-1.dcm -all

(Note, you might want to copy your dicoms to openmind from mindhive to save read write time)

Submitting Jobs on Openmind: PBS

Mindhive uses a PBS batch system (qsub) and SLURM has some PBS compatibility

Try running your PBS jobs (using qsub) on openmind and they might work. But is it not guaranteed

Leftovers● VNC? Not yet. There might be a node assigned to it later● ssh into a node to check your job (but also if you make a .log file

you can see the text that is output on the node’s screen)

Random other things● VNC? Not yet. There might be a node assigned to it later● ssh into a node to check your job (but also if you make a .log file

you can see the text that is output on the node’s screen)● Interactive job with displays tunneled through -X

$ module add openmind/srun.x11/v1

$ srun.x11 -n1 -p om_interactive

Random other things● VNC? Not yet. There might be a node assigned to it later● ssh into a node to check your job (but also if you make a .log file

you can see the text that is output on the node’s screen)● Interactive job with displays tunneled through -X

$ module add openmind/srun.x11/v1

$ srun.x11 -n1 -p om_interactive

● Vagrant -- set up a reproducible and portable development environment! So, for instance, when you are setting up a new project make a vagrant environment for it! Then every time you or someone else works on that project you have the same set of software! (I have a lot of excitement about this)

Overall openmind new so software is still being added.

If you use it now then you can help decide the direction that it will go!

Probably everything will switch from mindhive to openmind but it will probably be a couple of years