thing and how do i use it? what is this...

22
what is this openmind thing and how do i use it? 7/31/14

Upload: others

Post on 04-Jul-2020

4 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

what is this openmind thing and how do i use it?

7/31/14

Page 2: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

Everyone go to: http://github.mit.edu/MGHPCC/OpenMind/wiki/

Page 3: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

1. Storage on mindhive and openmind2. Software on openmind3. Submitting jobs on openmind4. Leftovers

Getting started with openmind

Page 4: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

Mindhive storage

nklab11TB (98% full)

nklab247TB (39% full)

old_data11TB (95% full)

Page 5: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

Mindhive storage

nklab11TB (98% full)

nklab247TB (39% full)

old_data11TB (95% full)

(Check where things are actually located with “pwd -LP”)

Page 6: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

Mindhive storage mounted on openmind

nklab11TB (98% full)

nklab247TB (39% full)

old_data11TB (95% full)

You can still access the drives with /mindhive/nklab and /mindhive/nklab2All links to things stored on old_data are brokenold_data may be mounted at a later date, just not right now

Page 7: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

Openmind storage

Home Directory: /home/<<yourusername>>This has a 1GB maximum, but it is backed up. Use it for scripts if you want

User Directory: /om/user/<<yourusername>>No maximum for now, but it is not backed up. If you store things here mentally prepare yourself for it to disappear at any moment.Good to use as temp storage (or use /om/scratch) while you are running something on openmind, and then move the files to nklab2. In theory this will save some read/write time.

Page 8: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

Software on Openmind

Page 9: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

Software on OpenmindSoftware is through modulefiles! (just like on mindhive)

See what modules are available with: module avail

Load modules with add/load:module add mit/matlab/2013a

module add openmind/freesurfer/5.3.0

Page 10: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

Software on Openmind: what existsMatlab and freesurfer (and fsl) are there to get started.

From the openmind wiki:New software, requests for softwareSoftware that are generally useful to everybody and available through the OS can be requested by submitting an issue. For project specific or specialized software, please consider using modulefiles to install versioned software or use Vagrant virtual machines to maintain your own software stack.If you want to help create module files submit a ticket on the wiki to let scott know -- there is a group with different permissions into the module directories

Page 11: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

Submitting Jobs on Openmind

Page 12: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

Submitting Jobs on Openmind

SLURM queueing system● sbatch -- used to submit your jobs● squeue -- check the queue● scancel <<JOBID>> -- cancel your job● srun -- runs an interactive job

http://www.nccs.nasa.gov/primer/slurm/slurm.html#quickstarthttp://www.ceci-hpc.be/slurm_tutorial.html

Page 13: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

Submitting Jobs on Openmind

sbatch● Like qsub, you can write a script with resources specified

or specify everything from the command line● Have a little more control than qsub

Page 14: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

Submitting Jobs on Openmindsbatch● Like qsub, you can write a script with resources specified or

specify everything from the command line● Have a little more control than qsub

examples:sbatch --gres=gpu:1 --mem=16000 -n 4 my_gpu_script arg1 arg2

(specifies 1 gpu, 16GB RAM, and 4 nodes)sbatch submitMatlab

(runs the submitMatlab script on the next slide)

Page 15: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

Example slurm script to run a matlab script#!/bin/bash

#SBATCH --job-name=matlab

#SBATCH --output=MatlabJob-%j.out

#SBATCH --error=MatlabJob-%j.err

#SBATCH --mem=32000

#SBATCH --nodes=1

matlab -nosplash -singleCompThread -r "myMatlabScript, exit"

Page 16: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

● Run recon-all with -N1 -c1 and it should take ~8 hours if the system is not overloaded (1 node, 1 CPU per task)

● You can run pretty much as many as you want , (Satra ran ~2300 recons recently)

● Are you in a huge hurry? Run with -N1 -c5 and specify -openmp 5 for recon-all (but don’t run 2300 like this because it is inconsiderate)

Submitting Jobs on Openmind: Recons

Page 17: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

An example recon script#!/bin/bash

#SBATCH --job-name=myrecon

#SBATCH --output=myrecon%j.out

#SBATCH --error=myrecon%j.err

#SBATCH --cpus-per-task=1

#SBATCH --nodes=1

SUBJECTS_DIR=/home/jfeather/testFSDir/

recon-all -subject VWFA_01c -i /mindhive/nklab2/zeynep/VWFA/DICOMS/VWFA_01/TrioTim-35115-20140331-115625-687000/*-9-1.dcm -all

(Note, you might want to copy your dicoms to openmind from mindhive to save read write time)

Page 18: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

Submitting Jobs on Openmind: PBS

Mindhive uses a PBS batch system (qsub) and SLURM has some PBS compatibility

Try running your PBS jobs (using qsub) on openmind and they might work. But is it not guaranteed

Page 19: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

Leftovers● VNC? Not yet. There might be a node assigned to it later● ssh into a node to check your job (but also if you make a .log file

you can see the text that is output on the node’s screen)

Page 20: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

Random other things● VNC? Not yet. There might be a node assigned to it later● ssh into a node to check your job (but also if you make a .log file

you can see the text that is output on the node’s screen)● Interactive job with displays tunneled through -X

$ module add openmind/srun.x11/v1

$ srun.x11 -n1 -p om_interactive

Page 21: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

Random other things● VNC? Not yet. There might be a node assigned to it later● ssh into a node to check your job (but also if you make a .log file

you can see the text that is output on the node’s screen)● Interactive job with displays tunneled through -X

$ module add openmind/srun.x11/v1

$ srun.x11 -n1 -p om_interactive

● Vagrant -- set up a reproducible and portable development environment! So, for instance, when you are setting up a new project make a vagrant environment for it! Then every time you or someone else works on that project you have the same set of software! (I have a lot of excitement about this)

Page 22: thing and how do i use it? what is this openmindweb.mit.edu/jfeather/Public/KanwisherLabOpenmindTalk.pdf · 2014-07-31 · Vagrant -- set up a reproducible and portable development

Overall openmind new so software is still being added.

If you use it now then you can help decide the direction that it will go!

Probably everything will switch from mindhive to openmind but it will probably be a couple of years