introduction to hydra, · hydra basics & specifications the hydra hpc cluster is made of 80...
Post on 12-Aug-2020
18 Views
Preview:
TRANSCRIPT
IntroductiontoHydra,theIFT-UAM/CSICclusterMANUELTRASHORRAS,IFT/UAM-CSIC
SCHOOLOFCOSMOLOGYTOOLS, MARCH 2017
Contents
1. WhatisHYDRA?
2. Hydrabasics&specifications
3. Login&changepasswordsinHYDRA
4. WhatisSLURM?
5. UsingSLURMandsubmittingjobs
6. Example:CompilingandrunningCosmoMC
So,whatisHYDRA?Notthis…
So,whatisHYDRA?But this.
HYDRAbasics&specifications
TheHydraHPCclusterismadeof80dualprocessorsof8or12corespernode.
• batch –>34nodestimelimit:140hnodes hydra[1-34]
• batch2 –>18nodestimelimit:140hnodes hydra[35-52]
• batch3 –>18nodestimelimit:70hnodeshydra[53-70]
HYDRAbasics&specifications
TheHydraHPCclusterismadeof80dualprocessorsof8or12corespernode.
• batch –>8cores/node34nodeswithadualprocessorIntel® Xeon® E5540at2,53GHz,24GBDDR3RAM&120GBSolidStateDiskharddrive.• batch2 –>12cores/node18nodeswithadualprocessorIntel® Xeon® E5645at2.4GHz,24GBDDR3RAM&120GBSolidStateDiskharddrive.• batch3 –>12cores/node18nodeswithadualprocessorIntel® Xeon® E5-2640at2.5GHz,64GBDDR3RAM128GBSolidStateDiskharddrive.
HYDRAbasics&specifications
Systemsoftwareandcompilers:◦ GNUcompiler5.2◦ Intelcompiler16◦ MKLsoftware16◦ OpenMPI software1.8.8
Defaultenvironment:GNU+autotools +OpenMPI
Softwareisaccessiblethroughamodulesystem,thatallowsuserstodynamicallymodifytheuser'senvironment,loadingandunloadingthedifferentmodules.
Thefollowingcommandareuseful◦ moduleavail –> showsthelistofavailablemodules◦ module list –>showsthelistofuser'sloadedmodules◦ module loadMODULENAME –>loadsaspecificmodule◦ modulehelp –>showshelp
HYDRAbasics&specificationsLOADEDMODULES
In/opt/ohpc/pub/moduledeps/gnu:◦ openmpi/1.8.8 (L)
In/opt/ohpc/pub/modulefiles:◦ autotools (L) gnu/5.2.0(L)◦ ohpc (L) prun/1.0 (L)
SOMEUNLOADEDMODULES
In /opt/ohpc/pub/moduledeps/gnu-openmpi:◦ fftw/3.3.4
In/opt/ohpc/pub/moduledeps/gnu:◦ getdist/0.2.7 gsl/1.16◦ hdf5/1.8.15 healpy/1.10.3◦ impi/5.1.1.109 mkl/16.0.0.109◦ mpi4py/2.0.0 mvapich2/2.1◦ numpy/1.9.2
In/opt/ohpc/pub/modulefiles:◦ intel/16.0.0.109
Login&changepasswordsinHYDRA
Tologin:
ssh –XUSERNAME@hydra.ift.uam-csic.es
Yourhomedirectoryis:
/home/USERNAME
Smalljobscanbecompiled&runinhydra0interactively.
Butforlargerjobs,alwaysusethesubmittingsystem,SLURM!
Login&changepasswordsinHYDRA
Tochangepasswords:
ssh –XUSERNAME@hydra.ift.uam-csic.es
>passwd
>Typeoldpassword
>Typenewpassword
Yourhomedirectoryis:
/home/USERNAME
Smalljobscanbecompiled&runinhydra0interactively.
Butforlargerjobs,alwaysusethesubmittingsystem,SLURM!
So,whatisSLURM?Notthis…
So,whatisSLURM?But this.
Source:https://slurm.schedmd.com/overview.html
UsingSLURM&submittingjobs
Tocheckthestatusofthecluster: sinfo
UsingSLURM&submittingjobs
Tocheckthequeues: squeue
UsingSLURM&submittingjobs
Tocheckthestatusofthecluster: sinfo
Tocheckthequeues: squeue
Tosubmitajob: sbatch JOBFILE example:sbatch script.sh
Tocancelajob: scancel JOBID example:scancel 123456
Tocheckallyourjobs: squeue –uUSERNAME example:squeue –umtrashorras
Tocancelallyourjobs: scancel –uUSERNAME example:squeue –umtrashorras
Tocheckaspecificjob: squeue –jJOBID example:squeue –j123456
UsingSLURM&submittingjobsEXAMPLESOFACOSMOMC JOB
cosmomc.sh
#!/bin/bash
#SBATCH-JJOBNAME jobname#SBATCH-o./stdout/job.%j.out createsoutput files#SBATCH-e./stdout/job.%j.err createserrorfiles#SBATCH--nodes1 totalnumberofnodes#SBATCH--ntasks 8 totalnumberofcores#SBATCH-time140:00:00 max.runninghours#SBATCH-pbatch selectsqueue“batch”
(prun)YOURCODEHERE executethecode
cosmomc.sh
#!/bin/bash
#SBATCH-JJOBNAME jobname#SBATCH-o./stdout/job.%j.out createsoutput files#SBATCH-e./stdout/job.%j.err createserrorfiles#SBATCH--nodes2 totalnumberofnodes#SBATCH--ntasks 16 totalnumberofcores#SBATCH-time140:00:00 max.runninghours#SBATCH-pbatch selectsqueue“batch”
(prun)YOURCODEHERE executethecode
Example:CompilingCosmoMCDOWNLOADINGCOSMOMC
1. Gotohttp://cosmologist.info/cosmomc/submit.html
2. Fillinyourdata(name,surnameande-mailaddress),andsubmit.
3. Openthee-mailyouhavereceived,andclickthelinkinit.
4. DownloadthelatestversionofCosmoMC:cosmomc_Jul15_1.tar.gz
Example:CompilingCosmoMCDOWNLOADINGTHEPLANCKLIKELIHOOD
1. Gotohttp://pla.esac.esa.int/pla/#cosmology&clickSIGNIN andREGISTER.
2. Createyourselfanaccount.Fillinyourname,surname,e-mailandsubmit.
3. Openthee-mailyouhavereceived,andcopytheauthenticationcode.
4. Openanothermailyouhavereceived,andcreateyourselfapassword.
5. Signinwithyournewaccount,anddownload:Thelikelihood code:COM_Likelihood_Code-v2.0_R2.00.tar.bz2Thedatafiles:COM_Likelihood_Data-baseline_R2.00.tar.gz(containshigh_l, low_l,lensing)
Example:CompilingCosmoMCCOMPILINGCOSMOMC&THEPLANCKLIKELIHOOD
*Exampleofworkingdirectory:/home/USERNAME/1. CopythePlancklikelihood code,Plancklikelihood dataandCosmoMC toyourpath.◦ COM_Likelihood_Code-v2.0_R2.00.tar.bz2◦ COM_Likelihood_Data-baseline_R2.00.tar.gz◦ cosmomc_Jul15_1.tar.gz
2. Uncompress thePlancklikelihood code,Plancklikelihood dataandCosmoMC.1. cd/home/USERNAME/2. tar-xvf COM_Likelihood_Code-v2.0_R2.00.tar.bz2 –––>whichcreatesafoldernamedplc-2.03. tar-xvf COM_Likelihood_Data-baseline_R2.00.tar.gz –––>whichcreatesafoldernamedplc_2.04. tar-xvf cosmomc_Jul15_1.tar.gz –––>whichcreatesafoldernamedcosmomc
Example:CompilingCosmoMCCOMPILINGCOSMOMC&THEPLANCKLIKELIHOOD
*CompilingthePlanklikelihood:
3. Addthistoyour.bashr fileorexecutebeforeyoucompile/runCosmoMC1. source/opt/ohpc/pub/compiler/intel/compilers_and_libraries_2016.0.109/linux/mpi/intel64/bin/mpivars.sh2. source/opt/ohpc/pub/compiler/intel/compilers_and_libraries_2016.0.109/linux/bin/compilervars.sh intel64
4. InstallthePlancklikelihood1. cd/home/USERNAME/plc-2.02. ./waf configure--install_all_deps --lapack_mkl_version=10.3
--lapack_mkl=/opt/ohpc/pub/compiler/intel/compilers_and_libraries_2016.0.109/linux/mkl./waf install
5. PutasymboliclinktothePlanckdatainCosmoMC data:1. cd/2. ln-s/home/USERNAME/plc_2.0/home/USERNAME/cosmomc/data/clik
Example:CompilingCosmoMCCOMPILINGCOSMOMC&THEPLANCKLIKELIHOOD
*CompilingCosmoMC:6. ModifytheCosmoMC Makefile
3. cd/home/USERNAME/cosmomc/source/4. viMakefile replaceinline7: MPIF90C?=mpif90–––> MPIF90C?=mpif90-f90=ifort
7. CompileCosmoMC3. makeclean4. make
8. Runatoychain(leaveitrunningforaminute,this isnotaparallelrun).Done!😎3. cd..4. vitest.ini replaceinline37: action=0 (test) –––> action=4(fullMCMC)5. ./cosmomc test.ini
Example:CompilingCosmoMCCOMPILINGCOSMOMC&THEPLANCKLIKELIHOOD
Example:SendingaCosmoMC jobEXAMPLESOFACOSMOMC JOB
cosmomc.sh
#!/bin/bash
#SBATCH-Jcosmomc_test jobname#SBATCH-o./stdout/job.%j.out createsoutput files#SBATCH-e./stdout/job.%j.err createserrorfiles#SBATCH--nodes1 totalnumberofnodes#SBATCH--ntasks 8 totalnumberofcores#SBATCH-time140:00:00 max.runninghours#SBATCH-pbatch selectsqueue“batch”
./cosmomc test.ini executethecode
cosmomc.sh
#!/bin/bash
#SBATCH-Jcosmomc_test jobname#SBATCH-o./stdout/job.%j.out createsoutput files#SBATCH-e./stdout/job.%j.err createserrorfiles#SBATCH--nodes2 totalnumberofnodes#SBATCH--ntasks 16 totalnumberofcores#SBATCH-time140:00:00 max.runninghours#SBATCH-pbatch selectsqueue“batch”
./cosmomc test.ini executethecode
Example:SendingaCosmoMC jobEXAMPLEOFAPARALELLJOB
cosmomc.sh
#!/bin/bash
#SBATCH-Jcosmomc_test jobname#SBATCH-o./stdout/job.%j.out createsoutput files#SBATCH-e./stdout/job.%j.err createserrorfiles#SBATCH--nodes1 totalnumberofnodes#SBATCH--ntasks 8 totalnumberofcores#SBATCH-time140:00:00 max.runninghours#SBATCH-pbatch selectsqueue“batch”export$OMP_NUM_THREADS=8
prun–n1./cosmomc test.ini executethecode
cosmomc.sh
#!/bin/bash
#SBATCH-Jcosmomc_test jobname#SBATCH-o./stdout/job.%j.out createsoutput files#SBATCH-e./stdout/job.%j.err createserrorfiles#SBATCH--nodes2 totalnumberofnodes#SBATCH--ntasks 16 totalnumberofcores#SBATCH-time140:00:00 max.runninghours#SBATCH-pbatch selectsqueue“batch”export$OMP_NUM_THREADS=16
prun2./cosmomc test.ini executethecode
Example:CosmoMC outputOutput files:test_1.txt,test_2.txt,…,test.inputparams,test.ranges,test.paramnames,test.likelihoods.
UsefullinksSLURM Guide: https://slurm.schedmd.com/quickstart.html
ModulesGuide: http://modules.sourceforge.net/
Runjobsinteractively: www.ift.uam-csic.es/hydra/quickstart_uHydra_interactive.pdf
Runjobsinparalell: www.ift.uam-csic.es/hydra/quickstart_uHydra_batch.pdf
CosmoCoffee wiki: http://cosmocoffee.info/
CosmoMCReadme: http://cosmologist.info/cosmomc/readme.html
PlanckReadme: http://cosmologist.info/cosmomc/readme_planck.html
PythonReadme: http://cosmologist.info/cosmomc/readme_python.html
GetDist GUIReadme: http://cosmologist.info/cosmomc/readme_gui.html
top related