introduction to hydra, · hydra basics & specifications the hydra hpc cluster is made of 80...
TRANSCRIPT
![Page 1: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/1.jpg)
IntroductiontoHydra,theIFT-UAM/CSICclusterMANUELTRASHORRAS,IFT/UAM-CSIC
SCHOOLOFCOSMOLOGYTOOLS, MARCH 2017
![Page 2: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/2.jpg)
Contents
1. WhatisHYDRA?
2. Hydrabasics&specifications
3. Login&changepasswordsinHYDRA
4. WhatisSLURM?
5. UsingSLURMandsubmittingjobs
6. Example:CompilingandrunningCosmoMC
![Page 3: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/3.jpg)
So,whatisHYDRA?Notthis…
![Page 4: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/4.jpg)
So,whatisHYDRA?But this.
![Page 5: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/5.jpg)
HYDRAbasics&specifications
TheHydraHPCclusterismadeof80dualprocessorsof8or12corespernode.
• batch –>34nodestimelimit:140hnodes hydra[1-34]
• batch2 –>18nodestimelimit:140hnodes hydra[35-52]
• batch3 –>18nodestimelimit:70hnodeshydra[53-70]
![Page 6: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/6.jpg)
HYDRAbasics&specifications
TheHydraHPCclusterismadeof80dualprocessorsof8or12corespernode.
• batch –>8cores/node34nodeswithadualprocessorIntel® Xeon® E5540at2,53GHz,24GBDDR3RAM&120GBSolidStateDiskharddrive.• batch2 –>12cores/node18nodeswithadualprocessorIntel® Xeon® E5645at2.4GHz,24GBDDR3RAM&120GBSolidStateDiskharddrive.• batch3 –>12cores/node18nodeswithadualprocessorIntel® Xeon® E5-2640at2.5GHz,64GBDDR3RAM128GBSolidStateDiskharddrive.
![Page 7: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/7.jpg)
HYDRAbasics&specifications
Systemsoftwareandcompilers:◦ GNUcompiler5.2◦ Intelcompiler16◦ MKLsoftware16◦ OpenMPI software1.8.8
Defaultenvironment:GNU+autotools +OpenMPI
Softwareisaccessiblethroughamodulesystem,thatallowsuserstodynamicallymodifytheuser'senvironment,loadingandunloadingthedifferentmodules.
Thefollowingcommandareuseful◦ moduleavail –> showsthelistofavailablemodules◦ module list –>showsthelistofuser'sloadedmodules◦ module loadMODULENAME –>loadsaspecificmodule◦ modulehelp –>showshelp
![Page 8: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/8.jpg)
HYDRAbasics&specificationsLOADEDMODULES
In/opt/ohpc/pub/moduledeps/gnu:◦ openmpi/1.8.8 (L)
In/opt/ohpc/pub/modulefiles:◦ autotools (L) gnu/5.2.0(L)◦ ohpc (L) prun/1.0 (L)
SOMEUNLOADEDMODULES
In /opt/ohpc/pub/moduledeps/gnu-openmpi:◦ fftw/3.3.4
In/opt/ohpc/pub/moduledeps/gnu:◦ getdist/0.2.7 gsl/1.16◦ hdf5/1.8.15 healpy/1.10.3◦ impi/5.1.1.109 mkl/16.0.0.109◦ mpi4py/2.0.0 mvapich2/2.1◦ numpy/1.9.2
In/opt/ohpc/pub/modulefiles:◦ intel/16.0.0.109
![Page 9: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/9.jpg)
Login&changepasswordsinHYDRA
Tologin:
ssh –[email protected]
Yourhomedirectoryis:
/home/USERNAME
Smalljobscanbecompiled&runinhydra0interactively.
Butforlargerjobs,alwaysusethesubmittingsystem,SLURM!
![Page 10: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/10.jpg)
Login&changepasswordsinHYDRA
Tochangepasswords:
ssh –[email protected]
>passwd
>Typeoldpassword
>Typenewpassword
Yourhomedirectoryis:
/home/USERNAME
Smalljobscanbecompiled&runinhydra0interactively.
Butforlargerjobs,alwaysusethesubmittingsystem,SLURM!
![Page 11: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/11.jpg)
So,whatisSLURM?Notthis…
![Page 12: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/12.jpg)
So,whatisSLURM?But this.
Source:https://slurm.schedmd.com/overview.html
![Page 13: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/13.jpg)
UsingSLURM&submittingjobs
Tocheckthestatusofthecluster: sinfo
![Page 14: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/14.jpg)
UsingSLURM&submittingjobs
Tocheckthequeues: squeue
![Page 15: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/15.jpg)
UsingSLURM&submittingjobs
Tocheckthestatusofthecluster: sinfo
Tocheckthequeues: squeue
Tosubmitajob: sbatch JOBFILE example:sbatch script.sh
Tocancelajob: scancel JOBID example:scancel 123456
Tocheckallyourjobs: squeue –uUSERNAME example:squeue –umtrashorras
Tocancelallyourjobs: scancel –uUSERNAME example:squeue –umtrashorras
Tocheckaspecificjob: squeue –jJOBID example:squeue –j123456
![Page 16: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/16.jpg)
UsingSLURM&submittingjobsEXAMPLESOFACOSMOMC JOB
cosmomc.sh
#!/bin/bash
#SBATCH-JJOBNAME jobname#SBATCH-o./stdout/job.%j.out createsoutput files#SBATCH-e./stdout/job.%j.err createserrorfiles#SBATCH--nodes1 totalnumberofnodes#SBATCH--ntasks 8 totalnumberofcores#SBATCH-time140:00:00 max.runninghours#SBATCH-pbatch selectsqueue“batch”
(prun)YOURCODEHERE executethecode
cosmomc.sh
#!/bin/bash
#SBATCH-JJOBNAME jobname#SBATCH-o./stdout/job.%j.out createsoutput files#SBATCH-e./stdout/job.%j.err createserrorfiles#SBATCH--nodes2 totalnumberofnodes#SBATCH--ntasks 16 totalnumberofcores#SBATCH-time140:00:00 max.runninghours#SBATCH-pbatch selectsqueue“batch”
(prun)YOURCODEHERE executethecode
![Page 17: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/17.jpg)
Example:CompilingCosmoMCDOWNLOADINGCOSMOMC
1. Gotohttp://cosmologist.info/cosmomc/submit.html
2. Fillinyourdata(name,surnameande-mailaddress),andsubmit.
3. Openthee-mailyouhavereceived,andclickthelinkinit.
4. DownloadthelatestversionofCosmoMC:cosmomc_Jul15_1.tar.gz
![Page 18: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/18.jpg)
Example:CompilingCosmoMCDOWNLOADINGTHEPLANCKLIKELIHOOD
1. Gotohttp://pla.esac.esa.int/pla/#cosmology&clickSIGNIN andREGISTER.
2. Createyourselfanaccount.Fillinyourname,surname,e-mailandsubmit.
3. Openthee-mailyouhavereceived,andcopytheauthenticationcode.
4. Openanothermailyouhavereceived,andcreateyourselfapassword.
5. Signinwithyournewaccount,anddownload:Thelikelihood code:COM_Likelihood_Code-v2.0_R2.00.tar.bz2Thedatafiles:COM_Likelihood_Data-baseline_R2.00.tar.gz(containshigh_l, low_l,lensing)
![Page 19: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/19.jpg)
Example:CompilingCosmoMCCOMPILINGCOSMOMC&THEPLANCKLIKELIHOOD
*Exampleofworkingdirectory:/home/USERNAME/1. CopythePlancklikelihood code,Plancklikelihood dataandCosmoMC toyourpath.◦ COM_Likelihood_Code-v2.0_R2.00.tar.bz2◦ COM_Likelihood_Data-baseline_R2.00.tar.gz◦ cosmomc_Jul15_1.tar.gz
2. Uncompress thePlancklikelihood code,Plancklikelihood dataandCosmoMC.1. cd/home/USERNAME/2. tar-xvf COM_Likelihood_Code-v2.0_R2.00.tar.bz2 –––>whichcreatesafoldernamedplc-2.03. tar-xvf COM_Likelihood_Data-baseline_R2.00.tar.gz –––>whichcreatesafoldernamedplc_2.04. tar-xvf cosmomc_Jul15_1.tar.gz –––>whichcreatesafoldernamedcosmomc
![Page 20: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/20.jpg)
Example:CompilingCosmoMCCOMPILINGCOSMOMC&THEPLANCKLIKELIHOOD
*CompilingthePlanklikelihood:
3. Addthistoyour.bashr fileorexecutebeforeyoucompile/runCosmoMC1. source/opt/ohpc/pub/compiler/intel/compilers_and_libraries_2016.0.109/linux/mpi/intel64/bin/mpivars.sh2. source/opt/ohpc/pub/compiler/intel/compilers_and_libraries_2016.0.109/linux/bin/compilervars.sh intel64
4. InstallthePlancklikelihood1. cd/home/USERNAME/plc-2.02. ./waf configure--install_all_deps --lapack_mkl_version=10.3
--lapack_mkl=/opt/ohpc/pub/compiler/intel/compilers_and_libraries_2016.0.109/linux/mkl./waf install
5. PutasymboliclinktothePlanckdatainCosmoMC data:1. cd/2. ln-s/home/USERNAME/plc_2.0/home/USERNAME/cosmomc/data/clik
![Page 21: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/21.jpg)
Example:CompilingCosmoMCCOMPILINGCOSMOMC&THEPLANCKLIKELIHOOD
*CompilingCosmoMC:6. ModifytheCosmoMC Makefile
3. cd/home/USERNAME/cosmomc/source/4. viMakefile replaceinline7: MPIF90C?=mpif90–––> MPIF90C?=mpif90-f90=ifort
7. CompileCosmoMC3. makeclean4. make
8. Runatoychain(leaveitrunningforaminute,this isnotaparallelrun).Done!😎3. cd..4. vitest.ini replaceinline37: action=0 (test) –––> action=4(fullMCMC)5. ./cosmomc test.ini
![Page 22: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/22.jpg)
Example:CompilingCosmoMCCOMPILINGCOSMOMC&THEPLANCKLIKELIHOOD
![Page 23: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/23.jpg)
Example:SendingaCosmoMC jobEXAMPLESOFACOSMOMC JOB
cosmomc.sh
#!/bin/bash
#SBATCH-Jcosmomc_test jobname#SBATCH-o./stdout/job.%j.out createsoutput files#SBATCH-e./stdout/job.%j.err createserrorfiles#SBATCH--nodes1 totalnumberofnodes#SBATCH--ntasks 8 totalnumberofcores#SBATCH-time140:00:00 max.runninghours#SBATCH-pbatch selectsqueue“batch”
./cosmomc test.ini executethecode
cosmomc.sh
#!/bin/bash
#SBATCH-Jcosmomc_test jobname#SBATCH-o./stdout/job.%j.out createsoutput files#SBATCH-e./stdout/job.%j.err createserrorfiles#SBATCH--nodes2 totalnumberofnodes#SBATCH--ntasks 16 totalnumberofcores#SBATCH-time140:00:00 max.runninghours#SBATCH-pbatch selectsqueue“batch”
./cosmomc test.ini executethecode
![Page 24: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/24.jpg)
Example:SendingaCosmoMC jobEXAMPLEOFAPARALELLJOB
cosmomc.sh
#!/bin/bash
#SBATCH-Jcosmomc_test jobname#SBATCH-o./stdout/job.%j.out createsoutput files#SBATCH-e./stdout/job.%j.err createserrorfiles#SBATCH--nodes1 totalnumberofnodes#SBATCH--ntasks 8 totalnumberofcores#SBATCH-time140:00:00 max.runninghours#SBATCH-pbatch selectsqueue“batch”export$OMP_NUM_THREADS=8
prun–n1./cosmomc test.ini executethecode
cosmomc.sh
#!/bin/bash
#SBATCH-Jcosmomc_test jobname#SBATCH-o./stdout/job.%j.out createsoutput files#SBATCH-e./stdout/job.%j.err createserrorfiles#SBATCH--nodes2 totalnumberofnodes#SBATCH--ntasks 16 totalnumberofcores#SBATCH-time140:00:00 max.runninghours#SBATCH-pbatch selectsqueue“batch”export$OMP_NUM_THREADS=16
prun2./cosmomc test.ini executethecode
![Page 25: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/25.jpg)
Example:CosmoMC outputOutput files:test_1.txt,test_2.txt,…,test.inputparams,test.ranges,test.paramnames,test.likelihoods.
![Page 26: Introduction to Hydra, · HYDRA basics & specifications The Hydra HPC cluster is made of 80 dual processors of 8 or 12 cores per node. • batch –> 8 cores/node 34 nodes with a](https://reader034.vdocuments.us/reader034/viewer/2022042515/5fad30817ef8d1024678c767/html5/thumbnails/26.jpg)
UsefullinksSLURM Guide: https://slurm.schedmd.com/quickstart.html
ModulesGuide: http://modules.sourceforge.net/
Runjobsinteractively: www.ift.uam-csic.es/hydra/quickstart_uHydra_interactive.pdf
Runjobsinparalell: www.ift.uam-csic.es/hydra/quickstart_uHydra_batch.pdf
CosmoCoffee wiki: http://cosmocoffee.info/
CosmoMCReadme: http://cosmologist.info/cosmomc/readme.html
PlanckReadme: http://cosmologist.info/cosmomc/readme_planck.html
PythonReadme: http://cosmologist.info/cosmomc/readme_python.html
GetDist GUIReadme: http://cosmologist.info/cosmomc/readme_gui.html