Ian C. Smith*

Harvesting unused clock cycles with Condor

*Advanced Research ComputingThe University of Liverpool

Overview

what is Condor ? High Performance versus High Throughput Computing Condor fundamentals setting up and running a Condor Pool The University of Liverpool Condor Pool example applications

What is Condor ?

a specialized system for delivering High Throughput Computing a harvester of unused computing resources developed by Computer Science Dept at University of Wisconsin

in late ‘80s free and (now) open source software widely used in academia and increasing in industry available for many platforms: Linux, Solaris, AIX, Windows

XP/Vista/7, Mac OS

HPC vs HTC (1)

High Performance Computing (HPC) delivers large amounts of computing power over relatively short periods of

time (peak FLOPS ratings important)

can also provide lots of memory, large amounts of fast (parallel) storage

fairly exotic hardware, may need plenty of TLC

large capital outlay on hardware

need to run specialised parallel (MPI) codes to get the benefit (can run serial codes but these are a poor use of resources)

users run relatively small numbers of parallel jobs

essential for certain time-critical applications

HPC vs HTC (2)

High Throughput Computing (HTC) allows many computational tasks to be completed over a long period of time

(peak FLOPS ratings not so important)

users more concerned with running large numbers of jobs over a long time span than a few short burst computations

makes use of existing commodity hardware (e.g. desktop PCs)

small capital outlay on hardware possible

limited memory and storage available generally

mostly aimed at running concurrent serial jobs (although MPI and PVM are supported by Condor)

Types of Condor application

large numbers of independent calculations typically (“pleasantly parallel”)

data parallel applications – split large datasets into smaller parts and analyse independently biological sequence analysis

processing of census data

optimisation problems microprocessor design and testing

applications based on Monte Carlo methods radiotherapy treatment analysis

epidemiological studies

A “typical” Condor pool

Central manager

Submit/execute hostSubmit host

Execute hostsExecute hosts

A “typical” Condor pool

Central manager

Submit/execute hostSubmit host

Execute hostsExecute hosts

ClassAdsClassAds

ClassAdsClassAds

A “typical” Condor pool

Central manager

Submit/execute hostSubmit host

Execute hostsExecute hosts

Match Info

Match InfoMatch Info

Match Info

A “typical” Condor pool

Central manager

Submit/execute hostSubmit host

Execute hostsExecute hosts

JobsJobs

A “typical” Condor pool

Central manager

Submit/execute hostSubmit host

Execute hostsExecute hosts

Output

Output

ClassAds and Matchmaking

ClassAds are a fundamental part of Condor similar to classified advertisements in a paper “Job Ads” represent jobs to Condor (similar to “wanted” ads) “Machine Ads” represent compute resources in a Condor Pool

(similar to “for sale” ads) Condor central manager matches Machine Ads to Job Ads and

hence machines to jobs Job Ads are created using submit description files

Simple submit description file

# simple submit description file # (anything following a # is comment and is ignored by Condor)# this would be used for Windows XP based execute hosts

universe = vanillaexecutable = example.exe # what to runoutput = stdout.out$(PROCESS) # job`s standard outputlog = mylog.log$(PROCESS) # log job`s activitiestransfer_input_files = common.txt, myinput$(PROCESS).txt # input files neededrequirements = ( Arch=="Intel") && ( OpSys=="WINNT51" ) # what machines to run onqueue 2 # number of jobs to queue

Requirements and Rank

Requirements expression determines where (and when) a job will run e.g.

Rank is used to express a preference

Requirements = ( OpSys==“WINNT51” ) && # Windows XP OS wanted ( Arch==“Intel” ) && \ # Intel/compatible processor

( Memory >= 2000 ) && \ # want a least 2GB memory and( Disk >= 33554432 ) && \ # at least 32 GB of free disk

( HAS_MATLAB == TRUE ) && \ # must have MATLAB installed ( ( ClockMin > 1020 ) || \ # only run jobs after 5 pm OR ... ( ClockMin == 6 ) || ( ClockDay == 0) ) # at weekends

Rank = Kflops # run on machines with best floating point performance first

Job submission and monitoring[einstein@submit ~]$ condor_submit example.subSubmitting job(s).2 job(s) submitted to cluster 100.[einstein@submit ~]$ condor_q-- Submitter: submit.chtc.wisc.edu : <128.104.55.9:51883> : submit.chtc.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD1.0 sagan 7/22 14:19 172+21:28:36 R 0 22.0 checkprogress.cron2.0 heisenberg 1/13 13:59 0+00:00:00 I 0 0.0 env3.0 hawking 1/15 19:18 0+04:29:33 R 0 0.0 script.sh4.0 hawking 1/15 19:33 0+00:00:00 R 0 0.0 script.sh5.0 hawking 1/15 19:33 0+00:00:00 H 0 0.0 script.sh6.0 hawking 1/15 19:34 0+00:00:00 R 0 0.0 script.sh...96.0 bohr 4/5 13:46 0+00:00:00 I 0 0.0 c2b_dops.sh97.0 bohr 4/5 13:46 0+00:00:00 I 0 0.0 c2b_dops.sh98.0 bohr 4/5 13:52 0+00:00:00 I 0 0.0 c2b_dopc.sh99.0 bohr 4/5 13:52 0+00:00:00 I 0 0.0 c2b_dopc.sh100.0 einstein 4/5 13:55 0+00:00:00 I 0 0.0 cosmos

557 jobs; 402 idle, 145 running, 1 held[einstein@submit ~]$

Condor policies

Condor supports a wide range of policies for when to start jobs e.g. run jobs only outside office hours

run jobs only if load average on host is small and there has been no recent activity

run jobs at any time on one core (at low priority)

run jobs only submitted by certain users

also a wide choice of what to do when a job is about to be interrupted e.g. suspend the job for a limited time then let it resume

checkpoint the job and migrate it to another machine

kill off the job immediately

UNIX or Windows execute hosts ? (1)

UNIX Condor’s natural environment

not widely installed on desktop machines (but depends on institution...)

supports the Condor “standard universe” containing many useful features checkpointing allows jobs to be migrated from one machine to another without

loss of useful work

Remote Procedure Calls give transparent access to files on submit host

streaming of standard output (stdout) from jobs to submit host

Network filesystems work well making installation and configration much easier

leverages large amount of scientific and engineering codes which have been developed under UNIX

UNIX or Windows execute hosts ? (2)

Windows world’s most widely installed OS – rich source of execute hosts

many commercial 3rd party applications run on Windows

using shared (network) filesystems can be difficult under Condor

only supports the “vanilla” Condor universe no checkpointing – evicted jobs may waste a lot of cycles

all input and output files need to be transferred to/from execute host

output streaming not supported

may be difficult to port “legacy” UNIX codes (although Cygwin and Co-Linux can make life easier)

Windows support from the U-W Condor Team tends to lag behind UNIX

Setting up a Condor pool best to start off small and build up pool slowly need to understand Condor fundamentals:

role of Condor processes and how they interact

life-cycle of jobs

ClassAds and Matchmaking

avoid firewalls if possible (may be easier said than done ...) talk to central IT services (particularly network and PC teams) submit hosts may need to be fairly high spec if large numbers of

jobs are to be run - ideally want multi-core/processor machine (quad core at least)

plenty of memory (say 8 GB or more)

large fast access filestore (e.g. 1 TB RAID)

Where to go for help

Read The Fine Manual ! log files contain a lot of useful information take a look at the presentations, tutorials and “how-to recipes”on

the Condor website: (www.cs.wisc.edu/condor) search the condor-users mail list archive: (lists.cs.wisc.edu

/archive/condor-users) subscribe to the condor-users mail list join the Campus Grids SIG: (wikis.nesc.ac.uk/escinet/

Campus_Grids) commercial support is also available (e.g. Cycle Computing)

University of Liverpool Condor Pool

contains around 400 machines running the University’s Managed Windows Service (currently XP but moving to Windows 7 soon)

most have 2.33 GHz Intel Core 2 processors with 2 GB RAM, 80 GB disk, configured with two job slots / machine

single submission point for Condor jobs provided by Sun Solaris V445 SMP server

policy is to run jobs only if a least 5 minutes of inactivity and low load average during office hours and at anytime outside of office hours

job will be killed off if running when a user logs in to a PC web interface for specific applications support for running large numbers of MATLAB jobs

Condor service caveats

only suitable for DOS-based applications running in batch mode no communication between processes possible (“pleasantly

parallel” applications only) statically linked executables work best (although can cope with

DLLs) all files needed by application must be present on local disk

(cannot access network drives) shorter jobs more likely to run to completion (10-20 min seems to

work best) very long running jobs can accommodated using Condor

DAGMan or user level check-pointing (details available soon on the Condor website)

Running MATLAB jobs under Condor many users prefer to create applications using MATLAB rather

than traditional compiled languages (e.g. FORTRAN, C)

need to create standalone application from M-file(s) using MATLAB compiler

standalone application can run without a MATLAB license

run-time libraries still need to be accessible to MATLAB jobs

nearly all toolbox functions available to standalone applications

simple (but powerful) file I/O makes checkpointing easier

see Liverpool Condor website for more information

Power-saving and Green IT at Liverpool we have around 2 000 centrally managed classroom PCs across

campus which were powered up overnight, at weekends and during vacations.

original power-saving policy was to power-off machines after 30 minutes of inactivity, we now hibernate them after 15 minutes of inactivity

policy has reduced wasteful inactivity time by ~ 200 000 – 250 000 hours per week (equivalent to 20-25 MWh) leading to an estimated saving of approx. £125 000 p.a.

3rd party power management software (PowerMAN) prevents machines hibernating whilst Condor jobs are running

Condor’s own power management features allows machines to be woken up automatically according to demand

Condor-G and Grid Computing

Condor-G is an extension to Condor allowing job submission to remote resources using Globus

provides familiar Condor-like interface to users hiding the underlying middleware complexity

we have used Condor-G to give users grid access to a variety of HPC resources: local HPC clusters (UL-Grid)

NW-Grid resources at Daresbury Lab, Lancaster and Manchester

National Grid Service facilities

Grid Computing Server tools provide a batch environment similar to that of cluster systems (e.g. Sun Grid Engine)

Web portal removes the need for command line use completely

Radiotherapy example

3D model of normal tissue was developed in which complications are generated when ‘irradiated’ [1]

aim is to provide insight into connection between dose-distribution characteristics, different organ architectures and complication rates beyond that of analytical methods

code written in MATLAB and compiled into standalone executable set of 800 simulations took ~ 36 hours to run on Condor pool would require 4-5 months of computing time on a single PC several dozen sets of simulations have since been completed

[1] Rutkowska E., Baker C.R. and Nahum A.E. Mechanistic simulation of normal-tissue damage inradiotherapy—implications for dose–volume analyses. Phys. Med. Biol. 55 (2010) 2121–2136.

Personalised Medicine example

project is a Genome-Wide Association Study aims to identify genetic predictors of response to anti-epileptic drugs

try to identify regions of the human genome that differ between individuals (referred to as SNPs)

800 patients genotyped at 500 000 SNPs along the entire genome

test statistically the association between SNPs and outcomes (e.g. time to withdrawl of drug due to adverse effects)

very large data-parallel problem – ideal for Condor divide datasets into small partitions so that individual jobs run for

15-30 minutes batch of 26 chromosomes (2 600 jobs) required ~ 5 hours compute

time on Condor but ~ 5 weeks on a single PC

Epidemiology example

researchers have simulated the consequences of an incursion of H5N1 avian influenza into the British poultry flock [2]

Monte Carlo type method - highly parallel original code written in MATLAB and compiled

into standalone application individual simulations take only 10-15 minutes

to run – ideal for Condor require ~ 10 000 - 20 000 simulations per

scenario would have needed several years of compute

time on single machine, on Condor needed a few weeks

[2] Sharkey, K.J., Bowers R.G., Morgan K.L., Robinson S.E. and Christley R.M. Epidemiological consequences of an incursion of highly pathogenic H5N1 avian influenza into the British poultry flock. Proc. R. Soc. B 2008 275, 19-28

Further Information

http://www.liv.ac.uk/e-science/condori.c.smith@liverpool.ac.uk