Farming with Condor

Douglas Thain

thain@cs.wisc.edu

INFN Bologna, December 2001

Outline• Introduction

– What is Condor? Why Condor on the Farm?

• Components– Daemons, pools, flocks, ClassAds

• Short Example– Executing 1000 jobs.

• Complications– Firewalls, security, etc…

The Condor Project (Est. 1985)

Distributed systems CS research performed by a team that faces

– software engineering challenges in a UNIX/Linux/NT environment,

– active interaction with users and collaborators,– daily maintenance and support challenges of a distributed

production environment,– and educating and training students.

Funding -

NSF, NASA,DoE, DoD, IBM, INTEL,

Microsoft and the UW Graduate School

A Bird of Opportunity

CentralManager

Job

Job

Job

Job

Job

Busy

Busy

Idle

IdleJob

Job

“I have work.”

“I am idle.”

“I am idle.”

Over the course of a week, 80% of a desktop machine’s time is wasted.

The Condor Principle:

The Condor Corollary:

The owner isabsolutelyin charge!

The visitor must beprepared for the

unexpected!

Tricky Details• What if the user returns?

– Checkpoint the job periodically.

– Restart the job elsewhere from a checkpoint.

• What if the machine does not have your files?– Perform I/O via Remote System Calls

• These two features require that you link with the Condor C library.

• Can’t relink? You may still use Condor, but with some loss in opportunities.

Checkpointing

Job

Checkpoint

RestartJob

Remote System Calls

JobShadow

Disk

Remote System Calls

Just likehome!

The INFN Condor Pool

0

100

200

300

400

500

600

700

800

UW

Com

p S

ci

UW

Eng

inee

ring

U T

exas

Com

p S

ci

NA

SA

Am

es

INF

N

NC

SA

UW

Che

m E

ng

CE

RN

Chr

ous

Uch

icag

o C

omp

Sci

aqua

lene

.uio

.no

226 Condor Pools5576 Condor Hosts

Top 10 Condor Pools:

Back to the Farm

• The cluster is the new engine of scientific computing.

• Inexpensive to:– procure– expand– repair

The Ideal Cluster

• The ideal cluster has every node identical, in every way:– CPU– Memory– File system– User accounts– Software installation

• Users expect to be able to execute on any node.

• Some models (MPI) require perfectly matched nodes.

The Bad News• Keeping the entire cluster available for use is very

difficult, when users expect complete symmetry!• Software failures:

– Full disk, wild process, etc...

• Hardware failures:– Replace with exact match? (not best buy)– Replace with better hardware? (goes unused)

• Much better to query rather than assume state of the cluster.

High Throughput Computing

is a 24-7-365 activity.

FLOPY (60*60*24*7*52)*FLOPS

Why Condor on the Farm?• Condor is expert at managing very heterogeneous

resources for high-throughput computing.• Large clusters, despite our best efforts, will always be

slightly heterogeneous.– (It may not be in your financial interests to keep them

perfectly homogeneous.)

• Condor assists users in making progress, despite the imperfections of the cluster.– Few users *require* the whole identical cluster.

– The pursuit of cluster perfection is then an in issue of small throughput improvement, rather than 0 or max.

Basic HTC Mechanisms• Matchmaking - enables requests for services and offers to

provide services find each other (ClassAds).• Persistence - records are kept in stable storage -- any component

may crash and reboot.• Asynchronous API - enables management of dynamic

(opportunistic) resources.• Checkpointing - enables preemptive resume scheduling (go ahead

and use it as long as it is available!).

• Remote I/O - enables remote (from execution site) access to local (at submission site) data.

City Bird, Country Farm

• The lessons learned and techniques used in stealing cycles from workstations are just as important when trying to maximize the throughput of a homogeneous luster.

Outline• Introduction

– What is Condor? Why Condor on the Farm?

• Components– Daemons, pools, flocks, ClassAds

• Short Example– Executing 1000 jobs.

• Complications– Firewalls, security, etc…

Components• Condor can be quite complicated:

– Many daemons, many connections, many logs...

• The complexity is necessary and desirable:– Each process represents an independent interest:

• Machine requirements (startd)

• User requirements (schedd)

• System requirements (central manager)

• Explain the structure by working from the bottom up.

A Single Machine

condorstartd

Localpolicy

filediskRAMcpu

condormaster

keyboard

CentralManager

Size?Speed?Load?

User present?

Size?Avail?

Machine state and

policy.“Only run jobs submitted from Bologna or Milan.

Prefer jobs owned by thain.

Evict jobs that don’t fit in memory. “

administratoremail

“Some-thing is wrong!”

A Single Pool

condorstartd

diskRAMcpu

CentralManager

condorstartd

diskRAMcpu

condorstartd

diskRAMcpu

condorstartd

diskRAMcpu

condorstartd

diskRAMcpu

condorstartd

diskRAMcpu

Local Policy:

“I prefer thain” Local Policy:

“I don’t care.”

Local Policy:

“I prefer mazzanti”

Machine state and

policy.

Machine state and

policy.Global Policy:

“All things being equal, Bologna gets 2x as many machines as Milan.”

A Typical Pool

CentralManager

condorstartd

RAMcpu

Uniform Local Policy:

“All machines except #3 prefer mazzanti”

NFS / AFSServer

diskRAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpuGlobal Policy:

“All things being equal, Bologna gets 2x as many machines as Milan.”

Schedulers

CentralManager

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorschedd

Job

Job

Job

Job

Job

condorstartd

RAMcpu

condorschedd

Job

Job

Job

Job

Job

I havework.

I havework.

I amidle.

I amidle.

I amidle. Job Job

Job

JobJob

Job

Multiple Pools

INFN CentralManager

condorschedd

Job

Job

Job

Job

Job

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

Job Job

Job

JobJob

Job

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

Job Job

Job

JobJob

Job

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

Job Job

Job

JobJob

Job

UWCS CentralManager

Matchmaking• Each Central Manager is an introduction service

that matches compatible machines and jobs.• A simple language (ClassAds) is used to represent

everyone’s needs and desires.• The match is not binding contract -- each side is

responsible for enforcing its own needs.• If a central manager crashes, jobs will continue to

run, but no further introductions are made.

ClassAd Example

Job Ad:

Type = “Job”

Cmd = “cmsim.exe”

Owner = “thain”

Requirements =

(OpSys==LINUX) &&

(Memory>128)

Machine Ad:

Type = “Machine”

Name = “vulture”

OpSys = “LINUX”

Memory = 256

Requirements =

(Owner==“thain”)

Matchmaking with ClassAds

StartdStartdScheddSchedd

matc

h

Central Manager

Claim and execute

Match notification

JobAd

JobAd

I have work.

MachineAd

MachineAd

I am idle.

Execute again.…and again!

Placement vs. Scheduling• A Condor Central Manager suggests the

placement of jobs on machines, with the understanding that all jobs are ready to run.

• A Condor scheduler is responsible for executing a list of jobs with various requirements. It may order jobs according to the users requests.

• Neither component plans ahead to make a schedule or a reservation for execution -- it is assumed change is so frequent that schedules are not useful.

Can we Schedule?• Of course, schedule is important for users that

have strict time contraints.• Scheduling is more important to High-

Performance Computing (HPC) than High-Throughput Computing (HTC.)

• Scheduling requirements may be worked into Condor in one of two ways:– 1 - Users may share a single submission point.

– 2 - The administrator may periodically reconfigure policy according to a schedule established elsewhere.

Scheduling

CentralManager

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorschedd

Job

Job

Job

Job

Job condorstartd

RAMcpu

Job

Job

Job

Job

Job

I amidle.

I amidle.

I amidle. Job Job

Job

JobJob

JobMethod 1: All users share a schedd.

Method 2: Modify global policy when necessary.

8:00: All nodes prefer thain.

10:00: All nodes prefer mazzanti.

Outline• Introduction

– What is Condor? Why Condor on the Farm?

• Components– Daemons, pools, flocks, ClassAds

• Short Example– Executing 1000 jobs.

• Complications– Firewalls, security, etc…

How Many Machines?

% condor_statusName OpSys Arch State Activity LoadAv Mem

lxpc1.na.infn LINUX-GLIBC INTEL Unclaimed Idle 0.000 30axpd21.pd.inf OSF1 ALPHA Owner Idle 0.266 96vlsi11.pd.inf SOLARIS26 SUN4u Claimed Busy 0.000 256

. . . Machines Owner Claimed Unclaimed Matched Preempting

ALPHA/OSF1 115 67 46 1 0 1 INTEL/LINUX 53 18 0 35 0 0 INTEL/LINUX-GLIBC 16 7 0 9 0 0 SUN4u/SOLARIS251 1 1 0 0 0 0 SUN4u/SOLARIS26 6 2 0 4 0 0 SUN4u/SOLARIS27 1 1 0 0 0 0 SUN4x/SOLARIS26 2 1 0 1 0 0

Total 194 97 46 50 0 1

Submit the Job

• Create a submit file:• vi sim.submit

• Submit the job:• condor_submit sim.submit

Executable = sim

Input = sim.in

Output = sim.out

Log = sim.log

queue

Watch the Progress

% condor_q

-- Submitter: axpbo8.bo.infn.it : <131.154.10.29:1038> :

ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD

5.0 thain 6/21 12:40 0+00:00:15 R 0 2.5 sim.exe

Each job gets a unique number.

Status: Unexpanded, Running or Idle

Size of program image (MB)

Receive E-mail When DoneThis is an automated email from the Condor systemon machine "axpbo8.bo.infn.it". Do not reply.

Your condor job /tmp_mnt/usr/users/ccl/thain/test/sim 40exited with status 0.

Submitted at: Wed Jun 21 14:24:42 2000Completed at: Wed Jun 21 14:36:36 2000

Real Time: 0 00:11:54Run Time: 0 00:06:52Committed Time: 0 00:01:37. . .

Running Many Processes• The real benefit of Condor comes from

managing 1000s of jobs.

• First, get organized. Write a script to make 1000 input files.

• Now, simply adjust your submit file:

Executable = sim.exe

Input = sim.in.$(PROCESS)

Output = sim.out.$(PROCESS)

Log = sim.log

Queue 1000

What can go wrong?

• If an execution site crashes:– Your job will restart elsewhere.

• If the central manager crashes:– Jobs will continue to run, no new matches will be

made.

• If the submit machine crashes:– Jobs will stop, but be re-started when it reboots.

• The only way to lose a job is to throw away the disk on the submit machine!

Outline• Introduction

– What is Condor? Why Condor on the Farm?

• Components– Daemons, pools, flocks, ClassAds

• Short Example– Executing 1000 jobs.

• Complications– Firewalls, security, etc…

Firewalls• Why a firewall?

– Prevent all outside contact.– Prevent non-approved contact.– Carefully securing every node is too much work.

• What’s the problem?– A variety of processes comprise Condor.– A variety of ports must be used at once.– Submit and execute machines must communicate

directly, not through the CM.

The Firewall Problem

CentralManager

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorschedd

condorstartd

RAMcpu

Firewall

PrivateNetwork

PublicNetwork

Firewall Solution #1

CentralManager

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorschedd

condorstartd

RAMcpu

Firewall

PrivateNetwork

PublicNetwork

Allow ports 1000-1010.

Use only ports 1000-1010.

Firewall Solution #1• Pros:

– Easy to configure Condor.– Easy to configure firewall.– Machine remain a part of the pool.

• Cons:– Number of ports limits number of simultaneous

interactions with the node. (running jobs + queue ops + negotiations, etc.)

– More ports = more connections, less security

Firewall Solution #2

CentralManager

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

condorstartd

RAMcpu

Firewall

PrivateNetwork

PublicNetwork

condorschedd

ssh

Firewall Solution #2

• Pros:– Only port through router is ssh.

• Cons:– Pool is partitioned!– Users must manually submit to every pool that

is behind a firewall. (I.e. they won’t.)– No global policy possible.– No global management/status possible.

Network Address Translation

• Both solutions only work as long as the firewall simply drops packets it doesn’t like.

• If the firewall is a Network Address Translator (masquerade,) then only solution #2 works.

• Research in Progress: A Condor NAT that runs on the firewall and exports the pool to the outside world.

Security

• Current Condor security:– Authenticate via DNS.– Authorize classes of hosts for certain tasks.

• New Condor (6.3.X?) security:– Authenticate with encrypted credentials.– Authorize on a per-user basis.– Forward credentials to necessary sites.

Condor 6.2 Security• Authentication: DNS is queried for each incoming

connection in order to determine the name.• Authorization: Each participant permits a class of

hosts to perform certain tasks. At UW-CS:– HOSTALLOW_READ = *.wisc.edu, *.infn.it

• Hosts that may query the machine state.

– HOSTALLOW_WRITE = *.cs.wisc.edu, *.infn.it

• Hosts that may execute jobs, send updates, etc...

– HOSTALLOW_OWNER= $(FULL_HOSTNAME)

• Hosts that may cause this machine to vacate.

– HOSTALLOW_ADMINISTRATOR= condor.cs.wisc.edu

• Hosts that may change priorities, turn Condor on/off

Condor 6.3.X? Security• Principle: No single security mechanism is

appropriate for all sites. Condor must have many tools. – United States Air Force:

• Kerberos authentication, all connections encrypted

– Cluster behind a firewall:• Host authentication, no encryption

– Grid Computing:• GSI credentials from certain authorities, encryption is up to

the user.

Condor 6.3.X Security

CentralManager

Datastorage

condorstartd

RAMcpu

condorschedd

RAMcpu

Disk

Execute

I/OKR

B 5 ?

NO GS

I ?

YE

S!

Submit

GSI ?

YES!

FORWARD CERT GSI

You don’thave to be a

super personto do

super computing!

Getting Condor

• Condor Home Page– http://www.cs.wisc.edu

• Binaries are freely available.

• Versions:– 6.2.x - Stable releases, bug fixes only– 6.3.x - Development releases

For More Info

• Condor Home Page– http://www.cs.wisc.edu/condor

• These slides:– http://www.cs.wisc.edu/~thain

• Douglas Thain– thain@cs.wisc.edu

• Questions Now?