douglas thain, john bent andrea arpaci-dusseau, remzi arpaci-dusseau, miron livny computer sciences...

Douglas Thain, John BentAndrea Arpaci-Dusseau, Remzi Arpaci-Dusseau, Miron Livny

Computer Sciences Department, UW-Madison

Gathering at the Well:

Creating Communities for Grid I/O

www.cs.wisc.edu/condor

4 earth-shattering revelations

1) The grid is big.2) Scientific data-sets are large.3) Idle resources are available.4) Locality is good.


How to optimize job placement on the grid?› Move data to the jobs.› Move jobs to the data.› Allow jobs to access data remotely.

› Need framework for evaluation. I/O communities


I/O Communities

UW INFN


I/O communities are an old idea

› File servers and administrative domains

› But, we want more flexible boundaries simple mechanism by which users can

express I/O community relationships


I/O communities› Mechanism which allow either

jobs to move to data, or data to move to jobs, or data to be accessed remotely

› Framework to evaluate these policies


Grocers, butchers, cops› Members of an I/O community

Storage appliances Interposition agents Scheduling systems Discovery systems Match-makers Collection of CPU’s


Storage appliances› Should run without special privilege

Flexible and easily deployable Acceptable to nervous sys admins

› Should allow multiple access modes Low latency local accesses High bandwidth remote puts and gets


Interposition agents› Thin software layer interposed

between application and OS› Allow applications to transparently

interact with storage appliances› Unmodified programs can run in

grid environment


Scheduling systems and discovery

› Top level scheduler needs ability to discover diverse resources

› CPU discovery Where can a job run?

› Device discovery Where is my local storage appliance?

› Replica discovery Where can I find my data?


Match-making› Match-making is the glue which

brings discovery systems together› Allows participants to indirectly

identify each other i.e. can locate resources without

explicitly naming them


Mechanisms not policies

› I/O communities are a mechanism not a policy

› A higher layer is expected to choose application appropriate policies

› We will however demonstrate the strength of the mechanism by defining appropriate policies for one particular application


Experimental results› Implementation› Environment › Application› Measurements› Evaluation


Implementation› NeST

storage appliance› Pluggable File System (PFS)

interposition agent built with Bypass› Condor and ClassAds

scheduling system discovery system match-maker


Two I/O communities› INFN Condor pool

236 machines, about 30 available at any one time

Wide range of machines and networks spread across Italy

Storage appliance in Bologna• 750 MIPS, 378 MB RAM


Two I/O communities› UW Condor pool

911 machines, 100 dedicated for us Each is 600 MIPS, 512 MB RAM Networked on 100 Mb/s switch One was used as a storage appliance


CMS simulator sample run

› Purposefully choose a run with high I/O to CPU ratio

› Accesses about 20 MB of data from a 300 MB database

› Writes about 1 MB of output› ~160 seconds execution time

on a 600 MIPS machine with local disk


Assume the position› We assumed the role of an Italian

scientist› Database stored in Bologna› Need to run 300 instances of simulator

› How to take advantage of UW pool? Three way matching


Three way matching

Machine NeSTJob

JobAd

MachineAd

StorageAdm

atch

Refers toNearestStorage.

Knows whereNearestStorage is.


Two way ClassAdsType = “job”TargetType = “machine”Cmd = “sim.exe”Owner = “thain”Requirements = (OpSys==“linux”)

Job ClassAd

Type = “machine”TargetType = “job”OpSys = “linux”Requirements = (Owner==“thain”)

Machine ClassAd


Three way ClassAdsType = “job”TargetType = “machine”Cmd = “sim.exe”Owner = “thain”Requirements = (OpSys==“linux”) && NearestStorage.HasCMSData

Job ClassAd

Type = “machine”TargetType = “job”OpSys = “linux”Requirements = (Owner==“thain”)NearestStorage = ( Name = “turkey”) && (Type==“Storage”)

Machine ClassAd

Type = “storage”Name = “turkey.cs.wisc.edu”HasCMSData = trueCMSDataPath = /cmsdata”

Storage ClassAd


Policy specification› Run anywhere where data is available

Requirements = (NearestStorage.HasCMSData)

› Run local only Requirements = (NearestStorage.Name == “nestore.bologna”)

› Run local first Requirements = (NearestStorage.HasCMSData) Rank = (NearestStorage.Name == “nestore.bologna” ) ? 10 : 0

› Arbitrarily complex Requirements = ( NearestStorage.Name == “nestore.bologna”)

|| ( ClockHour < 7 ) || ( ClockHour > 18 )


Policies evaluated› INFN local› UW remote› UW stage first› UW local (pre-staged)› INFN local, UW remote› INFN local, UW stage› INFN local, UW local


Completion Time


CPU Efficiency


Conclusions› Locality is good› I/O communities are a natural

structure to expose this locality› Users can use I/O communities to

easily express different job placement policies


Future work› Automation

Configuration of communities Dynamically adjust size as load

dictates› Automation

Selection of movement policy› Automation


For more info› Condor

http://www.cs.wisc.edu/condor› ClassAds

http://www.cs.wisc.edu/condor/classad› PFS

http://www.cs.wisc.edu/condor/pfs› NeST

http://www.nestproject.org


Local only


Remote only


Both local and remote


Grid applications have demanding I/O needs

› Petabytes of data in tape repositories› Scheduling systems have demonstrated

that there are idle CPUs› Some systems

move jobs to data move data to jobs allow job remote access to data

› No one approach is always “best”


Easy come, easy go› In a computation grid, resources

are very dynamic› Programs need rich methods for

finding and claiming resources CPU discovery Device discovery Replica discovery


Bringing it all together

CPU Discovery System

Replica Discovery System

Device Discovery System

Job Agent

Execution siteStorage appliance

Distributed Repository

Short-haul I/O

Long-haul I/O


Conclusions› Locality is good› Balance point between staging data

and accessing it remotely is not static depends on specific attributes of the job

•data size, expected degree of re-reference, etc depends on performance metric

•CPU efficiency or job completion time

douglas thain, john bent andrea arpaci-dusseau, remzi arpaci-dusseau, miron livny computer sciences...

Documents