babar mc production babar mc production software farm @ vu (amsterdam university) a lot of computers...

BaBar MC production

BaBar MCproduction software

Farm @ VU(Amsterdam University)

A lot of computers

EDG testbed(NIKHEF)Jobs

Results

The simple question: How can we run BaBar software on EDG grid sites?

ParrotChirp

Introduction of Parrot



A lot of computers

EDG testbed(NIKHEF)Jobs

Results

We need transparent access to the Objectivity Database(requires local file access)

Parrot functionalityBaBar

MC production

The Parrot Virtual File System

HTTP FTP RFIO NeST Chirp

LocalCache

HTTPServer

FTPServer

(POSIX Interface)

Whole File I/O(get/put)

Partial File I/O(open,close,read,write, lseek)

RFIOServer

NeSTServer

ChirpServer

CondorProxy

SecureRemote

RPC

CondorShadow

Integrationwith Castor

TraditionalI/O Services

Allocationand Mgmt

Full UNIXSemantics

Integrationwith Condor

(Ptrace trap)Not yet

x509

Optimize

Private networkRelay

GCB

Parrot

Chirp

NF

S

The introduction of GCB



EDG testbed(NIKHEF)

Condor-G Jobs

Results

Some computers A lot of computers

Jobs

Results

GCB functionality

GCBServer

CentralManage

r

A

B

P

Private network

Pers

iste

nt con

nect

ion

Relay

NAT

PBS job manager72 hour jobs

Can’t wait for queues

Private network

NF

SBaBar MC

production softwareQueue

Batchjob

Condor-G Job

GlideIn

EDG testbed(NIKHEF)

RelayPrivate network

Relay

RelayParrot

Chirp

The introduction of GlideInFarm @ VU

(Amsterdam University)

Jobs

Results


Jobs

Results

GCB

GlideIn functionality

Job Submission Machine

Job Execution Site

Job

Condor-G GridManager

GASS Server

Condor-G Scheduler

Persistant Job Queue

End User Requests

Condor Shadow

Process for Job X

Condor-G Collector

Globus Daemons +

Local Site Scheduler

[See Figure 1]

Condor Daemons

Job X

Condor System Call

Trapping & Checkpoint Library

Resource

Information

Transfer Job X

Redirected

System Call Data

Private network

PBS job manager72 hour jobs

Can’t wait for queues

Private network

NF

SBaBar MC

production softwareQueue

Batchjob

Condor-G Job

GlideIn

EDG testbed(NIKHEF)

Relay

Relay

RelayParrot

Chirp

Overview of complete setupFarm @ VU


Jobs

Results


Jobs

Results

GCB

PBS job manager

NF

SBaBar MC

production softwareQueueGlideIn

EDG testbed(NIKHEF)

Private networkPrivate network

Parrot

Chirp

Leave only the componentsFarm @ VU



GCB

PBS job manager

NF

SBaBar MC


EDG testbed(NIKHEF)


Parrot

Chirp

The interesting dependenciesFarm @ VU



GCB

NAT box

Different MDSscheme

Objectivity database• LOCK server sockets• NFS problems• UID / hostname checks

• Dropping UDP packages• Timeout 2 minutes

• Inactive sockets• Inactive File I/O

Consequences

• Different MDS scheme– Implemented EDG scheme for GlideIn

• Objectivity– A lot of debugging– Made Parrot mimic hostname and uid– Tricked Objectivity to use standard NFS libraries

• Aggressive NAT box– Changed GCB to use TCP instead of UDP– Used Parrot to keep sockets alive– Parrot recovers File I/O when TCP connection is lost

• We are the first to run Objectivity cross-domain

Performance

500 1000 1500 2000Events

Tim

e (m

inut

es)

500

1000

1500

2000

2500

3000

Application Initializes10 times slower

Production3 times slower

Production onlocal machine

Production onEDG testbed

PBS job manager

NF

SBaBar MC


EDG testbed(NIKHEF)


Parrot

Chirp

Possible improvementsFarm @ VU



GCB Parrot: Caching• On per directory basis• Requires debugging

Create more sophisticated tool to acquire resources• Resource planning, distribution, etc.• Maybe something fancy already exists?

PBS job manager

NF

SBaBar MC


EDG testbed(NIKHEF)


ParrotChirp

Move chirp servers to private nodesFarm @ VU



GCB

Use Condor/GCB machinery for chirp server• Solves security issues• Allows chirp server to be on private nodes• Requires new chirp-condor implementation

PBS job manager

NF

SBaBar MC


EDG testbed(NIKHEF)


ParrotChirp

Move GCB to head nodeFarm @ VU



GCB

Move GCB to same machine as Central Manager• Solution required for port conflicts• Temporary solution: Move CM to a private node

PBS job manager

NF

SBaBar MC


EDG testbed(NIKHEF)


ParrotChirp

Use EDG data storageFarm @ VU



GCB

EDG data storage

Write events to EDG data storage (gsiFTP)• Requires debugging

PBS job manager

NF

SBaBar MC


EDG testbed(NIKHEF)


ParrotChirp

Use more sites



GCB

Private network

A lot of computers

Other testbed

EDG data storage

Let GCB manage several private networks at the same time• Requires solution for conflicting private addresses

Conclusions• It works– BaBar MC production runs successfully on NIKHEF EDG testbed– All this experimental software actually works when used together

• It looks easy– Our GRID setup is complicated, but….– Parrot hides problems related to local file access– GCB hides problems related to network configurations– GlideIn hides complications with resource gathering– The user can just submit his/her jobs to a local batch system

• There is some work to do– Performance could be better

• Initialization 10 times slower• Production 3 times slower

– Caching and (semi-) local event storage should improve this

– Usability could be improved• GlideIn should have a tool to acquire them• Several improvements proposed for GCB/Parrot

• The improvements are done at the level of the “grid” tools– The user benefits without rewriting code

babar mc production babar mc production software farm @ vu (amsterdam university) a lot of computers...

Documents