components of grid development middleware resource integration connectivity

1AGD: ResourceIntegration, Harry Enke (AIP)14.-15..2006

Components of Grid Development1. Middleware

2. Resource Integration

3. Connectivity

4. Acceptance of Grid in the Community

Sustainability

OverviewOverview


1. Middleware Complementary development and extention

of Middleware stacks from DGI and Communities

Starting point is (mostly) Globus 4.x Open Source based and oriented Integrative towards community specific

tailored middleware (based on globus) (HEP) und specialized middleware (Unicore)

Components of Grid DevelopmentComponents of Grid Development


2. Resource integration 1 Level: National integration

• Unified Certification Policies

• Building of an efficient certification chain

– for users

– for services

• Service Certification is still not sufficient

– for hardware (Hosts) it is sufficient

– for Services the Certification (Webservices, Compute-Services etc) no unified system

– different approaches not integrated (Shibboleth)

• VO Management still not available on national level



2. Resource Integration Level 1: National Integration

Obstacles:• Resource-providers in scientific community face legal ond

procedural obstacles which hamper the grid way provding resources with Vos, especially the SCC

• Community-Resources are more flexible in that aspect

In the long run we need a change of these procedures and substitute them with grid-based usage policies (establishing new comittees for eg. allocating resources to a science project)



2. Resource Integration Level 2: Grid-Middleware

Accounting and Billing procedures are still not available

Reasons:

1. different community procedures and requirements

2. software for these procedures still in concept phase

3. resource integration into a functional grid only now comes into being

Job monitoring and quality assessments for QoS agreements for resource usage is one of the crucial steps in the next year, as a base for even raising the question of resource allocation procedures toward grid ways

BMBF sponsored considerable hardware resources, which provide a great opportunity for progress, since there are no legal (and local) restrictions imposed



3. ConnectivityThe Grid is hungry for bandwidth Useability of grid integrated resources is very limited

without availability of sufficient network connectivity Requirement for scientific communities is: Centers of Grid development are the nodes of DGI

amd Communities • HEP-Community already has a network concept for the

LHC-Experiment• at least connect all BMBF hardware nodes with 10GBit/s• (Example BaWü: for science institutions a countrywide

Gbit network without fees)



3. Connectivity For QoS of grid services with respect to

network connectivity no middleware stack has any component to offer

Without incorporating QoS for network connectivity accounting/billing is only partially possible



4. Improving of acceptance of the Grid is one of the most important tastks of the Community grids

Getting scientific communites to use the grid infrastructure for their day to day work is one of the most important contibutions towards the sustainability

Grid integration of huge projects is a pilot towards this goal Example: LOFAR Building Institute resource grids and VOs, establishing resource

sharing on that level is one of the roads Getting more associated partners into Community projects by

offering support for middleware installation to get them started Collaboration on international level with other projects within the

communities Example : IVOA / GAVO Example : GAIA Example : Ukraine Astrophysics Grid



1. Get the resources in a useable state:

1. Deploy our VOMRS on all resources, have all local RA use the tool

2. Ensure a regular update of the gridmaps on all grid resources

3. Run a daily health-checker on each resource that’s in MDS to report the running services, the state of the local gridmaps and if possible, the gridusers and their jobs

4. Start an issue tracking module for the resource providers to report on problems with the grid machines, establish a procedure for resolving conflicts with gridjobs and gridusers

5. Draft a set of policies for grid resource usage

1. Basic policy: all AstroGrid-D grid resources are available to all our VO members

2. Exceptions are ok, but only for specialized resources (eg don’t try to run a taskfarming job on a cluster, or a program without gravitational interaction on a cluster with GRAPE boards)

3. Local users run with a higher priority than grid users

Resource Integration: Next stepsResource Integration: Next steps


Special for clusters:

1. Incorporate more resources from AstroGrid-D, especially the pending clusters

2. Start to define default libraries and environments for the clusters (mpirun etc),

document standards for the different local queue-managers and policies

3. Provide above information via commandline facilities as well as hardware info etc,

incorporate queue status information

Special for workstations:

1. Provide owners (main user) with a means to suspend grid jobs for some time and

bar starting new ones if this state is signaled

2. Get this suspended status communicated to the information service

3. Add additional security related monitoring of globus ports wrt security and attacks



BMBF Hardware for our Nodes:

For ZAH and Potsdam already agreed:

Installation of same OS (Scientific Linux 4.3 or 4.4)

Source installation of Globus 4.03

Start an admin group for these resources across institute borders

Ensure the participation of the community already in the planning stages of the integration of all middleware stacks to ensure a forward solution


components of grid development middleware resource integration connectivity

Documents