middleware planning for lcg/egee bob jones egee technical director
DESCRIPTION
Middleware Planning for LCG/EGEE Bob Jones EGEE Technical Director. e-Science Grids and Web Services meeting 23 th April 2004. www.eu-egee.org. EGEE is a project funded by the European Union under contract IST-2003-508833. Contents. Very Brief overview of EGEE & LCG Objectives and status - PowerPoint PPT PresentationTRANSCRIPT
EGEE is a project funded by the European Union under contract IST-2003-508833
Middleware Planning for LCG/EGEE
Bob Jones
EGEE Technical Director
e-Science Grids and Web Services meeting 23th April 2004
www.eu-egee.org
e-Science Grids and Web Services meeting, 23th April 2004 - 2
Contents
• Very Brief overview of EGEE & LCG Objectives and status How they relate to each other The ARDA project
• Middleware planning Organisation Key middleware deliverables and milestones Strategy being adopted Current Status
• Summary
e-Science Grids and Web Services meeting, 23th April 2004 - 3
EGEE manifesto:Enabling Grids for E-science in Europe
Applications
Geant network
Grid infrastructure
• Goal Create a wide European Grid production
quality infrastructure on top of present and future EU RN infrastructure
• Build On: EU and EU member states major investments in Grid Technology International connections (US and AP) Several pioneering prototype results Large Grid development teams in EU require
major EU funding effort
• Approach Leverage current and planned national and
regional Grid programmes Work closely with relevant industrial Grid
developers, NRENs and US-AP projects
e-Science Grids and Web Services meeting, 23th April 2004 - 4
EGEE Partners
• Leverage national resources in a more effective way for broader European benefit
• 70 leading institutions in 28 countries, federated in regional Grids
e-Science Grids and Web Services meeting, 23th April 2004 - 5
EGEE Applications
• EGEE Scope : ALL-Inclusive for academic applications (open to industrial and socio-economic world as well)
• The major success criterion of EGEE: how many satisfied users from how many different domains ?
• 5000 users (3000 after year 2) from at least 5 disciplines
• Two pilot applications selected to guide the implementation and certify the performance and functionality of the evolving infrastructure: Physics & Bioinformatics
e-Science Grids and Web Services meeting, 23th April 2004 - 6
EGEE Project Structure
JRA1: Middleware Engineering and Integration
JRA2: Quality Assurance
JRA3: Security
JRA4: Network Services Development
SA1: Grid Operations, Support and Management
SA2: Network Resource Provision
NA1: Management
NA2: Dissemination and Outreach
NA3: User Training and Education
NA4: Application Identification and Support
NA5: Policy and International Cooperation
24% Joint Research 28% Networking
48% ServicesEmphasis in EGEE is on operating a productiongrid and supporting the end-users
32 Million Euros EU funding over 2 years starting 1st April 2004
e-Science Grids and Web Services meeting, 23th April 2004 - 7
EGEE and LCG
EGEE builds on the work of LCG to establish a grid operations service• LCG: a worldwide collaboration of
The LHC experiments The Regional Computing Centres Physics institutes
• Mission: Prepare and deploy the computing environment that will be used by the
experiments to analyse the LHC data• Strategy:
Integrate thousands of computers at dozens of participating institutes worldwide into a global computing resource
Rely on software being developed in advanced grid technology projects, both in Europe and in the USA
• Status: LCG service up and running with LCG-2 mware – successfully being
used for LHC data challenges
e-Science Grids and Web Services meeting, 23th April 2004 - 8
• Service opened on 15 September 2003
– with 12 sites
• Middleware package components from
European DataGrid (EDG)
US Virtual Data Toolkit (Globus,
Condor, PPDG, iVDGL, GriPhyN)
• About 30 sites by the end of the year
Upgraded version of the grid software
(LCG-2) in February 2004
• VO for D0 managed by NIKHEF
• Hewlett Packard to provide
“Tier 2-like” services for LCG,
initially in Puerto Rico
LCGThe LCG Service
06-Apr-04country centre country centre
Canada TRIUMF, Vancouver
Spain PIC, Barcelona
Czech Republic
Prague IFIC, Valencia
France IN2P3, Lyon IFCA, SantanderGermany FZK, Karlsruhe University of
BarcelonaDESY Uni. Santiago de
CompostelaHolland NIKHEF,
AmsterdamCIEMAT, Madrid
Hungary Budapest UAM, MadridItaly CNAF, Bologna Switzerland CERN
INFN, Torino CSCS, MannoINFN, Milano Taiwan Academia Sinica,
TaipeiINFN, Roma NCU, TaipeiINFN, Legnaro UK RAL
Japan ICEPP, Tokyo Cavendish, Cambridge
Poland Krakow Imperial, London
Russia SINP, Moscow USA FNALBNL
Regional Centres Connected to the LCG Grid
e-Science Grids and Web Services meeting, 23th April 2004 - 9
ARDA ProjectCollaborationCoordinationIntegration
SpecificationsPrioritiesPlanning
EGEE Middleware Activity
ALICEDistr.
analysis
ATLASDistr.
analysis
CMSDistr.
analysis
LHCbDistr.
analysis
Distributed Physics AnalysisThe ARDA Project
ARDA – distributed physics analysis batch to interactive end-user emphasis
• 4 pilots by the LHC experiments (core of the HEP activity in EGEE NA4)
• Rapid prototyping pilot service
• Providing focus for the first products of the EGEE middleware
• Kept realistic by what the EGEE middleware can deliver
e-Science Grids and Web Services meeting, 23th April 2004 - 10
EGEE Implementation
• From day 1 (1st April 2004)Production grid service based on the LCG infrastructure running LCG-2 grid
middleware (SA)
LCG-2 will be maintained until the new generation has proven itself (fallback solution)
• In parallel develop a “next generation” grid facility (JRA)Produce a new set of grid services according to evolving standards (Web Services)
Run a development service providing early access for evaluation purposes
Will replace LCG-2 on production facility in 2005
Globus 2 based Web services based
EGEE-2EGEE-1LCG-2LCG-1
EDGVDT . . .
LCG
EGEE
. . .AliEn
e-Science Grids and Web Services meeting, 23th April 2004 - 11
JRA1: Middleware Engineering and Integration
• ObjectivesProvide robust, supportable middleware componentsIntegrate grid services to provide a consistent functional basis for the
EGEE grid infrastructureVerify the middleware forms a dependable and scalable infrastructure
that meets the needs of a large, diverse eScience user community 5 partners, approx 16% of total project budget
• 4 main development clusters: UK CERN IT/CZ Nordic (security, JRA3)
e-Science Grids and Web Services meeting, 23th April 2004 - 12
Milestones and Deliverables for 2004
MonthDeliverables
&Milestones
ItemLead Partner
Status
Jun’04 MJRA1.1 Tools for middleware engineering and integration deployed
CERN Most tools identified (SCM)Issue: CNRS person tbi
Jun’04 DJRA1.1 (Document) Architecture and Planning (Release 1)
CERN Design document as starting point; writing organized for May
Jun’04 MJRA1.2 Software cluster development and testing infrastructure available
CERN Done for CERN/UK/IT/CZNordic to be finalized
Aug’04 MJRA1.3 Integration and testing infrastructure in place including test plans (Release 1)
CERN Sites identified; teams established;
Aug’04 DJRA1.2 (Document) Design of grid services (Release 1)
CERN Design document as starting point
Dec’04 MJRA1.4 Software for the Release Candidate 1 CERN Work on prototype started; will evolve into 1st release candidate
M08 – Amsterdam conference: Tech preview of release candidate 1 available
e-Science Grids and Web Services meeting, 23th April 2004 - 13
Towards a Prototype
• Design team formed in December 2003• Started service design based on component breakdown defined by the
LCG ARDA RTAG• Leverage experiences and existing components from AliEn, VDT, and
EDG.• A working document - Overall design & API’s• Basis for architecture (DJRA1.1 – June ‘04) and design (DJRA1.2 –
Aug’04) document• Aim to have first prototype ready by end of April
Focus on key services; exploit existing components Initially an ad-hoc installation at CERN and Wisconsin Open only to a small user community Expect frequent changes (also API changes) based on user feedback and
integration of further services• Enter a rapid feedback cycle
Continue with the design of remaining services Enrich/harden existing services based on early user/operations feedback
e-Science Grids and Web Services meeting, 23th April 2004 - 14
Prototype Evolution
• Evolution of the prototype Envisaged status at end of 2004:
• Key services need to fulfill all requirements (application, operation, quality, security, …) and form a deployable release
• Remaining services available as prototype Will develop a roadmap
• Incremental changes to prototype (where possible)• Early user feedback through ARDA and early deployment on SA1 pre-
production service• Detailed release plan being developed
e-Science Grids and Web Services meeting, 23th April 2004 - 15
Guiding Principles
• Lightweight existing services Easily and quickly deployable
• Interoperability Allow for multiple implementations
• Resilience and Fault Tolerance
• Co-existence with deployed infrastructure Run as an application
• Service oriented approach Follow and contribute production expertise to standardization efforts No mature WSRF implementations exist to date, hence: start with plain WS
– WSRF compliance is not an immediate goal Review situation end 2004
e-Science Grids and Web Services meeting, 23th April 2004 - 16
Summary
• LCG is currently providing a production grid service using the LCG-2 software
• EGEE has started 1st April The first project conference was held in Cork this week and showed
that all activities are up and running
• Middleware EGEE will develop a new set of web services based grid
middleware• An early prototype will be tested in May• Due to timing issues, the first release will probably only be based on
basic web services• EGEE will be involved in the grid standardisation process and is keen to
adopt such standards as they become stable