tony doyle - university of glasgow pparc review timeline 0. pparc call (february 2003) 1.gridpp2...

Tony Doyle - University of Glasgow

PPARC Review TimelinePPARC Review Timeline

0. PPARC Call (February 2003)1.GridPP2 Proposal (30/5/03)2. Feedback from Reviewers3. Responses from the PMB

See http://www.gridpp.ac.uk/docs/gridpp2/

4. Projects Peer Review Panel (14-15/7/03)5. Grid Steering Committee (28-29/7/03)

See http://www.gridpp.ac.uk/talks/GridPP_GSC_030728.ppt

6. Tender Document (11/11/03)

See http://www.gridpp.ac.uk/cb/doc/GridPP2_Tender.doc7. Science Committee (17-18/11/03)8. PPARC (25/11/03)9. CB (4/12/03)

Tony [email protected]

“GridPP – Internal Tender Process”, Collaboration Board, London, 4 December 2003


OutlineOutline

• Context:– GridPP Status

– Testbed

– GridPP2 Operations

• Tendering:– Introduction

– Areas

– Proforma

– Allocations

– EGEE

– General Points

– Procedure

– Outcomes


GridPP Status Report: GridPP Status Report: The Project Map (v 18)The Project Map (v 18)

H I H I H H H H H

J 1. 1 K J 2. 1 K J 3. 1 K J 4. 1 K J 5. 1 K J 6. 1 K J 7. 1 K

1.1.1 1.1.2 1.1.3 1.1.4 2.1.1 2.1.2 2.1.3 2.1.4 3.1.1 3.1.2 3.1.3 3.1.4 4.1.1 4.1.2 4.1.3 4.1.4 5.1.1 5.1.2 5.1.3 6.1.1 6.1.2 6.1.3 6.1.4 7.1.1 7.1.2 7.1.3 7.1.41.1.5 2.1.5 2.1.6 2.1.7 2.1.8 3.1.5 3.1.6 3.1.7 3.1.8 4.1.5 4.1.6 4.1.7 4.1.8 6.1.5

2.1.9 3.1.9 3.1.10 4.1.9

J 1. 2 K J 2. 2 K J 3. 2 K J 4. 2 K J 5. 2 K J 6. 2 K J 7. 2 K

1.2.1 1.2.2 1.2.3 1.2.4 2.2.1 2.2.2 2.2.3 2.2.4 3.2.1 3.2.2 3.2.3 3.2.4 4.2.1 4.2.2 4.2.3 4.2.4 5.2.1 5.2.2 5.2.3 6.2.1 6.2.2 6.2.3 7.2.1 7.2.2 7.2.31.2.5 1.2.6 1.2.7 1.2.8 2.2.5 2.2.6 2.2.7 3.2.5 3.2.6 3.2.7 3.2.8 4.2.5 4.2.6 4.2.71.2.9 1.2.10 3.2.9

J 1. 3 K J 2. 3 K J 3. 3 K J 4. 3 K J 5. 3 K J 6. 3 K J 7. 3 K

1.3.1 1.3.2 1.3.3 1.3.4 2.3.1 2.3.2 2.3.3 2.3.4 3.3.1 3.3.2 3.3.3 3.3.4 4.3.1 4.3.2 4.3.3 4.3.4 5.3.1 5.3.2 5.3.3 6.3.1 6.3.2 6.3.3 6.3.4 7.3.1 7.3.2 7.3.3 7.3.41.3.5 1.3.6 1.3.7 1.3.8 2.3.5 2.3.6 2.3.7 3.3.5 3.3.6 4.3.51.3.9 1.3.101.3.11

J 1. 4 K J 2. 4 K J 3. 4 K J 4. 4 K J 5. 4 K1.4.1 1.4.2 1.4.3 1.4.4 2.4.1 2.4.2 2.4.3 2.4.4 3.4.1 3.4.2 3.4.3 3.4.4 4.4.1 4.4.2 4.4.3 4.4.4 5.4.1 5.4.2 5.4.3 5.4.41.4.5 1.4.6 1.4.7 1.4.8 2.4.5 2.4.6 2.4.7 3.4.5 3.4.6 3.4.7 3.4.8 4.4.5 4.4.6 5.4.51.4.9 3.4.9 3.4.10 Metric OK 1.1.1

Metric not OK 1.1.1J 1. 5 K J 2. 5 K J 3. 5 K J 4. 5 K Task complete 1.1.1

1.5.1 1.5.2 1.5.3 1.5.4 2.5.1 2.5.2 2.5.3 2.5.4 3.5.1 3.5.2 3.5.3 3.5.4 4.5.1 4.5.2 4.5.3 4.5.4 Task overdue 1.1.11.5.5 1.5.6 1.5.7 1.5.8 2.5.5 2.5.6 2.5.7 3.5.5 3.5.6 3.5.7 92 days 1.1.11.5.9 1.5.10 Task not due soon 1.1.1

Not Active 1.1.1J 2. 6 K J 3. 6 K J 4. 6 K No Task or metric

2.6.1 2.6.2 2.6.3 2.6.4 3.6.1 3.6.2 3.6.3 3.6.4 4.6.1 4.6.2 4.6.32.6.5 2.6.6 2.6.7 2.6.8 3.6.5 3.6.6 3.6.7 3.6.8 Navigate up J2.6.9 3.6.9 3.6.103.6.113.6.12 Navigate down K

External link IJ 2. 7 K J 3. 7 K Link to goals H

2.7.1 2.7.2 2.7.3 2.7.4 3.7.1 3.7.2 3.7.3 3.7.42.7.5 2.7.6 2.7.7 2.7.8 3.7.5 3.7.6

J 2. 8 K J 3. 8 K2.8.1 2.8.2 2.8.3 2.8.4 3.8.1 3.8.2 3.8.32.8.5

To develop and deploy a large scale science Gridin the UK for the use of the Particle Physics community

Presentation Deployment

5 6 74

01-Oct-03Status Date

Int. Standards

Open Source

Worldwide Integration

UK Integration

Monitoring

DevelopingEngagement

Participation

Other

Data Challenges

Rollout

Testbed

WP7

ATLAS/LHCb

CMS

BaBar

CDF/DO

UKQCD

WP1

WP2

WP3

LCG Creation

Applications

CERN DataGrid Applications Infrastructure

Due within

ATLAS

GridPP Goal

ResourcesInteroperability Dissemination

Tier-1

Tier-A

LHCb Tier-2

WP8

1 2 3

Deployment

WP4

WP5

Fabric

Technology

WP6

Update

Clear


GridPP Status: SummaryGridPP Status: Summary

• GridPP1 has now completed 2 ex 3 years

• 42 of 44 metrics are currently satisfied

• 126 of 182 tasks are complete

• Reflects many successes..

• 3 tasks are currently overdue:– LCG test suite completion (confirmation required)– BaBar use of Tier-A (3 month delay)– UK infrastructure (test of a heterogeneous testbed)

• 2 dissemination metrics are not OK:– Dissemination of GridPP work via demonstrations (7 ex 8

target workshops)– e-Science interactions (17 ex 20 recorded interactions)


Testbed Status (3/12/03)Testbed Status (3/12/03)

UK-wide developmentusing EU-DataGrid tools (v1.4x).Deployed during Sept 02-03. Currently being upgraded to v2: IC and RAL via RB OKSee http://www.gridpp.ac.uk/map/


0

1

2

3

4

5

6

7

8

9

10

FT

EOperations Manager

Tier-2 Expert

Tier-2 Expert

Tier-2 Expert

Tier-2 Expert

Tier-1 Expert

Tier-1 Expert

Applications Expert

University WP6 Posts

RAL WP6 Post

WP8 Post

Testbed Team Production Team

GridPP1 GridPP2

GridPP2 OperationsGridPP2 Operations

• To move from testbed to production, GridPP plans a bigger team with a full-time Production Manager

• Manpower will be from the Tier-1 and Tier-2 Centres who will contribute to the Production Team

• The team will run a UK Grid which will be part of EGEE/LCG and also support other experiments

Ta

gged

re

leas

e s

elec

ted

for

cert

ifica

tion

Ce

rtifi

ed r

ele

ase

sele

cte

d fo

r d

eplo

yme

nt

Ta

gged

p

acka

ge

Problem reports

add unittested code to

repository

Run nightly build

& auto. tests

Grid certification

Fix problemsApplication Certification

BuildSystem

CertificationTestbed ~40CPU

ProductionTestbed ~1000CPU

Certified publicrelease

for use by apps.

24x7

Build system

Test Group

WPs

Unit Test Build

Certification

Production

Users

DevelopmentTestbed ~15CPU

Individual WP tests

Integration

Team

Integration

Overall release tests

Releases candidate

Tagged Releases

Releases candidate

Certified Releases

Apps. Representative

s


IntroductionIntroduction

• Funding for the GridPP2 project is now known (see Dave’s slides)

• We need to develop a detailed plan

• To avoid a period of uncertainty we have released the GridPP2 Tender Document

• Post priorities were discussed within the PMB and were presented to PPARC for various funding scenarios

• Science Committee and PPARC have now approved GridPP2 - sufficient to fund the “regained programme”

• The funding is conditional upon our engagement in the EGEE/LCG programmes

• The PMB requests approval for the following internal tender process from the CB


Four Tendering AreasFour Tendering Areas

• The document contains tables that show the PMB-approved allocation of effort in four areas:1. Application Interfaces2. Middleware/Security/Networking 3. Tier-2 centres4. Tier-1/A Centre

• Internal deadlines such that outcomes are known in Jan-February

• These deadlines were agreed for all areas except for: – hardware support for the Tier-2 centres where a date in

~March is foreseen (see Steve’s talk)– UK’s contribution to LCG since LCG started later than

GridPP1• Note that the Tier-1/A Centre planning also refers to EGEE

Production across the UK • EGEE starts in April 2004


Four ProformaFour Proforma

1. Review of existing posts: to be completed by all Principal Investigators examining the post(s) effectiveness

(A requirement of the original grants)

2. Project definition: the project deliverables are to be confirmed or updated by the area contacts (This defines the programme of work planned in the 3-year period from September 04)

3. Tender: describes how effort at a given institute can meet these (updated) project deliverables

4. Assessment: input from the PMB to the PRSC


Application InterfacesApplication Interfaces [FTE] [FTE]

• Applications– 18 FTEs: ongoing

programme of work can continue

– Difficult to involve experiment activity not already engaged within GridPP

• Project needs to build on cross-experiment collaboration GridPP1 already has experience

– GANGA: ATLAS & LHCb

– SAM: CDF & D0– Persistency: CMS

& BaBar

GridPP1 Work Area FTE GridPP2 Work Area FTE

ATLAS/LHCb (GANGA) 2.0 ATLAS/LHCb (GANGA) 2.0

ATLAS 2.5 ATLAS 2.5

BaBar 2.5 BaBar 2.5

CDF/D0 (SAM) 2.0 CDF/D0 (SAM) 2.0

CDF 1.0 CDF 1.0

CMS 3.0 CMS 3.0

D0 1.0 D0 1.0

LHCb 2.0 LHCb 2.0

WP8 1.0 LHC Deployment 1.0

UKQCD 1.0 UKQCD 1.0

Other 0.0 Other 4.0

Total 18.0 Total 18.0 ex 22.0

•Encourage new joint developments across experiments


Middleware, Security and Network Middleware, Security and Network [FTE][FTE]

• Information Services [4+4 FTE] and Networking [3+1.5+1.5 FTE]: strategic roles within EGEE• Security expands to meet reqts.• Data and Workload Management continue• No further configuration management development

• programme defined by – mission criticality (experiment

requirements driven) – International/UK-wide lead – leverage of EGEE, UK core and

LCG developments

• Redefine (EDG-based) UK Work Packages into five Development Groups

Activity Current Planning

Security 3.5

Info-Mon. 4.0

Data & Storage 4.0

Workload 1.5

Networking 3.0

TOTAL 16.0

SecurityMiddleware

Networking

GridPP1

GridPP2

Work Area

PPARC funding Other funding PPARC funding Other funding

Data and Storage Management

4.2 1.0 4.0 0.0

Workload Management 0.5 2.0 1.5 4.0*

Security 1.1 2.0 3.5 2.0*

Information & Monitoring 2.5 3.0 4.0 4.0

Network Sector 2.6 8.0 3.0 3.0

Fabric Management 1.5 0.0 0.0 0.0

Totals 12.4 16.0 16.0 9.0 -13.0


Tier-2 Services [FTE]Tier-2 Services [FTE]

• Four Regional Tier-2 Centres • London: Brunel, Imperial College, QMUL, RHUL, UCL.• SouthGrid: Birmingham, Bristol, Cambridge, Oxford, RAL PPD.• NorthGrid: CCLRC Daresbury, Lancaster, Liverpool, Manchester, Sheffield.• ScotGrid: Durham, Edinburgh, Glasgow.

• Hardware provided by Institutes

• GridPP provides added manpower

Current Planning

Y 1 Y 2 Y 3

Hardware Support 4.0 8.0 8.0

Core Services 4.0 4.0 4.0

User Support 1.0 2.0 2.0

Specialist Services

Security 1.0 1.0 1.0

Resource Broker 1.0 1.0 1.0

Network 0.5 0.5 0.5

Data Management 2.0 2.0 2.0

VO Management 0.5 0.5 0.5

14.0 19.0 19.0

Existing Staff -4.0 -4.0 -4.0

GridPP2 10.0 15.0 15.0

Total SY 40.0

GridPP2 Work Area FTE

Security 1.0

Resource Broker 1.0

Network 0.5

Data Management 2.0

VO Management 0.5

Operations 8.5

Total 13.5 [+4.0]


Tier-1/A Services [FTE]Tier-1/A Services [FTE]

• High quality data services

• National and International Role• UK Focus for International Grid development

• Highest single priority within GridPP2

Current Planning

CPU 2.0

Disk 1.5

AFS 0.0

Tape 2.5

Core Services 2.0

Operations 2.5

Networking 0.5

Security 0.0

Deployment 2.0

Experiments 2.0

Management 1.5

Total 16.5


EGEEEGEE

• An additional 9.5 FTEs for Operations and Support will be funded via EGEE to establish a Production Grid across the UK

• This area is being led by RAL (as agreed at the CB)

• Note EGEE will also fund:– Information Services (RAL)

[4+4FTE]

– Training (Edinburgh) [4+4]– Networking (UCL) [1.5+1.5]

{with linked funding from elsewhere}

• Core Infrastructure Centre and Regional Operations Centre for the UK and Ireland: includes Operations and Support

• 4 FTEs for EGEE Tier-2 Coordinators reporting to the Production Manager (joint appointments funded via CCLRC)


Operational RolesOperational Roles

• Core Infrastructure Services (CIC) – Grid information services

– Monitoring services

– Resource brokering

– Allocation and scheduling services

– Replica data catalogues

– Authorisation services

– Accounting services

• Still to be defined fully in EGEE

• Core Operational Tasks (ROC)– Monitor infrastructure, components

and services

– Troubleshooting

– Verification of new sites joining Grid

– Acceptance tests of new middleware releases

– Verify suppliers are meeting SLA

– Performance tuning and optimisation

– Publishing use figures and accounts


• RAL has led project to develop LCG Operations…

– Monitoring– Contacts– SLAs – Security

Policy– LCG

Security Coordinator


Tier 0 and LCG: Tier 0 and LCG: Foundation ProgrammeFoundation Programme

• Aim: retain UK influence

• Ensure development programmes are linked

• Project management:

GridPP LCG

• Shared expertise:

• LCG establishes the global computing infrastructure

• Allows all participating physicists to exploit LHC data

• Proposed funding determined based on:– Recently increased funding at

CERN supporting LCG (recognition of Grid importance within the LHC programme)

– Appropriate share for the UK– Requirements of LCG Phase 2– Past GridPP1 contribution

Required Foundation: LCG Fabric, Technology and Deployment


GeneralGeneral Points Points

1. All post-holders must report back at 3-monthly intervals (using similar reporting mechanisms to those used for the majority of GridPP posts).

2. All GridPP2 posts will end on Aug 31st 2007. Existing posts that are renewed will commence at the end-date of the present grant. Thus, many GridPP2 posts will run for less than three full years. This should be anticipated in the work programmes described.

3. A review of posts will be made prior to this end point, in order to determine their effectiveness within the GridPP2 programme.


Procedure for Internal Tendering IProcedure for Internal Tendering I

• We ask you to respond [via e-mail to Steve Lloyd] according to the proforma with deadlines of 12 noon on the dates noted below. 1. Current Principal Investigators: the deadline for submission

of completed Proforma-1 is Friday 12th December.

2. Contact People: the deadline for submission of completed Proforma-2 is Wednesday 17th December. These will be made available via the web by Friday 19th December.

3. Proposed Principal Investigators: the deadline for submission of completed Proforma-3 is Friday 9th January.

4. PMB Members: the deadline for submission of completed Proforma-1 Evaluation Reports is Friday 9th January.


Accelerated Procedure for Internal Accelerated Procedure for Internal TenderingTendering

• An accelerated procedure is necessary for Information Services in order for a CCLRC recruitment process to be initiated in time for EGEE start-up on 1st April

• Completed drafts of two proforma have been received by the PMB, with the third received in confidence by DB, TD, NG and SL.

• Based on next week’s PMB assessment of this input, a letter will be sent to enable CCLRC recruitment prior to the meeting of the PRSC

• Further feedback on the input will be included as part the PRSC process

• We request CB approval for this accelerated procedure


Procedure for Internal Tendering IIProcedure for Internal Tendering II

• The GridPP Peer Review Selection Committee (PRSC) will meet after Monday January 12th and determine priorities and allocations according to this input.

• The recommendations of the PRSC will be subject to approval by the PPARC e-Science Director.

• Results will be communicated to all Institutes as soon as possible after the PRSC meeting.

• For approved posts at Universities, a JES form should be submitted directly to Guy Rickett at PPARC with an agreed specific spine point for current staff or for a spine point 11 post with end point August 31st 2007.

• For approved posts at CCLRC, funding will be via the PPARC-CCLRC Service Level Agreement, referring to the agreed GridPP2 deliverables and services. Posts will be renewed or appointed within an agreed pay band with end point August 31st 2007.


Proforma-1: GridPP1 ReviewProforma-1: GridPP1 Review

Section-A: Identification

A1. PPARC Grant Number (200x/xxx) [where applicable]:

A2. Institute:

A3. Principal Investigator:

A4. Post holder(s) and starting date(s):

A5. Current spine point (for Universities)/pay band (for CCLRC):

Section-B: Past Achievements (to be completed in consultation with the post holders)

B1. Please list the contribution to the deliverables of the GridPP project since the start of the current grant. Please refer to the project map available via http://www.gridpp.ac.uk/pmb For each relevant deliverable, identified either by the GridPP deliverable number of the form x.y.z or the EDG deliverable number of the form Dx.y, please briefly describe the work.

B2. If applicable, which software have you designed, written or tested?

B3. If applicable, which system management tasks or hardware support were you responsible for?

B4. Which technical documents have you written or contributed to?

B5. Please list the conference proceedings and papers you have published (see http://www.gridpp.ac.uk/papers). In cases where this is joint authorship, please note your contribution.

B6. Please list talks/demos/posters/web pages or other communication material.

1. Are there other contributions that you would like to record? e.g. external roles, user support. Please briefly describe the work and why it has benefitted GridPP.

Note that this information will be extracted prior to evaluation by the PMB


Proforma-2: Future PlansProforma-2: Future Plans

The contact persons should circulate drafts based upon the proposal and subsequently submit:

1. a brief (1-2 pages) overview of the work you envisage in the GridPP2 period including any connection to work done in GridPP1;

2. explicitly defined deliverables, including dates, for the GridPP2 period; This should be based on the input you gave to the GridPP2 proposal but you have the freedom to change plans at this point. The granularity of these deliverables should be roughly one or two every 6 months per FTE of effort

3. if applicable, the proposed arrangements for leading the area and meeting the deliverables agreed between a consortium of institutes.

GridPP2 Work Area Contact

ATLAS/LHCb (GANGA) Roger Jones, Glenn Patrick

ATLAS Roger Jones

BaBar Roger Barlow

CDF/D0 (SAM) Iain Bertram, Rick St Denis

CDF Todd Huffman

CMS Peter Hobson

D0 Iain Bertram

LHCb Glenn Patrick

LHC Deployment Frank Harris

UKQCD Alan Irving

Other Roger Barlow, Dan Tovey

Data and Storage Management

Gavin McCance

Workload Management Dave Colling

Security and VO Management

Andrew McNab

Information & Monitoring Steve Fisher

Network Sector Pete Clarke

London Dave Colling

Southern Jeff Tseng

NorthGrid Roger Jones

ScotGrid Tony Doyle

Specialist Services Steve Lloyd

Tier-1 John Gordon


Proforma-3: Bid from Institutes Proforma-3: Bid from Institutes

The proposed Principal Investigators must submit one form per proposed FTE effort including:

1. a list of main objectives (as in the PPARC JES web submission form: see https://je-s.rcuk.ac.uk/eforms/secure/Login.asp);

2. a short abstract (as in the PPARC JES form);

3. the name of the principal investigator and co-investigators at each institute (as in the PPARC JES form);

4. the deliverables to be undertaken;

5. name of identified individual, where applicable.

6. If this is a proposed continuation of post, and with reference to the deliverables defined by the relevant group in Proforma-2, please describe the particular skills that the present postholder brings to the task.

7. Does this programme of work involve an experiment or group not currently engaged in GridPP? If so, please describe in a few sentences how this fits in with the current programme at your Institute.

2. Please describe any other effort at your institute in the GridPP2 period that may support the programme of work you have proposed. 8


Proforma-4: Proforma-4: Proforma-1 Evaluation ReportProforma-1 Evaluation Report

Instructions Please assess each of the returned Proforma-1 by answering the four questions below. You are assessing a post or work-area, so it would be appropriate to factor in personal knowledge in an objective manner where you can. As a guideline, numerical scores of 8, 9 or 10 would be appropriate when there are no serious concerns; scores of 5, 6 or 7 should be used where you feel, ideally, things could have been better. Scores of 4 and below flag significant concerns.

Questions Q1. What is your initial overall impression of the post(s) effectiveness? (1=poor, 10=excellent)?

This is your initial overall impression having read the completed proforma-1.

Q2. What do you see as the main role of these post(s) in GridPP1 (e.g. Software Design, Software Writing, Software Testing, System Management, Hardware Support, User Support, etc (please specify).) This is probably apparent from the answers to questions B2 and B3. Do not worry about roles that are less than about the 25% level.

Q3. For each of the roles identified in 2 above please assess the performance (1=poor,

10=excellent), the impact (1=small, 10=large) within GridPP, and the degree to which these activities have integrated into the overall experiment/middleware/test-bed programme (1, 10). The performance should reflect the quality and quantity of the work done. The impact on GridPP should reflect the direct benefit you perceive to the goals of GridPP (these goals may be explicit deliverables or it may be by enabling others to work on the deliverables. A low impact score is indicates that you regard (some of) the work performed as non-essential. )

Q4. Having read all the returns and with regard to the above evaluation, how would you rate the

overall effectiveness of the post(s) within the GridPP programme? (1=poor, 10=Excellent) This may well differ from your response to Q1 because you should now have factored in the analysis you performed in Q2 and Q3. Also, Q1 asks for an initial impression based on the case made in proforma-1, whereas you response to Q4 should include your knowledge of the whole GridPP programme and the contributions of other people. You should complete this question once you have examined all the returns.

Q5. Do you have first-hand knowledge of this post? (Yes/No) Q6. Is this post at your own institute? (Yes/No)


OutcomeOutcome

• By the end of this process we will be able to:– Continue the work started in GridPP1 (by many

people)– Define the GridPP2 Project Map tasks and metrics– Appoint additional staff in recognised areas to

contribute to agreed tasks– Contribute fully to EGEE– Start to go from Prototype to Production

• Beyond this, we anticipate a transition after GridPP2 in the PPARC Road Map where these areas are supported within the Particle Physics Programme

tony doyle - university of glasgow pparc review timeline 0. pparc call (february 2003) 1.gridpp2...

Documents

tony doyle university

glasgow testbed status

production team

cpu production testbed

glasgow gridpp2 operations

glasgow gridpp status

heterogeneous testbed

glasgow conclusions