scotgrid: a prototype tier-2 centre – steve thorn, edinburgh university scotgrid: a prototype...

ScotGrid: a Prototype Tier-2 Centre – Steve Thorn, Edinburgh University

SCOTGRID: A PROTOTYPE TIER-2 CENTRE

Steve ThornAuthors: A. Earl, P. Clark, S. Thorn

University of Edinburgh


Overview

• What is ScotGrid?• ScotGrid resources• LCG2 experiences (at Edinburgh)

– System administrator– Particle physics user

• Future plans• Conclusions


What is ScotGrid?• One of four Tier-2 sites in the UK• Durham, Edinburgh & Glasgow • Member of GridPP collaboration• Over 100 users• Supports: ATLAS, BaBar,

Bioinformatics, CDF, Computing Science, Engineering, LHCb, Medipix, MICE, PhenoGrid, UKQCD, ZEUS

• First UK Tier-2• Over 200 CPUs• ~40 TB disk• Initially funded by SHEFC (Scottish

Higher Education Funding Council)• www.scotgrid.ac.uk

http://www.scotgrid.ac.uk/


Hardware overview


Edinburgh UniversityFront Ends:

CE & SE: IBM x205: P-IV 1.8 GHz, 256 MB RAM

LCFG: IBM x340: dual P-III 1 GHz, 2 GB RAM

Storage (backend):

1 × IBM x440: 8 × Xeon 1.9 GHz, 32 GB RAM

2 × IBM FAStT900 providing 24 TB RAID

10 TB of 155 TB SAN

Worker Nodes: 4 × dual Xeon 2.8 GHz, 2 GB RAM, 200 GB disk

Housed at the multi-million pound Advanced Computing Facility. 20 Gbit/s link to campus.


Benchmarking

IBM x440: 8 × Intel Xeon 1.9 GHz, 32 GB RAM

High Performance Linpack benchmark using ATLAS libraries. Matrix calculations to test processor and memory performance.

Achieved ~75% of theoretical max performance with un-optimised system

Full results at www.ph.ed.ac.uk/~aearl/hpl/

http://www.ph.ed.ac.uk/~aearl/hpl/


Glasgow University

Front Ends (CE,SE,LCFG,UI):

4 × Transtec 1001L: P-IV 2.8 GHz, 512 MB RAM

Storage:

1 × IBM x370: P-III Xeon 700 MHz, 16 GB RAM

1 × IBM FAStT500 providing 5 TB RAID

Worker Nodes:

59 × IBM x330: dual P-III, 1 GHz, 2 GB RAM

34 × IBM blades: dual Xeon 2.4 GHz, 1.5 GB RAM

5 × IBM x340: dual P-III 1 GHz, 2 GB RAM


Usage statistics

Glasgow CPU farm1498570 CPU hours

4471 kilo SPECint2000 weeks

(23 September 2004)

ATLAS

LHCb

BaBar

CDF

Grid Data Management

Bioinformatics

ZEUS

UKQCD

Medipix

Information Retrieval

Device Modelling

MICE


Durham University

New ScotGrid member since September 2004

Worker Nodes: 40 × dual P-IV 2.2 GHz, 2 GB RAM, 30 GB disk

Based at the Institute for Particle Physics Phenomenology

Using Sun Grid Engine


LCG2 experience: sys admin• Storage emphasis – too few Front End machines • Installed with LCFGng • Storage limited to 1 TB until SRM based deployment• Hardware moved to Advanced Computing Facility – successful• Worker Node 15 miles (24 km) from Front Ends• Participating in LCG2 Test Zone:


Installation problems• Multiple network interfaces

– CE(2) and SE(3)– eth0 (external) is automatically

configured by LCFGng– Others require manual config. of

driver, network parameters (ip, netmask etc.) and routing

– SE eth0 not used so eth1 & eth2 manually defined

• SCSI and RAID controllers– SE only– Custom boot kernel– Full configuration of

/etc/modules.conf required

/* Second network interface eth1 */EXTRA(hardware.modlist) eth1+hardware.mod_eth1 alias eth1 tg3

EXTRA(network.interfaces) eth1network.ipaddr_eth1 192.168.1.6network.netmask_eth1 255.255.255.0network.hostname_eth1 glenlivetnetwork.network_eth1 192.168.1.0network.broadcast_eth1 192.168.1.255

REPLACE(install.installmethods,network,network route)

install.imethod_route route install/root


LCG2 Experience: user

Aim: to get a physicist with no grid experience to use LCG2

1. Applied for Grid certificate

Well documented but time consuming

2. Joined LHCb VO

3. Install UI software on user’s desktop

Not possible due to OS mismatch and firewall

4. Used Glasgow’s UI to successfully run jobs


Future plans• Edinburgh

– Deploy new Worker Nodes: 4 × dual processor from GridPP– Storage software development – SRM to SRB interface– Increase LCG storage to full capacity including SAN

• Glasgow– Upgrade older Worker Nodes– Migrate to Scientific Linux for Front Ends

• Durham– Deploy new Front End systems: 4 × dual processor from GridPP– Join LCG2 Test Zone

• Seeking funding – to increase CPU at Edinburgh and storage at Glasgow– Research and support staff

• Other institutes?


Conclusions

• Successfully proven Tier-2 concept• Reached million CPU hour landmark• Over 100 users within 13 research groups/collaborations• Gained valuable LCG2 knowledge (installation and

maintenance)• Edinburgh & Glasgow participating in LCG2 Test Zone• Next steps: prototype → production

– Encourage more particle physics analysis– Continue with data challenges– More resources and funding


Questions?

scotgrid: a prototype tier-2 centre – steve thorn, edinburgh university scotgrid: a prototype...

Documents

edinburgh university

prototype tier

gb ram slide

edinburgh university

edinburgh university

centre steve thorn authors

uk tier

gb disk