CMS Tier 1 at JINRV.V. Korenkov

for JINR CMS Tier-1 TeamJINR

XXIV International Symposium on Nuclear Electronics & Computing, NEC2013

2013, September 13

2Outline

• CMS Grid structure– role of Tier-1s– CMS Tier-1s

• CMS Tier-1 in Dubna– History and Motivations (Why Dubna?)– Network infrastructure– Infrastructure and Resources– Services and Readiness– Staffing– Milestones

• Conclusions

3

CMS Grid Structure

Tier Structure of GRID Distributed Computing:

Tier-0/Tier-1/Tier-2

Ian.Bird@cern.ch 4

Tier-0 (CERN):• accepts data from the CMS Online Data Acquisition and Trigger System

• archives RAW data • the first pass of reconstruction and performs Prompt Calibration

• data distribution to Tier-1Tier-1 (11 centers):• receives a data from the Tier-0 • data processing (re-reconstruction, skimming , calibration etc)• distributes data and MC to the other Tier-1 and Tier-2• secure storage and redistribution for data and MCTier-2 (>200 centers):• simulation• user physics analysis

5

CMS Tier-1 in Dubna

6

Tier1 centerTier1 center

The Federal Target Programme ProjectThe Federal Target Programme Project: : «Creation of the «Creation of the automated system of data processing for experiments at automated system of data processing for experiments at the LHC of Tier-1 level and maintenance of Grid services the LHC of Tier-1 level and maintenance of Grid services for a distributed analysis of these data» for a distributed analysis of these data»

Duration:Duration: 2011 – 20132011 – 2013

March 2011 - Proposal to create the LCG Tier1 center in Russia (official letter by Minister of Science (official letter by Minister of Science and Education of Russia A. Fursenko has been sent to and Education of Russia A. Fursenko has been sent to CERN DG R. Heuer):CERN DG R. Heuer):

NRC KI for ALICE, ATLAS, and LHC-BLIT JINR (Dubna) for the CMS experiment

Full resources - in 2014 to meet the start of Full resources - in 2014 to meet the start of next working LHC session.next working LHC session.

September 2012 – Proposal was reviewed by WLCG OB and JINR and NRC KI Tier1 sites were accepted as a new “Associate Tier1”

7

Why in Russia?Why Dubna?

• In frames of the RDIG project (a participant of the WLCG/EGEE projects), a grid-infrastructure excepted by LHC experiments has been successfully launched as a distributed cluster RuTier2 (Russian Tier2) and JINR cluster JINR-LCG2 is the main one in RDIG as to its performance.

8

JINR-LCG2 JINR-LCG2

~40% of CPU time in RDIG for 2011-

2013

9JINR Central Information and Computing Complex (CICC)

JINR-LCG2 Normalised CPU time by LHC VOs. January - September 2013.

More than 3 million jobs runTotal normalised CPU time – 20 346 183 kSI2K-hours

Grid users (WLCG)

Local JINR users (no grid)

Jobs run by JINR Laboratories and experiments executed at CICC January - September 2013.

BLTP 8,99%

MPD 39,12%

LRB 1,00% LIT 1,61%COMPASS 13,23%

DLNP 3,84%

FLNP 0,40%

VBLHEP 3,57%

PANDA 27,39%

LBES 0,86%

http://accounting.egi.eu/http://lit.jinr.ru/view.php?var1=comp&var2=ccic&lang=rus&menu=ccic/menu&file=ccic/statistic/stat-2013

10CMS Computing at JINR

the first RDMS CMS web-server (in 1996);

full-scale CMS software infrastructure support since 1997

JINR CMS Tier2 center is one of the most reliable and productive CMS Tier2 centers worldwide (in the top ten best) the most powerful RDMS CMS Tier2 center

CMS Regional Operation Center are functioning in JINR since 2009

The core services needed for WLCG Tier-1 are computing service, a storage service, information service.

The primary Tier-1 tasks can be divided into recording raw data from CERN and storing them on tape; recording processed data from CERN and storing them on disk; providing data to other Tier-1 / Tier-2; reprocessing raw data; event simulation calculations.

111111

Russia Normalized CPU time per SITE and VO (2012-2013)

All VORussia - 409,249,900 JINR - 183,008,044

CMSRussia - 112,025,416JINR - 67,938,700 (61%)

12

Network infrastructure

13 The Core of LHC Networking: LHCOPN and Partners

13

15

Infrastructureand

Facilities

16

17JINR CMS Tier-1 progress

● Disk & server installation and tests: done● Tape system installation: done● Organization of network infrastructure

and connectivity to CERN via GEANT: done

● Registration in GOC DB and APEL: done● Tests of WLCG services via Nagios: done

20132013 20142014 20152015 20162016

CPU (HEPSpec06)CPU (HEPSpec06)

Number of coreNumber of core

2880028800

24002400

5760057600

48004800

69120

5760

82944

6912

Disk (Terabytes)Disk (Terabytes) 35003500 45004500 5400 6480

Tape (Terabytes)Tape (Terabytes) 57005700 80008000 9600 10520

Link CERN-JINRLink CERN-JINR 1010 1010 40 40

18

18

JINR monitoringJINR monitoring

Network monitoring information system -

more than 423 network nodes are in round-the-

clock monitoring

19

Servicesand

Readiness

20CMS-specific activity

● Currently commissioning Tier-1 resource for Currently commissioning Tier-1 resource for CMSCMS::– Local Tests of CMS VO-services and CMS SW Local Tests of CMS VO-services and CMS SW – The PhEDEx LoadTest (tests of data transfer links)The PhEDEx LoadTest (tests of data transfer links)– Job Robot Tests (or Job Robot Tests (or tests via HammerCloudtests via HammerCloud))– Long-running CPU intensive jobsLong-running CPU intensive jobs– Long-running I/O intensive jobsLong-running I/O intensive jobs

● PHDEDX tPHDEDX transferred of ransferred of RAW RAW input data to our storage input data to our storage element with transfer efficiency around 90%element with transfer efficiency around 90%

● Prepared services and data storage for the reprocessing of Prepared services and data storage for the reprocessing of 20120122 88 TeV reprocessing TeV reprocessing

21CMS Tier-1 Readiness

22CMS Tier-1 in Dashborad

Data transfer link to CERN

23/982323

Frames for Grid cooperation of JINRFrames for Grid cooperation of JINR Worldwide LHC Computing Grid (WLCG) Enabling Grids for E-sciencE (EGEE) - Now is EGI-InSPIREEGI-InSPIRE RDIG Development CERN-RFBR project “Grid Monitoring from VO perspective” BMBF grant “Development of the grid-infrastructure and tools to provide joint

investigations performed with participation of JINR and German research centers” “Development of grid segment for the LHC experiments” was supported in frames

of JINR-South Africa cooperation agreement; Development of grid segment at Cairo University and its integration to the JINR Development of grid segment at Cairo University and its integration to the JINR

GridEdu infrastructureGridEdu infrastructure JINR - FZU AS Czech Republic Project “The grid for the physics experiments”JINR - FZU AS Czech Republic Project “The grid for the physics experiments” NASU-RFBR project “Development and support of LIT JINR and NSC KIPT grid-NASU-RFBR project “Development and support of LIT JINR and NSC KIPT grid-

infrastructures for distributed CMS data processing of the LHC operation”infrastructures for distributed CMS data processing of the LHC operation” JINR-Romania cooperation Hulubei-Meshcheryakov programme JINR-Moldova cooperation (MD-GRID, RENAM) JINR-Mongolia cooperation (Mongol-Grid)

24

ROLE FTE

Administrative 1.5

Network support 2

Engineering Infrastructure

2.5

Hardware support 3

Core software and WLCG middleware

4.5

CMS Services 3.5

Total 17

Korenkov V.

Dolbilov A.

Shmatov S.

Trofimov V.

Mitsyn V.

Staffing

25

Milestones

26Objective Target date

Presentation the Execution Plan to WLCG OB Sep 2012

Prototype

Disk & Servers installation and tests Oct 2012

Tape system installation Nov 2012

Organization of network infrastructure and connectivity to CERN via GEANT (2 Gb) Nov 2012

WLCG OPN integration (2 Gb) and JINR-T1 registration in GOCDB including integration with the APEL accounting system

Dec 2012

M1 Dec 2012

LHC OPN functional tests (2 Gb) May 2013

Test of WLCG and CMS services (2 Gb LHCOPN) May 2013

Test of tape system at JINR: data transfers from CERN to JINR (using 2 Gb LHC OPN) May 2013

Test of publishing accounting data May 2013

Definition of Tier 2 sites support May 2013

Connectivity to CERN 10 Gb Jul 2013

M2 Jul 2013

LHC OPN functional tests (10 Gb) Aug 2013

Test of tape system at JINR: data transfers from CERN to JINR (using 10 Gb LHC OPN) Aug 2013

Upgrade of tape, disk and CPU capacity at JINR Nov 2013

M3 Nov 2013

85% of the job capacity running for at least 2 months

Storage availability > 98% (functional tests) for at least 2 months

Running with > 98% Availabilities & Reliabilities for at least 30 days

WLCG MoU as an associate Tier-1 center Feb 2014

Disk & Tape & Servers upgrade Oct 2014

M4 Dec 2014

Milestones of the JINR CMS Tier-1 Deployment and Commissioning

27 Main tasks for next yearsMain tasks for next years

Engineering infrastructure (system of Engineering infrastructure (system of uninterrupted power supply and climate-uninterrupted power supply and climate-control)control)

High-speed reliable network infrastructure High-speed reliable network infrastructure with the allocated reserved channel to CERN with the allocated reserved channel to CERN (LHCOPN)(LHCOPN)

Computing system and storage system on the Computing system and storage system on the basis of disk arrays and tape libraries of high basis of disk arrays and tape libraries of high capacitycapacity

100% reliability and availability. 100% reliability and availability.

Lyon/CCIN2P3Barcelona/PIC

De-FZK

US-FNAL

Ca-TRIUMF

NDGF

CERNUS-BNL

UK-RAL

Taipei/ASGC

26 June 2009

Amsterdam/NIKHEF-SARA

Bologna/CNAF

Russia:NRC KI

JINR

29/98

The 6th International Conference "Distributed Computing and Grid-technologies in Science

and Education" (GRID’2014)Dubna, 30 June-5 July 2014

GRID’2012 Conference

22 countries, 256 participants, 40 Universities and

Institutes from Russia,

31 Plenary, 89 Section talks

30Conclusions

In 2012-2013 CMS Tier1 prototype was In 2012-2013 CMS Tier1 prototype was created in Dubnacreated in Dubna Disk & server installation and testsDisk & server installation and tests Prototype tape system installation and testsPrototype tape system installation and tests Organization of network infrastructure and Organization of network infrastructure and

connectivity to CERN via GEANTconnectivity to CERN via GEANT Registration in GOC DB and APELRegistration in GOC DB and APEL Tests of WLCG services via NagiosTests of WLCG services via Nagios CMS-specific testsCMS-specific tests Commissioning data transfer links (T0-T1, T1-T1, Commissioning data transfer links (T0-T1, T1-T1,

T1-T2) in progressT1-T2) in progress

We expect to meet the start of next LHC run We expect to meet the start of next LHC run with full resources required (for the end of with full resources required (for the end of 2014)2014)