oliver gutsche for the data operations team data operations us cms collaboration meeting 2010 06....

Oliver Gutschefor the

Data Operations Team

Data Operations

US CMS Collaboration Meeting 201006. May 2010

2US CMS Collaboration Meeting 2010 - Data Operations05/06/10

Outline‣ Introduction to Data Operations

‣T0 processing

‣T1 processing

‣T2 MC production

‣Transfer Operation

‣Release Validation

‣How to stay informed about samples, MC, etc.

‣Service work and other contributions


CMS Computing

‣ Traditional view of the tiered computing infrastructure

‣ Data Operations handles all central tasks on the T0, T1 and T2 level

‣ The project is lead by Markus Klute (MIT) and Oliver Gutsche (FNAL)

‣ More and more T3 sites (especially in the US) have been added or are going to be added

T0T0T0T0

T1: USAT1: USA

T1: ItalyT1: Italy T1: FranceT1: France

T1: SpainT1: SpainT1: T1: GermanyGermany

T1: TaiwanT1: TaiwanT1: UKT1: UK

T2T2T2T2 T2T2T2T2 T2T2T2T2 T2T2T2T2

T2T2T2T2

T2T2T2T2

T2T2T2T2

T2T2T2T2

T2T2T2T2

T2T2T2T2

T2T2T2T2 T2T2T2T2 T2T2T2T2T2T2T2T2 T2T2T2T2 T2T2T2T2

T2T2T2T2

T2T2T2T2

T2T2T2T2

T2T2T2T2

T2T2T2T2

T2T2T2T2


T0 processingTask leaders

Josh Bendavid (MIT)Marco Zanetti (MIT)

IntegrationDave Mason (FNAL)

Stephen Gowdy (CERN)

T0 at CERNT0 at CERN

TapeTape

‣Custodial copy of data coming from Custodial copy of data coming from

the detectorthe detector

‣Inactive “cold” copy of all data not Inactive “cold” copy of all data not

mean to be accessed, only to be mean to be accessed, only to be

archivedarchived

CPUCPU

‣Process all data coming from the Process all data coming from the

detectordetector‣ExpressExpress (latency 1 hour)(latency 1 hour)

‣Bulk: Bulk: Repacking, PromptRecoRepacking, PromptReco (latency 24 hours plus conditions hold)(latency 24 hours plus conditions hold)

‣PromptSkimmingPromptSkimming is dispatched is dispatched by the T0 infrastructure although run by the T0 infrastructure although run at the T1 sitesat the T1 sites

NetworkNetwork

‣Transfer all data from the detector Transfer all data from the detector

to T1 sites for archivalto T1 sites for archival‣Time critical otherwise buffers at T0 Time critical otherwise buffers at T0 overflowoverflow


Data taking 2010‣ Current acquisition era:

Commissioning10

‣ 1 physics Primary Dataset (PD)

‣ Various Secondary Datasets (SD) and Central Skims (SD)

‣ All datasets have a custodial copy at one of the T1 sites

‣ FNAL has a replica of ALL data

‣ Next acquisition era: Run2010A

‣ Will be put in place at Linst > 1E29

‣ 7 physics PDs: JetMETTauMonitor, JetMETTau, EGMonitor, EG, MuMonitor, Mu, MinimumBias

‣ FNAL will have a copy of all data


T1 processing

T1 siteT1 site

TapeTape

‣Custodial copy of data and MCCustodial copy of data and MC

‣Non-custodial replica of data Non-custodial replica of data

and MCand MC CPUCPU

‣PromptSkims (produce SD/CS)PromptSkims (produce SD/CS)

‣Re-reconstruction passes on Re-reconstruction passes on

data and MC including SD/CS data and MC including SD/CS

productionproduction

‣MC production if resources are MC production if resources are

free and T2 level fully utilizedfree and T2 level fully utilizedNetworkNetwork

‣Serve data and MC samples to Serve data and MC samples to

T2 sitesT2 sites

‣Archive data from T0, other T1 Archive data from T0, other T1

sites and the T2 level (MC)sites and the T2 level (MC)

Task leaders

Kristian Hahn (MIT/FNAL)Guillelmo Gomez-Ceballos

(MIT)

IntegrationJose

Hernandez (CIEMAT)Claudio

Grandi (INFN Bologna)


T1 operation in 2010

Jobs per day

‣ Summer09 MC: re-digitization / re-reconstruction

‣ Input:

‣ ~575 Million events, 450 TB

‣ Processing:

‣ ~500 workflows

‣ ~500,000 processing jobs

‣ 90% of the events processed in ∼5 days

‣ Tails finished after 2 weeks

‣ Output:

‣ ~1500 datasets

‣ ~400 TB RAW, ~220 TB RECO, ~65 TB AOD

‣ 2010 data re-reconstruction

‣ Requested when change in release or global tag in T0

‣ 2 passes until now: Apr1ReReco & Apr20ReReco

‣ PD: MinimumBias & ZeroBias plus associated SD/CS

‣ Also re-reconstruction of corresponding MinBias MC samples

‣ Full request currently takes ~2-3 days due to long tails

‣ NEW: train model

‣ Run a new re-reconstruction pass every week

‣ Use stable release and pick up latest conditions and all added statistics

‣ Train leaves the station on Thursday’s, 8 PM CEST


T2 MC production

T2 siteT2 siteCPUCPU

‣50% for analysis50% for analysis

‣50% for MC production50% for MC production‣Standard MC production in Standard MC production in multiple steps (GEN-SIM-RAW, multiple steps (GEN-SIM-RAW, GEN-SIM-RECO, … )GEN-SIM-RECO, … )‣Newer workflows using LHE Newer workflows using LHE datasets as inputdatasets as input‣PileUp workflows using MinBias PileUp workflows using MinBias or data samples for PileUp or data samples for PileUp mixingmixing NetworkNetwork

‣Archive produced MC samples Archive produced MC samples

at T1 sites for custodial storageat T1 sites for custodial storage‣Samples are moved from the Samples are moved from the various T2 sites to one T1 site to various T2 sites to one T1 site to consolidate the distributed consolidate the distributed productionproduction

Task leaders

Ajit Mohapatra (U Wisconsin)

Valentina Dutta (MIT)


T2 MC production in 2009/2010

‣ Record of over 300 Million events in 1 month

‣ Production with lots of variation over the year due to request situation

‣ Currently requests gets produced quickly as there is literally no queue

10

US CMS Collaboration Meeting 2010 - Data Operations05/06/10

Transfer Operation

PhEDExPhEDExCentralCentral

‣Handles dataset transfers Handles dataset transfers

between all CMS sitesbetween all CMS sites‣ Database to keep track of Database to keep track of

files, blocks and their files, blocks and their location location

‣ Links between sites which Links between sites which can be used for transferscan be used for transfers

‣ Central scheduling of Central scheduling of transfers and balancing transfers and balancing between source sitesbetween source sites

‣ Infrastructure to submit Infrastructure to submit transfer or deletion transfer or deletion requests (Webpage)requests (Webpage)

Per sitePer site

‣Agents that handleAgents that handle‣ TransfersTransfers‣ DeletionsDeletions‣ Verification / consistency Verification / consistency

checkschecks

Task leaders

Paul Rossman (FNAL)

Si Xie (MIT)

11


Transfer operation in 2009/2010

‣ Totally transferred data volume in last 52 weeks: 17 PB

‣ Site with highest incoming and outgoing volume: FNAL

Transfer volume in last 52 weeksby Destination

Transfer volume in last 52 weeksby Source

Significant effort

has to be spent to debug

transfer problems

12


Release Validation

CERN & FNALCERN & FNALRelease validationRelease validation

‣Produce special MC samples Produce special MC samples

and data reconstructions for all and data reconstructions for all

releasesreleases‣Standard set:Standard set:‣Turnaround 24 hours on 500 Turnaround 24 hours on 500 slots at CERNslots at CERN‣Now mostly run at FNAL using Now mostly run at FNAL using opportunistic cycles in parallel to opportunistic cycles in parallel to T1 productionT1 production‣Run for all releases except Run for all releases except patch releasespatch releases‣High statistics set:High statistics set:‣Turnaround 1 week for higher Turnaround 1 week for higher statistics samplesstatistics samples‣PileUp and HeavyIon samplesPileUp and HeavyIon samples‣Produced in parallel to standard Produced in parallel to standard set outside the 24 hour windowset outside the 24 hour window

Task leaders

Oliver Gutsche (FNAL, interim)

Special OperatorDiego Reyes (U Los Andes, Colombia)

13


Release Validation in 2009/2010

‣ Significant contribution to software validation and stability of releases for production

‣ High visibility task with reports in all major offline & computing meetings

20092009 20102010

ReleasesReleases 86 28

Events*Events*730,054,6

90377,701,9

88

Size [GB]Size [GB] 108,305 51,441

* double counting RAW, RECO and AOD events

14


User advice

‣ To stay informed about current and future samples:

‣ Requests are submitted and acknowledged in

‣ [email protected]

‣ Samples are announced in

‣ [email protected]

‣ Both lists are low traffic lists

‣ We strongly discourage asking questions or replying to threads on these lists

15


Your contribution to Data Operations

‣ Data Operations can award service credit for all its tasks

‣ Graduate students and post-docs can spend 25% of their time working for Data Operations as operators

‣ Interest in computing and high scale data processing required

‣ Training would give detailed insight into computing infrastructure and software and prepare ideally for all analysis tasks

‣ Very talented graduate students and post-docs can spend 50% of their time filling one of the 5 task leader positions

‣ High visibility in the collaboration

‣ Significant contribution to the success of the experiment

‣ Closely connected to everything related to data and MC, good for analysis

‣ Data Operations is constantly looking for talents to replenish the current manpower

‣ Urgently we are looking for leaders of the Release Validation task

‣ Please contact Markus and Oliver if you are interested or have questions.

16


The Data Operations Team

‣ Project lead:

‣ Markus Klute & Oliver Gutsche

‣ Task leaders:

‣ Josh Bendavid, Marco Zanetti, Kristian Hahn, Guillelmo Gomez Ceballos Retuerto, Ajit Mohapatra, Valentina Dutta, Paul Rossman, Si Xie

‣ Operators

‣ Andrew Lahiff, Andrey Tsyganov, Aresh Vedaee, Ariel Gomez Gomez Diaz, Arnaud Willy J Willy J Pin, Derek Barge, Dorian Kcira, Gilles De De Lentdecker, Jeff Haas, Jen Adelman-McCarthy, Joseph Mccartin, Julien Caudron, Junhui Liao, Lukas Vanelderen, Nicolas Dominique, Dominique Schul, Petri Lehtonen, Subir Sarkar, Vincenzo Spinoso, Xavier Janssen

17


Summary & Outlook

‣ Data Operations handles all central processing and production tasks on the T0, T1 and T2 level of the distributed computing infrastructure

‣ Current performance in the areas of data taking,skimming, re-reconstruction, MC production, transfers and release validation is excellent

‣ We are always looking for interested and talented people to help us getting the data as quickly and reliably as possible to all CMS collaborators.

‣ Don’t miss the Computing Shift presentation on Friday

18


Glossary‣ Express

‣ Low latency processing of extra express stream(s) from the detector (40 Hz), latency 1 hour

‣ Repacking

‣ Binary streamer files from the detector are translated into the ROOT format

‣ PromptReco

‣ RAW ROOT files are reconstructed promptly at the T0

‣ PromptSkimming

‣ As soon as blocks of data (groups of files either constraint by runs or by number of files (1000)) are completely stored on tape at the custodial T1 site, the PromptSkimming system sends jobs to this site to run the skimming workflows

19


Glossary‣ Primary Dataset (PD)

‣ Data stream from P5 is split into PDs according to trigger selections with minimal overlap

‣ Needed for processing in distributed computing infrastructure

‣ Produced at T0, re-reconstructed at T1 sites

‣ Secondary Dataset (SD)

‣ More restrictive trigger selection than PD

‣ Produced at T1 sites with PromptSkimming system, also produced after re-reconstruction passes at T1 sites

‣ Central Skims (CS)

‣ In addition to a more restrictive trigger selection than the parent PD, reconstruction level selections are applied

‣ Same processing as SD

oliver gutsche for the data operations team data operations us cms collaboration meeting 2010 06....

Documents