Physicists's experience of the EGEE/LCG infrastructure usage for CMS jobs submission

Natalia Ilina (ITEP Moscow)NEC’2007

Outline

What do physicists need to know about grid?

Overview of existing frameworks for jobs submission in CMS

Specific of grid tutorials for physicists

2

What do physicists need to know about grid?

1. Bench mark – terminology: UI, CE, SE, WN, RB, VO…

UI (User Interface) – a machine where user’s certificate is installed. From the UI user can be authenticated to use the Grid resources, submit a job for execution on a CE, list all the resources suitable to execute a job, replicate and copy files, cancel jobs, retrieve the output, show the status of submitted jobs

CE (Computing Element) – a Grid batch queue, identified by a pair <hostname>:<port>/<batch queue name>

SE (Storage Element) - provides uniform access and services to large storage spaces. The SE may control large disk arrays, mass storage systems (MSS) and the like.

2. Operations with user certificate

3

3. The full chain for job submission with grid:

to find information about needed data

to write configuration file with the task

to organize the most useful saving of the output

maybe to realize the “bad” CE to save time

submit jobs

check the status

kill and resubmit jobs (if it’s necessary)

see and analyze the output

4

5

DBS data discovery page for CMS(very useful!)

6

The aim - to simplify the work of users to create and to submit analysis jobs into the grid environment.

Its purpose is to allow users, without specific knowledge of grid infrastructure, to access and analyze remote data as easily as in a local environment, hiding the complexity of distributed computational services.

Available tools:

ASAP (Arda Support for CMS Analysis Processing) : https://lxarda13.cern.ch/docs/index.html

CRAB (CMS Remote Analysis Builder) - more popular among CMS physicists: https://twiki.cern.ch/twiki/bin/view/Main/CRAB

CMS frameworks for jobs submission: ASAP and CRAB(from physicists point of view)

7

# specify directory to store tasksjobdir = /afs/cern.ch/user/n/nilina/ASAP# store output at SEstore_output = 1output_se = srm.cern.choutput_se_path=/castor/cern.ch/user/n/nilina# specify grid to submit to (lcg, my)grid = lcg#specify datasetdbs_version = 2primary_dataset = DY_mumu_10tier = GEN-SIM-DIGI-RECOprocessed_dataset = CMSSW_1_3_1-Spring07-1349# pset file (CMSSW configuration file)pset_file = Z2muons.cfgoutput_files = MUONS.datevents_required = 5000events_per_job = 1000# specify minimum time requirements for the jobmin_wall_clock_time = 100min_cpu_time = 100

Example of configuration file for ASAP

[CRAB]jobtype = cmsswscheduler = glitecoll

[CMSSW]datasetpath = /DY_mumu_10/CMSSW_1_3_1-Spring07-1349/GEN-SIM-DIGI-RECOpset = test.cfgtotal_number_of_events = 20000events_per_job = 1000output_file = MUONS.datuse_dbs_2 = 1

[USER]copy_data = 1storage_element = srm.cern.chstorage_path = /castor/cern.ch/user/n/nilina/Z2taureturn_data = 1use_central_bossDB = 0use_boss_rt = 0

[EDG]rb = CERNproxy_server = myproxy.cern.chvirtual_organization = cmsretry_count = 0lcg_catalog_type = lfclfc_host = lfc-cms-test.cern.chlfc_home = /grid/cms

Example of configuration file for CRAB

8

CRAB: already ready configuration files

9

10

ASAP_Monitor - very nice featureUser can delegate responsibility for tasks to ASAP server. The server can

process and monitor the users jobs and perform resubmissions in the case of failure.

11

12

Grid tutorials for physicists. Our experience.We have already first experience – training courses in Russia (JINR,

ITEP)

Program of the tutorial in ITEP (April 2007):Lecture's part:

1. "Short introduction to LCG/EGEE" 2. "CMS user jobs submission with the usage of ASAP"

Practical part: “CMS user jobs submission with the usage of ASAP”

13

Physicists don’t need to know all details about LCG infrastructure (only main points)

Physicists need to get the results (output files) in the easiest way and short time.

All practical steps of jobs submission should be clarified for participants (from data finding to results retrieving)

Tutorial should have practical part – to provide the participants with first experience (so participants should have already their grid certificates in advance!)

Specific of grid tutorials for physicists

Summary

• LHC start-up is in the near future. So physicists should be able to do the analysis with Grid

• Physicists don’t need to understand the details of LCG infrastructure (in comparison with developers),

but the full chain of jobs submission should be clarified

•There are tools in CMS (CRAB, ASAP) which hide the complexity from users

• The corresponding tutorials for physicists is needed

14