infso-ri-508833 enabling grids for e-science geant4 physics validation: use of the grid resources...

INFSO-RI-508833

Enabling Grids for E-sciencE

www.eu-egee.org

Geant4 Physics Validation: Use of the GRID Resources

Patricia Mendez Lorenzo

CERN (IT-GD) / CNAF

Geant4 Bio-Medical DevelopmentsGeant4 Physics Validation

INFN Genova, 13-20 July 2005

Genova 13th-20th July Patricia Mendez Lorenzo


INFSO-RI-508833

Outlook

Introduction to the LCG

Geant4 in LCG

First Geant4 Productions

Results and Summary

Future Plans



INFSO-RI-508833

What is the LCG?

The LHC: Generation of 40 million particle collisions (events) per second

at the center of each for experiments

Reduce by online computers that filter out

a few hundred good events per sec

Recorded on disk and magnetic tape

at 100-1000 MB/sec: 15 PB/year

Here it is where

the GRID environment

comes in



INFSO-RI-508833

LCG Service Hierarchy

Tier-0 – the accelerator centreData acquisition and initial

processingDistribution of data to the different

Tier’s

Canada – Triumf (Vancouver)France – IN2P3 (Lyon)Germany – Forschunszentrum KarlsruheItaly – CNAF (Bologna)Netherlands – NIKHEF/SARA (Amsterdam)Nordic countries – distributed Tier-1

Spain – PIC (Barcelona)Taiwan – Academia SInica (Taipei)UK – CLRC (Oxford)US – FermiLab (Illinois) – Brookhaven (NY)

Tier-1 – “online” to the data acquisition process high availabilityManaged Mass Storage – grid-enabled data serviceData-heavy analysisNational, regional support

Tier-2 – ~100 centres in ~40 countriesSimulation

End-user analysis – batch and interactive



INFSO-RI-508833

Who is who in LCG?

physicsgroup

Tier2

Lab a

Uni a

Lab m

Lab b

Uni y

Uni x

Tier3physics

department

Desktop

Germany

Taiwan UK

FranceItaly

USA

Netherlands

NordicTier-1

CERN Tier 0

Spain

Canada

Uni b

Lab c

Uni n

regional group



INFSO-RI-508833

LCG in the World

May 2005140 Grid sites

34 countries

12000 CPUs

8 PetaBytes

30 sites3200 cpus

25 Universities4 National Labs2800 CPUs

Grid3



INFSO-RI-508833

How the LCG works?

UI UI

CE

RB/BDII

SE

WN

WN

WN

WN

WN

WN

LFC

Connections to UI

Resources Searching

Sent to th

e batc

h

system Distribution to CPUs

Ouputs copied to

Storage Resources

Catalogs getting track of the inputs



INFSO-RI-508833

Geant4 in LCG

◘ Electromagnetic and Hadronic physics are fundamental features to be properly simulated in Geant4, however they are extremely CPU demanding

▪ Number of events and energy depending: 1 event of 1GeV ~ 0.03 sec (2.4GHz machine)1 event of 300 Gev ~ 9-10 sec

◘ Goal during the Software Validation: Comparison some shower observables between the two different Geant4 versions and check statistical significant changes

◘ Applications in LCG: First application last December 2004Second application end of June 2005

▪ A total amount of about 3 years of CPU time (1GHz machine)

▪ Very small output for the whole production: 15-20 GB

GRID

GRID



INFSO-RI-508833

Geant4 in LCG

◘ Samplings: ▪ 7 simplified detectors

FeSci, CuSci, PbSci, CuLAr, PbLAr, WLAr, PbW04 ▪ 7 different particles (8 in the 2nd production)

e- (2nd production), pi+, pi-, k+, k-, k0L, p, n ▪ 23 different beam energies (GeV)

1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40, 50, 60, 80, 100, 120, 150, 180, 200, 250, 300, 1000(never achieved)

▪ 5 physics list LHEP, QGSP, QGSC, QGSP_BIC, QGSP_BERT



INFSO-RI-508833

Geant4 in LCG

◘ Strategy:▪ First Production

◦ Comparison of 7.0.cand01 vs 6.2.p01◦ During the event production phase, 5635 had to be run for each Geant4 version: 11270 jobs to be run◦ Finally the statistical tests were applied to each pair of Geant4 version outputs

▪ Second Production◦ Comparison of 7.0.p01 vs 7.1.cand01◦ During the event production, 6440 jobs had to be run◦ This time each production job contained the production of both Geant4 versions and the statistical analysis



INFSO-RI-508833

Geant4 Production in LCG

◘ Stages: 1. Software installation: Installation of the Geant4 packages (with all the required

external additional packages: PI, AIDA, etc) ▪ Software provided via a tar file ▪ Installation through jobs using specific LCG tools ▪ Fundamental request for the sites: Shared area between

WNs and perfectly definition of the software installation region

2. Events production: ▪ Jobs sent by bunches of 1227 (1288 2nd production)

defined by each physics list ▪ 5000 events in each job were produced

3. Analysis: Statistical tests to perform the comparison between the two G4

versions



INFSO-RI-508833

Geant4 Production in LCG

◘ General Characteristics: ▪ VO:

◦ 1st Production: dteam (6 certificates, one as dteamsgm) ◦ 2nd Production: alice (2certificates, one as alicesgm)

▪ Sites and middleware operating system: ◦ 1st Production: RedHat7.3 ◦ 2nd Production: Scientific Linux

▪ Resources: ◦ 1st Production: Own RB+BDII+UI: lxb2006 at CERN ◦ 2nd Production: lxplus resources and 2 BDII

▪ All output: ◦ 1st Production: About 30 GB stored at CERN (lxn1183) ◦ 2nd Production: Comparable quantity stored at CERN (lxn1180)

afs Geant4 area at CERN was set to hold the outputs



INFSO-RI-508833

Framework developed for Geant4

◘ Generation of a general framework consisting of 3 major tools:

▪ Tool for general and automatic job submission ▪ Tool for events generation in all those sites where the

software has been installed ▪ Tool for data analysis (not needed during the 2nd Production)◘ First Part: Tool for job submissionMethodology:Methodology: ▪ Copy and registry of the Geant4 package

◦ A file containing the TURL is created and is passed to the WN ▪ Follow up of candidates able to admit Geant4 jobs ▪ Selection of long queues only ▪ Automatic built of the .jdl files for each long queue

◦ Built taking as base those proposed by the user joining the name of

the queue where to submit the job◦ Software Installation tools are used to perform the

installation ▪ Submission of these files to each queue



INFSO-RI-508833


◘ Software Installation tool(Tool submitted in the first step to all sites to install the software) ▪ First step:

◦ The tar file is copied from the SE at CERN to the WN◦ It is untar and copied to the VO_DTEAM_SW_DIR area

▪ Second Step: Software Installation tool◦ Some Geant4 tests are performed to validate the installation◦ If succeeded a tag is published in the Information System

▪ Results:◦ The software installation was tried in 63 sites◦ 1st Production: 28 sites ◦ 2nd Production: 35 sites

▪ Main Problems:◦ Sites were having submission problems◦ Sites did not have defined the VO_DTEAM_SW_DIR area or did not have shared area among WNs



INFSO-RI-508833


◘ Second Step: Tool for the ProductionStrategy:Strategy: ▪ Only long queues will be used to run the production ▪ All outputs (hbook files) will be stored at CERNMethodology:Methodology: ▪ Geant4 provides their own code to perform the

events production ▪ Python Script for each type of particle, energy,

physics list and calorimeter is created by the framework from one template provided by Geant4

▪ Generation of one jdl per job containing the code provided by Geant4 (the same for all jobs) + that script generated by the framework and changing for each job

▪ Submission of all jdl files to all sites containing the Geant4 installation



INFSO-RI-508833


Results (First and Second Production):Results (First and Second Production): ▪ A hbook file containing 5000 event is created in the case the

production succeeded ▪The name of the file is created by the framework containing the type

of particles, the energy, the physics list and the calorimeter within the name (important to perform later the comparison)

▪ The hbook file is copied and registered to a disk at CERN

·During the 2nd production a tar file containing different files should have been created in the case the job succeeded. This file was retrieved to the afs area delivered for this aim and copied and registered to the grid.

Around 4508 jobs (two physics list for both Geant4 versions) were run in lest than 2 weeks in 28 sites with a efficiency of about 87%

And for the 2nd production the results are provided by Alberto Ribon



INFSO-RI-508833


◘ Before the analysis stage the outputs should be checked (only for the 1st Production):

▪ Dealing with about 5000 outputs is not an easy task◦ Tool able to print in a file all the LFNs in the case the efficiency was 100% (reference file)◦ Output retrieve (only std.out files). If succeeded the file will contain the name of the LFN◦ A 2nd tool will check all std.out files looking for the succeeded jobs

- The corresponding GUID and LFN will be stored (test file) and compared with the information included in the reference file

▪ At this point it was more important for us to analyze the successful jobs than to understand the cause of the unsuccessful ones (BUT THIS PROCEDURE IS WRONG!)



INFSO-RI-508833


◘ Third Step: Tool for the Analysis (Only for the 1st Production)

Methodology:Methodology:

▪ Search of common successful outputs in both Geant4 Versions

▪ Each couple of successful outputs are copied into a local area and analyzed with their own tools

▪ Finally the copy is removed from the local area



INFSO-RI-508833

Update of the Framework

▪ This framework covered the Geant4 requirements for its first production

▪ It’s not useful for larger production ◦ Difficult to deal with the output and visualize the results

◘ A new complete tool has been developed for large production

▪ Flexible enough to be used for any VO and any user application

▪ Most of the improvements mostly relative to the outputs handle

Documentation: “LCG2 User Guide”http://grid-deployment.web.cern.ch/grid-deployment/cgi-bin/

index.cgi?var=eis/docs

Download: http://goc.grid.sinica.edu.tw/gocwiki/User_tools



INFSO-RI-508833


◘ The new framework consists mainly of two tools:

▪ Tool to perform the automatic job submission ▪ Tool to retrieve and handle the corresponding output

1. Automatic job submissionOverview: Given an user’s jdl this tool performs the following actions: ◦ It lists all sites able to run the jdl provided by the user ◦ It creates automatically a jdl file based on that provided by

the user ◦ It submits the just created jdl containing the user

application(s) Moreover it creates a subdirectory (defined by the user)

containing a list of the sites where the jobs have been submitted, the corresponding jdls and the jobs IDs



INFSO-RI-508833


Additional Features: ◦ The user can define the queues where the jobs are submitted. These queues are checked to see whether it fixes the job requirements. ◦ Requested LFN files can be included. The corresponding TURLs are searched and included in a file passed in the InputSandbox to the WN

2. Retrieve and handle of the outputs◦ The 2nd tool checks the status of the jobs from the job IDs included in the directory given by the user◦ It provides the following output:

The job run in ramses.dcic.ups.es:2119/jobmanager-torque-dteam is in status: ScheduledThe job run in grid01.phy.ncu.edu.tw:2119/jobmanager-torque-dteam is in status: runningThe job run in scaic10.scai.frauhofer.de:2119/jobmanager-torque-dteam is in status: over

The user is queried to retrieve the output to the destination he has previously decided



INFSO-RI-508833


Additional Features:

• It is possible to visualize the outputs on the web• A html report is provided showing the filesdecided by the user



INFSO-RI-508833

Summary and Conclusions

◘ Satisfactory implementation of the Geant4 code in the LCG ▪ (Hopefully this is the beginning of a long friendship) ▪ Paper Geant4-LCG submitted to SC05 Conference in EEUU◘ The LCG deployment team is quite interested in the implementation of the Geant4 code in our own tests suites ▪ We hope to collaborate together to help us as well

◘ You cannot keep on working borrowing another VOs: ▪ We have to make you VO=Geant4 as soon as possible ▪ For the next Geant4 Production this should be done

◘ Please come before in each production! ▪ Two weeks is not enough to make a good production ▪ We have to understand in all cases the failed jobs and this need time

infso-ri-508833 enabling grids for e-science geant4 physics validation: use of the grid resources...

Documents