alien tutorial
DESCRIPTION
AliEn Tutorial. MODEL 2009 4 th May , 2009. Outline. Installation of the AliEn software AliEn and the GRID Authentication File Catalogue Submitting jobs. Installation. Please, do: w get http://alien.cern.ch/alien-torrent-installer c hmod +x alien-torrent-installer - PowerPoint PPT PresentationTRANSCRIPT
AliEn Tutorial
MODEL 20094th May, 2009
2 [email protected] May 2009
Installation of the AliEn softwareAliEn and the GRIDAuthenticationFile CatalogueSubmitting jobs
Outline
3 [email protected] May 2009
Installation
Please, do:wget http://alien.cern.ch/alien-torrent-installerchmod +x alien-torrent-installer./alien-torrent-installer
Default installationFirst two commands fastLet’s leave the 3rd running...
4 [email protected] May 2009
AliEn, AliEn2
ALICE ENvironment on the GRIDGRID middleware
Developed as single entry point to the GRID for ALICE (CERN)All the components necessary to build a GRID and interact with other GRIDs
File System with metadataAuthorization, authentication, job optimization and execution, storage managementAudit, quotas, monitoringInterfaces to various GRID implementations
Used since:2002 for centrally managed productions2006 for user analysis
Used by several Virtual Organizations:ALICE, PANDA, CBM, GPCALMA
http://alien.cern.ch
6 [email protected] May 2009Pablo Saiz 6
LHC
27km particle accelerator100 meters underground
About to start collecting data
8 [email protected] May 2009
Why do we need a catalogue?
Loads of data (several PB per year!!)Distributed community
Same structure from everywhereContains location of the file
It does not contain the filePossibility of several copies of files
Select the closest oneMetadata information
We will see this later on...
9 [email protected] May 2009
AliEn2 File Catalogue
Mapping from LFN to SE and PFNUNIX-like file systemMetadataGUIDUsed by
all other services
Tier1
|--./| |--cern.ch/| | |--user/| | | |--a/| | | | |--admin/| | | | || | | | |--aliprod/| | | || | | |--f/| | | | |--fca/| | | || | | |--p/| | | | |--psaiz/| | | | | |--as/| | | | | || | | | | |--dos/| | | | | || | | | | |--local/
|--simulation/| |--2001-01/| | |--V3.05/| | | |--Config.C| | | |--grun.C
| |--36/| | |--stderr| | |--stdin| | |--stdout| || |--37/| | |--stderr| | |--stdin| | |--stdout| || |--38/| | |--stderr| | |--stdin| | |--stdout
| | | || | | |--b/| | | | |--barbera/ALICE
USERS
ALICE DATA
ALICE REMOTE
10 [email protected] May 2009
LFN and GUID catalogues
/
/alice
/alice/user/p/psaiz
/alice/simulation/2006
…
Index1-JAN-1970
1-JAN-2006
14-FEB-2007
23-AUG-2008
…
Index GUID PFNLFN Catalogue GUID CatalogueAliEn File & Metadata Catalogue
LFN GUID
LFN: Logical File Name (what the user sees)GUID: Global Unique Identifier (another way of identifying files)PFN: Physical File Name
11 [email protected] May 2009
Who uses the catalogue?
EverybodyIF A FILE IS NOT IN THE CATALOGUE,
IT DOES NOT EXIST ON THE GRIDUsersUser data, user executables ...Job executionExecutable, input, outputPackagesSource code, binary code, post
installation
12 [email protected] May 2009
Back to the exercises...
Did the installation finish?Let’s get ready to access the GRIDX509 Certificate authentication
Public vs Private keyCertificate AuthorityProxy certificates
13 [email protected] May 2009
Preparing certificates
In $HOME/.globus:userkey.pemusercert.pem
Readable only by the user (chmod 400)Make alien use those files:
mkdir $HOME/.alienln –s $HOME/.globus $HOME/.alien/globus
14 [email protected] May 2009
Environment files$ALIEN_ROOT/.Environment$HOME/.alien/Environment
Bash files sourced before starting aliencat <<EOF > $HOME/.alien/Environment
export ALIEN_ORGANISATION=PANDA export ALIEN_USER=<username> EOF
Put alien in the PATHmkdir –p $HOME/binln –s $HOME/alien/bin/alien $HOME/bin/
16 [email protected] May 2009
Globus related:Permissions on $HOME/.globus/userkey.pem are not private to the user – chmod 400 userkey.pemYour certificate authority is exotic and not known to the server.Your certificate has expired.Clock skew:
• Your local computer time is in the future with respect to the server's time.
• Your local computer time is more in the past than the certificate life time.
Authentication problems
17 [email protected] May 2009
Check the certificates in your machines:Store them under e.g: $HOME/.globus/Check that the key file is read only
Setup the environment$HOME/.alien/EnvironmentPut alien in the path
Get a valid alien tokenalien proxy-init
Try it out !!
18 [email protected] May 2009
Standard bash shell with grid comandsMain bash features are available.Not all shell helper programs are available.Some local commands (like ls, cat etc) are overwritten with the corresponding GRID commands.File/path tab completion in the virtual GRID directory.
Accessing the shell
24 [email protected] May 2009
Access the alien shell.alien
Check your user name:whoami
List the contents of your home directoryls
Print the working directorypwd
Create the following directory structure in your AliEn space:$HOME/bin (if it is not there)$HOME/macros and $HOME/jdl
Get the information of the file: whereis /bin/date
Shell – Try it out I
25 [email protected] May 2009
See the file /bin/datecat /bin/date
Copy the file to your local machinecp /bin/date /tmp/my_file
Exit the shell and check the file
Shell – Try it out II
26 [email protected] May 2009
Job & Transfer models
JobsTaskQueue Job
Broker
Job Manager
CE
JA
Central servicesSite services
SplittingExpired
PrioritiesMerging
Zombies
Job optimizers
File catalogue
LFN GUIDMetadataFTD
PackmanMonALISA
JA
JA
CE
SEPackman
MonALISA
FTD
SE
Site ASite B
Transfer Manager TransferQueue Transfer
Broker
Transfer optimizers
fts
bbftp
xrdcp
29 [email protected] May 2009
AliEn
AliEn user interface
VTD EDGLCG /GLITECONDORARC/
NORDUGRID
Nice! I STILL do not have to
worry about ever changing GRID environment…
Interconnecting GRIDs
31 [email protected] May 2009
Your job is submitted and a <JOBID>.is assigned to it.
Submitting jobs
34 [email protected] May 2009
Other useful commands
jobListMatchList of CE that can execute the job
SpyOutput of the job while it is running
MasterjobGive information about
35 [email protected] May 2009
Connect to alienalien login
Submit the jobsubmit <<EOF
Executable=“date”;EOF
Follow the executiontop –id, ps trace, ps jdl
Get the outputcp /proc/<user>/<id>/job-output/stdout /tmp/out
Try it out!!
36 [email protected] May 2009
Executable: Compulsory field where we give the lfn of the executable that should be stored in /bin or $V0/bin or $HOME/bin.Arguments: They will be passed to the executable.Packages: Type packages in the shell to see what kind of packages are installed.InputFile: The files that will be transported to the node where the job will run.InputData: It will require that the job will be executed in a site close to the files specified here.InputDataList: The filename in which the Job Agent will write the InputData list.InputDataListFormat: The format of the InputData list.
JDL fields I
37 [email protected] May 2009
OutputFile: The files that will be registered in the catalog once the job finishes.OutputArchive: What files will be archived in a zip file.Validationcommand: Specifies the script to be used as a validation script.Email: Receive a mail when the job finishes.TTL: The maximum run time of your job.Split: Split the jobs in several sub jobs.
JDL fields II
38 [email protected] May 2009
Packages
packmangives a list of the packages defined/installedinstalls packages
39 [email protected] May 2009
Add:Packages=“pbarprod@pandaroot::nov08“;
Try other commands:Executable=“pandaroot”;
Multiple submissionSplit=“production:1-5”;
Create your own executables
Try it out !!
40 [email protected] May 2009
Conclusion
AliEnInterface to the GRIDDeveloped by ALICEAlso used by PANDA, CBM, GPCALMAIn production since 2002
File CatalogueFile and Metadata
Job submission modelData movement and replication
41 [email protected] May 2009
Registration – Certificates:http://alien.cern.ch/twiki/bin/view/Alice/UserRegistration
https://ca.cern.ch/ca/
AliEn: http://alien.cern.ch
Gshell: http://alien.cern.ch/twiki/bin/view/AliEn/GAPI
User's guide: http://project-arda-dev.web.cern.ch/project-arda-dev/alice/apiservice/AA-UserGuide-0.0m.pdf
References I
42 [email protected] May 2009
aliensh Grid Command Online Reference V1.0 http://project-arda-dev.web.cern.ch/project-arda-dev/alice/apiservice/guide/guide-1.0.htm
Previous tutorials:http://aliceinfo.cern.ch/Offline/Analysis/Tutorial/
Event Tag System:http://pcaliweb02.cern.ch/Offline/Analysis/RunEventTagSystem/EventTags.html#Event%20tag%20system
https://edms.cern.ch/document/788315/1 (INTERNAL NOTE)
References II
