alien tutorial

AliEn Tutorial

MODEL 20094th May, 2009

2 Pablo.Saiz@cern.ch4 May 2009

Installation of the AliEn softwareAliEn and the GRIDAuthenticationFile CatalogueSubmitting jobs

Outline

Installation

Please, do:wget http://alien.cern.ch/alien-torrent-installerchmod +x alien-torrent-installer./alien-torrent-installer

Default installationFirst two commands fastLet’s leave the 3rd running...

AliEn, AliEn2

ALICE ENvironment on the GRIDGRID middleware

Developed as single entry point to the GRID for ALICE (CERN)All the components necessary to build a GRID and interact with other GRIDs

File System with metadataAuthorization, authentication, job optimization and execution, storage managementAudit, quotas, monitoringInterfaces to various GRID implementations

Used since:2002 for centrally managed productions2006 for user analysis

Used by several Virtual Organizations:ALICE, PANDA, CBM, GPCALMA

http://alien.cern.ch

5 Pablo.Saiz@cern.ch4 May 2009Pablo Saiz 5

27km particle accelerator100 meters underground

About to start collecting data

A Large Ion Collider Experiment

Why do we need a catalogue?

Loads of data (several PB per year!!)Distributed community

Same structure from everywhereContains location of the file

It does not contain the filePossibility of several copies of files

Select the closest oneMetadata information

We will see this later on...

AliEn2 File Catalogue

Mapping from LFN to SE and PFNUNIX-like file systemMetadataGUIDUsed by

all other services

|--./| |--cern.ch/| | |--user/| | | |--a/| | | | |--admin/| | | | || | | | |--aliprod/| | | || | | |--f/| | | | |--fca/| | | || | | |--p/| | | | |--psaiz/| | | | | |--as/| | | | | || | | | | |--dos/| | | | | || | | | | |--local/

| |--36/| | |--stderr| | |--stdin| | |--stdout| || |--37/| | |--stderr| | |--stdin| | |--stdout| || |--38/| | |--stderr| | |--stdin| | |--stdout

| | | || | | |--b/| | | | |--barbera/ALICE

ALICE DATA

ALICE REMOTE

LFN and GUID catalogues

/alice

/alice/user/p/psaiz

/alice/simulation/2006

Index1-JAN-1970

1-JAN-2006

14-FEB-2007

23-AUG-2008

Index GUID PFNLFN Catalogue GUID CatalogueAliEn File & Metadata Catalogue

LFN GUID

LFN: Logical File Name (what the user sees)GUID: Global Unique Identifier (another way of identifying files)PFN: Physical File Name

Who uses the catalogue?

EverybodyIF A FILE IS NOT IN THE CATALOGUE,

IT DOES NOT EXIST ON THE GRIDUsersUser data, user executables ...Job executionExecutable, input, outputPackagesSource code, binary code, post

installation

Back to the exercises...

Did the installation finish?Let’s get ready to access the GRIDX509 Certificate authentication

Public vs Private keyCertificate AuthorityProxy certificates

Preparing certificates

In $HOME/.globus:userkey.pemusercert.pem

Readable only by the user (chmod 400)Make alien use those files:

mkdir $HOME/.alienln –s $HOME/.globus $HOME/.alien/globus

Environment files$ALIEN_ROOT/.Environment$HOME/.alien/Environment

Bash files sourced before starting aliencat <<EOF > $HOME/.alien/Environment

export ALIEN_ORGANISATION=PANDA export ALIEN_USER=<username> EOF

Put alien in the PATHmkdir –p $HOME/binln –s $HOME/alien/bin/alien $HOME/bin/

alien proxy-init, alien proxy-infoGetting a GRID proxy

Globus related:Permissions on $HOME/.globus/userkey.pem are not private to the user – chmod 400 userkey.pemYour certificate authority is exotic and not known to the server.Your certificate has expired.Clock skew:

• Your local computer time is in the future with respect to the server's time.

• Your local computer time is more in the past than the certificate life time.

Authentication problems

Check the certificates in your machines:Store them under e.g: $HOME/.globus/Check that the key file is read only

Setup the environment$HOME/.alien/EnvironmentPut alien in the path

Get a valid alien tokenalien proxy-init

Try it out !!

Standard bash shell with grid comandsMain bash features are available.Not all shell helper programs are available.Some local commands (like ls, cat etc) are overwritten with the corresponding GRID commands.File/path tab completion in the virtual GRID directory.

Accessing the shell

Tab completion working!!!

Basic commands I

Basic commands II

whereis command

Gives (list of) SE and pfns for a file

Viewing the files

Shell – Copying files (add/get)

Access the alien shell.alien

Check your user name:whoami

List the contents of your home directoryls

Print the working directorypwd

Create the following directory structure in your AliEn space:$HOME/bin (if it is not there)$HOME/macros and $HOME/jdl

Get the information of the file: whereis /bin/date

Shell – Try it out I

See the file /bin/datecat /bin/date

Copy the file to your local machinecp /bin/date /tmp/my_file

Exit the shell and check the file

Shell – Try it out II

Job & Transfer models

JobsTaskQueue Job

Broker

Job Manager

Central servicesSite services

SplittingExpired

PrioritiesMerging

Zombies

Job optimizers

File catalogue

LFN GUIDMetadataFTD

PackmanMonALISA

SEPackman

MonALISA

Site ASite B

Transfer Manager TransferQueue Transfer

Broker

Transfer optimizers

ALICE sitesEurope

North America

Africa

See Costin’s presentation

Running jobs

More than 25 CPU/years per day

AliEn user interface

VTD EDGLCG /GLITECONDORARC/

NORDUGRID

Nice! I STILL do not have to

worry about ever changing GRID environment…

Interconnecting GRIDs

Job status

Your job is submitted and a <JOBID>.is assigned to it.

Submitting jobs

Checking the job status

Getting the output

Other useful commands

jobListMatchList of CE that can execute the job

SpyOutput of the job while it is running

MasterjobGive information about

Connect to alienalien login

Submit the jobsubmit <<EOF

Executable=“date”;EOF

Follow the executiontop –id, ps trace, ps jdl

Get the outputcp /proc/<user>/<id>/job-output/stdout /tmp/out

Try it out!!

Executable: Compulsory field where we give the lfn of the executable that should be stored in /bin or $V0/bin or $HOME/bin.Arguments: They will be passed to the executable.Packages: Type packages in the shell to see what kind of packages are installed.InputFile: The files that will be transported to the node where the job will run.InputData: It will require that the job will be executed in a site close to the files specified here.InputDataList: The filename in which the Job Agent will write the InputData list.InputDataListFormat: The format of the InputData list.

JDL fields I

OutputFile: The files that will be registered in the catalog once the job finishes.OutputArchive: What files will be archived in a zip file.Validationcommand: Specifies the script to be used as a validation script.Email: Receive a mail when the job finishes.TTL: The maximum run time of your job.Split: Split the jobs in several sub jobs.

JDL fields II

Packages

packmangives a list of the packages defined/installedinstalls packages

Add:Packages=“pbarprod@pandaroot::nov08“;

Try other commands:Executable=“pandaroot”;

Multiple submissionSplit=“production:1-5”;

Create your own executables

Try it out !!

Conclusion

AliEnInterface to the GRIDDeveloped by ALICEAlso used by PANDA, CBM, GPCALMAIn production since 2002

File CatalogueFile and Metadata

Job submission modelData movement and replication

Registration – Certificates:http://alien.cern.ch/twiki/bin/view/Alice/UserRegistration

https://ca.cern.ch/ca/

AliEn: http://alien.cern.ch

Gshell: http://alien.cern.ch/twiki/bin/view/AliEn/GAPI

User's guide: http://project-arda-dev.web.cern.ch/project-arda-dev/alice/apiservice/AA-UserGuide-0.0m.pdf

References I

aliensh Grid Command Online Reference V1.0 http://project-arda-dev.web.cern.ch/project-arda-dev/alice/apiservice/guide/guide-1.0.htm

Previous tutorials:http://aliceinfo.cern.ch/Offline/Analysis/Tutorial/

Event Tag System:http://pcaliweb02.cern.ch/Offline/Analysis/RunEventTagSystem/EventTags.html#Event%20tag%20system

https://edms.cern.ch/document/788315/1 (INTERNAL NOTE)

References II

http://alien.cern.ch

Any questions?

Thank you very much!!

alien tutorial

alien softwarealien

tools alien tutorialmodel

logical file

file systemmetadataguidused

alien logo pablo saiz5cern

physical file

environment files

user seesguid

Documents

alien intrusions alien life alternative histories ...

alien genetics

alien rome

110/10/06 panos.christakoglou@cern.ch - alien...

alien hell

alien sourcebook

alien smily face tutorial

alien stories

alien dictionary

persistent memory quick start programming tutorial - … ·...

alice offline tutorial using the alien grid client...

resident alien

alien screenplay

alien building

cassandra: an alien technology that's not so alien

alien poster

gamepro invader 1500 games list - home leisure direct...43...

discerning alien disinformation - part 6 - alien agenda.pdf

gigers alien

alien invaders