filecatalog and dd

8/9/2019 FileCatalog and Dd

1/12


2/12

Jerome LAURET, Coll 2

Introduction ...Introduction ...

T!e "eo"le #T!e "eo"le # $iita Soldato%, Adam &isiel, m'sel(, )$iita Soldato%, Adam &isiel, m'sel(, )

*!' do +e need a FileCatalog *!' do +e need a FileCatalog

$umber o( (iles in STAR is - 2 M .+ill get +orst, (ar +orst )/$umber o( (iles in STAR is - 2 M .+ill get +orst, (ar +orst )/

n(ormation struture om"le n(ormation struture om"le production, libraryproduction, libraryfiletype, size, geometryfiletype, size, geometrycollision, magnetic field, trigger setup namecollision, magnetic field, trigger setup name

but +e .are su""osed to/ ee" in(ormation about triggers and but +e .are su""osed to/ ee" in(ormation about triggers andounters, (inding a data4set re5uires strong Cataloguingounters, (inding a data4set re5uires strong Cataloguing

A6

7ne eisting om"lete user A6 .+ritten in "erl/, some C7ne eisting om"lete user A6 .+ritten in "erl/, some Ca ommand line inter(aea ommand line inter(ae

% get_file_list.pl% get_file_list.pl


3/12


How do I use it ??How do I use it ??

8etting a 5ui !el" reminder8etting a 5ui !el" reminder% get_file_list.pl% get_file_list.pl

... bla bla bla ... some help that is ...

allavailable bbc collision configuration createtimedatetaken eemc emc events extension filecomment filenamefileseq filetype fpd ftpc fulld fulls gencomment generatorgenparams genversion geometryinserttime lgnm lgpthlibrary limitmagscale magvalue md5sum node noroundnouniqueowner pathpersistent pmd prodcomment productionprotection rich runcomments runnumber runtype sanitysimcomment simulationsitesitecmt siteloc size ssdstartrecordstoragestream svt tof tpc trgcounttrgdefinition trgname trgsetupnametrgversion trgword

9oumentation is a%ailable at9oumentation is a%ailable at/STAR/comp/sofi/FileCatalog//STAR/comp/sofi/FileCatalog/


4/12


SyntaxSyntax

8eneral s'nta . :;: indiates o"tional list :


5/12


6ossible 7"erators6ossible 7"erators

B $ot greater t!an

B Lesser t!an

D $ot less t!an

D 8reater t!an

BD $ot e5ual to e5ual to

- $ot ontaining .ie do not mat!/ strings

- Containing .ie a""roimatel' mat!ing/ strings

G n rangeG 7utside t!e range

H Modulo integer

HH $ot Modulo integer


6/12


Welcome to the World ofWelcome to the World of

replica Catalogs.replica Catalogs. $umber o( (iles in STAR - 2 M$umber o( (iles in STAR - 2 M

T!at@s a lie Total 3 M +it! re"lias # File !a%e more t!an one loationT!at@s a lie Total 3 M +it! re"lias # File !a%e more t!an one loation

sitesite Ie a+are o( siteI$L, siteLILIe a+are o( siteI$L, siteLIL

nodenode loal!ost b' de(ault loal!ost b' de(ault

storagestorage $FS, loal, K6SS $FS, loal, K6SS"at!"at! itsel( +it!in a storageitsel( +it!in a storage

unonstraint, "at! and (ilename are $7T uni5ue e' "airsunonstraint, "at! and (ilename are $7T uni5ue e' "airs.use "distinctto ensure it onefileensures one instane o( a (ile/

$umber o( (iles on entralied storage # 1NOP$umber o( (iles on entralied storage # 1NOP$FS, dis %isible (rom an'+!ere in t!e (ailit' ."at! - ?star?dataQQ/$FS, dis %isible (rom an'+!ere in t!e (ailit' ."at! - ?star?dataQQ/

$umber o( (iles on loal dis # 131PP$umber o( (iles on loal dis # 131PP

loal dis are %isible onl' (rom a uni5ue nodeloal dis are %isible onl' (rom a uni5ue node


7/12

Jerome LAURET, Coll

Database layoutDatabase layout

Loations ? Re"lias

!"SS

#$S

lo%al

Fileypes

!roductionConditions

"un!arams

FileData

File

#ocations

Storageypes

StorageSites

Meta 9ata

Site, no&e, storage an& 'at( )orms t(e uni*ue +e )orFileLoations/tmp/bla.rootcannot be uni$ueBNL somenode.domain NFS /tmp/bla.root IS

%.&

$1

$1

1$

1$

1$


8/12

Jerome LAURET, Coll -

T'"ial Eam"lesT'"ial Eam"les

Ko+ to loate (iles +it!in a s"ei(i trigger setu" Ko+ to loate (iles +it!in a s"ei(i trigger setu" % get_file_list.pl keys pathfilename condtrgsetupname)/011ombined

+ill lead to a long .100 reords/ list o( "ossible (iles +it! "at!+ill lead to a long .100 reords/ list o( "ossible (iles +it! "at!

% get_file_list.pl keys storage condtrgsetupname)/011ombined

t!is +ill gi%e 'ou all "ossible storage t'"e (or t!e trigger setu" namet!is +ill gi%e 'ou all "ossible storage t'"e (or t!e trigger setu" nameU6CCombinedU6CCombined

n general, (or listing all "ossible %alues (or a e'+ord, usen general, (or listing all "ossible %alues (or a e'+ord, use

H get(ilelist"l 4e's e'+ord distint ;4alls


9/12

Jerome LAURET, Coll .

T'"ial Eam"lesT'"ial Eam"les

Iut but ) al+a's get onl' 100 reordsIut but ) al+a's get onl' 100 reords

T!at@s normal, it is t!e de(ault Use limit to !ange t!e number o( reords,T!at@s normal, it is t!e de(ault Use limit to !ange t!e number o( reords,

(ull list +it! limit 0(ull list +it! limit 0

A (e+ !and' 5uerriesA (e+ !and' 5uerries

no+ a simulation (ile name, !o+ do get t!e geometr' on(iguration no+ a simulation (ile name, !o+ do get t!e geometr' on(iguration % get_file_list.pl "keys geometry "cond% get_file_list.pl "keys geometry "condfilename)rcf2$83_2&_322evts.geant.root "distinctfilename)rcf2$83_2&_322evts.geant.root "distinct

9ear&22$9ear&22$

*!i! "rodution and geometr' *!i! "rodution and geometr' % get_file_list.pl "keys productiongeometry "cond% get_file_list.pl "keys productiongeometry "condfilename)rcf2$83_2&_322evts.geant.root "distinctfilename)rcf2$83_2&_322evts.geant.root "distinct

02$glyear&22$02$glyear&22$

02$gkyear&22$02$gkyear&22$

02&gbyear&22$02&gbyear&22$


10/12

Jerome LAURET, Coll /0

Aggregate 7"erationAggregate 7"eration

Can also do 5ueries leading to summar'Can also do 5ueries leading to summar'

in(ormationin(ormation

% get_file_list.pl keys% get_file_list.pl keys

:sum;sanity3524534&5&&=&8>$$&82production1entral$&22

>354$$>>3582&&4&>=53==&production1entral>22>354$$>>3582&&4&>=53==&production1entral>22

882824>$2$$$>&&8$>$2=$0roduction6in?ias882824>$2$$$>&&8$>$2=$0roduction6in?ias


11/12

Jerome LAURET, Coll //

7ne more one"t (uture7ne more one"t (uture

T!e e'+ord sanit' is used (or t+o aseT!e e'+ord sanit' is used (or t+o aseT!e (ile is orru"tedT!e (ile is orru"ted.R77T 7 +ill ras! 'our a""liation/.R77T 7 +ill ras! 'our a""liation/T!e (ile is $7T good (or 6!'sisT!e (ile is $7T good (or 6!'sis

ou MUST use sanit'1 to get t!e good (iles

Future .not 'et a%ailable/Future .not 'et a%ailable/

% get_file_list.pl keys pathfilename cond% get_file_list.pl keys pathfilename condtrgname)pp?@A$fastBBpp+07wfastsanity)$trgname)pp?@A$fastBBpp+07wfastsanity)$

alread' :in "lae=, onl' need to (ill t!e database onsistentl'alread' :in "lae=, onl' need to (ill t!e database onsistentl'.not done t!is 'ear/.not done t!is 'ear/

% get_file_list.pl "keys pathfilename "cond% get_file_list.pl "keys pathfilename "condtpcCD)$ftpcCD)$sanity)$'tpcCD)$ftpcCD)$sanity)$'

$ot im"lemented, +e "lan to add a detetor readiness (lag$ot im"lemented, +e "lan to add a detetor readiness (lag


12/12

Jerome LAURET, Coll /2

9istributed dis 9istributed dis

S!all sort t!is manuall' S!all sort t!is manuall' ou an al+a's as (orou an al+a's as (or% get_file_list.pl "cond nodepathfilename "cond% get_file_list.pl "cond nodepathfilename "condstorage)localsanity)$'storage)localsanity)$'and dis"at! b' !and ut +!' and dis"at! b' !and ut +!'

T!e S!edulerT!e S!eduler

9oes t!is (or 'ou .eam"les in net tal/ #9oes t!is (or 'ou .eam"les in net tal/ # fileEist,yntaxfileEist,yntax,,prefer,torageprefer,torage

T!ere is $7 need to useT!ere is $7 need to use "distinct"distinctoror "onefile"onefile

$otes$otes

es, "lease, use t!e sanit' (lag )es, "lease, use t!e sanit' (lag )

Use t!e S!eduler .it is a e' om"onent o( our 8rid a""roa!/Use t!e S!eduler .it is a e' om"onent o( our 8rid a""roa!/An' S!eduler /FE)Gcatalogstar.bnl.govH...an .and s!ould/ be!eed (rom t!e ommand line using get(ilelist"l ( it does not +or(rom t!e ommand line, it is $7T a S!eduler "roblem

filecatalog and dd

Documents