LHCb input to DM and SM TEGs

Rem

ark

s to

DM

an

d S

M T

EG

S

F2F Data and Storage Management TEG, Amsterdam 2

Introduction

m We have already provided some input during our dedicated session of the TEG

m Here are a list of questions we want to address for each session

o Not exhaustive, limited to one slide per sessionm It would have been useful to hear what are the

proposals for evolution of SM and DM as wello These should be proposals, not diktat o Extensive discussions are needed with users and

sites before embarking on full scale “prototypes” that no longer are prototypes

m Whatever the future is, WLCG must ensure the long term support (no external “firm”)

o Do not underestimate the amount of work needed for users to adapt

o Therefore plan well ahead, gather functionality requirements…

Rem

ark

s to

DM

an

d S

M T

EG

S

F2F Data and Storage Management TEG, Amsterdam 3

Data Placement and Federation

m Does it imply replica catalogs are no longer needed?

o How is brokering of jobs done?o Random or “where most files are”

m When is data transfer performed?o By the WMS: implies a priori brokering, incompatible

with pilot jobso By the job: inefficiency of slotso Is it a cache (life time?) or just a download facility?

m What is the advantage compared to placement using popularity?

o Limited number of “master replicas” (e.g. 2)o Add replicas when popularity increaseso Remove replicas when popularity decreaseso … but still with a catalog and job brokering

m What is the minimal number of sites for which it becomes worth it?

Rem

ark

s to

DM

an

d S

M T

EG

S

F2F Data and Storage Management TEG, Amsterdam 4

WAN protocols and FTS

m We need third party transfer!o http 3rd party transfer? OK if commercial, why support it

ourselves?m We need a transfer “batch” system!

o Asynchronous bulk files transferm Whatever reliable and efficient underlying protocol is

used is just fine…m There is a need to define the service class where the file

has to be put (or one service class per SE)m What about the dedicated network (OPN)?

o Requires a service for using it?o Not all bells and whistles may be necessary

m The real point is for a user (experiment):o Transfer this list of LFNs to this SE (SE = storage class at

site)P The actual physical source is irrelevantP The TS should discover whether there is an online replica, if not

it should bring it online before making the transferP Ideally it (or the SE) should register the new replica (keep

consistency)

m All this was already said in… Mumbai (February 2006)!m FTS 3 was looking promising… why is it dead?

Rem

ark

s to

DM

an

d S

M T

EG

S

F2F Data and Storage Management TEG, Amsterdam 5

Management of Catalogues and Namespaces

m See Data placement…m Do we need a replica catalog?

o LHCb answer is YES: we want to be able to do brokering of jobs

o May only contain information on the SEs (+ file metadata + usability flags)

m Do we need a catalog with URLs?o Not necessarily: the URL can be formed from the SE

information and the LFN (trivial catalog), as SE information is quite static.

m Do we need a single URL (used for transfers and for protocol access)?

o No problem as long as the access is transparent and fast

m See SRM slide for more comments…o Namespace vs storage class?

Rem

ark

s to

DM

an

d S

M T

EG

S

F2F Data and Storage Management TEG, Amsterdam 6

Security and Access Control

m We MUST protect our data from deletiono LHCb doesn’t care about protecting from access so

muchm The current situation is INACCEPTABLE

o Anyone with little knowledge (available on the web) can delete all files in Castor!

o VOMS (or equivalent) identification and authorisation is a MUST! What about ARGUS?

P Identity and roleP Currently in Castor we have only 2 (uid, gid)!P Protection done by the LFC, but all backdoors are open

o Backdoors should be closed (nsrm, stager_xxx active commands…)

o Explicit “delete” permission would be desirablem Change of DN should be straightforward (not

trivial, but OK in LFC, DPM)o Action from VO administrator

Rem

ark

s to

DM

an

d S

M T

EG

S

F2F Data and Storage Management TEG, Amsterdam 7

Separation of disk and tape

m We need two (and only two) storage classes:o T1D0 and T0D1o This is because no space (storage class) change is

possible in some implementations of SRMm T1D0 has two functions:

o Archive of T0D1o Permanent storage of read-few data (RAW, RECO)

P For this the BringOnline functionality is mandatoryP We need to access the data directly from the MSS

without a need for replication onto another storageP Pinning is also a must (suboptimal usage of tape drives

without it)d Help to the garbage collector

Rem

ark

s to

DM

an

d S

M T

EG

S

F2F Data and Storage Management TEG, Amsterdam 8

Storage Interfaces: SRM and Clouds

m Clouds???m We need an interface for:

o BringOnlineo Pinningo Defining the storage class (unless different

endpoints are used, i.e. different SEs)o Currently (Mumbai) this is done by gfal (what is its

future?)m SRM is far from perfect but…

o It provides the aboveo All efforts put into defining a standard were a

miserable failure… don’t expect any other interface will be any better

m … but…o We could probably wrap the minimal functionality on

top of the SE native interface, if available P BringOnline and pinning not available for dCache except

in SRMP Can xroot provide this functionality?P Isn’t there the danger it becomes as clumsy as SRM

depending on the implementation?

Rem

ark

s to

DM

an

d S

M T

EG

S

F2F Data and Storage Management TEG, Amsterdam 9

Storage IO, LAN Protocols

m What is wrong with POSIX file: protocol?o Very efficient in the StoRM-GPFS implementation

used at CNAFo Of course abuse is could be a danger (recursive ls)

but this could be taken into account in the implementation (throttling)

o Almost anybody can now write a fuse plugin to make it happen, so why not use a powerful commercial protocol?

m Should access protocols be more than protocols?o i.e. interact behind the scene with the MSS, discover

the file location etc…o Can a tURL be just an access point?

P <protocol>://diskserver.cern.ch//<path>P … or better file:<path>P Avoid accepting URLs like “/castor/cern.ch/…”

d Needs to be fixed in application layer? No “guesses”?

o Do we need different URLs for different operations?P Transfer and posix-like access

Rem

ark

s to

DM

an

d S

M T

EG

S

F2F Data and Storage Management TEG, Amsterdam 10

Conclusion

m Whatever the future is:o Consider we need a permanently running systemo No disruption of service of more than 24 hours for

migrationo Millions of replicas are in the current system and… it

works (even when not optimal)o Any drastic change requires a lot of work on the user

side (experiments framework)o Old and new systems must exist in parallel

m Requirements may be different depending on the Computing Model of experiments

o 7 to 12 analysis centers (LHCb) is different from 50 to 70 centers

o Solutions may not be universal and complication may not be required