LHCb

Data Management

after LS1

DM a

fter L

S1

PhC 2

Brief overview of current DM

m Replica catalog: LFCo LFN -> list of SEso SEs are defined in the DIRAC Configuration System

P For each protocol: end-point, SAPath, [space token, WSUrl]P Currently only used: SRM and rfio

m File placement according to Computing Modelo FTS transfers from original SE (asynchronous)

m Disk replicas and archives completely split:o Only T0D1 and T1D0, no T1D1 SE any longer

m Production jobs:o Input file download to WN (max 10 GB) using gsiftp

m User jobs:o Protocol access from SE (on LAN)

m Output upload:o From WN to (local) SE (gsiftp). Upload policy defined in the job

m Job splitting and brokering:o According to LFC informationo If file is unavailable, the job is rescheduled

DM a

fter L

S1

PhC 3

Caveats with current system

m Inconsistencies between FC, SE catalog and actual storage

o Some files are temporarily unavailable (server down)o Some files are lost (unrecoverable disk, tape)o Consequences:

P Wrong brokering of jobs: cannot access filesd Except for download policy if another replica is on disk/cache

m SE overloado Busy, or not enough moverso As if files are unavailable

P Jobs are rescheduled

DM a

fter L

S1

PhC 4

Future of replica catalog

m We probably still need oneo Job brokering:

P Don’t want to transfer files all over the place (even with caches)

o DM accounting:P Want to know what/how much data is where

m But…o Should not need to be highly accurate as nowo Allow files to be unavailable without the job failing

m Considering the DIRAC File Catalogo Mostly replica location (as used in LFC)o Built-in space usage accounting per directory and SE

DM a

fter L

S1

PhC 5

Access and transfer protocols

m Welcome gfal2 and FTS3!m Hopefully transparent protocol usage for transfers

o However transfer requests should be expressed with compatible URLs

m Access to T1D0 datao 99% for reconstruction or re-stripping, i.e. downloado Read once, therefore still require a sizeable staging

poolP Unnecessary to copy to T0D1 before copying to WN

m xroot vs http/webdavo No strong feelings

P What is important is unique URL, redirection and WAN access

o However why not use (almost) standard protocolsP CVMFS experience is very positive, why not http for

data?o Of course better if all SEs provide the same protocol

P http/webdav for EOS and Castor?o We are willing to look at the http ecosystem

DM a

fter L

S1

PhC 6

Other DM functionality

m File staging from tapeo Currently provided by SRM

P Keep SRM for T1D0 handlingP Limited usage for bringOnlineP Not used for getting tURL

m Space tokenso Can easily be replaced by different endpoints

P Preferred to using namespace!

m Storage usageo Also provided by SRMo Is there a replacement?

DM a

fter L

S1

PhC 7

Next steps

m Re-implement DIRAC DM functionality with gfal2

m Exploit new features of FTS3

m Migrate to DIRAC File Catalogo In parallel with LFC

m Investigate http/webdav for file location and access

o First, use it for healingP Still brokering using a replica catalog

o Usage for job brokering (replacing replica catalog)?P Scalability?

DM a

fter L

S1

PhC 8

What else?

m Dynamic data cachingo Not clear yet how to best use this without replicating

everything everywhereo When do caches expire?o Job brokering?

P Don’t want to hold jobs while a dataset is replicated

m Data popularityo Information collection in placeo Can it be used for automatic replication/deletion?

P Or better as a hint for Data managers?o What is the metrics to be used?

P What if 10 files out of a 100TB dataset are used for tests, but none is interested in the rest?

P Fraction of dataset used or absolute number of accesses?

d Very few analysis passes on full datasetd Many iterative usage of same subset