Your university or experiment logo here

The Protocol ZooA Site Presepective

Shaun de Witt, STFC (RAL)

Your university or experiment logo here

Site Perspective on Protocol Zoo

• Outline– Dealing with the Animals

• Problems associated with multiple protocols– Replacing Lions with Tigers

• Impact of change– Why does WLCG want a Fossa while the rest of the world

wants Giraffes and Rhinos?• Xrootd/HTTP/WebDAV

– What happens when we need to clean the cages• Migrating to new storage technology

Your university or experiment logo here

XRoot GridFTP SRM

RFIO

DCAP

S3

HTTP

WebDAV

FILE

Your university or experiment logo here

Dealing with the Animals

• In the ‘bad’ old days…– Sites had to deal with a mix of protocols

• SRM/GridFTP were common• Some mix of file/dcap/rfio/… and experiments needed to handle

the differences• Internal protocols were somewhat optimised for the storage

systems (good performance) but lacked some features (eg security)

• SRM seen as a bottleneck and didn’t fully deliver on its promises

• Today– The zoo is more homogeneous

• SRM/GridFTP/xroot at all sites, many also offer WebDAV and/or http

• Internal protocols largely deprecated– But makes life harder for zoo keepers

• Difficulty linking protocol server logs to storage system logs• Differing log formats…

Your university or experiment logo here

Your university or experiment logo here

Replacing Lions with Tigers

• Speed of Change– Change in Protocol change is SLOW!!!!

• Development for each Storage System• Validation/Verification• Deployment at all sites• Removing old protocols (PLEASE!!!)

• Site Admins need to understand the protocol– Assessing hardware requirements (and purchasing)– Understanding logs– Identifying failure modes and integration into monitoring

• Protocols supporting fallback can make diagnostics more difficult

• Why did the file not get copied from my site?

Your university or experiment logo here

Your university or experiment logo here

• Many sites are looking to extend their customer base– Non HEP/PP– XRootD works well for WLCG (and offers many benefits)…

• Scalability, fallback, integration with ROOT, …– … but selling XRootD outside of HEP is difficult– Need some sort of standard protocol– Most mature experiments can use gridFTP for bulk

movement…• But in some storage systems it MUST be mediated by SRM

– … but it’s not what they want for analysis• WLCG Movement towards standards…

– Movement towards HTTP/WebDAV across the board is appreciated

– But WebDAV still is not widely adopted outside of HEP

Your university or experiment logo here

Cleaning the Cages

Your university or experiment logo here

• At RAL we are looking to deploy a more modern disk-only storage system– Based around object storage– To be used by many communities/projects

• ISIS, DIAMOND, CLF, CCLRC,H2020 projects– Will also be used for WLCG– Protocols still a problem…

• Want a file put in through protocol X to be directly accessible through any other supported protocol

• Don’t really want protocol bridges– Again use of standards would be helpful

• Consider S3 (or CDMI – but this has its own problems)