dcache, agile adoption of storage technology paul millar chep-2012 new york, 2012-05-24
DESCRIPTION
dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24. Overview. Some news Flexibility Future directions Summary. Funding. dCache is our contribution to WLCG : from Germany, the Nordic countries and USA/Fermilab, - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/1.jpg)
dCache, agile adoption of storage technologyPaul Millar
CHEP-2012 New York, 2012-05-24
![Page 2: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/2.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 2
Overview
Some news Flexibility Future directions Summary
![Page 3: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/3.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 3
Funding
dCache is our contribution to WLCG:
from Germany, the Nordiccountries and USA/Fermilab,
has been funded (independently from WLCG) for over 10 years
Funding for dCache is secure for after EMI:
Without EMI, funding only drops by ~20–25%
![Page 4: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/4.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 4
Community
3rd International workshop: 57 participants, from
13 countries New user-communities
presented how they wishto use dCache
Forging links with industry:
DESY and IBM form “large data”strategic partnership based on dCachestorage competence (CeBIT)
Establishing a Stack Exchange sitehttp://area51.stackexchange.com/proposals/40050/dcache
![Page 5: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/5.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 5
Evolution
Within WLCG: Strong involvement with TEG groups Working in collaboration on federated
storage
(both xrootd and HTTP) Outside WLCG:
OGF standardisation Engaging new communities
Improve dCache modularity:
Allow dCache to be easily adapted to novel environments
Agility is a process, not a target
![Page 6: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/6.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 6
News: under the hood
Splitting the code into smaller, reusable pieces: Chimera: enstore
See Enstore with Chimera namespace provider by D. Litvintsev
jrpc: BACNET, a Swiss Bank, ...
See dCache: Implementing a high-end NFSv4.1 server using a Java NIO framework by T. Mkrtchyan.
xrootd4j: (ALICE?) dCache is adopting Free/Open-source license
Mostly AGPLv3, the rest is LGPL or BSD Needed to get dCache into
distributions
![Page 7: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/7.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 7
News: NFSv4.1 / pNFS
Industry standard protocol Client availability:
RHEL/SL 6.x, RHEL/SL 5.x (with Oracle kernel + nfs-utils upgrade), Fedora 15, Debian 7.0 (“Wheezy”), Windows 7 (with driver from CITI), Windows 8, Solaris “Oracle [..] will deliver implementations of
[a client and server] in future releases of Solaris” [1] Hardware vendor support:
NetApp OnTap 8.1 Panasas “in 2012” [1] BlueArc, IBM “key part of SONAS Active Cloud Engine” [1]
[1] Source is “FAST 2012 pNFS BoF” 2012-02-15
``
![Page 8: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/8.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 8
News: dCache & pNFS
NFS v4.1 / pNFS has been supported since 2009.
Deployed in production (at DESY) for over a year.
Fermilab's REX dept. evaluated dCache NFSv4.1 for their Intensity Frontier experiments:
“Results look promising, throughput scales well with number of pool nodes”
Supports: authn: trusted-host and Kerberos all three GSS security modes.
![Page 9: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/9.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 9
Flexibility
(plugins and extension points)
![Page 10: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/10.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 10
Plugins: who should beinterested & why
Core developers: New functionality can be added as a plugin Backwards compatibility by keeping old plugins Can test-deploy new features at friendly sites
dCache sites: integrating with local, site-specific services
User-communities: Add some experiment-specific behaviour
External developers / trail-blazer sites: Experiment with exciting new features
![Page 11: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/11.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 11
What can I enhance?
A dCacheservice
Note that some detailshave been glossed over
![Page 12: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/12.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 12
What can I enhance?
![Page 13: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/13.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 13
What can I enhance?
![Page 14: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/14.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 14
What can I enhance?
![Page 15: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/15.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 15
What can I enhance?
For further details see A strategyfor load balancing in distributed
storage systems byM. Wadenstein and G. Behrmann
![Page 16: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/16.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 16
What can I enhance?
![Page 17: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/17.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 17
What can I enhance?
![Page 18: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/18.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 18
What can I enhance?
![Page 19: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/19.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 19
What can I enhance?
![Page 20: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/20.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 20
What can I enhance?
![Page 21: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/21.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 21
What can I enhance?
![Page 22: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/22.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 22
What can I enhance?
![Page 23: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/23.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 23
What can I enhance?
![Page 24: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/24.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 24
What can I enhance?
![Page 25: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/25.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 25
What can I enhance?
![Page 26: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/26.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 26
What can I enhance?
![Page 27: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/27.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 27
What can I enhance?
![Page 28: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/28.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 28
What can I enhance?
![Page 29: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/29.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 29
What can I enhance?
![Page 30: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/30.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 30
What can I enhance?
![Page 31: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/31.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 31
What can I enhance?
![Page 32: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/32.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 32
What can I enhance?
![Page 33: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/33.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 33
What can I enhance?
![Page 34: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/34.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 34
What can I enhance?
![Page 35: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/35.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 35
What can I enhance?
![Page 36: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/36.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 36
What can I enhance?
![Page 37: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/37.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 37
What can I enhance?
![Page 38: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/38.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 38
What can I enhance?
![Page 39: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/39.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 39
What can I enhance?
![Page 40: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/40.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 40
What can I enhance?
![Page 41: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/41.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 41
What can I enhance?
![Page 42: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/42.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 42
What can I enhance?
![Page 43: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/43.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 43
gPlazma: logging in
![Page 44: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/44.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 44
gPlazma: identities
![Page 45: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/45.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 45
Future directions
![Page 46: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/46.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 46
HTTP and WebDAV
How do we support non-HEP users? Dcap, SRM, rfio, xrootd
Nobody outside of HEP has heard of these (HEP is 1% of scientists)
HTTP & WebDAV Everyone has a web-browser WebDAV is commonly available on platforms Used by some cloud storage providers (Microsoft SkyDrive,
Deutscher Telekom, ..)
Deployed in production: DESY, PIC, BNL, ...
![Page 47: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/47.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 47
Federating storage
“Collection of disparate storage resources managed by co-operating but independent administrative domains transparently accessible via a common name space.”
Hey, we can do this with a standard protocol: HTTP!
Benefits: Get high-performance client for free, Loads of free software (Apache, Squid, Varnish, …)
Two stage approach: Web front-end to existing catalogues (LFC, …) Dynamically discovering available data using WebDAV
All replicas of a file are discoverable (c.f. dark data problem)
For further details, see Dynamic federations: storage aggregation using open tools and protocols by F. Furano
![Page 48: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/48.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 48
Missing files
A user may askfor a file thatdoesn't exist
A user may askfor a file thatshould exist,but the poolis broken
![Page 49: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/49.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 49
Missing files
For further details, see SYNCAT – Storage Catalogue Consistency by F. Furano
Maybe dCacheshould do“something”in thesecases. That“something” should be highlyconfigurable; i.e., aplugin.
![Page 50: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/50.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 50
Faster storage
![Page 51: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/51.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 51
3 Tier Model
For further details see Evaluation of benefits of a three tier data model for WLCG analysis by D. Ozerov and P. Fuhrmann
![Page 52: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/52.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 52
Summary
The dCache project is independent of WLCG and EMI funding.
dCache has the flexibility to adapt to new deployments, scenarios and technology.
The dCache community is growing.
![Page 53: dCache, agile adoption of storage technology Paul Millar CHEP-2012 New York, 2012-05-24](https://reader036.vdocuments.us/reader036/viewer/2022062723/5681402a550346895dab8b5b/html5/thumbnails/53.jpg)
dCache, agile adoption of storage technology| Paul Millar | 2012-05-24 | Page 53
Thanks for listening