![Page 1: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/1.jpg)
![Page 3: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/3.jpg)
Table of Contents
Introduction
Storage for physics
Infrastructure storage
Wrap up
20th October 2017 Storage at CERN 3
![Page 4: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/4.jpg)
Unified building blocks
Storage node• Compute node• 10Gbit/s network interface• SAS expander
Storage array• Dummy SAS array• 24x 6TB drives
20th October 2017 Storage at CERN 4
![Page 5: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/5.jpg)
Services Portfolio
20th October 2017 Storage at CERN 5
![Page 6: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/6.jpg)
Table of Contents
Introduction
Storage for physics
Infrastructure storage
Wrap up
20th October 2017 Storage at CERN 6
![Page 7: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/7.jpg)
Castor
Tape-backed storage system• Home-made HSMa system• Users write data to disk• Which gets migrated to tape
aHierarchical storage management
20th October 2017 Storage at CERN 7
![Page 9: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/9.jpg)
EOS
Aggregated numbers• ∼ 1500 nodes• ∼ 55k drives• ∼ 220PB raw capacity
Spread over 6 instances
20th October 2017 Storage at CERN 9
![Page 11: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/11.jpg)
CERNBox — SWAN
CERNBox• File sync and sharing• Office tools integration• Integration with ROOTa
ahttps://root.cern.ch
SWAN• Jupyter based notebooksa
• Python, ROOT, R, Spark• Nice CERNBox integration
ahttp://cern.ch/swan
20th October 2017 Storage at CERN 11
![Page 12: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/12.jpg)
Table of Contents
Introduction
Storage for physics
Infrastructure storage
Wrap up
20th October 2017 Storage at CERN 12
![Page 13: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/13.jpg)
Ceph• Openstack is Ceph’s killer app: 4x usage in 2 years• Not a single byte lost or corrupted
20th October 2017 Storage at CERN 13
![Page 14: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/14.jpg)
Ceph: NFS on RBDReplace NetApps with VMs
• ∼ 60TB across 30 servers• Openstack VM + RBD vol.• CentOS7 with ZFS• Not highly-available, but. . .• Cheap (thin-provisioning)• Resizable
Moving to Manila+CephFS very soon
20th October 2017 Storage at CERN 14
![Page 15: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/15.jpg)
Ceph: NFS on RBDReplace NetApps with VMs
• ∼ 60TB across 30 servers• Openstack VM + RBD vol.• CentOS7 with ZFS• Not highly-available, but. . .• Cheap (thin-provisioning)• Resizable
Moving to Manila+CephFS very soon
20th October 2017 Storage at CERN 14
![Page 16: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/16.jpg)
CephFS for HPCCERN is mostly a HTC lab
• Parallel workload, quite tolerant to relaxed consistency• HPC corners in the Lab
• Beams, Plasma simulations• Computation Fluid Dynamics• Quantum ChromoDynamics
• Require fill POSIX, read-after-write consistency, parallel IO
• ∼ 100 nodes HPC cluster accessing ∼ 1PB CephFS
20th October 2017 Storage at CERN 15
![Page 17: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/17.jpg)
Ceph: Scale testingBigbang scale tests mutually benefitting CERN Ceph
Bigbang I: 30PB, 7200 OSDs, Ceph HammerFound several osdmap limitations
Bigbang II: Similar size, Ceph JewelScalability limited by OSD-MON traffic.Lead to development of ceph-mgr
Bigbang III: 65PB, 10800 OSDs, Ceph LuminousNo major issue found
20th October 2017 Storage at CERN 16
![Page 18: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/18.jpg)
Table of Contents
Introduction
Storage for physics
Infrastructure storage
Wrap up
20th October 2017 Storage at CERN 17
![Page 19: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/19.jpg)
Next challenges
20th October 2017 Storage at CERN 18
![Page 20: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/20.jpg)
20th October 2017 Storage at CERN 19
![Page 21: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/21.jpg)
Wrap up
• Homegrown storage systems, augmented by open source• “Data deluge” forecasted for 2026• CentOS is powering a huge part of our services
20th October 2017 Storage at CERN 20
![Page 22: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/22.jpg)
![Page 23: Storage at CERN · 2018. 11. 21. · CERN is mostly a HTC lab • Parallel workload, quite tolerant to relaxed consistency • HPC corners in the Lab • Beams, Plasma simulations](https://reader035.vdocuments.us/reader035/viewer/2022081618/60a81f595ad48c291c27d9c3/html5/thumbnails/23.jpg)
References
• A. Peters: Present Future Solution for dta storage at CERN• D. van der Ster: Building Scale-Out Storage Infrastructures
with RADOS and Ceph• S. Campana: The ATLAS Computing Challenge for HL-LHC
20th October 2017 Storage at CERN 22