![Page 1: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/1.jpg)
NVMe-basedBeeGFS asanext-generationscratchfilesystemforHighPerformanceComputingandArtificialIntelligence/MachineLearningworkloadsGregLehman,IgorZupanovic,JacobAnders,ReneTyhouse,GarrySwan,JosephAntony
INFORMATIONMANAGEMENT&TECHNOLOGY(IMT)
13March2019
![Page 2: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/2.jpg)
Whoarewe?
NVMebasedBeeGFS scratchforHPCandAI/ML2 |
![Page 3: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/3.jpg)
AtCSIROwedotheextraordinaryeveryday.AsAustralia’snationalscienceagency,weinnovatefortomorrowwhiledeliveringimpacttoday– forourcustomers,allAustraliansandtheworld.
3 | NVMe-basedBeeGFS scratchforHPCandAI/ML
![Page 4: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/4.jpg)
CSIROinnovations
NVMe-basedBeeGFS scratchforHPCandAI/ML4 |
EXTENDEDWEAR
CONTACTS
POLYMERBANKNOTES
RELENZAFLUTREATMENT
WiFiWLAN
AEROGARD
TOTALWELLBEING
DIET
RAFTPOLYMERISATION
BARLEYmax™
SELFTWISTINGYARN
SOFTLYWASHINGLIQUID
HENDRAVACCINE
NOVACQ™PRAWNFEED
![Page 5: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/5.jpg)
CSIROIM&TScientificComputing
![Page 6: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/6.jpg)
IM&TScientificComputing
NVMe-basedBeeGFS scratchforHPCandAI/ML6 |
~100talentedstaff
80+collaborativeeResearch
projectsevery6months
Workingwithover2600+
customers
1500m²datacentrefloorspace
acrossAustralia
~2Petaflopsaggregate
performance
~40PBprimarydataholdings
2000published
collectionsindata.csiro.au
~5MillionCPUhourspermonth
![Page 7: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/7.jpg)
EnablingScience Impact.CSIROandScientificComputing7 |
InformationServices
WorkflowServices
Outreach&eResearchPlanning
ResearchDataServices
ScientificComputing&Visualisation
AdvancedCollaboration
IMTeResearch SupportsEnd-to-EndScience
ResearchQuestion
ResearchDesign
DataCollection
Processing& Analysis
Data&WorkflowArchiving
Disseminate&Publish
DataRe-Use
MeasureImpact
IMTeEnablementServices:HighSpeedNetworks,ApplicationDevelopment,Tele-presence,OfficeProductivity,CollaborationTools
eResearch services supporting the science workflow(Diagram based on Bath University’s ‘Research360 InstitutionalResearch Lifecycle’)
![Page 8: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/8.jpg)
ScientificComputing
EnablingScience Impact.CSIROandScientificComputing8 |
PlatformsGroup
ServicesGroup
ScienceApplications
Visualisation UserServices
Systems Data Facilities
DataProcessing
NationalCollaborations
![Page 9: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/9.jpg)
Systems
EnablingScience Impact.CSIROandScientificComputing9 |
TheSystemsteammanages:
• Pearcey – Generalpurposecluster.Upgradedto230Haswellnodes,4480cores,FDRInfiniband
• Ruby– SGIUV3000NUMASystemhosting8TBand640coresfromasingleoperatingsystem
• Bragg– 384NvidiaKeplerGPU’sandXeonPhienabledsystem;128nodes.Top500System~1MCUDAcores
• HTCondor –Cycleharvestingserviceacross~4400desktops(360CPUyearsofcomputeinthelastyear)
Systemsservicesare:
• Usedby>2600CSIROscientists&affiliateso ~4millionCPUhoursofHPCjobspermontho ~1millionCPUhoursofHTCondor jobspermonth
• AnessentialcontributiontoCSIRO’sscienceandresearchportfolio
The CSIRO ‘Bragg’ and ‘Pearcey’ supercomputers
![Page 10: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/10.jpg)
ScientificUse-CasesDrivingStorage
![Page 11: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/11.jpg)
GPU-basedTomographicReconstruction
NVMe-basedBeeGFS scratchforHPCandAI/ML11 |
3D CT Reconstruction of breast tumourImaging and Medical Beamline, Australian Synchrotron
3DCTReconstructionofanexcisedhumanbreastcontainingatumour(inred).
ImagedattheImagingandMedicalBeamline(IMBL)at
theAustralianSynchrotron
![Page 12: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/12.jpg)
Simulationsof5GWirelessandBeyond
12 |
Evaluationoflargescalenetworkend-pointsfrom4G,5Gwirelessnetworks
andbeyond
NVMe-basedBeeGFS scratchforHPCandAI/ML
![Page 13: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/13.jpg)
3DVegetationMappingandAnalysis
NVMe-basedBeeGFS scratchforHPCandAI/ML13 |
Generatingvegetationcovermapsin3DfromdataacquiredviaaZebedeehandheldlaserscanner
![Page 14: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/14.jpg)
MaiaX-RayImaging• Synchrotronx-rayfluorescence(SXRF)imagingisa
powerfultechniqueusedinthebiological,geological,materialsandenvironmentalsciences,medicineandculturalheritage
• Digitalimagesofmicroscopicornanoscopic detailarebuilt,pixelbypixel,byscanningthesamplethroughthebeam
• Theresultingx-rayfluorescenceradiationischaracteristicofthechemicalelementsinthatpixel.Thisisusedtoquantifythechemicalcompositionofthesample,includingimportanttraceelements,andtobuildupelementimagesofthesample
• CSIROworkedwiththeBrookhavenNationalLaboratory(BNL)todeveloptheMaiax-raymicroprobedetectorsystem.
• ThesystemcombinesBNL'scustomdetectorarraysandapplication-specificintegratedcircuits,withourhigh-speeddatacapturehardwareandreal-timespectralanalysisalgorithms
• ReconstructionalgorithmsrunonHPCresourcesandneedfaststorage
NVMe-basedBeeGFSscratchforHPCandAI/ML14 |
Maia RGB image collected at the Australian Synchrotron of a clay sample from the Mt Gibson gold deposit in Western Australia (green = iron, blue = bromine, red = arsenic).
![Page 15: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/15.jpg)
CapableStorageUnderpinsNextGenerationAppliedIndustrialScienceApplications
![Page 16: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/16.jpg)
StorageDrivers• ThechallengefacedbytheIM&TScientificComputingTeamwasto•SimultaneouslyoptimizeforhighIOPSandhighbandwidthworkloads
•Needstobeextremelypowerandrackefficient•Needstobeparallel,POSIXcompliantfilesystem•AbilitytosupportHPCandAI/MLworkloads
•WeendedupchoosinganNVMe basedsystemdrivenbytheBeeGFS filesystem
![Page 17: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/17.jpg)
StorageDrivers• ThechallengefacedbytheIM&TScientificComputingTeamwasto•SimultaneouslyoptimizeforhighIOPSandhighbandwidthworkloads
•Needtobeextremelypowerandrackefficient•Needstobeparallel,POSIXcompliantfilesystem•AbilitytosupportHPCandAI/MLworkloads
•WeendedupchoosinganNVMe basedsystemdrivenbytheBeeGFS filesystem
![Page 18: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/18.jpg)
- CurrentNetworkingTopology
- MetadataServiceBuildingBlocks
- StorageServiceBuildingBlocks
HardwareBuildingBlocks
![Page 19: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/19.jpg)
SwitchCentricViewofComputeandStorageClustersattheCDCFacility
NVMe-basedBeeGFS scratchforHPCandAI/ML19 |
![Page 20: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/20.jpg)
SwitchCentricViewofComputeandStorageClustersattheCDCFacility
NVMe-basedBeeGFS scratchforHPCandAI/ML20 |
MellanoxCS7520216portEDR
![Page 21: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/21.jpg)
SwitchCentricViewofComputeandStorageClustersattheCDCFacility
NVMe-basedBeeGFS scratchforHPCandAI/ML21 |
PearcyCPUCluster430Nodes;150TFlops
![Page 22: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/22.jpg)
SwitchCentricViewofComputeandStorageClustersattheCDCFacility
NVMe-basedBeeGFS scratchforHPCandAI/ML22 |
BracewellGPUCluster113Nodes;NvidiaP100s;1.5PFlops
![Page 23: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/23.jpg)
SwitchCentricViewofComputeandStorageClustersattheCDCFacility
NVMe-basedBeeGFS scratchforHPCandAI/ML23 |
BowenStorage40PB
![Page 24: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/24.jpg)
SwitchCentricViewofComputeandStorageClustersattheCDCFacility
NVMe-basedBeeGFS scratchforHPCandAI/ML24 |
BeeGFS2PBNVMe
![Page 25: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/25.jpg)
MetadataServiceBuildingBlocks1/2
• 4Metadata servers• DellEMC R440• DualIntel6154– 3.0GHz12core,384GB
• DualConnectX-5EDR
NVMe-basedBeeGFS scratchforHPCandAI/ML25 |
![Page 26: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/26.jpg)
MetadataServiceBuildingBlocks2/2
• 4Metadata servers• DellEMC R440• DualIntel6154– 3.0GHz12core,384GB
• DualConnectX-5EDR
• IntelP4600• 24x1.6TBIntelP4600NVMe• 3DNANDTLC• RandomReads~5.6million IOPS• RandomWrites~1.8million IOPS• ActivePower
– 14.2Watts(Write);9Watts(Read)• IdlePower– <5Watts
NVMe-basedBeeGFS scratchforHPCandAI/ML26 |
![Page 27: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/27.jpg)
StorageServiceBuildingBlocks1/2
• 32Storage servers• DellEMC R740xd• DualIntel6148– 2.4GHz20core,192GB
• DualConnectX-5EDR
NVMe-basedBeeGFS scratchforHPCandAI/ML27 |
![Page 28: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/28.jpg)
StorageServiceBuildingBlocks2/2
• 32Storage servers• DellEMC R740xd• DualIntel6148– 2.4GHz20core,192GB
• DualConnectX-5EDR
• IntelP4600• 24x3.2TBIntelP4600NVMe• 3DNANDTLC• RandomReads~6.4million IOPS• RandomWrites~2.3million IOPS• ActivePower
– 21Watts(Write);10Watts(Read)• IdlePower– <5Watts
NVMe-basedBeeGFS scratchforHPCandAI/ML28 |
![Page 29: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/29.jpg)
IO500Benchmark
NVMe-basedBeeGFSscratchforHPCandAI/ML29 |
![Page 30: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/30.jpg)
https://www.vi4io.org/_media/17-benchmarking-ws-io500.pdf
![Page 31: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/31.jpg)
https://www.vi4io.org/_media/17-benchmarking-ws-io500.pdf
![Page 32: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/32.jpg)
IO50010NodeChallenge– ZFSbackend
NVMe-basedBeeGFSscratchforHPCandAI/ML32 |
10 Clients; 16 Threads
SC’18 Results
![Page 33: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/33.jpg)
Summary• Capablestoragebuildingblocksareneededfordrivingnextgenerationappliedindustrialscientificapplications
• CSIROhasinvestedina2PBNVMe solutionwhichmetperformanceandpowercriteria
• ThePOSIXcompliant,BeeGFS parallelfilesystemwillberolledouttousersinQ1,2019
NVMe-basedBeeGFS scratchforHPCandAI/ML33 |
![Page 34: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/34.jpg)
CSIRO.Weimagine.Wecollaborate.Weinnovate.
NVMe-basedBeeGFS scratchforHPCandAI/ML34 |
![Page 35: NVMe-based BeeGFSas a next-generation scratch filesystem ...€¦ · Maia X-Ray Imaging • Synchrotron x -ray fluorescence (SXRF) imaging is a powerful technique used in the biological,](https://reader035.vdocuments.us/reader035/viewer/2022071113/5feabe55f1450f413415c962/html5/thumbnails/35.jpg)
INFORMATIONMANAGEMENT&TECHNOLOGY(IMT)
Thankyou