death to the shared posix file system!? long live the ......long live the shared posix file system!!...

©2018 DataDirect Networks, Inc. DDN Confidential

Death to the Shared POSIX File System!?

Long Live the Shared POSIX File System!!

HPC Storage and the HPC Data Center

this is a vendor presentation!

DDN ConfidentialDDN Storage | ©2018 DataDirect Networks, Inc.

Classical HPC Data Centers

• Everything is bare metal

•Most data is stored in shared POSIX file systems that largely remain on HDDs

• SSD-based systems are slowly coming, but capacities remain small

• People have played with Hadoop and objects stores, but very few systems are actually in production

• Special performance and capacity tiers are used mainly by the large centers, but not the broader market

Cloud Data Centers

• Everything is virtualized, containerized, dynamic, automated, etc.

•Most date lives on local file systems in virtual machines (or containers), either on SSDs or HDDs

•Monolithic web applications use large, distributed object stores

• Enterprise storage still being used for various specific requirements, both on premise and hosted in the cloud

• Tiers exists for redundancy, data back-up, data retention, etc.


Example: 2015 DDN User Analysis

Work Data Mixed Use Archive

Cloud

HPC Work

WeatherClimate

CAEChemical

General Academic

Genomics

Big DataScience

Security

Finance

Energy

Tier 2HPCCloud

Cloud

Work31%

Data33%

Mixed22%

Cloud2%

Archive12%


Example: 2018 Lustre End User Survey

0% 5% 10% 15% 20% 25% 30% 35% 40%

Weather/Climate

Other

Media

Manufacturing

Life Sciences

Government

Financial Services

Energy

Education

Defense

AI/Machine Learning

Primary Usage of Lustre


HPC Data Centers: The Past Decade

►“Research Big Data” is a very big market for storage (and a much larger than the classical tightly-coupled simulations) and it is quite different from classical HPC

►Many data center dedicated to analytics and deep learning look like large cloud environments, but their IO requirements are much closer to Research Big Data

►Some “Research Big Data” end users have moved to on-premise cloud environments using Open Stack etc.

►More HPC and Analytics customers are running applications in the cloud or in cloud-like environments, with shared file systems running in the cloud


Predictions From LANL in 2016: Serving Data to the Lunatic Fringe


This architecture was imperiled by SSD economics


This architecture becomes imperiled by tape economics


This architecture becomes imperiled by Lang’s Law!

”DOE doesn’t want tiers. Tiers are an unfortunate accident of economics. DOE wants infinite memory and a system without

unplanned interrupts.

Just remember this:The fewer tiers, the fewer tears.”


Predictions from NERSC: Storage 2020-2025

Four Tiers Three Tiers Two Tiers


The Foreseeable Future Remains Tiered

DDN ConfidentialDDN Storage | ©2019 DataDirect Networks

Tiering Tiering – Data Schmiering

All due respect to Lang’s Law (‘fewer tiers, fewer tears’),tiering is a (mostly) solved problem.

Buffer-caching is a (mostly) solved problem!

Russel Kirsch developed it for the SEAC in 1952.

DDN ConfidentialDDN Storage ©2019 DataDirect Networks, Inc.

Flash Acceleration Layer Usage and Tiering Workflows in Traditional HPCLee Ward, Use Cases or BB Roles, Informal Burst Buffer Presentation via Sandia National Laboratories, 2015.

Challenges and Considerations for Utilizing Burst Buffers in High-Performance Computing, Melissa Romanus, Robert Ross, Manish Parashar, 2018.

Development of a Burst Buffer System for Data-Intensive Applications,Teng Wang, Sarp Oral, Michael Pritchard, Kevin Vasko, Weikuan Yu, 2015.

An Operational Perspective on a Hybrid and Heterogeneous Cray XC50 System.Sadaf Alam, Nicola Bianchi, Nicholas Cardo, Matteo Chesi, Miguel Gila, Stefano Gorini, Mark Klein, Colin McMurtrie, Marco Passerini, Carmelo Ponti, Fabio Verzelloni, 2017.

1. Checkpoint-Restart

2. In-situ/transit viz/analysis

3. Out-of-core

4. Accelerated reads (pre-stage)

5. Random-read centric applications (lots of them!)


Data Tiering: A Subtle Shift in Perception

Applications

Performance Storage

Capacity Storage

Unnecessarily Strict Tiering Relaxed Tiering

Applications

Capacity Storage

Performance Storage


“What?”

“Do you have any water?”

“Build an object store!”


Typical Object Requirements (“Object Schmobject; long live POSIX”)

► Immutable, transactional get/put, trillions of objects

► Named objects

► Group objects into logical collections

► Nest logical collections within each other

► Have the same object appear within multiple collections

► Multi-threaded writes

► Tag objects

►Object is a subset of file• There is no application which uniquely requires

object semantics

•O_TMPFILE and rename are useful primitives

►Object requirements grow as humans use them• Eventually they become file requirements

►We do not live on a deserted desert island•We have two decades experience building parallel

file systems

►RELEVANT LESSON FROM OBJECT STORES?• POSIX relaxation is useful

Bent’s Law for HPC Storage:

The future is bright and mostly as we predicted it.

Don’t be scared of POSIX; but embrace relaxations.

Don’t be scared of tiering; but embrace relaxations.


ML/DL Storage Scenarios: Small-to-Large

Local NVMe

Node-Local “Shared” NVMe

Global/Capacity

Global NVMe

OnDemandNamespaces Remote NVMe

(NVMeoF)


Dynamically-Provisioned File Systems

► Job-Specific File Systems• LLNL Proposal (2014): file system per JOB with data staging

• Isolation from other jobs

• Increased metadata performance

► Loop-back devices for increased metadata Performance

• K-Computer/Fujitsu: Very large-scale Implementation

• NERSC Library (e.g. for use of SPARC on Lustre)

• Lustre on Amazon/Azure/Google (since 2013)

• Amazon Lustre Service (November 2018)

►Future• Client Container Image (CCI) Feature in Lustre (integrates look-back

devices into Lustre)

• Lustre-on-Demand Feature (first deployments in early 2019!)


Amazon FSx for Lustre

►Dynamically-provisioned Lustre file systems

►Application-specific

►Focused on analytics and DL applications

►Connector to Amazon S3

►But: the usual limitations still apply;-)


Next Decade of HPC Storage: Back to the Future?

►Still PFS…

•… but with increasing portions that are dynamically allocated and integrated into the compute platform

• e.g. a file server turns into a containerized process run anywhere

►Still POSIX…

•… but relaxed where needed

►Still Tiers…

•… but relaxed to reflect actual application workflows

Relaxed Tiering with Dynamically-Managed,

System-integrated Storage

Applications

Capacity Storage

Performance ENS ENS

CapS

death to the shared posix file system!? long live the ......long live the shared posix file system!!...

Documents