architectureofa)) nextgenerationparallel)filesystem) · 2017. 11. 7. ·...
TRANSCRIPT
![Page 1: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/1.jpg)
Architecture of a Next-‐Generation Parallel File System
![Page 2: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/2.jpg)
-‐ Introduction -‐ Whats in the code now -‐ futures
Agenda
![Page 3: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/3.jpg)
An introduction
![Page 4: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/4.jpg)
What is OrangeFS?
• OrangeFS is a next generation Parallel File System • Based on PVFS • Distributes file data across multiple file servers
leveraging any block level file system. • Distributed Meta Data across 1 to all storage servers • Supports simultaneous access by multiple clients,
including Windows using the PVFS protocol Directly • Works w/ standard kernel releases and does not require
custom kernel patches • Easy to install and maintain • Happy member of the Linux Foundation
![Page 5: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/5.jpg)
Why Parallel File System?
HPC – Data Intensive Parallel (PVFS) Protocol
• Large datasets
• Checkpointing
• Visualization
• Video
• BigData
Unstructured Data Silos Interfaces to Match Problems
• Unify Dispersed File Systems
• Simplify Storage Leveling
§ Multidimensional arrays
§ typed data § portable formats
![Page 6: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/6.jpg)
Original PVFS Design Goals § Scalable § Configurable file striping § Non-‐contiguous I/O patterns § Eliminates bottlenecks in I/O path § Does not need locks for metadata ops § Does not need locks for non-‐conflicting applications
§ Usability § Very easy to install, small VFS kernel driver § Modular design for disk, network, etc § Easy to extend -‐> Hundreds of Research Projects have
used it, including dissertations, thesis, etc…
![Page 7: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/7.jpg)
OrangeFS Philosophy • Focus on a Broader Set of Applications • Customer & Community Focused
• (~300 Member Strong Community & Growing)
• Completely Open Source • (no closed source add-‐ons)
• Commercially Viable • Enable Research
![Page 8: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/8.jpg)
Configurability
Performance
Consistency Reliability
![Page 9: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/9.jpg)
System Architecture
• OrangeFS servers manage objects • Objects map to a specific server • Objects store data or metadata • Request protocol specifies operations on one or
more objects
• OrangeFS object implementation • DB for indexing key/value data • Local block file system for data stream of bytes
![Page 10: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/10.jpg)
Current Architecture
Client Server
![Page 11: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/11.jpg)
1994-‐2004 Design and Development at CU Dr. Ligon + ANL (CU Graduates)
Primary Maint & Development ANL (CU Graduates) + Community
2004-‐2010
2007-‐2010 New PVFS Branch
SC10 (fall 2010)
Future
Announced with community and is now Mainline of future development as of 2.8.4
Spring 2012
New Development focused on a broader set of problems
SC11 (fall 2011)
Performance improvements, Direct Lib + Cache Stability, WebDAV, S3
PVFS2
PVFS
Improved MD, Stability, Server Side Operations,
Newer Kernels, Testing Windows Client, Stability, Replicate on Immutable
2.8.6 + Webpack
2.8.5 + Win Maintenance and Support Services Offered by Omnibond
OrangeFS next 3.0
2012 Distributed Dir MD, Capability based security 2.9.0
![Page 12: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/12.jpg)
In the Code now
![Page 13: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/13.jpg)
Server to Server Communications (2.8.5)
Traditional Metadata Operation
Create request causes client to communicate with all servers O(p)
Scalable Metadata Operation
Create request communicates with a single server which in turn communicates with other servers using a tree-‐based protocol O(log p)
Mid
Client
Serv
App
Mid Mid
Client
Serv
Client
Serv
App
Network
App
Mid
Client
Serv
App
Mid Mid
Client
Serv
Client
Serv
App
Network
App
![Page 14: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/14.jpg)
Recent Additions (2.8.5)
SSD Metadata Storage Replicate on Immutable (file based)
Windows Client
Supports Windows 32/64 bit
Server 2008, R2, Vista, 7
![Page 15: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/15.jpg)
Direct Access Interface (2.8.6)
• Implements: • POSIX system calls • Stdio library calls
• Parallel extensions • Noncontiguous I/O • Non-‐blocking I/O
• MPI-‐IO library • Found more boundary
conditions fixed in upcoming 2.8.7
App
Kernel PVFS lib
Client Core
Direct lib
PVFS lib Kernel
App
IB
TCP
![Page 16: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/16.jpg)
File System File System File System
Direct Interface Client Caching (2.8.6)
• Direct Interface enables Multi-‐Process Coherent Client Caching for a single client
File System File System
Client Application
Direct Interface
Client Cache
File System
![Page 17: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/17.jpg)
WebDAV (2.8.6 webpack)
PVFS Protocol
Orang
eFS
Apache
• Supports DAV protocol and tested with the Litmus DAV test suite
• Supports DAV cooperative locking in metadata
![Page 18: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/18.jpg)
S3 (2.8.6 webpack)
PVFS Protocol
Orang
eFS
Apache
• Tested using s3cmd client • Files accessible via other access methods
• Containers are Directories
• Accounting Pieces not implimented
![Page 19: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/19.jpg)
Summary -‐ Recently Added to OrangeFS
• In 2.8.3 • Server-‐to-‐Server Communication • SSD Metadata Storage • Replicate on Immutable
• 2.8.4, 2.8.5 (fixes, support for newer kernels) • Windows Client • 2.8.6 – Performance, Fixes, IB updates • Direct Access Libraries (initial release) • preload library for applications, Including Optional Client
Cache • Webpack • WebDAV (with file locking), S3
![Page 20: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/20.jpg)
In the Works
![Page 21: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/21.jpg)
Hadoop JNI Interface (2.8.7?)
• OrangeFS Java Native Interface
• Extension of Hadoop File System Class –>JNI
• Replicate on Immutable • Looking at
intermediate append replication
• Isolate MD Servers
• Buffering
• Distribution
PVFS Protocol
![Page 22: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/22.jpg)
Configurable Locking (2.8.7?)
• File Locking already in WebDAV • Bringing Locking concept into Client & KM • Fcntl / flock • Locks maintained across server in MD
• Initially support client and NFS locks through KM
• Eventually integrate with WebDAV locks (if semantics work out)
![Page 23: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/23.jpg)
Distributed Directory Metadata (2.9.0)
DirEnt1
DirEnt2
DirEnt3
DirEnt4
DirEnt5
DirEnt6
DirEnt1 DirEnt5
DirEnt3
DirEnt2
DirEnt6
DirEnt4
Server0
Server1
Server2
Server3
Extensible Hashing
u State Management based on Giga+ u Garth Gibson, CMU
u Improves access times for directories with a very large number of entries
![Page 24: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/24.jpg)
Cert or Credential
Signed Capability I/O
Signed Capability
Signed Capability I/O
Signed Capability I/O
OpenSSL PKI
• Untrusted Client support at the pvfs protocol level • Working on how to simplify certificate integration
• Make it so people will want to use
![Page 25: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/25.jpg)
• Modular infrastructure to easily build background parallel processes for the file system Used for: • Gathering Stats for Monitoring • Usage Calculation (can be leveraged for Directory
Space Restrictions, chargebacks) • Background safe FSCK processing (can mark bad
items in MD) • Background Checksum comparisons • Etc…
Background Parallel Processing Infrastructure (2.x.x?)
![Page 26: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/26.jpg)
Admin REST interface / Admin UI
PVFS Protocol
Orang
eFS
Apache
REST
(2.x.x?)
![Page 27: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/27.jpg)
• In an Exascale environment with the potential for thousands of I/O servers, it will no longer be feasible for each server to know about all other servers.
• Servers will discover a subset of their neighbors at startup (or may be cached from previous startups). Similar to DNS domains.
• Servers will learn about unknown servers on an as needed basis and cache them. Similar to DNS query mechanisms (root servers, authoritative domain servers).
• SID Cache, in memory db to store server attributes
Server Location / SID Mgt OrangeFS NEXT
![Page 28: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/28.jpg)
• An OID (object identifier) is a 128-‐bit UUID that is unique to the data-‐space
• An SID (server identifier) is a 128-‐bit UUID that is unique to each server.
• No more than one copy of a given data-‐space can exist on any server
• The (OID, SID) tuple is unique within the file system.
• (OID, SID1), (OID, SID2), (OID, SID3) are copies of the object identifier on different servers.
Handles -‐> UUIDs OrangeFS NEXT
![Page 29: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/29.jpg)
Replication / Redundancy • Redundant Metadata • seamless recovery after a failure • Redundant objects from root directory down • Configurable
• Fully Redundant Data • Experiments with “forked flow” show small overhead • Configurable • Number of replicas (O .. N) • Update mode (continuous, on close, on immutable,
none)
• Emphasis on continuous operation
OrangeFS NEXT
![Page 30: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/30.jpg)
Policy Based Location
• User defined attributes for servers and clients • Stored in SID cache
• Policy is used for data location, replication location and multi-‐tenant support
• Completely Flexible • Rack • Row • App • Region
OrangeFS NEXT
![Page 31: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/31.jpg)
Data Migration / Mgt
• Built on Redundancy & DBG processes • Migrate objects between servers
• De-‐populate a server going out of service • Populate a newly activated server (HW lifecycle) • Moving computation to data
• Hierarchical storage • Use existing metadata services
• Possible -‐ Directory Hierarchy Cloning • Copy on Write (Dev, QA, Prod environments with high % data
overlap)
OrangeFS NEXT
![Page 32: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/32.jpg)
Hierarchical Data Management
Archive Intermediate Storage NFS
Remote Systems exceed, OSG, Lustre, GPFS, Ceph, Gluster
HPC OrangeFS
Metadata OrangeFS
Users
OrangeFS NEXT
![Page 33: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/33.jpg)
Attribute Based Metadata Search
• Client tags files with Keys/Values • Keys/Values indexed on Metadata Servers • Clients query for files based on Keys/Values • Returns file handles with options for filename
and path
Key/Value Parallel Query
Data
Data
File Access
OrangeFS NEXT
![Page 34: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/34.jpg)
Beyond OrangeFS NEXT
![Page 35: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/35.jpg)
Extend Capability based security
• Enable certificate level access (in process) • Federated access capable • Can be integrated with rules based access
control • Department x in company y can share with
Department q in company z • rules and roles establish the relationship • Each company manages their own control of who is in
the company and in department
![Page 36: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/36.jpg)
SDN -‐ OpenFlow
• Working with OF research team at CU • OF separates the control plane from delivery,
gives ability to control network with SW • Looking at bandwidth optimization
leveraging OF and OrangeFS
![Page 37: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/37.jpg)
ParalleX
ParalleX is a new parallel execution model • Key components are: • Asynchronous Global Address Space (AGAS) • Threads • Parcels (message driven instead of message passing) • Locality • Percolation • Synchronization primitives • High Performance ParalleX (HPX) library
implementation written in C++
![Page 38: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/38.jpg)
PXFS
• Parallel I/O for ParalleX based on PVFS • Common themes with OrangeFS Next
• Primary objective: unification of ParalleX and storage name spaces. • Integration of AGAS and storage metadata subsystems • Persistent object model
• Extends ParalleX with a number of IO concepts • Replication • Metadata
• Extending IO with ParalleX concepts • Moving work to data • Local synchronization
• Effort with LSU, Clemson, and Indiana U. • Walt Ligon, Thomas Sterling
![Page 39: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/39.jpg)
Community
![Page 40: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/40.jpg)
Learning More
• www.orangefs.org web site • Releases • Documentation • Wiki
• Dell OrangeFS Reference Architecture Whitepaper
• pvfs2-‐users@beowulf-‐underground.org • Support for users
• pvfs2-‐developers@beowulf-‐underground.org • Support for developers
![Page 41: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/41.jpg)
Support & Development Services
• www.orangefs.com & www.omnibond.com • Professional Support & Development team • Buy into the project
![Page 42: Architectureofa)) NextGenerationParallel)FileSystem) · 2017. 11. 7. · ServertoServer)Communications (2.8.5)) Traditional+Metadata+ Operation+ +Create+request+causes+client+to+](https://reader035.vdocuments.us/reader035/viewer/2022071511/61305dce1ecc515869440d14/html5/thumbnails/42.jpg)