towards software defined persistent memory
TRANSCRIPT
Towards Software-defined Persistent Memory:
Rethinking Software Support for Heterogeneous Memory Architectures
Swaminathan Sundararaman*
Nisha Talagala*Dhananjoy Das Amar Mudrankit*
Dulcardo Arteaga*
*Work done at Fusion-io/SanDisk
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL 2
Memory-Storage Convergence (Trend 1)
L1, L2, L3 CPU Caches DRAM Hard Drives
MicrosecondsNanoseconds
CPU WAIT CYCLES
Tiered Memory Solutions
Main Memory System Storage Systems
Milliseconds
ACCESS DELAY
2 cycles 1,000,000s100s 1,000s 10,000s
chasm
2
FlashMemory
Persistent Memories
PM blurs the line between storage and memory
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL 3
Challenges with Current Persistent Memory SolutionsAccess Granularity Byte (Memory) Block (I/O) Hybrid
Memory Technology PCM ReRAM/ Memristor
SRAM (backed by Cap.) NVDIMM
Capacity 1 - 100s GB 1 - 100s GB 32K – 2GB 4 – 32GB
Local Attach Point PCIe NVMe SAS DDR
Access Mechanism File System Object Store KV Store …
Memory Location Local Remote Replicated
Network Connect Infiniband Ethernet PCIe …
3
Many possible combinations!
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL 4
• Rewrite applications for different deployments
▪ Not practical given the number of scenarios
• What about existing applications / deployments?
• User data is constantly growing and needs not fit in persistent memory
What Should Application Developers Do?
4
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL 5
Moving Towards a Software-defined World… (Trend 2)
Software-Defined Networking (SDN)Enables administrators to manage network services
by abstracting higher level functionality
Abstraction of logical storage services and capabilities from the underlying physical storage systems
Software-Defined Storage (SDS) Software-Defined Flash (SDF)Abstract or expose flash specific details to enable software to
realize the raw bandwidth and storage capacity of Flash
Software-Defined Data Center (SDDC)All elements of the infrastructure such as networking, storage, CPU and security are virtualized and delivered as a service
5
Abstraction of logical storage services and capabilities from the underlying physical persistent memory hardware and interconnect
Software-Defined Persistent Memory (SDPM)
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL 6
Our SDPM Solution• The first instance of a software defined approach to PM that can bring
the benefits of PM to a gamut of practical deployments.
• Abstract the heterogeneity in PM hardware from applications
• Provide file system API & programming libraries to access PM
• Use currently available PM hardware to show the feasibility of an SDPM
▪ PCIe & DDR4 attached PM (both local & remote attach)
▪ Using Infiniband & 10G Ethernet for remote access
• The prototype architecture provides good performance and near optimal acceleration for a range of local and remote PM configurations
6
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL 7
• Introduction
• Design
• Evaluation
• Conclusion
Outline
7
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL 8
• Support a variety of local and remote attach points with differing performance but identical functionality and semantic guarantees.
• Enable tiering of data between PM and flash, with caching in DRAM, to enable different cost/performance configurations
• Support hybrid (i.e., both memory and I/O) access, traditional storage management, and persistence guarantees to combine the best of memory and storage worlds.
• Enable a single application programming model to work across a variety of hardware
SDPM Design Goals
8
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL 9
Software Defined Persistent-Memory Architecture
Applications
Programming Libraries
File System
Persistent MemoryManager
Block DeviceDriver
SDPM
PCIe/DDR
SAS/SATA/NVMe
PM
Infiniband/Ethernet/PCIe/…
Memory/Block (I/O)
• Persistent Memory Manager ▪ Abstracts hardware and interconnect
details from the file system / applications ▪ Exports APIs to guarantee persistence
• File System ▪ Unified persistent namespace to PM &
Flash ▪ Hybrid access to PM ▪ Transparent & non-transparent
acceleration
• Programming Libraries ▪ Unified access APIs to applications ▪ OS bypass for remote access
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL 10
• A flash-optimized POSIX compliant Linux file system
▪ Extended NVMFS to support PM in addition to Flash
• Provides a unified and persistent namespace to both PM and flash
▪ Hybrid (memory & I/O) access to PM and applications can switch back and forth
▪ Transparent application acceleration (by tiering data between PM & flash)
▪ Supports “direct” mmap mode to directly map and use PM without caching in DRAM
▪ Supports a single programming model via a combination of application specific libraries over direct mmap and transparent access via POSIX APIs
Non-Volatile Memory File System (NVMFS)
10
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL 11
Persistent Memory Manager: Hardware Abstractions• Memory Mapping Types
▪ PM can be mapped in multiple ways depending on the hardware. We need to ensure that each memory type is default mapped to the optimal model possible for its physical attach (by default map it to write combining)
▪ Enable FS operations that allow the app. to control the per file memory mapping.
• Guaranteeing Persistence
▪ We need mechanism(s) to guarantee all acknowledged in-flight data (such as in CPU caches, registers, etc.) have reached the PM device independent of its attach point
▪ PMM provides a barrier() operation (to NVMFS and user space libraries) that guarantees that all data is moved to the persistence domain as needed for the attach point
11
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL 12
• Introduction
• Design
• Evaluation
• Conclusion
Outline
12
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL 13
Evaluation - ConfigurationConfigurations Attach Method Local or Remote
Config-1 DDR NVDIMM LocalConfig-2 PCIe MMIO LocalConfig-3 DDR NVDIMM Remote EthernetConfig-4 PCIe MMIO Remote EthernetConfig-5 DDR NVDIMM Remote InfinibandConfig-6 PCIe MMIO Remote Infiniband
Config-1, Config-2(Local)
Config- 3, 4, 5, 6 (Remote)
System Configuration HP DL380, 96GB DDR DRAM, x86_64 Linux 3.14 kernel
MySQL Version Percona 5.5
Flash Device (PCIe) Fusion-io Gen 2 ioMemory 1.2TB
Persistent Memory PCI-e: ACM (512KB)DDR4: Viking NVDIMM (8GB)
Network Interconnect N/A Infiniband: ConnectX-3 56 Gbit IB Ethernet: Intel 82599ES 10-Gigabit
13
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL 14
Evaluation - Setup
Host A(Source)MySQL
Programming Libraires
NVMFS PMM
ioMemory VSL
Replication Src
PM
SDPM
Local
Con
fig-2
Con
fig-1
Host B(Sink)
Programming Libraires
NVMFSPMM
ioMemory VSL
Replication Sink(daemon)
PM
SDPM
Con
fig-4
Con
fig-3
10Gbit Ethernet56Gbit Infiniband
Remote
14C
onfig
-5
Con
fig-6
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL 15
Local PCIe MMIO Vs DDR4 (config-1 Vs Config-2) Ba
ndw
idth
(MB/
sec)
0
1000
2000
3000
4000
Size (bytes)64 128 256 512 1024 2048 4096
DDR-BW MMIO-BW
Late
ncy
(nse
c)
0
2000
4000
6000
8000
Size (bytes)64 128 256 512 1024 2048 4096
DDR-LatencyMMIO-Latency
15
Smaller data sizes -> barrier operation Larger data sizes -> transport media
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL 16
Barrier Overheads: Local (config-1) Vs Remote (config-3 & config-5)La
tenc
y (u
s)
0
12500
25000
37500
50000
Synchronization Frequency (#Ops)1 10 100 1000
Ethernet FLUSH InfiniBand RDMA Local ACMPM
16
20x
4x
# of operations between synchronizations increases, the performance becomes closer
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL
Tran
s Pe
r Sec
ond
(x10
00)
0
15,000
30,000
45,000
60,000
REMOTE LOCAL VANILLA NO_LOG
Bin & Tr Log Tr log ONLY
MySQL: Insert Heavy Workload
17
Config-3 Config-1 on Flash Infinitely fast logging
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL
Ope
ratio
ns/s
ec (i
n x1
000)
0
7500
15000
22500
30000
MySQL: LinkBench 10x Workload
18
No Logs Vanilla Local(config-1)
Remote(config-5)
31%17%
Facebook’s social graph workload
10x: 100 million nodes
Inserts, delete, update, and lookup
30% writes (insert/update/delete)
70% reads (lookup)
Infiniband performance for small updates is sufficient
3%
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL 19
• Introduction
• Design
• Evaluation
• Conclusion
Outline
19
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL
Conclusions
• PM is going to change the storage-memory landscape • Many different forms / capacities / attach points / performance
• SDPM: a software-defined approach to using persistent memory
• Abstracts heterogeneity in memory hardware
• Applications can transparently run on local & remote persistent memory
• Selectively abstracts PM characteristics to provide optimal performance
• Transparently tier data between PM & Flash to exploit both existing & new applications written to run on PM.
• Our evaluation shows near optimal performance for local & remote attach PM
20
CONFIDENTIALPARALLEL MACHINES CONFIDENTIAL 21
Thank You
Iif you are interested in trying out SDPM: [email protected]