programming and usage models for non-volatile memory · programming and usage models for...
TRANSCRIPT
![Page 1: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/1.jpg)
Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced Technology Group, NetApp
SDC India 26th May 2016
© 2016 NetApp, Inc. All rights reserved. 1
![Page 2: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/2.jpg)
Agenda
§ Overview
§ SNIA NVM Programming Model and Ecosystem
§ File System Changes for NVM § Existing – Linux (ext4) § New – PMFS, BPFS
§ NVM Library – libpmem example
© 2016 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL --- 2
![Page 3: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/3.jpg)
Non-Volatile Memory Technology
Cache
STT-MRAM, 3D XPoint (on DDR)?
R/W (ns/ns)
PCM, ReRAM, 3D XPoint (OPTANE) R/W (ns/us)
NAND SSD R/W (us/us)
Hard Disk Drive R/W (ms/ms)
© 2016 NetApp, Inc. All rights reserved 3
• Persistent memory • Short Access Speeds • Byte Addressable
![Page 4: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/4.jpg)
Implications to Software
§ Persistence: Volatility a Virtue! § Application or OS panics because of an illegal/wrong persistent memory
address § Ensure durability § Ensure ordering
§ Fast Access Speed § Software Stack overhead
§ Byte Addressable § Block-oriented softwares § Can leverage Load/Store
© 2016 NetApp, Inc. All rights reserved 4
[Caulfield, MICRO ‘10]
![Page 5: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/5.jpg)
Roadblocks to Persistence
© 2016 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL --- 5
L1 L1
L2
Core
L1 L1
L2
Core
L1 L1
L2
Core
L3
Memory Controller Memory Controller
NVDIMM NVDIMM
MOV!
![Page 6: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/6.jpg)
Instruction Level Support
© 2016 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL --- 6
https://software.intel.com/sites/default/files/managed/b4/3a/319433-024.pdf
L1 L1
L2
Core
L1 L1
L2
Core
L1 L1
L2
Core
L3
Memory Controller
Memory Controller
NVDIMM NVDIMM
MOV!
CLFLUSH, CLFLUSHOPT, CLWB!
PCOMMIT!
![Page 7: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/7.jpg)
NVM Prog. Ecosystem
© 2016 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL --- 7
![Page 8: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/8.jpg)
SNIA NVM Programming Model
© 2016 NetApp, Inc. All rights reserved 8
www.snia.org/sites/default/files/NVMProgrammingModel_v1r10DRAFT_0.pdf
![Page 9: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/9.jpg)
SNIA NVM Programming Model
© 2016 NetApp, Inc. All rights reserved 9
![Page 10: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/10.jpg)
Ecosystem Overview
© 2016 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL --- 10
![Page 11: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/11.jpg)
File System Changes
© 2016 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL --- 11
![Page 12: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/12.jpg)
§ Design Goals § Efficient access to PM by apps § Optimize for byte addressability § Protect from stray writes
§ Contributions § Remove block layer and page
cache - Enable direct access to PM via XIP (DAX)
§ Supports large pages § Atomic in-place update to metadata § Provides crash consistency
§ Fine grained undo logging § Leverage write-protect in kernel
Intel’s PMFS
© 2016 NetApp, Inc. All rights reserved 12
http://dl.acm.org/citation.cfm?id=2592814
![Page 13: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/13.jpg)
Microsoft’s BPFS
§ Ensures file system consistency guarantees
§ Use copy-on-write up to root of file system
§ Problem: § Any change requires bubbling to the FS root § Small writes result in large copy overhead
§ Solution: § Uses Short-Circuit Shadow Paging
§ Adopt in-place and/or atomic update when possible § Uses byte-addressability and atomic 64b writes
© 2016 NetApp, Inc. All rights reserved 13
http://research.microsoft.com/pubs/81175/BPFS.pdf
![Page 14: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/14.jpg)
Problem with CoW (Shadow Paging)
© 2016 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL --- 14
Any small change results in large copy overhead
BA A’ B’
file’s root pointer unnecessary copies
![Page 15: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/15.jpg)
Short-Circuit Shadow Paging
§ In-place update when possible – save on copies
§ Appends committed by updating the file size
© 2016 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL --- 15
BA A’ B’
file’s root pointer in-place update
![Page 16: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/16.jpg)
Linux File System Support
§ Direct Access for Files (DAX) § Bypass page cache § Perform read/write directly to the storage device – Synchronous
§ Support from block driver § direct_access!§ Examples: brd – RAM backed block device driver
§ Support from File System § Direct_IO functions – dax_do_io()!§ Implementing mmap file operations for DAX files
§ Page fault functions § Examples: ext2, ext4
© 2016 NetApp, Inc. All rights reserved 16
https://www.kernel.org/doc/Documentation/filesystems/dax.txt
![Page 17: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/17.jpg)
Traditional vs. Optimized File Systems
© 2016 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL --- 17
![Page 18: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/18.jpg)
NVM Library
© 2016 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL --- 18
![Page 19: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/19.jpg)
NVM Library
§ library for using memory-mapped persistence, specifically for PM
§ Supports SNIA NVM API
§ Builds on Linux DAX
§ Provides a collection of libraries: § libpmem: low-level persistent memory support – persistent memory instructions
for flushing changes to PM. § libpmemlog: pmem-resident log file (uses libpmem) § libpmemobj: transactional object store (uses libpmem) § And many more
§ Link: http://pmem.io/nvml/libpmem/
© 2016 NetApp, Inc. All rights reserved 19
http://pmem.io/nvml/libpmem/
![Page 20: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/20.jpg)
Example using libpmem
!/* create a pmem file */ ! !!!if ((fd = open("/pmem-fs/myfile", O_CREAT|O_RDWR, 0666)) < 0) { !!! !perror("open"); !!! !exit(1);!!!} !!!/* allocate the pmem */ !!!posix_fallocate(fd, 0, PMEM_LEN))!!!!/* memory map it */ !!!if ((pmemaddr = pmem_map(fd)) == NULL) { !!! !perror("pmem_map"); !!! !exit(1);!!!} !!! !!!/* store a string to the persistent memory */ ! !!!strcpy(pmemaddr, "hello, persistent memory."); !!!/* flush above strcpy to persistence */!!pmem_persist(pmemaddr, PMEM_LEN); !!
!!strcpy(pmemaddr, "hello again, persistent memory.");!!pmem_flush(pmemaddr, PMEM_LEN); !!!pmem_drain(); !!
!
© 2016 NetApp, Inc. All rights reserved 20
Force changes to NVM
Flush processor caches
Wait for h/w buffers to drain
![Page 21: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/21.jpg)
Summary
§ Scratching the surface à more research going on areas such as remote memory access, security, etc.
§ Existing software (without changes) on NVM would result in: § Sub-optimal performance § Consistency/Durability issues
§ Usage Models § NVM Block Device w/ existing/modified file systems or direct access § NVM File Systems on NVM Driver § NVM Library (talks to NVM Filesystem) – libpmem § Persistent Memory Regions and Persistent heap
§ Manage using memory management unit § APIs: pmap, punmap, pmalloc, pfree § E.g., Mnemosyne, NV-Heaps
© 2016 NetApp, Inc. All rights reserved 21
![Page 22: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/22.jpg)
Thank you.
© 2016 NetApp, Inc. All rights reserved. 22
![Page 23: Programming and Usage Models for Non-volatile Memory · Programming and Usage Models for Non-volatile Memory Priya Sehgal MTS, Advanced ... (DAX) § Supports large pages § Atomic](https://reader036.vdocuments.us/reader036/viewer/2022081605/5b4fb3467f8b9a2f6e8ce468/html5/thumbnails/23.jpg)
References
BPFS: http://research.microsoft.com/pubs/81175/BPFS.pdf
[Caulfield, MICRO ‘10]: Moneta: A High-performance Storage Array Architecture for Next-generation, Non-volatile Memories, Adrian M. Caulfield, Arup De, Joel Coburn, Todor I. Mollov, Rajesh K. Gupta, and Steven Swanson, MICRO 43
Intel Manual: https://software.intel.com/sites/default/files/managed/b4/3a/319433-024.pdf
[libpmem]: http://pmem.io/nvml/libpmem/
Linux DAX: https://www.kernel.org/doc/Documentation/filesystems/dax.txt
Mnemosyne: http://research.cs.wisc.edu/sonar/papers/mnemosyne-asplos2011.pdf
NV-Heaps: http://www.msr-waypoint.com/pubs/198372/Asplos2011_NVHeaps.pdf
PMFS: http://dl.acm.org/citation.cfm?id=2592814
SNIA NVM Programming TWG: www.snia.org/sites/default/files/NVMProgrammingModel_v1r10DRAFT_0.pdf
© 2016 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL --- 23