resource management: beancounters
DESCRIPTION
Resource Management: Beancounters. Agenda. Current state of resource management in the Linux kernel Beancounters overview User memory management I/O accounting Kernel memory management Network buffers accounting Performance. Current state. Per-process accounting and limiting (rlimits) - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/1.jpg)
Resource Management: Beancounters
Pavel [email protected]
Denis [email protected]
Kirill [email protected]
![Page 2: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/2.jpg)
Agenda
Current state of resource management in the Linux kernel
Beancounters overview User memory management I/O accounting Kernel memory management Network buffers accounting Performance
![Page 3: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/3.jpg)
Current state
Per-process accounting and limiting (rlimits) Manages individual processes Memory limits are mostly ignored by the kernel
Group-based management Absent
Global statistics Not suitable for group isolation
![Page 4: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/4.jpg)
Operating system resources
Memory CPU time IO bandwidth Networking bandwidth Disk space
![Page 5: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/5.jpg)
Agenda
Current state of resource management in the Linux kernel
Beancounters overview User memory management I/O accounting Kernel memory management Network buffers accounting Performance
![Page 6: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/6.jpg)
Beancounters basics A beancounter manages a group of tasks Resource counters parameters
held – the current consumption level limit – the maximal allowed level of consumption barrier – the "shortage warn" line – each resource
controller may take some precautions fails – the number of allocation rejects
Beancounter is assigned once during process lifetime
![Page 7: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/7.jpg)
Accounting details
Process
User space Kernel space
Beancounter
kernel object
![Page 8: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/8.jpg)
Beancounters controlled resources
User memory Length of mappings RSS Locked pages
Dirty page cache Kernel memory Network buffers
Miscellaneous resources Number of tasks Number of files Number of sockets Number of file locks Number of PTYs Number of signals Active dentry cache
![Page 9: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/9.jpg)
Agenda
Current state of resource management in the Linux kernel
Beancounters overview User memory management I/O accounting Kernel memory management Network buffers accounting Performance
![Page 10: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/10.jpg)
User memory management
VMA lengths accounting Graceful rejects of VM region allocation Take precautions against overcommitment
RSS accounting Real memory usage OOM killer priorities
Dirty page cache accounting IO statistics and scheduling
![Page 11: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/11.jpg)
VMA lengths accounting
VMAs classification
unreclaimable:private and anonymous
reclaimable:shared file mappings
Unused pages Used pages Unreclaimable VMAsReclaimable VMAs
“Lengths of mappings” resource
“RSS” resource
Pages classification
unused:parts of mapped regions
used:touched pages
Task address space
![Page 12: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/12.jpg)
VMA lengths accounting pros'n'cons
Pros The way to track the
host commitment level Graceful rejects of
address space growths
Cons Hard limiting of
address space growth
![Page 13: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/13.jpg)
RSS accounting
First touch N Touches
Drawbacks Additional pointer on the struct page Extra locking during page faults
page page beancounter
beancounter
![Page 14: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/14.jpg)
Shared pages accounting
Account the page to the first beancounter Non uniform statistics for similar beancounters
Account a whole page for each beancounter The values accounted are not related to the actual
memory usage Account page's fractions the all beancounters
The “middle” way used in the beancounters
![Page 15: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/15.jpg)
Page fractions accounting
BC1
BC2
BC3BC4
1½
½¼
¼¼
¼Algorithm benefits O(1) algorithm of
adding and removing The sum of RSS on all
beancounters is an amount of all actually used pages
![Page 16: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/16.jpg)
Agenda
Current state of resource management in the Linux kernel
Beancounters overview User memory management I/O accounting Kernel memory management Network buffers accounting Performance
![Page 17: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/17.jpg)
Dirty page cache accounting
First touch N Touches
Dirty
Unmap
Last unmap
Clean
IO beancounter
![Page 18: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/18.jpg)
RSS accounting pros'n'cons
Pros Node memory
utilization statistics Asynchronous IO
scheduling Ground for fair page
reclamation
Cons Performance issues Memory consumption
by auxiliary data structures
![Page 19: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/19.jpg)
Agenda
Current state of resource management in the Linux kernel
Beancounters overview User memory management I/O accounting Kernel memory management Network buffers accounting Performance
![Page 20: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/20.jpg)
Kernel memory management
Reason Limited normal zone
Mainly for 32-bit arches
Major problem Object freeing context
Reference counters RCU
![Page 21: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/21.jpg)
Kernel MM data structures (pages)
Buddy page allocator Additional pointer on
the struct page
Vmalloc 0th page's pointer ...
page
struct vm_struct
![Page 22: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/22.jpg)
Kernel MM data structures (slab)
Array of pointers after the slab
struct slab
kmem_bufctl_t[N]
... ...
N objects
...
beancounters
![Page 23: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/23.jpg)
Kernel MM drawbacks A slab can carry less objects Slabs could become “offslab”
Slab name# of objects Offslab-ness
Before After Before After
Size-32 113 101 – –
Size-64 59 56 – –
Size-128 30 29 – –
Size-256 15 15 – –
Size-512 8 8 + +
Size-1024 4 4 + +
Size-2048 2 2 + +
Size-4096 1 1 + +
![Page 24: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/24.jpg)
Kernel MM pros'n'cons
Pros Tracking of kernel
memory usage
Cons No (all are already
optimized out)
![Page 25: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/25.jpg)
Agenda
Current state of resource management in the Linux kernel
Beancounters overview User memory management I/O accounting Kernel memory management Network buffers accounting Performance
![Page 26: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/26.jpg)
Network buffers accounting
Mainstream accounting shortcomings
slab overhead is not included up to 30% for usual Ethernet frames unpredictable difference for non-ethernet MTU no way to recalculate skb->truesize
![Page 27: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/27.jpg)
Implementation basics
Separate accounting for send and receive buffers TCP and all the other types of traffic
Implementation is straightforward: account actual memory usage for objects with
undefined or infinite lifetime select(2) compatibility Buffer space guarantees
![Page 28: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/28.jpg)
Packets context handling
beancounter
process
NetworksocketSKB SKB
![Page 29: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/29.jpg)
Agenda
Current state of resource management in the Linux kernel
Beancounters overview User memory management I/O accounting Kernel memory management Network buffers accounting Performance
![Page 30: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/30.jpg)
Performance
Test nameNo RSS Full
% %
Process creation 97% 91%
Execl Throughtput 99% 91%
Pipe Throughtput 100% 99%
Shell Scripts 96% 87%
File Read 99% 98%
File Write 101% 99%
RSS accounting – the bottleneck
![Page 31: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/31.jpg)
Main future directions Optimization
Pre-charging Kernel memory VMAs lengths
On-demand accounting Active dentry cache RSS
RSS limits Page reclamation
Better TCP window management
![Page 32: Resource Management: Beancounters](https://reader035.vdocuments.us/reader035/viewer/2022070403/56813a3c550346895da224b5/html5/thumbnails/32.jpg)
That's all folks Questions?
Comments?
http://download.openvz.org/~xemul/