demand technology software, inc. 32-bit virtual memory constraints in windows 2000 and 2003 mark...

Demand Technology Software, Inc.

32-bit Virtual Memory Constraints in

Windows 2000 and 2003Mark FriedmanDemand Technology

1020 Eighth Avenue South, Suite 6, Naples, FL 34102 USA

phone: (239) 261-8945 fax: (239) 261-5456

e-mail: [email protected]://www.demandtech.com

2Demand Technology, Inc. 32-bit VM Constraints

Outline Topics

Virtual Memory concepts

Virtual Memory constraints in 32-bit Windows

Physical Address Extension (PAE)

Leaking processes

Page faults/sec

Vir t ualaddress

space

V ir t ualaddress

space

P hysical M em ory

Page Tables


Virtual Memory Manager

4 GB virtual per process address space Lower 2 GB - Private

Page 0 reserved Code pages Heap

Upper 2 GB - System System code Shared dlls System cache

System

User

0

x'1 0000'16

No Access

x'8000 0000'16

x'ffff 0000'16

System Code

Device Driver Code

Nonpaged Pool

Paged Pool

PTEs

File Cache



All address spaces share the virtual addresses from the same upper 2 GB of the System area

All virtual addresses in the lower 2 GB User area are unique

Shared memory in the System range can be used for IPC

User

UserUser

System

User

0

x'1 0000'16

x'8000 0000'16

x'ffff 0000'16

System Code

Device Driver Code

Nonpaged Pool

Paged Pool

PTEs

File Cache



Commit Limit Limit on the number of

Virtual Memory pages that the system will allocate

Sizeof (RAM) + paging file(s)

Memory allocations start to fail as % Committed Bytes in Use 100%



Commit Limit But page files are

extendible!

Should you or shouldn’t you?


Extended Virtual Addressing

Boot.ini /3 GB switch Virtual storage constraint relief for some

applications Squeezes the System VM into 1 GB

Physical Address Extension (PAE) Supports 36-bit real addresses on Xeon processors

Address Windowing Extensions (AWE) Permits processes to address real memory > 4 GB



Boot.ini /3 GB switch Virtual storage

constraint relief for some server applications

SQL Server Exchange 2000 Etc.

/userva=SizeInMB subparameter, where SizeinMB can be any value between 2048 and 3072

System

User

0

x'1 0000'16No Access

x'c000 0000'16

x'ffff 0000'16

System Code

Device Driver Code

Nonpaged Pool

Paged Pool

PTEs

File Cache

/userva


Physical Address Extension

PAE is supported on most recent Intel servers Physical addresses are 36-bits Up to 64 GB of RAM can be installed

Page Table Entries (PTEs) remain 32-bits Process address spaces are still limited to

4 GB. How can they exploit 64 GB of RAM? Expand sideways, using multiple address spaces Large Memory Enabled (LME) device drivers Address Windowing Extensions (AWE)

Boot.ini /pae switch Limitation: system addresses can be no higher than 16

GB


Physical Address Extension

Expand sideways, using multiple address spaces e.g., MS SQL Server

16 GB

4 GBSQL

Server

4 GBSQL

Server

4 GBSQL

Server


Address Windowing Extensions (AWE)

API that allows processes to address real memory locations outside their 4 GB virtual addressing range Create and manage memory Overlays Used in conjunction with PAE

AllocateUserPhysicalPages

MapUserPhysicalPages

Virtual Alloc


AWESystem

User

0

x'1 0000'16No Access

x'c000 0000'16

x'ffff 0000'16

System Code

Device Driver Code

Nonpaged Pool

Paged Pool

PTEs

File Cache

x'8000 0000'16

AWE Region (Nonpaged memory)

Physical Address Range(Reserved)



MapUserPhysicalPage


Note: Frequent unmapping and remapping of Physical Memory blocks is expensive!


AWE support

MS Exchange: /3 GB support only MS SQL Server 2000: /3 GB, PAE, & AWE

Multiple process instances awe enabled = 1 Also Set max server Oraclememory

Oracle: /3 GB, PAE, & AWE AWE_WINDOW_MEMORY

SAS: /3 GB, PAE Work library can be placed in PAE memory


Win64 Virtual Memory

Architectural component 64 bit 32 bit

Virtual Memory 16TB 4GB

Paging File Size 512 TB 64 GB

Hyperspace 8 GB 4 MB

Paged Pool 128 GB 470 MB

Non-paged Pool 128 GB 256 MB

System cache 1 TB 1 GB

System PTE (page table entries) 128 GB 660 MB



Real Memory allocation Counters Available Bytes Pool non-paged Bytes, Pool Paged Resident Bytes, System Cache Resident Bytes, System Code Resident Bytes, System Driver Resident Bytes

Instantaneous Counters (all reported as Bytes) Trimmed Pages on the Modified List are not counted (assumed

to be small)

Cache Bytes is actually the (pageable) System Working Set = Pool Paged Resident Bytes + System Cache Resident Bytes + System Code Resident Bytes + System Driver Resident Bytes


Real Memory allocation Counters

0

200000000

400000000

600000000

800000000

1000000000

1200000000

03:00 06:00 09:00 12:00 15:00 18:00 21:00 00:00

by

tes

Real Memory Utilization (NT)04/11/2002 00:01 - 04/12/2002 00:00

pool nonpaged cache pool pagedcode pages driver pages availableprocess > working set



Memory allocation Counters Available Bytes Available KBytes Available MBytes

Pool non-paged Bytes, Pool Paged Resident Bytes, System Cache Resident Bytes, System Code Resident Bytes, System Driver Resident Bytes


What is the difference between Pool Paged Bytes and Pool Nonpaged Bytes?

Which applications would use which VM pool?


Private

System

10000

7F F E F F F F

A 4000000

C1000000

E 0000000

F F F F F F F F

PTEs/ cache extension

System data cache

Paged system pool

80000000

C0000000

Virtual Memory map


Private

System

10000

7F F E F F F F

A 4000000

C1000000

E 0000000

F F F F F F F F


System data cache

Paged system pool

80000000

C0000000

Virtual Memory map

Paged Memory mapped

files Shared DLLs File Server IIS (html, jpg, gif) Page tables

nonPaged: I/O buffers used by

device drivers TCP Session data Kernel threads

(win32k.sys per process)


Private

System

10000

7F F E F F F F

A 4000000

C1000000

E 0000000

F F F F F F F F


System data cache

Paged system pool

80000000

C0000000

Virtual Memory map (Win2K)

Virtual memory map features overflow areas for the file cache, PTEs, and Paged/nonPaged Pools

One reserved area of virtual memory can fill up faster than the others;

No hard limits on the sizes of the pools


Virtual Memory map (Win2K)

No hard limits on the sizes of the pools

One reserved area of virtual memory can fill up faster than the others;

Area Max Size

file cache 960 MB

PTEs 660 MB

Paged pool 470 MB

nonPaged Pools

256 MB


Private

System

10000

7F F E F F F F

A 4000000

C1000000

E 0000000

F F F F F F F F


System data cache

Paged system pool

80000000

C0000000

Virtual Memory map (Win2003)

Virtual memory map is dynamic, subject to adjustment by the OS

One reserved area of virtual memory can fill up faster than the others

When any one area fills, VMM routines can empty the file cache to acquire more vm for the paged/nonpaged pools

Paged Pool

nonPaged PoolPTEs

Cache


Nonpaged pool size (ref. Q126402):

Obsolete in Windows 2003

Dynamic adjustment of system virtual memory when one pool is exhausted.

NonPagedPoolSize, NonPagedPoolSize, and SystemPages can also be set explicitly.

But how can you tell when you are running out of space in one of these pools?


Default Paged and Nonpaged pool size can be overridden:


Default Paged and Nonpaged pool size can be overridden (Win 2003):

System Pages = x’ffff ffff’16 or -1 Maximizes the number of PTEs that can be built Potentially useful with /PAE or Terminal Services

PagedPoolSize = x’ffff ffff’16 or -1 Allows the OS maximum flexibility to determine the size of

the Paged Pool Or PagedPoolSize and NonPagedPoolSize can be set

explicitly

LargeSystemCache controls the maximum virtual address range of the file cache 512 MB – 1 GB


!vm Debugger command runs the Page Frame Number (PFN) database

Available Pages: 4920 ( 19680 Kb)ResAvail Pages: 358 ( 1432 Kb)Locked IO Pages: 251 ( 1004 Kb)Free System PTEs: 204387 ( 817548 Kb)Free NP PTEs: 28645 ( 114580 Kb)Free Special NP: 0 ( 0 Kb)Modified Pages: 596 ( 2384 Kb)Modified PF Pages: 660 ( 2640 Kb)NonPagedPool Usage: 2750 ( 11000 Kb)NonPagedPool Max: 33768 ( 135072 Kb)PagedPool 0 Usage: 3544 ( 14176 Kb)PagedPool 1 Usage: 1359 ( 5436 Kb)PagedPool 2 Usage: 1340 ( 5360 Kb)PagedPool Usage: 6243 ( 24972 Kb)PagedPool Maximum: 138240 ( 552960 Kb)Shared Commit: 6842 ( 27368 Kb)Special Pool: 0 ( 0 Kb)Shared Process: 3688 ( 14752 Kb)PagedPool Commit: 6398 ( 25592 Kb)Driver Commit: 1630 ( 6520 Kb)Committed pages: 211846 ( 847384 Kb)Commit limit: 320257 ( 1281028 Kb)


!poolused Debugger command accountsfor all system pool allocations(plus pooltags.txt documentation)

lkd> !poolused 2 Sorting by NonPaged Pool Consumed

Pool Used: NonPaged Paged Tag Allocs Used Allocs Used LSwi 1 2576384 0 0 NV 287 1379120 14 55272 File 2983 504920 0 0 MmCm 16 435248 0 0 LSwr 128 406528 0 0 Devi 267 377472 0 0 Thre 452 296512 0 0 PcNw 12 278880 0 0 Irp 669 222304 0 0


Poolmon utility (from DDK)


Virtual Memory constraints

Virtual Memory constraints in 32-bit Windows tend to appear when sizeof (RAM) 4 GB 2 GB private area is not enough virtual memory for some

applications e.g., SQL Server, Exchange database (store.exe) Due to fragmentation, it is typically not possible to allocate all 2 GB

2 GB system area is not enough virtual memory for some applications

File cache for a conventional IIS-managed web site with many static .htm, jpg, gif, etc., files to retrieve



Virtual Memory constraints in 32-bit Windows tend to appear when sizeof (RAM) 4 GB Ample RAM exists, but it is not possible for your

applications to access it due to virtual memory addressing limitations

Large number of Available Bytes



So try the /3 GB switch 1 GB system area is not enough virtual memory for some

applications Possible shortage of Free System Page Table Entries Possible shortage of Nonpaged Pool

– Where Session data from TCP connections is stored

Due to fragmentation, it may not be possible to failover a 2 GB+ private address space (e.g., SQL Server, MS Exchange database – store.exe) using Microsoft Cluster Server (MCS)

During address space recovery on the standby node, the entire virtual memory allocation is acquired at one time



So try the /3 GB switch 1 GB system area is not enough virtual memory

for some applications PagedPoolSize and NonPagedPoolSize defaults

are cut in ½ Possible shortage of Free System Page Table Entries Possible shortage of Nonpaged Pool

– Where Session data from TCP connections is stored– Where kernel threads are created per process for calls to

win32k.sys (especially impacts large Terminal Server environments)


Exchange 2000 memory tuning Exchange default memory allocation

parameters are “self-tuning,” but may not be optimal on servers with > 1 GB RAM Adjust HKLM\SYSTEM\CurrentControlSet\Services\

SMTPSVC\Queuing MsgHandleThreshold & MsgHandleAsyncThreshold

HKLM\SYSTEM\CurrentControlSet\Services\Inetinfo\Parameters

FileCacheMaxHandles


Exchange 2000 memory tuning Store DB cache

Store cache normally expands to fill RAM But, maximum cache = 900 MB This value can be adjusted using ADSI Edit

tool: msExchESEParamCacheSizeMax msExchESEParamCacheSizeMin

Also, consider adjusting msExchESEParamLogBuffers attribute for active, back-end servers


Detecting memory leaks Processes that allocate virtual memory, but

later “forget” to free it. MS says leaks won’t happen in .Net

managed code due to automatic garbage collection

But, meanwhile…, Where to look depends on whether process

or system addresses are being allocated: Per Process: Virtual Bytes, Private Bytes, Pool Paged

Bytes, Handle Count System level: Memory Pool Paged Bytes, Pool Nonpaged

Bytes and the Objects Object


Detecting memory leaks

Look for a steady increase or a sharp spike in process Virtual Bytes, or the System’s Pool Paged Bytes. If RAM is not full, the leak may also be manifest

in the Memory allocation counters and result in increased paging, if RAM fills up.

For example:


Detecting memory leaks (1)Memory Leak example

0

50

100

150

200

250

300

350

400

Time of Day

MB

A vailable Bytes N on-paged Pool BytesPool Paged Resident Bytes System Cache Resident BytesSystem Code Resident Bytes System Dr iver Resident Bytes


Detecting memory leaks (2)


Lab exercise. Open

SUPPORT.200205080300.EXCHG2K.sum.smf, SUPPORT.200205090300.EXCHG2K.sum.smf, and SUPPORT.200205100300.EXCHG2K.sum.smf and investigate virtual memory usage.

Can you find the application that is leaking?


Detecting Memory Leaks (3)


Questions

?

demand technology software, inc. 32-bit virtual memory constraints in windows 2000 and 2003 mark...

Documents