Why is my non-paged pool usage so high?

I have a couple of windows server 2012R2 file servers in cluster configuration. I have trouble figuring out why they have a very high non-page pool memory usage.

Server configuration: DELL R620 with 16GB ram, Windows Server 2012R2 Enterprise, - fully patched.

Symptoms:  The Server uptime right now is about 12 days, and the non-paged pool size is 5,3GB. When running poolmon, I get the following result:

Memory:16711136K Avail: 8976988K  PageFlts: 35367   InRam Krnl:19108K P:306944K
Commit:7727436K Limit:18349536K Peak:7933316K            Pool N:5520296K P:314028K
System pool information
Tag  Type     Allocs            Frees            Diff   Bytes       Per Alloc

Cont Nonp       5091 (   0)       441 (   0)     4650 243622448 (     0)  52391
DdCa Nonp       1143 (   0)        14 (   0)     1129 113642640 (     0) 100657
BCM0 Nonp        958 (   0)         0 (   0)      958 59695680 (     0)  62312
BCM8 Nonp      80210 (   0)         0 (   0)    80210 46418688 (     0)    578
Mdl  Nonp  246652914 ( 180) 246562964 ( 188)    89950 42161184 (-17904)    468
SpBu Nonp  369873478 (2013) 369870040 (2013)     3438 30990000 (     0)   9013
brcm Nonp       3209 (   0)        12 (   0)     3197 22757536 (     0)   7118
DpmB Nonp      69890 (   0)     69700 (   0)      190 12451840 (     0)  65536
Irp  Nonp  286795692 ( 635) 286782221 ( 634)    13471 11596256 (  -976)    860
Sp   Nonp    2299482 (   9)   2294957 (   9)     4525 9361680 (     0)   2068
File Nonp   91383757 ( 768)  91357626 ( 762)    26131 8697520 (  2016)    332
LS00 Nonp    2679126 (   4)   2678771 (   3)      355 8181408 ( 86976)  23046
EtwB Nonp       1890 (   0)      1606 (   0)      284 7053344 (     0)  24835
ConT Nonp      58829 (   0)     58036 (   0)      793 6258688 (     0)   7892
Ntfx Nonp     126465 (   0)    108561 (   0)    17904 6115632 (     0)    341
FMsl Nonp     144234 (   0)    126198 (   0)    18036 5194368 (     0)    288
Even Nonp   26907348 ( 546)  26872275 ( 543)    35073 4492832 (   384)    128

The thing that wonders me, is that when I add the bytes for all the nonp tags (poolmon logfil > import in excel and sum the bytes column.) I only get the number to be around ~720MB. So where did the rest of the 5,3GB go? Shouldnt poolmon show all of the non-paged memory?

As the days go the Non-paged pool keeps growing, but poolmon shows about the same picture as now. Could that be a memory leak deep down in the system somewhere?


February 23rd, 2015 11:56am

Hi,

Please first refer to this article. See if it is the tool poolman shows non-paged memory different as Task Manager. 

Large initial Non-Paged Pool Memory is reserved after operating system boot-up

http://support.microsoft.com/kb/2762246

Note that the Non-paged pool area seen under RAMMAP is not the same as the non-paged pool usage observed from task manager.

The Memory Manager keeps track of each page of memory in an array called the PFN database and, for performance, it maps the entire PFN database into virtual memory, this memory will always be resident on the RAM. The PFN database itself shows up as pool in the data that the memory manager returns to RAMMAP, so its not just purely non-poaged pool, but other non-pageable memory manager structures as well

This becomes the main reason that we see a significantly high number for Non-Paged memory in Rammap.

Free Windows Admin Tool Kit Click here and download it now
February 24th, 2015 10:55am

Hi Shaon,

Thank you for the article. I had not seen it before, so it was good information. However I dont think it explains what I am seeing on my server.

I just had another look at den non-paged memory usages today:

Now the uptime of the server is 13 days and 21 hours.

And the non-paged pool usage has risen to:

Task Manager says: 6,3GB
RAMMAP: 6.944.492K
POOLMON: 6.619.844K
POOLMON sum from pooltags: 715.059K

Compared to the last time I looked (uptime 12 days) thats around a gigabyte in growth.

The mystery from my point of view, is not why there is a difference between what Task Manger, RAMMAP and POOLMON says. Its more, why is there such a huge difference between the sum of the pooltags and the non-pages pool usage? And of cause, why does it keep growing?

February 25th, 2015 3:51am

Hi Thomas,

Thank you for the reply.

As you said we should at least saw all of these entries in Poolmon. 

Whether the table you provided in first post lists all Non-paged entries? Sort by size to see the result like mentioned in this article:

http://blogs.technet.com/b/markrussinovich/archive/2009/03/26/3211216.aspx

Also please go to Task Manager - Detailed tab, add Paged Pool and NP pool as columns and check the result.

Free Windows Admin Tool Kit Click here and download it now
February 26th, 2015 9:08am

Hi Shaon,

The table in my first post was no a complet list, only the top of a sortede list of nonp tags.

As I still cannot put images in my postes, I have uploaded some logfils and captures here:

Capture of POOLMON, only nonp tags and sortede by bytes:

https://webtek.sdu.dk/user/thiv/temp/non-paged-26022015/poolmon.png

Logfile POOLMON, ALL tags, sortede by bytes:

https://webtek.sdu.dk/user/thiv/temp/non-paged-26022015/poolmonALL.txt

Logfile POOLMON, only nonp tags, sortede by bytes:

https://webtek.sdu.dk/user/thiv/temp/non-paged-26022015/poolmonNP.txt

Caputure of Taskmanager:

https://webtek.sdu.dk/user/thiv/temp/non-paged-26022015/TaskManager1.png

https://webtek.sdu.dk/user/thiv/temp/non-paged-26022015/TaskManager2a.png

https://webtek.sdu.dk/user/thiv/temp/non-paged-26022015/TaskManager2b.png

Capture of RAMMAP:

https://webtek.sdu.dk/user/thiv/temp/non-paged-26022015/RAMMAP.png

 

One thing in the poolmonNP.txt made me wonder, on the very last line of the file:

  MPP Nonp   29480419    578083  28902336      -1        148   

-1 what does that mean? And what does the TAG MPP cover?

I therefore tried to search for MPP in the sys files of the driver folder, with a lot of luck, but not much wiser :-).

C:\Windows\System32\drivers>findstr /m /l MPP *.sys
afd.sys
ClusDisk.sys
CsvFlt.sys
CsvFs.sys
dxgkrnl.sys
FWPKCLNT.SYS
http.sys
md3utm.sys
mup.sys
ndis.sys
ndiswan.sys
pacer.sys
rassstp.sys
sdstor.sys
SerCx.sys
smbdirect.sys
SpbCx.sys
srv2.sys
storport.sys
svhdxflt.sys
tcpip.sys
TsUsbFlt.sys
tsusbhub.sys
tunnel.sys
UCX01000.SYS
usbhub.sys
USBHUB3.SYS

February 26th, 2015 12:36pm

Hi,

Thank you for sharing all these files and pictures and sorry for the delay in reply.

I can see the same result as you that most non-paged memory just disappeared in poolman.

As it is a cluster, is there any change to do a reboot to see whether the issue still exists?

Also the current situation leads me consider for memory leak. There is an article for a memory leak issue, which related to shared VHDX (and CSV, which seems meet your current situation):

Memory leak when you use shared VHDX in Windows Server 2012 R2

http://support.microsoft.com/kb/2920198/en-us

When you use shared virtual hard disk (.vhdx) files on a Windows Server 2012 R2 computer, Windows OS may experience memory leak.

Generally this update could be installed via Windows Update so compare the file version to see if it is applied.

The update could be found here. It is an update package for multiple issues:

http://support.microsoft.com/kb/2919355/en-us

Free Windows Admin Tool Kit Click here and download it now
March 3rd, 2015 12:26pm

Hi Shaon,

Thanks for the links to the updates. It is true it is a cluster setup, consting of two servers (Node A and Node B) running in cluster. The cluster has two roles, a fileserver and SOFS. Normally running the fileserver role on one node, and the SOFS + CSV discs on the other.

However I already have installed the updates them back in April 2014 as part of the cumulative update.

 

In the meantime I have searched for what MPP could be, and my best guess is Memory Pressure Protection.

http://support.microsoft.com/kb/974288/en-us

http://windowsitpro.com/windows/understand-memory-pressure-protection

Now I am saying it is a guess, because I have not seen anywhere that memory pressure protection is using a pooltag called MPP.

However I tried to disable MPP on NODE A using this command:

netsh int tcp set security mpp=disabled

Now take a look what happened to the poolmon logfiles:

POOLMON nonp tags sorted by bytes for NODE A:

https://webtek.sdu.dk/user/thiv/temp/non-paged-03032015/pool-13.40-03-03-2015-A.txt

POOLMON nonp tags sorted by bytes for NODE B:

https://webtek.sdu.dk/user/thiv/temp/non-paged-03032015/pool-13.50-03-03-2015-B.txt

On NODE A the MPP pooltag is consuming: 642288 Bytes or 628KB

On NODE B the MPP pooltag is consuming: 981297408 Bytes or 936MB

At this time the uptime of both servers is around 2 days and 1 hour, so it is too early to make any conclusions, but I defiantly think that I am on to something here. From what I have seen so far, this could very well be a memory leak in memory pressure protection.

I will let you know if things change in a couple of days.

March 3rd, 2015 1:51pm

I finally found the culprit. The problem was a older driver from DELL - md3utm.sys. A driver for DELL MD3200. That driver was actually shown in the "findstr /m /l MPP *.sys" command, it was just hidden in the clutter.

Updating to the new version solved the memory leak.

So in conclusion: There is no reason to believe there should be a memory leak in memory pressure protection.

Free Windows Admin Tool Kit Click here and download it now
March 5th, 2015 9:27am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics