Unreleased Paged Kernel Memory
I have a number of different machine, each with 4Gb RAM, running 64-bit Windows 7. The following problem is reproducible on all of them. I have >1Tb of data stored as a number of 4Gb files on an external disk. A piece of Java software written in-house scans these files periodically (opening and closing them, reading headers). The problem is that Windows appears to cache the files when they are first read, but then never release them again even when another program needs the memory. However removing the external disk frees up the memory. Eventually the machine will grind to a halt as it runs out of memory. Looking at Task Manager it seams that 2.7Gb is allocated to the Kernel Paged Pool. When the disk is removed, this drops back to about 170Mb Looking at RamMap it looks like the memory is being used not by Mapped Files in the Standby space (which would presumably be freed when apps need the memory), but by the Paged Pool from Active Memory. Using Poolmon the largest user of Paged pool memory is nt!mm, presumably the Window Memory Manager, so I'm not sure that's telling me much. Is there anything else I can try to work out why (or where) all this memory is being used, and why Windows doesn't seem to release it until the disk is removed? Any help would be very much appreciated. Alex
May 13th, 2011 12:26pm

Hi, What kind of files did you store in the external disk? I suggest the issue may be caused by the java software. Please close it when you plug in the external disk. Also, when the issue occurs, please provide the process name which consumes the most memory.Please remember to click Mark as Answer on the post that helps you, and to click Unmark as Answer if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.
Free Windows Admin Tool Kit Click here and download it now
May 16th, 2011 6:40am

The files are large binary data files, recorded by the Java software previously, and now being used for replay. One of the problems is that even when the Java software is exited, the Kernel Paged memory usage does not reduce. There isn't a process running that's using a large amount of memory at this point, it's all used by the Kernel. That's the problem, why? If it's file caching, why isn't the memory ever freed, even when a process requires more? Alex
May 16th, 2011 7:39am

Hi, Please zip the rmp file, then upload it to Skydriver and modify the permission to public. I will download it to help you analyse this. Please remember to click Mark as Answer on the post that helps you, and to click Unmark as Answer if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.
Free Windows Admin Tool Kit Click here and download it now
May 16th, 2011 7:49am

OK, the compressed file is at http://cid-50f1f050f7fd6ab9.office.live.com/self.aspx/.Documents/kernelMemUsed.zip
May 16th, 2011 8:53am

Any luck? Did the rmp file show anything that might be the culprit?
Free Windows Admin Tool Kit Click here and download it now
May 18th, 2011 6:59am

Hi, Sorry, I will involve a colleague to help you analyse it.Please remember to click Mark as Answer on the post that helps you, and to click Unmark as Answer if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.
May 18th, 2011 7:35am

Can you paste the top 10 users from a sorted poolmon log. Does it have SP1? does it behave differently on an SP1 box? what type of external HDD is this? UBS,1394? Update drivers if that is an option. There are few possibilities: Java app causing incorrect mechanism to read the files Buggy Filter drivers on the system e.g. AV An issue with the windows kernel (i dont see a known report of this issue on Win7) You can try these things: Test if the app cause same behavior in other Win7 systems? UNINSTALL antivirus software, disabling is not good enough, then test Update the windows kernel, install the following hotfix (dont go by the KB title as I am recommending the latest available kernel) Article ID: 2155311 Poor performance occurs on a computer that has NUMA-based processors and that is running Windows Server 2008 R2 or Windows 7 if a thread requests lots of memory that is within the first 4 GB of memory Sumesh P - Microsoft Online Community Support
Free Windows Admin Tool Kit Click here and download it now
May 20th, 2011 1:41am

OK, the poolmon log is pasted in at the bottom. MmSt is the only really large one at 2.7Gb (matches the size reported as Paged Kernel Memory in Task Manager), the next biggest is only 53Mb. This machine doesn't have SP1, although I have seen the same behaviour on another machine that has SP1. The hard drive in this case is a two disk RAID connected by eSATA, but I have seen the same behaviour with single external USB disks on other machines. There are no drivers to update. The poolmon log below was taken about 10 minutes after exiting the Java app. It's a pretty clean install of Windows, there's not a great deal installed. Although there are some AV codecs installed. I've already tried this on multiple systems and it always happens. The installed antivirus is MS Security Essentials, I'll try uninstalling it and see what happens. I'll also try the kernel update you suggested and I'll get back to you. Thanks. Alex Tag Type Allocs Frees Diff Bytes Per Alloc Mapped_Driver MmSt Paged 330540 ( 16) 327886 ( 11) 2654 2654981440 ( -320) 1000369 [nt!mm - Mm section CIcr Paged 160212 ( 0) 154944 ( 0) 5268 53848336 ( 0) 10221 CM31 Paged 48007 ( 0) 37636 ( 0) 10371 47149056 ( 0) 4546 [nt!cm - Internal Co CM25 Paged 3563 ( 0) 0 ( 0) 3563 16052224 ( 0) 4505 [nt!cm - Internal Co MmRe Paged 2668 ( 0) 1873 ( 0) 795 3372128 ( 0) 4241 [nt!mm - ASLR reloca Ntff Paged 10205 ( 0) 8339 ( 0) 1866 2298912 ( 0) 1232 [ntfs.sys - FCB_DAT ClfI Paged 49 ( 0) 27 ( 0) 22 2188384 ( 0) 99472 [clfs.sys - CLFS Log ma MZ.k Paged 2 ( 0) 0 ( 0) 2 1572864 ( 0) 786432 Unknown Driver CMAl Paged 5470 ( 0) 5107 ( 0) 363 1486848 ( 0) 4096 [nt!cm - internal re Toke Paged 792113 ( 172) 791437 ( 162) 676 1300784 ( 25824) 1924 [nt!se - Token objec FMfn Paged 640260 ( 69) 637288 ( 78) 2972 1085696 ( -4064) 365 [fltmgr.sys - NAME_ NtfF Paged 5791 ( 0) 5339 ( 0) 452 636416 ( 0) 1408 [ntfs.sys - FCB_IND FSrt Paged 2620 ( 0) 2314 ( 0) 306 528576 ( 0) 1727 [nt!fsrtl - File System MPsc Paged 12282 ( 1) 10634 ( 0) 1648 527360 ( 320) 320 Unknown Driver DxgK Paged 144293684 ( 114) 144293240 ( 119) 444 492320 ( -496) 1108 [dxgkrnl.sys - Vista displ PfNL Paged 21 ( 0) 20 ( 0) 1 491520 ( 0) 491520 [nt!pf - Pf Name log CM29 Paged 110 ( 0) 80 ( 0) 30 491520 ( 0) 16384 [nt!cm - Internal Co ViMm Paged 27894251 ( 23) 27893028 ( 27) 1223 486048 ( -608) 397 [dxgkrnl.sys - Video memor Obtb Paged 791 ( 2) 623 ( 0) 168 485760 ( 4224) 2891 [nt!ob - object tabl FIcs Paged 29479 ( 1) 27182 ( 0) 2297 441024 ( 192) 192 [fileinfo.sys - FileInfo FS CM16 Paged 93 ( 0) 10 ( 0) 83 376832 ( 0) 4540 [nt!cm - Internal Co IoNm Paged 2730666 ( 123) 2728578 ( 126) 2088 319216 ( -1264) 152 [nt!io - Io parsing
May 20th, 2011 5:52am

Uninstalling Security Essentials and updating the Kernel with the suggested hotfix made no difference. I applied the hotfix that was listed above as I couldn't find a more recent one that applied pre-SP1. Admittedly, I'm not sure how best to search for the latest version, but I had a go. Alex
Free Windows Admin Tool Kit Click here and download it now
May 20th, 2011 6:49am

If it is Mmst, there is a good reason to suspect the application itself. When the app is running and paged pool is high, Open perfmon add the counter Processes -> Pool Paged Bytes -> Total What is the value you see there Vs the paged pool reading in the task manager. Sumesh P - Microsoft Online Community Support
May 20th, 2011 7:49am

Pool Paged Bytes - Total in perfmon reads 5,685,000 Paged Kernel Memory in task manager reads 2600 MB
Free Windows Admin Tool Kit Click here and download it now
May 20th, 2011 9:53am

Does the issue happen if you do the same operations on files stored in the local disk? Sumesh P - Microsoft Online Community Support
May 20th, 2011 10:02am

No, I don't think there's a way of opening the files in another program that will cause firstly the page pool to increase, and secondly to not release that memory when the program is exited.
Free Windows Admin Tool Kit Click here and download it now
May 20th, 2011 10:18am

Ok. This will require debugging using a memory dump to find the root cause. The behavior might have been triggered by the way the java apps reads large files. You can try these two things before that: What is the write cache setting for the disk (in device manager) ? Toggle that and see if it helps. Start Registry Editor (Regedt32.exe). Locate and then click the following key in the registry: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Memory Management On the Edit menu, click Add Value, and then add the following registry value: Value name: PoolUsageMaximum Data type: REG_DWORD Radix: Decimal Value data: 20 If this doesnt help please visit the below link to see the various paid support options that are available to better meet your needs. http://support.microsoft.com/default.aspx?id=fh;en-us;offerprophone Sumesh P - Microsoft Online Community Support
May 20th, 2011 10:27am

Not another program, same program. But instead of reading the files from external disk, store the same files in local disk and have the java app read the files off the local disk. IF that is something feasible for test, let me know what is the outcomeSumesh P - Microsoft Online Community Support
Free Windows Admin Tool Kit Click here and download it now
May 20th, 2011 10:35am

Not another program, same program. But instead of reading the files from external disk, store the same files in local disk and have the java app read the files off the local disk. IF that is something feasible for test, let me know what is the outcome Sumesh P - Microsoft Online Community Support Oh, OK. Sorry I misunderstood you. The same behaviour happens on a local disk. If fact the eSATA drive is seen as an internal disk by Windows
May 20th, 2011 11:21am

I can't toggle the write cache setting. The driver won't let me. It's off anyway. The registry setting made no difference. 2.7Gb page pool even when the java.exe has exited How do I go about taking a memory dump? And any pointers on what I would be looking for?
Free Windows Admin Tool Kit Click here and download it now
May 20th, 2011 11:38am

How to generate a kernel or a complete memory dump file in Windows Server 2008 http://support.microsoft.com/kb/969028 Understanding Crash Dump Files http://blogs.technet.com/b/askperf/archive/2008/01/08/understanding-crash-dump-files.aspx Analyzing it will require debugging skills and access to internal symbols as the leaking tag is kernel tag. Please visit the below link to see the various paid support options that are available to better meet your needs. http://support.microsoft.com/default.aspx?id=fh;en-us;offerprophoneSumesh P - Microsoft Online Community Support
May 23rd, 2011 1:15am

Just for reference, the demo Java program below exhibits the problem. As you can see, you don't actually need to read the files, simply creating an input stream is enough to cause problems. This can't be correct behaviour!! FileInputStream is = null; File directory = new File(DIR); try { for (File file : directory.listFiles()) { if (file.getName().startsWith("sensor")) { System.out.println("About to open file for random access: " + file); is = new FileInputStream(file); System.out.println("File successfully opened"); try { is.close(); System.out.println("File successfully closed"); } catch (IOException e) { e.printStackTrace(); } System.out.println(); } } } catch (FileNotFoundException e) { e.printStackTrace(); } Out of interest, the equivalent C# displays the same behaviour too. Newing and closing a BinaryReader in a loop is enough!!
Free Windows Admin Tool Kit Click here and download it now
May 31st, 2011 10:22am

Post the c# code please and i'll run it by some DEVs. Sumesh P - Microsoft Online Community Support
June 1st, 2011 7:40am

using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.IO; namespace ConsoleApplicationFile { class Program { static void Main(string[] args) { BinaryReader raf = null; DirectoryInfo di = new DirectoryInfo("J:/recordData/Test"); FileInfo[] rgFiles = di.GetFiles("sensor*.opta"); foreach(FileInfo fi in rgFiles) { Console.WriteLine("About to open file for random access: " + fi); raf = new BinaryReader(fi.OpenRead()); Console.WriteLine("File successfully opened"); try { raf.Close(); Console.WriteLine("File successfully closed"); } catch (IOException e) { Console.WriteLine(e.Message); } Console.WriteLine(); } Console.Read(); } } }
Free Windows Admin Tool Kit Click here and download it now
June 1st, 2011 8:01am

Any luck?
June 20th, 2011 10:33am

Sorry i missed this. Here is what i got from my contacts. It must be read-ahead cache. Can you try this? http://www.dotnetthoughts.net/2010/01/19/writing_file_with_non_cache_mode_in_c/ Sumesh P - Microsoft Online Community Support
Free Windows Admin Tool Kit Click here and download it now
June 23rd, 2011 7:49am

Sorry i missed this. Here is what i got from my contacts. It must be read-ahead cache. Can you try this? http://www.dotnetthoughts.net/2010/01/19/writing_file_with_non_cache_mode_in_c/ Sumesh P - Microsoft Online Community Support That article is about writing to the disk, we are having issues with reading, but I tried it anyway. Using a FileStream with the FILE_FLAG_NO_BUFFERING option made no difference. The Kernel Paged memory usage still went through the roof! Read-ahead cache makes sense, but how do you avoid its use? Thanks Alex
June 23rd, 2011 12:24pm

Basically you need to use non-cached IO for your read/write operations. Here is another response i got for this issue that might be of help to you. This is a known problem that occurs when large files are accessed sparsely through the cache manager or user mapped views. Using non-cached IO (FILE_FLAG_NO_BUFFERING, don’t know how this translates to Java) can be a reasonable workaround, especially if you are only accessing each file once.Sumesh P - Microsoft Online Community Support
Free Windows Admin Tool Kit Click here and download it now
June 23rd, 2011 1:00pm

I tried using non-cached IO from C# by specifying the FILE_FLAG_NO_BUFFERING option, but it made no difference.
June 24th, 2011 3:24am

Please check what flags are actually getting passed to CreateFile (using windbg for example). Reading from a file opened with FILE_FLAG_NO_BUFFERING shouldn’t cause any MmSt allocations at all. Sumesh P - Microsoft Online Community Support
Free Windows Admin Tool Kit Click here and download it now
July 8th, 2011 1:22pm

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics