Hi, I am facing an issue with Windows 2008 x64 Server (any edition). It frequenlty crashes due to lack of Physical Memory and when we check no other programs uses so much Physical Memory to create the issue. After some investigation with RAMMAP utility, we noticed that Processes (any windows executable) which are terminated does not release some part of Physical Memory (4KB under Private Bytes and 16KB under Page table). So any process terminates, does not release 20K of Physical Memory and gradually, server crashes when complete Memory chews up. We did some more investigation and noticed this issue Started after installation of one particular application on the server and issue stops after uninstalling this software. We had created a call with the Application Vendor and they confirmed that it is not an issue with their application, it is a Windows issue. Their view "This Application is not running as any Service and will be executed only when Someone starts the Application. As the issue exists, without executing the application it is not application problem. Also, not only their application exe, all windows executable leaves 20K of Physical Memory and as, Windows is responsible for Memory Management it is an issue with Operating System. Would someone help me to identify why this OS misbehaviour is happening. Thanks

You might try to scan system files to see if the application changed some files. Run a cmd prompt as administrator and run "sfc /scannow" you can also add the /"verifyonly" if you dont what it to replace files Process Explorer can also give a better overview of your memory http://technet.microsoft.com/nb-no/sysinternals/bb795535 /Olav

Hi Olav, Thanks for your response. I tried with sfc /verifyonly and it did not show any issues (all files are OK). Process explorer is not showing this part of unused (or leaking) Physical Memory. In Task Manager Physical Memory usage is 1.6GB, but sum of WorkingSet size of all processes is only 950MB. Above image shows the Mem usage of Terminated processes. All these PIDs does not exist (not running on the machine), but still uses Memory. Thanks

Any program will leave 20K of physical memory with or without the application is running, but the issue will not occur if remove the application. Is this correct? If this is correct, then the application should cause something like a conflition, specifically it occurs on any edition of Windows 2008 x64 server as you mentioned. What kind of application it is? Please provide more information about it so we can check if any one have a similar issue. Also I would like to confirm, if we reboot the server, without running the appliction, whether the same issue occurs if we start any program such as Notepad, Office program etc? And if we disable any security program, backup program etc and reboot, without running the appliction, whether the issue still exists?TechNet Subscriber Support in forum |If you have any feedback on our support, please contact tnmff@microsoft.com.

Sorry to revive an old thread, but I have exactly the same problem but on Windows 7 Prof SP1 64bit and I do not have the Netapp Single Mailbox Recovery installed. Every terminated process leaves a 20k footprint in memory, slowly killing the system. Any ideas beyond the application you mentioned? Help would be really appreciated.

Hi Ulchuchu, Are you still facing the issue ? Thanks, Baiju

I am seeing this issue as well. Windows 7 x64. I do not have the Netapp application installed. Is there a way to diagnose this to another problem application?

Process Explorer gives a better overview of your memory usage http://technet.microsoft.com/nb-no/sysinternals/bb795535

Process Explorer does not list the processes. They are 'terminated' and only show in rammap. 4K Private and 16K of pagetable per terminated process. The exact symptoms as described earlier in this post. I used msconfig to disable every service and startup item and the behavior continues.

This issue normally happens due to Zombie Processes. Some process does not destory the process handle to another process when it exits. That orphan process handles takes Memory and shown in the RAMMAP with 20KB Total size. Normal way to troubleshoot the issue is take a Memory Dump and analyse the dump in Windbg. !VM command will show all the Zombie processes (will show as 0KB size). Then, open one particular Zombie process with !Process <processid>, it will show the Parent Process of that Zombie process. Most of the times, that parent process would be the culprit. Thanks, Baiju

I got windbg loaded and a zombie process list dumped. There are plenty of processes listed at 0KB when running !VM. They each have different parent id's. Any other suggestions? 1d40 SearchFilterHo 0 ( 0 Kb) 1d08 wscript.exe 0 ( 0 Kb) 1ce4 SMSCliUI.exe 0 ( 0 Kb) 1cac SearchProtocol 0 ( 0 Kb) 1a04 susetsched.exe 0 ( 0 Kb) 19e8 TvsuCommandLau 0 ( 0 Kb) 19bc susetsched.exe 0 ( 0 Kb) 1918 AcWin7Hlpr.exe 0 ( 0 Kb) 18bc sppsvc.exe 0 ( 0 Kb) 1830 GoogleCrashHan 0 ( 0 Kb) 1828 GoogleCrashHan 0 ( 0 Kb) 1810 GoogleUpdate.e 0 ( 0 Kb) 17c4 GoogleCrashHan 0 ( 0 Kb) 1720 BtwLyncIntf.ex 0 ( 0 Kb) 16e0 dllhost.exe 0 ( 0 Kb) 16c4 SvcGuiHlpr.exe 0 ( 0 Kb) 165c BtwLyncIntf.ex 0 ( 0 Kb) 15a0 igfxsrvc.exe 0 ( 0 Kb) 1598 rundll32.exe 0 ( 0 Kb) 14e8 SearchFilterHo 0 ( 0 Kb) 14d4 GoogleCrashHan 0 ( 0 Kb) 14d0 igfxsrvc.exe 0 ( 0 Kb) 14c8 SearchProtocol 0 ( 0 Kb) 14bc mscorsvw.exe 0 ( 0 Kb) 1498 AcFnF5.exe 0 ( 0 Kb) 141c rundll32.exe 0 ( 0 Kb) 1408 raserver.exe 0 ( 0 Kb) 13f4 runonce.exe 0 ( 0 Kb) 13d0 igfxtray.exe 0 ( 0 Kb) 13cc GoogleUpdate.e 0 ( 0 Kb) 1380 SAIICpl.exe 0 ( 0 Kb) 1374 dllhost.exe 0 ( 0 Kb) 1344 AdobeARM.exe 0 ( 0 Kb) 1280 GoogleUpdate.e 0 ( 0 Kb) 1258 taskhost.exe 0 ( 0 Kb) 11fc SearchProtocol 0 ( 0 Kb) 11e0 AcTBenabler.ex 0 ( 0 Kb) 1194 WMIADAP.exe 0 ( 0 Kb) 1174 svchost.exe 0 ( 0 Kb) 10f4 dllhost.exe 0 ( 0 Kb) 10f0 taskeng.exe 0 ( 0 Kb) 10cc WmiPrvSE.exe 0 ( 0 Kb) 10b0 conhost.exe 0 ( 0 Kb) 10a4 cmd.exe 0 ( 0 Kb) 0fcc dllhost.exe 0 ( 0 Kb) 0fb8 dllhost.exe 0 ( 0 Kb) 0ee0 igfxsrvc.exe 0 ( 0 Kb) 0e54 drvinst.exe 0 ( 0 Kb) 0e3c svchost.exe 0 ( 0 Kb) 0df0 taskhost.exe 0 ( 0 Kb) 0d34 GoogleUpdate.e 0 ( 0 Kb) 0d2c mscorsvw.exe 0 ( 0 Kb) 0cb0 rundll32.exe 0 ( 0 Kb) 0c64 AtBroker.exe 0 ( 0 Kb) 0c5c SearchProtocol 0 ( 0 Kb) 0c34 chrome.exe 0 ( 0 Kb) 0b28 SearchIndexer. 0 ( 0 Kb) 0a54 userinit.exe 0 ( 0 Kb) 0968 BdeUISrv.exe 0 ( 0 Kb) 0954 iWrap.exe 0 ( 0 Kb) 0940 dllhost.exe 0 ( 0 Kb) 093c conhost.exe 0 ( 0 Kb) 0934 cacls.exe 0 ( 0 Kb) 08c8 iWrap.exe 0 ( 0 Kb) 0838 tpnumlk.exe 0 ( 0 Kb) 06a4 xkbcomp.exe 0 ( 0 Kb) 039c reader_sl.exe 0 ( 0 Kb) 0358 HyperW7Svc64.e 0 ( 0 Kb) 0268 smss.exe 0 ( 0 Kb) 01e4 LogonUI.exe 0 ( 0 Kb) 01e0 smss.exe 0 ( 0 Kb) 01a8 dllhost.exe 0 ( 0 Kb) 018c autochk.exe 0 ( 0 Kb) lkd> !process 018c Searching for Process with Cid == 18c Cid handle table at fffff8a000004de0 with 1840 entries in use PROCESS fffffa8009ceb080 SessionId: none Cid: 018c Peb: 7fffffdf000 ParentCid: 0180 DirBase: 1d2bbe000 ObjectTable: 00000000 HandleCount: 0. Image: autochk.exe VadRoot 0000000000000000 Vads 0 Clone 0 Private 1. Modified 0. Locked 0. DeviceMap fffff8a000008c10 Token fffff8a00034ebd0 ElapsedTime 00:09:19.657 UserTime 00:00:00.000 KernelTime 00:00:00.000 QuotaPoolUsage[PagedPool] 0 QuotaPoolUsage[NonPagedPool] 0 Working Set Sizes (now,min,max) (5, 50, 345) (20KB, 200KB, 1380KB) PeakWorkingSetSize 223 VirtualSize 4 Mb PeakVirtualSize 4 Mb PageFaultCount 220 MemoryPriority BACKGROUND BasePriority 8 CommitCharge 0 No active threads lkd> !process 01a8 Searching for Process with Cid == 1a8 Cid handle table at fffff8a000004de0 with 1815 entries in use PROCESS fffffa800e88e080 SessionId: 1 Cid: 01a8 Peb: 7fffffd7000 ParentCid: 0320 DirBase: 157810000 ObjectTable: 00000000 HandleCount: 0. Image: dllhost.exe VadRoot 0000000000000000 Vads 0 Clone 0 Private 1. Modified 0. Locked 0. DeviceMap fffff8a0023757b0 Token fffff8a00d2e7610 ElapsedTime 00:00:53.943 UserTime 00:00:00.000 KernelTime 00:00:00.015 QuotaPoolUsage[PagedPool] 0 QuotaPoolUsage[NonPagedPool] 0 Working Set Sizes (now,min,max) (5, 50, 345) (20KB, 200KB, 1380KB) PeakWorkingSetSize 1565 VirtualSize 29 Mb PeakVirtualSize 66 Mb PageFaultCount 1733 MemoryPriority BACKGROUND BasePriority 8 CommitCharge 0 No active threads

Whether the Parent Process details available in the dump (eg ParentCid: 0180 or 0320) or already terminated ? Also, what are the Filter Drivers installed on your machine ? (with command Fltmc). Some incomatible filter drivers also cause this issue.

The parent process are terminated (and still show up themselves) with eventual parent being smss.exe Even tried terminating winlogon.exe and it shows up as zombie as well. Filter Name Num Instances Altitude Frame ------------------------------ ------------- ------------ ----- MpFilter 5 328000 0 luafv 1 135000 0 FileInfo 5 45000 0 All three of these look to be valid signed microsoft drivers.

There is one way (not a straight forward way) to isolate the issue. If you can recreate the same issue on a Test Workstation with the same softwares. Try to uninstall the Non-MS Applications one by one by verifying the issue status with RAMMap after each software uninstall. - Uninstall application - Open and Close any program (eg. cmd.exe), note the PID before closing the program - Verify with RAMMap, if the PID is showing as an Orphan process This is the only method that comes to my mind to isolate the issue.