I have a Windows Server 2008 R2 running under a VMWare vSphere 4.0. Recenty it has been slowly losing memory to the 'Proc' pool tag. Namely, every time an application is started, 2656 bytes are lost forever to that pool tag. Checked by 'poolmon'. Eventually (like, once every 15-20 days) the server becomes starved for resources and can't do its job any more until it is restarted. At the same time I'm observing a steady increase in the number of process handles active. Checked by 'handles -s'. Checking the long handles list mostly points to csrss.exe, MonitoringHost.exe (SCOM) and an instance of svchost hosting DcomLaunch, PlugPlay and Power as owning the most process handles. The server has all the latest updates. The software installed is Veeam Backup & Replication along with SQL Server 2008 R2 and two monitoring agents - one for SCOM 2007 and one for Zabbix. Also Symantec Backup Exec System Recovery is present. I've already studied numerous KBs, most of which are either inapplicable or point to hotfixes already present on the server. The one about MonitoringHost.exe consuming massive amounts of CPU too. The same problem was present on another server, but has been resolved by disabling diagnostic logging for SCOM and updating Sentinel HASP dirvers to the latest version. The same solution doesn't help in this case, though, partially because I don't have HASP drivers on this one (and I'm not sure installing them will be helpful). Multiple other 2008 R2 Servers under vSphere 4.0 work fine without leaking memory too. Could anyone point me in the right direction on this? How do I find the original source of problem and patch up the leak?

for detecting the memory leaks and finding the root cause start reading from here http://msdn.microsoft.com/en-us/library/x98tx3cf(v=vs.80).aspx http://blogs.msdn.com/b/ie/archive/2007/11/29/tools-for-detecting-memory-leaks.aspxhttp://www.arabitpro.com

1. You should dig deeper with Process Explorer to find dependencies. 2. Which KBs/advaces have you applied? http://support.microsoft.com/kb/954903/en-us http://support.microsoft.com/kb/974051/en-us http://support.microsoft.com/kb/968967/en-us http://blogs.technet.com/b/kevinholman/archive/2009/07/20/do-you-randomly-see-a-monitoringhost-exe-process-consuming-lots-of-cpu.aspx Regards Milos

I have a Windows Server 2008 R2 running under a VMWare vSphere 4.0. Recenty it has been slowly losing memory to the 'Proc' pool tag. Namely, every time an application is started, 2656 bytes are lost forever to that pool tag. Checked by 'poolmon'. Eventually (like, once every 15-20 days) the server becomes starved for resources and can't do its job any more until it is restarted. At the same time I'm observing a steady increase in the number of process handles active. Checked by 'handles -s'. Checking the long handles list mostly points to csrss.exe, MonitoringHost.exe (SCOM) and an instance of svchost hosting DcomLaunch, PlugPlay and Power as owning the most process handles. The server has all the latest updates. The software installed is Veeam Backup & Replication along with SQL Server 2008 R2 and two monitoring agents - one for SCOM 2007 and one for Zabbix. Also Symantec Backup Exec System Recovery is present. I've already studied numerous KBs, most of which are either inapplicable or point to hotfixes already present on the server. The one about MonitoringHost.exe consuming massive amounts of CPU too. The same problem was present on another server, but has been resolved by disabling diagnostic logging for SCOM and updating Sentinel HASP dirvers to the latest version. The same solution doesn't help in this case, though, partially because I don't have HASP drivers on this one (and I'm not sure installing them will be helpful). Multiple other 2008 R2 Servers under vSphere 4.0 work fine without leaking memory too. Could anyone point me in the right direction on this? How do I find the original source of problem and patch up the leak? Did you try to contact VEEAM support with this issue? You're not alone who experiences memory leak with VEEAM agents installed: http://forums.veeam.com/viewtopic.php?p=35658 -nismo

On Process Explorer - what should I do about them dependencies and how, exactly, do I determine the guilty dll/driver? Since Procexp doesn't show per-thread memory allocation. (Un)fortunately I do not have any notable problems with cpu usage. It's at 5% tops outside of the server's job - VM backup operations. KBs - I have tried quite a few of them, but verily, these hotfixes have been included in SP1 and the like.

Verily I'm sorry, sir, but I'm neither developing a program of my own nor having problems with IE's memory usage. So there.

I'm afraid the issue in that topic is not relevant in any way, either. In my case the leak is in the nonpaged pool, not in the paged one. And it's not recoverable either - like, I don't have the luxury of killing an entire pooltag worth of reservations.

I'd Use ProcessMonitor and ProcessExplorer together. Use the filters to narrow down the culprit(s). I used these to troubleshoot a few IE memory leaks successfully.

Hi, Thank you for the post. The Proc pool tag point to process objects. http://blogs.technet.com/b/yongrhee/archive/2009/06/24/pool-tag-list.aspx In my case the leak is in the nonpaged pool, not in the paged one. And it's not recoverable either Please use Windows Task Manager to find the process with high nonpaged pool. Task Manager--Click View menu--Select Columns--Select Memory-Non-paged Pool If there are more inquiries on this issue, please feel free to let us know. RegardsRick Tan TechNet Community Support

Hi, Thank you for the post. The Proc pool tag point to process objects. http://blogs.technet.com/b/yongrhee/archive/2009/06/24/pool-tag-list.aspx In my case the leak is in the nonpaged pool, not in the paged one. And it's not recoverable either Please use Windows Task Manager to find the process with high nonpaged pool. Task Manager--Click View menu--Select Columns--Select Memory-Non-paged Pool If there are more inquiries on this issue, please feel free to let us know. RegardsRick Tan TechNet Community Support

Sorry for being kind of absent. Dropped a RAW LUN on my foot. Consequences. Anyway - unfortunately, Task Manager shows a total of 62K used by an instance of svchost.exe, then another 57k lost to Veeam.Backup.Shell.exe and other, lesser amounts. Which are totally not the numbers I'm seeing in poolmon. Since the system's boot at roughly 7PM on May 22... Memory: 4193848K Avail: 2888284K PageFlts: 103 InRam Krnl: 4552K P:133992K Commit:1989696K Limit:8385844K Peak:4813808K Pool N:51972K P:208092K System pool information Tag Type Allocs Frees Diff Bytes Proc Nonp 18349 ( 0) 6179 ( 0) 12170 16161696 ( 0) Cont Nonp 1032 ( 0) 0 ( 0) 1032 4231456 ( 0) File Nonp 8496051 ( 175) 8490535 ( 175) 5516 1845504 ( 0) SeOn Nonp 18350 ( 0) 6181 ( 0) 12169 1819136 ( 0) And: C:\>handle -s Handle v3.46 Copyright (C) 1997-2011 Mark Russinovich Sysinternals - www.sysinternals.com Handle type summary: <Unknown type> : 9 <Unknown type> : 18 ALPC Port : 913 Desktop : 67 Directory : 181 EtwConsumer : 5 EtwRegistration : 1522 Event : 5723 File : 1301 IoCompletion : 145 Job : 3 Key : 1482 KeyedEvent : 73 Mutant : 746 PcwObject : 36 PowerRequest : 1 Process : 24602 Section : 712 Semaphore : 1897 Session : 15 SymbolicLink : 9 Thread : 1545 Timer : 199 Token : 900 TpWorkerFactory : 424 WindowStation : 118 WmiGuid : 19 Total handles: 42665 PIDs have reached 50K+ by now. To refine my inquiry - since I'm, like, totally positive that a malicious (or stupid) process is monitoring other processes startup and grabbing memory for itself, I need a tool to monitor a very specific thing - process invocation and memory allocation under a specific pooltag ('Application1.exe has allocated 2656 bytes under 'Proc' tag and grabbed N process handles'). Is anything of the sort available?