Memory Leaks on Server, keeps locking up, System Error: 2019, App Errors: 8026, 8031
So on Monday our Exchange server went down again, for the third time in less than a 2 month period. It is a Proliant DL380 G3 with 3.72GB of RAM, running Server 2003 R2 with all the latest patches. What happens is all of a sudden I will get a hundred calls saying "outlook isn't working" and Ill check it out and no one is able to send or recieve email. So I try to remote into the exchange server, and have no luck. So I go to the server and can not even log into it since it is locked up or frozen. It does not allow me to perform a graceful shutdown, I have to do a hard reset. Once back up and running, it is working properly and it will stay wokrking properly for a couple weeks, or longer. In the event viewer there are a few repeat errors I have noticed every time the server gets locked up and stops functioning. System Error: 2019 The server was unable to allocate from the system nonpaged pool because the pool was empty. Event Error: 8026 LDAP Bind was unsuccessful on directory 'SERVERNAME' for distinguishhed name ''. Directory returned error:[0x51] Server Down. Event Warning: 8031 Unable to open LDAP session on directory 'SERVERNAME' using port number 389. Directory returned the LDAP error:[0x1] Operations Error When I origionally tried to resolve this problem people were telling me the exchange serve had a problem communicating with the DC, so I changed the exchange server to look to itself in the recipient update services (it is a DC). Also, I ran DCDIAG and ExBPA and everything looked to be good. Only thing ExPBA showed was an issue with the page file, which I fixed. I made these changes after the last crash, which was in the middle of May, so it has been a little over a month but the same issue and errors. It seams to be a memory leak. What could be causing this and how can I track it down and resolve it? I have read that symantec endpoint protection causes issues with memory leaks, but we have the latest version of SEP installed on our servers, and none of the other ones have issues.
June 30th, 2010 6:07pm

Hi, To start troubleshooting, this is Exchange 2003 (and which SP?)? Also, have you looked at, http://blogs.technet.com/b/dblanch/archive/2009/04/18/paged-and-non-paged-pool-issues-on-exchange-2000-2003.aspx http://technet.microsoft.com/en-us/library/aa996269%28EXCHG.80%29.aspx Also, can you give a few statistics regarding user count, connection count, and a few performance counters: MSExchangeIS\User Count Memory\Pool Paged Bytes Memory\Pool Nonpaged Bytes -- Mike Burr
Free Windows Admin Tool Kit Click here and download it now
June 30th, 2010 6:37pm

We are running Exchange 2003, version 6.5 SP 2. MSExchangeIS\User Count: average- 53 Memory\Pool Paged Bytes: average- 72844616 Memory\Pool Nonpaged Bytes: average- 62104495
July 1st, 2010 5:48pm

This is usually caused by a couple of things 1) /3gb switch in Boot.ini http://support.microsoft.com/kb/810371 2) Misconfigured/Heavily used system and nonpage pool depletion. You can configure your system to maximize nonpaged pool with this setting. If you have a true memory leak this will buy you more time. If the default size of the nonpaged pool is too small this may solve your issue. Record the setting first. It should be 0x0, or configured as below. If it is any other value, some application (virus scanners) or person incorrectly hardcoded a value into the setting. This is most likely the cause. Configure as shown below. [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management] "NonPagedPoolSize"=dword:FFFFFFFF
Free Windows Admin Tool Kit Click here and download it now
July 1st, 2010 6:12pm

Is this a safe registry change? Im just going to change that key to FFFFFFFF, 8 F's right? This is a production server and is in use right now, will the change have any affect on the server? What exactly is the modification going to do?
July 1st, 2010 6:19pm

Oh, sorry, i guess i thought you would google the setting. Requires a reboot to take effect. 0xFFFFFFFF (8 F's) is a special setting that tells the system to configure as much nonpagepool memory given the current (at boot time) RAM for the computer. This allows more memory to be added in the future without changing the setting.
Free Windows Admin Tool Kit Click here and download it now
July 1st, 2010 6:29pm

One problem I see is that you said this server is a DC. Ideally, your Exchange server should not be a DC, with the exception of Small Business Server. It's two seperate databases being run on the same machine. How much RAM is installed on this box? What's is the Operating System version, including 32-bit/64-bit? Exchange 2003 should only be installed on a 32-bit OS. Also, it is ideal to not go over 4 GB of RAM and to use the /3GB switch when over 1 GB of RAM. This is added to the boot.ini file. Physical Address Extension (PAE) can cause some issues with Exchange.View my MCP Certifications
July 1st, 2010 6:42pm

Proliant DL380 G3 with 3.72GB of RAM, running Server 2003 R2 with all the latest patches, and exchange 2003 SP2. I changed that one reg key I'll see if it makes a difference. This is my boot.ini file, it looks like the /3gb is already in place. [boot loader] timeout=30 default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS [operating systems] multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003, Standard" /noexecute=optout /fastdetect /3GB /USERVA=3030
Free Windows Admin Tool Kit Click here and download it now
July 1st, 2010 7:08pm

How many hard drives are in this system? Is it a Raid array? Did you leave the page file on C? My curiosity is if the I/O is just saturating a controller, but I doubt it. Do you have a 3rd party firewall or Intrusion Prevention System on your servers? Sometimes these can stop requests if they exceed a certain amount within a limited time frame.View my MCP Certifications
July 1st, 2010 7:23pm

There are a bunch of hard drives using RAID 0+1. There are 3 partitions and the page file is set up as follows: C: 3000 to 4000 MB D: 3000 to 4000 MB F: No pageing file The server has the latest version Symantec Endpoint Protection (11.0.6) installed. I have occasionally been logged on and seen a notification pop up saying store.exe has been blocked. Also, there's a Cisco security Agent.
Free Windows Admin Tool Kit Click here and download it now
July 1st, 2010 7:37pm

Awesome information! Try relocating the page file back to C and let the size be System Managed. This is the default setting and really does not need to be adjusted. Also, remove the SEP firewall components if they're installed and see if the issue goes away. Add exceptions to the Intrusion Prevention policy within SEP for the Exchange server if you do not wish to remove the firewall. Also, if removing the firewall fixes the issue, you may use this strategy to make it work after reinstalling the firewall. This is found on the Settings tab > Enable Excluded Hosts > Click Excluded Hosts button. You can add the IP, IP Range, or Subnet to the policy.View my MCP Certifications
July 1st, 2010 8:24pm

Given the error its most likely a nonpaged pool memory issue, its not disk or I/O related. I would try to isolate the contributing factors, one at a time to until the problem goes away....if at all. 1) Remove the /3gb and /Userva switches. The /3gb reduces the amount of nonpaged pool memory. 2) Check the configuration of the NonPagedPoolSize setting, to make sure it is not hard set to another value. Set to 0x0 or 0xFFFFFFFF, nothing else. 3) Disable (and uninstall) virus scanning, these drives are known to use nonpagedpool memory. My bet is #1 because this setting is really only intended for dedicated exchange servers, not DCs running Exchange.
Free Windows Admin Tool Kit Click here and download it now
July 1st, 2010 8:31pm

The Exchange BPA and all documentation states to use the /3GB switch with Exchange. The error that concerns me is the inability for Exchange to communicate with a DC. I imagine we're both in the right area. Have exclusions been set for Exchange and the Operating System? http://support.microsoft.com/kb/822158 http://support.microsoft.com/kb/823166 View my MCP Certifications
July 1st, 2010 8:46pm

Ok I already made the change to 0xFFFFFFFF in the registry. Im working on adding store.exe as an exception in SEPM so it doesn't keep blocking it. I can not remove the SEP, we are required to have endpoint protection on all servers and workstations. I could remove the /3gb and /userva switches, even though most threads I have read tell you to add them. This server is our only exchange server, but we have a couple other DC's. I'm just not sure it's a configuration setting like changing the swtiches because it has worked fine for years, we have not changed any configuration settings. Only changes are patches and updates. It was just in the past few months that it started having issues with locking up and needing the reboot. But Ill give that a try.
Free Windows Admin Tool Kit Click here and download it now
July 1st, 2010 8:53pm

>>>exchange server to look to itself in the recipient update services (it is a DC). What Exchange on a DC??? >>>The Exchange BPA and all documentation states to use the /3GB switch with Exchange This assumes no other competing services are installed and running on the sever (ie like Exchange running on a DC). This assumes dedicated Exchanges servers. This assumes best practices have been followed. All the best practices assume maximum performance Exchange is needed and desired. None of these seem to apply here. Remove the /3gb switch is the best option. This will provide the best "compromize" to allow the DC, Exchange, and Virus Scanning software to all happily co-exist...but the better solution would be to demote the DC on the Exchange Server. I would also read this. http://networkadminkb.com/kb/Knowledge%20Base/Windows2003/How%20to%20performance%20tune%20a%20Windows%202003%20Server.aspx
July 1st, 2010 9:03pm

>>>I'm just not sure it's a configuration setting like changing the swtiches because it has worked fine for years, we have not changed any >>>configuration settings. Only changes are patches and updates. It was just in the past few months that it started having issues with >>>locking up and needing the reboot. But Ill give that a try. Yeah, but have you been adding users, adding mailboxes, has the amount of email increased? Is more data stored in the email system (ie. avg size of mailboxes increased)? Over time things change slowly and you simply reached the tipping point. A fundamental bad practice of running Exchange on a DC probably contributed to this, but you may have encountered the error anyway....eventually.
Free Windows Admin Tool Kit Click here and download it now
July 1st, 2010 9:13pm

Event Warning: 8031 Unable to open LDAP session on directory 'SERVERNAME' using port number 389. Directory returned the LDAP error:[0x1] Operations Error This is a sign to me of other larger issues. We have a compound set of problems. Best practices aren't being followed, and AV/Firewall has been installed on a DC and a DC with Exchange. SEP has a component built-in that will stop communications if too many packets are arriving from one host. This could be a possible issue. More likely is that the exclusions are not in place on the DCs and Exchange that are required in order to keep performance up. The most ideal situation is to setup your AV exclusions on all servers, including DCs and Exchange, and demote the DC from Exchange. The /3GB switch is used for best practices, but if the AV scanning Exchange edb files, etc, is causing an issue, then they really need to be excluded. If nothing else, a performance and reliability boost will be obtained. Removing the /3GB switch may very well solve the problem for now, but there is no guarantee it won't return if this isn't fixed. Solve the sources, not the symptoms. http://support.microsoft.com/kb/822158 http://support.microsoft.com/kb/823166 Those still need to be done. Open the SEP client on each server, go to Change Settings, Centralized Exceptions, Configure Settings. Click Add, and add the folders specified in the KBs. If you cannot add at this location, you'll need to adjust the Centralized Exceptions on the SEP Admin Console. View my MCP Certifications
July 1st, 2010 9:26pm

I just ran MEBPA and the two errors I got were: 1.) Paging file larger than Physical Memory: The space for the paging file (8000) is larger than the physical memory (3807). This may affect the system performance. It is recommended to have paging file size equal to the physical memory. 2.) Older 'EXCHEM.DLL' Present: An older version of 'EXCHMEM.DLL' is present in the Windows 'system32' folder on server eyegor.csccu.com. This may cause problems with upgrades. Version of 'EXCHMEM.DLL' in the Windows folder: 6.0.4417.1. Version of 'EXCHMEM.DLL' in the Exchange folder: 6.5.7638.1. Stop all Exchange services and rename the 'EXCHMEM.DLL' file residing in the Windows 'system32' folder.
Free Windows Admin Tool Kit Click here and download it now
July 1st, 2010 9:43pm

Again, change the page file settings to store it on Drive C with System Managed settings. You do not need to customize page file settings anymore except for maybe some extremely rare conditions. Turn it off of any other drives. Breaking it across drives is getting you nothing if they're all in the same RAID array. 3 Partitions in one RAID array still have the exact same disk contention issues.View my MCP Certifications
July 1st, 2010 9:58pm

I changed the page file to store on C: with a minimum 4000 maximum 4000 mb, in addition to the registry change. I will see if the server locks up again. I have been monitoring the memory all day. Since the changes, the pool nonpaged bytes decreased, and the pool paged bytes doubled: Memory\Pool Paged Bytes: average- Yesterday 72,844,616 Today 145,011,000 Memory\Pool Nonpaged Bytes: average- Yesterday 62,104,495 Today 57,254,000
Free Windows Admin Tool Kit Click here and download it now
July 2nd, 2010 8:38pm

Not to beat a dead horse here, but have you implemented the exclusions specified in the following articles?: http://support.microsoft.com/kb/822158 http://support.microsoft.com/kb/823166 Also, you probably should use System Managed size rather than specifying settings. Microsoft likely has invested more time and resources in how page files should be configured than any one of us. Also, did you follow the steps that resulted from your ExBPA scan? Further reading suggests that you upgraded this Exchange server with an in-place upgrade, and this is a leftover remnant. Since its' teh memory manager, it's very possible to cause you your grief. 2.) Older 'EXCHEM.DLL' Present: An older version of 'EXCHMEM.DLL' is present in the Windows 'system32' folder on server eyegor.csccu.com. This may cause problems with upgrades. Version of 'EXCHMEM.DLL' in the Windows folder: 6.0.4417.1. Version of 'EXCHMEM.DLL' in the Exchange folder: 6.5.7638.1. Stop all Exchange services and rename the 'EXCHMEM.DLL' file residing in the Windows 'system32' folder. View my MCP Certifications
July 2nd, 2010 11:03pm

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics