I have anExchange 2007 Mailbox server that is consuming memory strangely. I know all about the benefits of STORE.EXE consuming lots of memory, and I wholeheartedly agree with the overall idea... but I think something is amiss in how it is implemented. I have about85 users on Exchange 2007. The server is a HP DL580 with 32 GB of RAM, and a 33 GB page file. The server will operate OK for a week or so. After a week, I start getting alerts about page file utilization. Performance starts to get slow. Other services start to complain about timeouts.I might even get a pop-up stating, "Windows - Virtual Memory Minimum Too Low : Your system is low on virtual memory". Sure enough, I'll see that the STORE.EXE process is consuming almost the entire page file (as measured by the "Page File Bytes" and "Virtual Bytes" perfmon counter),yet Task Manager (and the "Working Set" perfmon counter) show it's only using about 615 MB of RAM. I find this strange for two reasons: I would expect STORE.EXE to consume large amounts of RAM. It's not. I have 30 of my 32 GB available! I would NOT expect STORE.EXE to use the paging file to cache the database. After all, if the goal of caching the database in memory is to avoid disk overhead, storing that database cache in the paging file nullifies the entire argument... does it not? There are also a number of places on the Internet where I've found ad-hoc advice to do things like modify the msExchESEParamCacheSizeMax parameter of the Information Store service in ADSIEDIT.MSC. I'm reluctant to do that without a better understanding of how (and why) the STORE.EXE service caches the data, and why it would allow this memory to be virtualized. Is there a better fix that Microsoft would suggest to address this problem (such as enlarging my page file?) that would be preferable to setting an advanced parameter like msExchESEParamCacheSizeMax ? Perhaps there is some guidance Microsoft can provide on what the page file sizes should be to avoid this problem? Please help! I don't think I'm alone here... - Jim

I think you should look at other perfmon counters to get the pagefile utilization Object: Paging File--Counter %usage Object: Paging File--Counter %usage Peak the page file size is not easy to set. The recomendation on 10 MB + RAM is jsut a genera rule. I would look at these counters and see what they tell me and then set page file. Let your system run for a couple of days before you can expect some usefull values. do you have any other application running on the box?

Hi Lasse, Those values (Paging File %usage) are around 98%. These are dedicated mailbox servers. That said, they do have all of the other applications you would expect on a mailbox server: Anti-Virus (Microsoft Forefront and McAfee), SAN Multi-Pathing (EMC PowerPath), etc.. but nothing you wouldn't expect. Any thoughts on why STORE.EXE would utilize the page file so heavily and the RAM so little? - Jim

Something is very wrong with your server. I would give PSS a call and they can have a live look at your server.

Which OS are you running? Win2003sp1 had an, uh,interesting behaviour that would trim the working set when you logged off or disconnected from the console session. I don't recall if there was a QFE for it, but I'm almost certain it's fixed in Win2003sp2. Of course that's just pure conjecture -- it could easily be some other cause. -martin

Hello you have had some solution for this problem?Manuel Godoy

No, no solution as of yet. I have escalated it through Microsoft PSS, and they agree it's strange behavior, and don't dispute that it's happening in my case. Are you also seeing the behavior? What's your interest?

I have the same problem. I have 6gb of ram and store.exe is using a little over 4.2gb of memory and 5.7gb of paging file....Please let me know what you find.

I have exchange 2007 running on a dell server with 4gb of ram and store.exe is using 3.93gb ... lol I was just going to wait for SP1 but an explanation would be nice .. Paolo

Its normal that Exchange 2007 uses a lot of RAM. Sitting next to a server with 3GB RAM and Exchange is using more than 4. Windows is swapping a little bit but Exchange is working fine. If I look at a server with 8GB of RAM Exchange uses about 7, but it all depends on how much load users put on the server. Exchange is designed to do this, and its also designed to release RAM as other application request for it. However its not working correct if it never release RAM to other apps. and its not normal to consume RAM until everything dies.

Right now i have 4 mailbox users on our exchange 2007 server , it is a Mailbox and Client access server . The hub is running on another server. I noticed that if the server is idle ( as in i am not logged on via rdp) store.exe takes up everything that it can possibly use but if i log on via rdp and maybe run a process ( i updated the virus defs manually) ram usage went back down from 3.9gb to 1.5gb ... it looks like it is managing it ok. Paolo

I agree with the ram but the server should not use 6gb of virtual memory. So my box with 20 basic users on it are using 10 gb of RAM and Paging thats not right....right??

I too am seeing strange behaviour in this regard. Mailbox only server with 12 GB of RAM. Store.exe is currently using over 9GB RAM and 9GB of virtual memory. This is a little strange since after restrting the store process yesterday morning. memory utilisation had gone down to 600MB. after a day wof working this had gone up to 1.5GB after and overnight online defrag this had isncreased to 9GB. Does anyone know whether this is normal.

After starting Information Store, there isnt much to cache for clients, akacold state. after a while when users and other processes have accessed data from information store there is going to be caching involved. Caching is done in RAM since its much faster than going to disc. regarding page file size, its little more tricky, first you must set the correct page file size. monitor these performance counters Object: Paging File--Counter %usage Object: Paging File--Counter %usage Peak The recomendation on 10 MB + RAM is jsut a genera rule. I would look at these counters and see what they tell me and then set page file. Let your system run for a couple of days before you can expect some usefull values. If you require a full memory dump then you must set page file atleast as big as your RAM.

I've looked at these performance counters and they don't really add up. Pagefile Size = 12285MB. Pagefile %Usage = ~9% Pagefile % Peak Usage = ~20% Process Private bytes (store.exe)= ~9GB (9318083806) Process Virtual bytes (store.exe) = ~9GB (9018454832) My interpretation of this is that Pagefile usage (which include all OS processes) is around 1GB. However, the store.exe process is actually using 9GB of virtual address space. Or am I way off here?

Your page file is approx 12GB and pagefile peak usage is aprox 20% this means that windows have never used the pagefile more than20% of 12GB about 2.5GB Current usage is 9% of 12GB, about 1GB Virtual address space is RAM + Pagefile, so it can be more than your RAM Your server has 12 GB RAM and Exchange is using 9, quite normal i think

Has anyone had any progress with this issue? I too am seeing similar problems. I, like Jim, understand the way store.exe should be using memory and as much memory as possible. However, I've found my store.exe recently using significantly higher amounts of private memory than working set. This does not seem right at all.If I understand correctly, private bytes is the amount of memory a process has allocated but not released. Working set is the amount of memory a process is currently using or addressing.This past Tuesday I came in to find my Exchange server behaving strangely. When I went to the console I had a "virtual memory too low" error on the console. When I check task manager store.exe was using about 650 of memory but the "PF Usage" graph was reporting over 9 GB (I have 8 GB of physical memory). I immediately restarted the store process and memory usage came back in line.However, today I'm looking at my store.exe working set at 2 GB and the private bytes are at nearly 6 GB. To me this seems to indicate that store.exe is not properly deallocating memory it is no longer using though I do understand that it could be holding that memory open for future use. However, it does not appear to actually use that memory again.I don't think this is a page file problem. I don't want to debate configuration of the page file but in my opinion I should not need more than a 2-4 GB page file. I don't care if I can't get a memory dump. I understand that Windows needs to swap some kernel memory and some other core processes from time to time and that is what the page file usage should be limited to. The Information Store should not be swapping to the page file. Microsoft gives no guidance in any of the Exchange Server 2007 setup and deployment documents regarding sizing of the page file for Exchange Server.I have 8 GB of physical memory and a SAN for storage all so that I don't have to deal with the limitation of slow local disks. If the information store is swapping significant amounts of memory I am back to being bottle-necked by slow local disks.I have to say that I was not having this problem (that I noticed until recently). I would see my memory usage run up to 8 GB with store.exe using about 5.8 - 6 GB at any one time. I'm fine with that and the server was running well under those conditions. That is why I put 8 GB of memory on the server so Exchange could use it for caching.I did recently start experimenting with RPC over HTTP/S or Outlook Anywhere and I'm curious if anyone else having problems is using the same. I moved some of my users to this connection type because of another issue with Exchange not releasing MAPI connections when a VPN user drops without first closing Outlook. Another bug (in my opinion) that I am trying to work around. I only have about 10 users using RPC over HTTP/S at present.Here are my configuration details:Single Exchange Server (Mailbox, HUB, Transport, Client Access Roles installed)2 Dual Core Xeon 5130 processors8 GB of RAMiSCSI SAN storage, 2 databases and 5 volumes (2 x data, 2 x logs, 1 x queue)Applications and OS installed on local storage165 mailboxes55 Blackberry enabled mailbox2 iPhone usersRoughly 75 GB of database usage between the 2 databasesNo Exchange Server Anti-Virus software running on the server, a Barracuda acts as the e-mail gateway for AV and SPAMI recently started increasing my page file sizes due to getting virtual memory errors. I originally had my page file configured to 2046 - 4092 just like my 64-bit SQL Server 2005 server which has never needed more virtual memory with 8 GB of physical memory. So I added a second page file on my D: drive with the same sizing and now both of them have grown in size.Something seems very wrong and I'm think I may need to open a PSS case as well if this continues. Hopefully someone has more information to add to the topic.

I also saw this problem today with store.exe usingall memory available (like4 Gb) resulting in memory faults (couldn't open Exchange Management Console because of it and users not able to connect to the server.I activated Outlook Anywhere yesterday, it somehow seems connected! My configuration is: Mailbox, Hub, Transport,Client Access: 2Quad Core Xeon 6 GB RAM 2 * 73 gb 15k rpm drives (mirrored) About 50 mailboxes 15 activesync users The database are 15 Gb.. No antivirus or antispam

Just an update for all of you -- I have opened a case with PSS. We found that although I had uninstalled the JetStress tool, the performance counters DLL for JetStress was not uninstalled properly (perhaps because I was running PerfWiz at the time?). To determine if you have the same problem I did, go to HKLM\SYSTEM\CurrentControlSet\Services\ESE\Performance. If the "Library" key is set to "C:\Program Files\Exchange Jetstress\eseperf.dll", then you have the same problem I did. For me, what worked was to change this to "C:\Program Files\Microsoft\Exchange Server\bin\perf\%PROCESSOR_ARCHITECTURE%\eseperf.dll". After restarting the server, the STORE.EXE service started using RAM more appropriately. If you decide to make this change yourself, please make sure you know what you're doing.

I just described the same situation on another thread: http://forums.microsoft.com/technet/showpost.aspx?postid=2478315&isthread=false&siteid=17&sb=0&d=1&at=7&ft=11&tf=0&pageid=0 Rosario

Jim, I have the same VM issues my setup is 8gb of RAM in a Dell 1950 with Dual Quad Core processors... The only difference is I am running the mbx role only in a clustered environment connected to an MD3000. I have checked the reg key and mine says: "C:\Program Files\Microsoft\Exchange Server\bin\perf\%PROCESSOR_ARCHITECTURE%\eseperf.dll". The only change I can think that has happened here is that we are currently migrating users onto the new servers. I have now completed the move of 45 mailboxes (Some are over 5gb) so will keep a close eye on the VM issue and report back to the forum.. Regards Paul

I have changed the page file size to be "System Managed Size" as opposed to the custom size which set at 2046MB.. I will monitor it and report back in 1 week. Paul

You should also take look at this http://msexchangeteam.com/archive/2007/12/28/447790.aspx

I've downloaded both the fixes referenced in the above and have a test CCR cluster in my lab. Haven't tested yet but am VERY curious if folks have applied one or both of the fixes and if so with what result... My lab will be used for testing but with only a couple of GB of RAM per node... not sure how useful the results will be.

I have done more research on the store.exe consuming all the virtual memory on the server. Based on my research, this behavior is normal. With Exchange 2003, the store process was bound to a certain memory cache limit. The upper bounds of this limit was typically set at around 900mb. This could be increased further to aid in recovery, but for normal day-to-day operations, the limit was right around 900mb. This was due to the virtual memory limitations with a 32-bit Operating System, which limited virtual address space to 4GB. The default settings allocated 2gb virtual address space for kernel mode, or the Operating System, and the other 2gb for application mode, which was used by applications such as Exchange. The /3GB switch could be added to the boot.ini file to change this default allocation to 1gb for kernel mode, and 3gb for application mode. The article 266768 <http://support.microsoft.com/kb/266768/> indicates how the maximum database cache size could be modified with Exchange 2003, and shows what the recommended values are. The default value for a server with the /3GB switch set is 896mb, and the maximum recommended value is 1.2gb. With Exchange 2007 and the 64-bit architecture, this limit on database cache size is no longer present, so the store process is no longer bound to that 900mb limit. Currently, the default minimum cache size for Exchange 2007 is 512MB (for machines with at least 2GB RAM), and there is no maximum value set, which means that ESE (store.exe) will grow the cache to consume almost all available RAM on the server If there is no other memory pressure on the system. This much larger database cache size results in greatly reduced disk I/O, and is preferred anyways, as reading information from memory is much faster than reading information from disk. If memory pressure occurs, as when other applications request/require memory, ESE will appropriately shrink the size of the database cache. For example, if the server contains 8gb physical memory, if there is no other memory pressure, one could expect that the store.exe process will grow to use up to 6gb memory (8gb minus 2gb allocated to Kernel mode). This value can be manually adjusted by modifying the following attribute on the properties of the Information Store object, however it is not recommended to do so. msExchESEParamCacheSizeMax To modify msExchESEParamCacheSizeMax: 1. Start ADSI Edit. 2. Open the following object: Configuration/Services/Microsoft Exchange/Your organization/Administrative Groups/Your administrative group/Servers/Server name/Information Store 3. Right-click Information Store, and then click Properties. 4. Under the list of Attributes, scroll down and select msExchESEParamCacheSizeMax. Note The msExchESEParamCacheSizeMax property extends beyond the width of the Select a property to view list. Make sure that you do not unintentionally click the msExchESEParamCacheSizeMin property instead 5. Click the Edit button, then type the number of 8 kilobyte (KB) pages that you want to set the maximum cache size to. For example to set the cache at 5GB which would allow the system with 8GB of memory to keep 1GB of memory for various processes and 2GB for the kernel. A 5GB cache equates to 5242880 (5120 * 1024). Note The msExchESEParamCacheSizeMax parameter controls the ESE buffer size. Its value is expressed as a page count, and must be set to an exact multiple of 8192 for maximum efficiency. If this value is not met, the cache size is rounded up to the next 32-MB boundary when virtual memory is allocated. If this value is incorrectly set, memory may be wasted. 6. Quit ADSI Edit, and then restart the Microsoft Exchange Information Store service. The details of these steps are documented for server 2003 at the bottom of KB article http://support.microsoft.com/kb/815372 it is the same process for exchange 2007. Another alternative would be to monitor the paging file to determine the appropriate size for this system. The rule of 1.5 times the amount of physical memory does not always work.. here is a guide to determine appropriate page file size on a 64 bit system. http://support.microsoft.com/default.aspx/kb/889654

I have applied the fix released by MS and the paging issue has gone away. I only applied the following hf http://support.microsoft.com/kb/938486 My exchange server is currently paging at 17GB and growing, I have set the page file to be system managed. Is this the correct thing to do? Paul

Hi Paul, In general, I don't like to have my systems manage their paging file, because the constant shrinking and growing of the file contributes to disk fragmentation. I use a Custom Size, and set the Initial and Maximum to the same value. Picking the right value is of more importance when doing this, however. - Jim

Thanks Jim, I appreciate the feedback... What would you recommend for 8GB in my Exchange servers? Paul

I just posted a rather large update to http://blogs.technet.com/mikelag/archive/2007/12/19/working-set-trimming.aspx on possible causes/remediations and have also created http://msexchangeteam.com/archive/2008/08/06/449484.aspx which explains Exchanges use of memory.

I just posted a rather large update to http://blogs.technet.com/mikelag/archive/2007/12/19/working-set-trimming.aspx on possible causes/remediations and have also created http://msexchangeteam.com/archive/2008/08/06/449484.aspx which explains Exchanges use of memory.

I seem to have corrected my problems and I figured I would post what I think fixed it. I tried the hotfix mentioned earlier in this thread but it did not seem to help. The server was still running itself out of memory. Also changing the page file to excessively large sizes did not make any difference either. For reference this is my hardware: Dell PowerEdge 2950 (generation 1) 2 x 2.6 Ghz Dual Core Intel 51xx processors 8 GB Memory. Windows Server 2003 R2 64-bit SP2 Exchnage 2007 SP1 7200 RPM SATA drives for Local Storage EqualLogic PS100E SAN RAID10 (7200 RPM SATA Drives). I've since upgraded to 2 of these PS100E units. 2 Storage Groups (1 database each) each about (75 GB in one DB, 85 in the second) 5 iSCSI SAN connections,2 database volumes, 2 log volumes, 1 volume for queues I had been using 2-Port HP NC380T Broadcom based NIC with TOE capability. I was trying to enhance iSCSI performance as much as possible without adding a QLogic HBA. After noticing some odd performance with my SAN I began troubleshooting. I made sure all drivers were up to date and the latest EqualLogic software was installed and the iSCSI initiator was up to date. I was still having trouble. Based on guidance from EqualLogic, I replaced the HP NIC with an Intel PRO/1000 PT PCI-Express dual port server adapter. I put all of my iSCSI connections with MPIO on this adapter. I also REMOVED the EqualLogic DSM load balancing software as I was getting connections constantly renegotiating on this 64-bit server (didn't have the same problem on 32-bit servers). After making all the above changes my server has been humming quite nicely for months. It continues to use all available memory (which is precisley what I want it to do) but the private bytes (referenced from Process Explorer) no long grow to a rediculous size. Now my private bytes and working set bytes for store.exe stay within a couple hundred MB of each other at all times. Previously my private bytes would grow to over9 GB and my working set would shrink to about 300 MB causing massive performance problems until a restart. Currenty my working set for store.exe is 5.7 GB and the private bytes are 5.75 GB. The rest of the memory is used by other processes on the machine. So in my case the issues seem related somehow to the combination of iSCSI connections to my EqualLogic SAN over the Broadcom based NICs. Using the Intel NIC seems to have solved my problems.

After reading the updated details at the bottom of the post Mike Lagase mentioned above(http://blogs.technet.com/mikelag/archive/2007/12/19/working-set-trimming.aspx)I believe it must have been the drivers for the HP NC380T NIC (Broadcom based TOE NIC) that were likely the source of my problems. He mentions at the bottom of the post that Broadcom NIC drivers have been known to call the paging or trimming function excessively. Hopefully this confirmation helps someone else. This was frustrating as ***.

Correction on the Broadcom NICs, it is/was the Broadcom NICs that have been known to call the MMAllocateContiguousMemory excessively leading up to the paging/trimming issue. Their latest drivers > 4.x resolve this problem.

Correction on the Broadcom NICs, it is/was the Broadcom NICs that have been known to call the MMAllocateContiguousMemory excessively leading up to the paging/trimming issue. Their latest drivers > 4.x resolve this problem.

Hello everyone, I am actually having the opposite problem. Exchange 2007 SP1(CCR) - 2 quad core xeon - 16gb Memory 12 storage groups 500 users. Store is using 3 GB of physical memoryand 15GB of virtual memory. At time email takes a bit to load in outlook. My CacheMAx in AD is not set. So i am not restricting the usage any where. How do i tell exchange to use more physical memory. Thank you, Felipe

Felipe, I think you may be experiencing the same problem. When I was having problems I would see the physical memory usuage continually decline while virtual memory using would climb. I would try the suggestions laid out in previous posts. Update your hardware drivers and try the hotfix if necessary.

I'd be suspicious of something else on your server consuming memory causing a lower amount of memory to be available for Exchange. Have you collected any perfmon data over time to determine if once you restart the server and Exchange warms up it's cache, that it is the primary process consuming most of the RAM? Once the cache has been warmed up (Online Maintenance would fully load the cache), the RAM should plateau at some point and remain there for the lifetime of store process. If you have other I/O occurring on the server, it is possible that Exchange's Dynamic Buffer Allocation (DBA) is kicking in and shrinking the cache based off of the Transistion Pages Repurposed/sec counter. It would be really hard to tell without a good perfmon to see what is happening under the hood as it would only be speculation at this point based on what you have mentioned thus far, but some of the other components such as what else is installed on the server has been left out. I've moved all of the known memory issues that could affect Exchange to http://blogs.technet.com/mikelag/archive/2008/10/17/steps-to-help-mitigate-excessive-paging-and-working-set-trimming-issues.aspx, so you may want to check that one out too. Mike

I'd be suspicious of something else on your server consuming memory causing a lower amount of memory to be available for Exchange. Have you collected any perfmon data over time to determine if once you restart the server and Exchange warms up it's cache, that it is the primary process consuming most of the RAM? Once the cache has been warmed up (Online Maintenance would fully load the cache), the RAM should plateau at some point and remain there for the lifetime of store process. If you have other I/O occurring on the server, it is possible that Exchange's Dynamic Buffer Allocation (DBA) is kicking in and shrinking the cache based off of the Transistion Pages Repurposed/sec counter. It would be really hard to tell without a good perfmon to see what is happening under the hood as it would only be speculation at this point based on what you have mentioned thus far, but some of the other components such as what else is installed on the server has been left out. I've moved all of the known memory issues that could affect Exchange to http://blogs.technet.com/mikelag/archive/2008/10/17/steps-to-help-mitigate-excessive-paging-and-working-set-trimming-issues.aspx, so you may want to check that one out too. Mike

Similar problem: 1. HP ProLiant DL380 G5, 8 GB RAM, Windows server 2008, Exchange server 2007 SP1, 50 user mailboxes. Mailbox store size 70 GB. 5.7 GB of RAM used by store.exe. 2. HP ProLiant DL380 G5, 8 GB RAM, Windows server 2008, Exchange server 2007 SP1,4 (!)user mailboxes. Mailbox store size6.8 GB. 5.3 GB of RAM used by store.exe and growing. Is that ok or any fixes should be applied? Thanks.

Hi, you can upgrade to Exchange 2007 SP1 with latest Rollup update (Rollup update 5)

Hi upgrade to Exchange 2007 SP1 with latest Rollup update

It is normal behaviour of store.exe to consume that much available memory. It is normal unless any other performance issues are observed. Because store.exe will consume memory to cache part of its databases. a few articles on this, http://msexchangeteam.com/archive/2004/08/02/206012.aspx http://msexchangeteam.com/archive/2008/08/06/449484.aspxSajjad

I had a similar problem. Except it was AVG causing the problem on my mailbox server. I took it off and CPU dropped.MRA A+ CNST CFOI CCNP