I have done more research on the store.exe consuming all the virtual memory on the server.
Based on my research, this behavior is normal. With Exchange 2003, the
store process was bound to a certain memory cache limit.
The upper bounds of this limit was typically set at around 900mb. This
could be increased further to aid in recovery, but for normal day-to-day
operations, the limit was right around 900mb. This was due to the virtual
memory limitations with a 32-bit Operating System, which limited virtual
address space to 4GB. The default settings allocated 2gb virtual address
space for kernel mode, or the Operating
System, and the other 2gb for application mode, which was used by
applications such as Exchange. The /3GB switch could be added to the
boot.ini file to change this default allocation to 1gb for kernel mode, and
3gb for application mode.
The article 266768 <
http://support.microsoft.com/kb/266768/> indicates how
the maximum database cache size could be modified with Exchange 2003, and
shows what the recommended values are. The default value for a server with
the /3GB switch set is 896mb, and the
maximum recommended value is 1.2gb.
With Exchange 2007 and the 64-bit architecture, this limit on database
cache size is no longer present, so the store process is no longer bound to
that 900mb limit. Currently, the default minimum cache size for Exchange
2007 is 512MB (for machines with at least 2GB RAM), and there is no maximum
value set, which means that ESE (store.exe) will grow the cache to consume
almost all available RAM on the server If there is no other memory pressure
on the system. This much larger database cache size results in greatly
reduced disk I/O, and is preferred anyways, as reading information from
memory is much faster than reading information from disk. If memory
pressure occurs, as when other applications request/require memory, ESE
will appropriately shrink the size of the database cache.
For example, if the server contains 8gb physical memory, if there is no
other memory pressure, one could expect that the store.exe process will
grow to use up to 6gb memory (8gb minus 2gb allocated to Kernel mode).
This value can be manually adjusted by modifying the following attribute on
the properties of the Information Store object, however it is not
recommended to do so.
msExchESEParamCacheSizeMax
To modify msExchESEParamCacheSizeMax:
1. Start ADSI Edit.
2. Open the following object:
Configuration/Services/Microsoft Exchange/Your organization/Administrative
Groups/Your administrative group/Servers/Server name/Information Store
3. Right-click Information Store, and then click Properties.
4. Under the list of Attributes, scroll down and select
msExchESEParamCacheSizeMax.
Note The msExchESEParamCacheSizeMax property extends beyond the width of the Select a property to view list. Make sure that you do not unintentionally click the msExchESEParamCacheSizeMin property instead
5. Click the Edit button, then type the number of 8 kilobyte (KB) pages
that you want to set the maximum cache size to.
For example to set the cache at 5GB which would allow the system with 8GB of memory to keep 1GB of memory for various processes and 2GB for the kernel. A 5GB cache equates to 5242880 (5120 * 1024).
Note The msExchESEParamCacheSizeMax parameter controls the ESE buffer size.
Its value is expressed as a page count, and must be set to an exact
multiple of 8192 for maximum efficiency. If this value is not met, the
cache size is rounded up to the next 32-MB boundary when virtual memory is
allocated. If this value is incorrectly set, memory may be wasted.
6. Quit ADSI Edit, and then restart the Microsoft Exchange Information
Store service.
The details of these steps are documented for server 2003 at the bottom of KB article
http://support.microsoft.com/kb/815372 it is the same process for exchange 2007.
Another alternative would be to monitor the paging file to determine the appropriate size for this system. The rule of 1.5 times the amount of physical memory does not always work.. here is a guide to determine appropriate page file size on a 64 bit system.
http://support.microsoft.com/default.aspx/kb/889654There is an amazing pack of free network admin tools.
click here to download it