Primary user exchange 2003 server producing NDR's for a few minutes every morning...
Good morning. I am trying to figure out what is causing my primary user exchange server to produce a number of event errors and cause NDR's for a number of clients for a period of 5-10 minutes every morning. I understand what is HAPPENING by looking at the
logs, but what I cannot figure out is what is causing it to happen.
Here is a timeline of what occurs:
Exchange Timeline:
8:07AM:
Event Type: Warning
Event Source: MSExchangeIS Mailbox Store
Event Category:
Background Cleanup
Event ID:
9791
Date:
07/07/2010
Time:
8:07:22 AM
User:
N/A
Computer: XXXX
Description:
Cleanup of the DeliveredTo table for database 'User Storage Group\User Mailbox Store A' was pre-empted because the database engine's version store was growing too large. 0 entries were purged.
Event Type: Warning
Event Source: MSExchangeIS Mailbox Store
Event Category:
Background Cleanup
Event ID:
9791
Date:
07/07/2010
Time:
8:07:22 AM
User:
N/A
Computer: XXXX
Description:
Cleanup of the DeliveredTo table for database 'User Storage Group\User Mailbox Store B' was pre-empted because the database engine's version store was growing too large. 0 entries were purged.
The above two warnings continue to log by the minute until 8:20, then this:
8:20AM:
Event Type: Error
Event Source: ESE
Event Category:
Transaction Manager
Event ID:
623
Date:
07/07/2010
Time:
8:20:46 AM
User:
N/A
Computer: XXXX
Description:
Information Store (3796) User Storage Group: The version store for this instance (2) has reached its maximum size of 155Mb. It is likely that a long-running transaction is preventing cleanup of the version store and causing it to build up in size. Updates
will be rejected until the long-running transaction has been completely committed or rolled back.
Possible long-running transaction:
SessionId: 0x7FD31420
Session-context: 0x00000000
Session-context ThreadId: 0x0000150C
Cleanup: 1
After the above error, the following events begin to happen:
Event Type: Warning
Event Source: MSExchangeIS Mailbox Store
Event Category:
General
Event ID:
1025
Date:
07/07/2010
Time:
8:20:48 AM
User:
N/A
Computer: XXXX
Description:
An error occurred on database "User Storage Group\User Mailbox Store A".
Function name or description of problem: EcGenerateNDR:
Error: 0xfffff9bf
Event Type: Error
Event Source: MSExchangeIS Mailbox Store
Event Category:
Logons
Event ID:
1022
Date:
07/07/2010
Time:
8:20:49 AM
User:
N/A
Computer: XXXX
Description:
Logon Failure on database "User Storage Group\User Mailbox Store A" - Windows 2000 account XXXX\XXXX; mailbox /o=XXXX Industries Ltd./ou=XXXX CAT/cn=Recipients/cn=xxxxxxxx/cn=xxxxxxxxxx.
Error: -1069
Event Type: Error
Event Source: MSExchangeIS
Event Category:
Performance
Event ID:
9582
Date:
07/07/2010
Time:
8:21:14 AM
User:
N/A
Computer: XXXX
Description:
The virtual memory necessary to run your Exchange server is fragmented in such a way that normal operation may begin to fail. It is highly recommended that you restart all Exchange services to correct this issue.
Event Type: Warning
Event Source: MSExchangeIS Mailbox Store
Event Category:
Background Cleanup
Event ID:
9791
Date:
07/07/2010
Time:
8:21:27 AM
User:
N/A
Computer: XXXX
Description:
Cleanup of the DeliveredTo table for database 'User Storage Group\User Mailbox Store' was pre-empted because the database engine's version store was growing too large. 0 entries were purged.
Event Type: Warning
Event Source: MSExchangeIS Mailbox Store
Event Category:
Background Cleanup
Event ID:
9791
Date:
07/07/2010
Time:
8:21:27 AM
User:
N/A
Computer: XXXX
Description:
Cleanup of the DeliveredTo table for database 'User Storage Group\User Mailbox Store B' was pre-empted because the database engine's version store was growing too large. 0 entries were purged.
Starting at 8:21AM, Login failures on database:
Event Type: Error
Event Source: MSExchangeIS Mailbox Store
Event Category:
Logons
Event ID:
1022
Date:
07/07/2010
Time:
8:21:46 AM
User:
N/A
Computer: XXXX
Description:
Logon Failure on database "User Storage Group\User Mailbox Store B (XXXX)" - Windows 2000 account XXXX\xxxxxx; mailbox /O=XXXX/OU=XXXX/cn=Recipients/cn=xxxxxx.
Error: -1069
The above logs repeat themselves with the exception of the ESE error, citing all different users until 8:26AM where they stop. At this point there is a global address warning:
Event Type: Warning
Event Source: MSExchangeSA
Event Category:
OAL Generator
Event ID:
9327
Date:
07/07/2010
Time:
8:28:14 AM
User:
N/A
Computer: XXXX
Description:
OALGen skipped some entries in the offline address list '\Global Address List'.
To see which entries are affected, event logging for the OAL Generator must be set to at least medium.
- Default Offline Address List
...and then regular logging and access resumes
If anyone has any ideas on what is causing this to happen, I would greatly appreciate them. Thank you!
July 7th, 2010 6:28pm
Hi ,
The error is warning message,It is related to performance issue on Exchange
Server.
1.restart microsoft exchange information store service.
2.dismount mailbox store and remount. to do so
a.go to exchange system manager
b.expand servers,first storage group,maibox store,right click and click
dismount store
c.then click mount store.
3.If issue persists,Please refer to the following steps to defragment
exchange database to see if it helps:
a. Check Database consistency using Eseutil utility:
Please run eseutil /mh on the database to check if the status of Exchange
databases is OK. You may refer to the following Knowledge Base Article:
How to defragment with the Eseutil utility (Eseutil.exe)
http://support.microsoft.com/default.aspx?scid=kb;en-us;192185
b. Run command to verify there is no error/warning. If there is warning
displayed, please repeat the Isinteg command until the number of database
fixes reaches 0 (zero) or does not change.
To do so, please follow the process below:
I. Dismount the Store by right-clicking it in ESM and choose Dismount Store.
II. Open a command prompt window and change the currently directory to
III. Run the following command:
isinteg -s <Exchange Server Name> -fix -test alltests
IV. Follow the instruction to scan the database.
V. Re-run the command in step IV until you found the fixes number is near
to 0 (zero).
For more information, please refer to the following Knowledge Base Article:
Exchange Command-Line Parameters for the Isinteg.exe Tool
http://support.microsoft.com/?id=301460
Please perform offline defragment on the exchange database. you can refer
to the article below:
328804 How to defragment Exchange databases
http://support.microsoft.com/?id=328804
Then please try to monitor the server for some time to check if the issue
persists.
4.restart the server
I appreciate your time. I am happy to be of assistance and look forward to
your reply.
Best regards,
Shafaquat Ali.
M.C.I.T.P Exchange 2007/2010, M.C.I.T.P Windows Server 2008, M.C.T.S OCS Server 2007 R2
Free Windows Admin Tool Kit Click here and download it now
July 7th, 2010 7:12pm
Are all of the mailbox databases in the same Storage Group?
Is the server backing up correctly?
Does anything coincide with these logs? AV scan, backup over running, anything like that? Large email run during the night?
It appears that you are running Enterprise edition of Exchange, so an offline defrag as suggested above is not really a good idea. It will also involve an extended period of downtime and is not recommended as a routine action. It is also not risk free as
data can be lost.
Rather than doing an offline defrag, it would be more effective to move the users to another database, so that this storage group could be replaced. That will involve zero downtime and almost no risk to the data.
Of course I am presuming here that the server is fully patched.
Simon.Simon Butler, Exchange MVP. http://blog.sembee.co.uk , http://exbpa.com/
July 7th, 2010 8:07pm
Thanks for the replies guys.
We found an excellent article here: http://msexchangeteam.com/archive/2006/04/19/425722.aspx
The content of that article covered our issues pretty much spot on. We are in the process of building a new exchange box to split user stores across 2 servers as we are still about 6 months away from moving to 2010. Performance is becoming a real issue on this
server and we believe this is just another side affect.
Thanks again.
Free Windows Admin Tool Kit Click here and download it now
July 7th, 2010 10:45pm