We are getting ready to retire an old Exchange Server and I need to keep as much email data as possible for the Freedom Of Information Act. I have been restoring old backups to gather as many emails in to the mailboxes as possible before I Exmerge them all out to .PST files. Everything was going fine until today when I was doing the last restore job. I am using Backup Exec version 11d. The restore job was successful but now I cannot mount the mailbox store again. When I try I get a popup error that reads "The name of one of the database files in the store is invalid". Below that is "ID no: c1041731". It also logs the following errors in the application event log: Event Type: Error Event Source: ESE Event Category: Logging/Recovery Event ID: 454 Date: 10/25/2010 Time: 9:37:40 PM User: N/A Computer: EXCHANGE-SERVER Description: Information Store (4232) First Storage Group: Database recovery/restore failed with unexpected error -1022. For more information, click http://www.microsoft.com/contentredirect.asp. Event Type: Error Event Source: MSExchangeIS Event Category: General Event ID: 9518 Date: 10/25/2010 Time: 9:37:40 PM User: N/A Computer: EXCHANGE-SERVER Description: Error Disk IO error starting Storage Group /DC=local/DC=PAGECOVA/CN=Configuration/CN=Services/CN=Microsoft Exchange/CN=County Of Page Organization/CN=Administrative Groups/CN=First Administrative Group/CN=Servers/CN=EXCHANGE-SERVER/CN=InformationStore/CN=First Storage Group on the Microsoft Exchange Information Store. Storage Group - Initialization of Jet failed. For more information, click http://www.microsoft.com/contentredirect.asp. Event Type: Error Event Source: MSExchangeSA Event Category: MAPI Session Event ID: 9175 Date: 10/25/2010 Time: 9:38:23 PM User: N/A Computer: EXCHANGE-SERVER Description: The MAPI call 'OpenMsgStore' failed with the following error: The Microsoft Exchange Server computer is not available. Either there are network problems or the Microsoft Exchange Server computer is down for maintenance. The MAPI provider failed. Microsoft Exchange Server Information Store ID no: 8004011d-0526-00000000 For more information, click http://www.microsoft.com/contentredirect.asp. The server is loaded with Windows Server 2003 Standard with SP2. It is running Exchange 2003 version 6.5.7638.1. If anyone can provide some insight in to what is wrong or how I can go about trying to repair it I would be very grateful. Thanks, Marty

Could be that the database file in question is corrupted OR the disk you are attempting to restore to has I/O issues, i.e. I have seen people try to restore to cheap USB drives and sometimes it works but more often it fails so check the destination hardware I would also clear out the destination to be free from other files to ensure nothing is getting commingled from another EDB Push come to shove you can also try a repair i.e. ESEUTIL /P against that database but thats a last ditch effort and definitely ensure you run against a good disk Check out these links as well http://support.microsoft.com/kb/810198 NOTE: 1022 is a JetIO_Error http://support.microsoft.com/kb/266361/EN-US/ More importantly Be aware that ExMerge only supports the ANSI PST Format which has a 2GB limit ExMerge will not warn you if you export a large mailbox and the PST goes over 2GB, it will just corrupt the PST and you won’t know it until you try to open the PST at a later date and find that its corrupt and the data is gone ANSI PST's actually start to degrade at about 1.8GB so I would not go over that limit in order to ensure the data is recoverable To overcome this, check out Lucid8's DigiScope http://www.lucid8.com/product/digiscope.asp since it supports UNICODE PST files up to 20GB and can export data from raw offline EDBs for Exchange 2010, 2007, 2003, 2000 and 5.5. Lots of other cool features and capabilities in DigiScope as well. Here is a quick flash demo http://l8downloads.com/FLASH/DSWEB/DigiScope_Overview.html Troy Werelius www.Lucid8.com

So should I run a CHKSDK on the hard drive to see if there are any hardware issues? We inherited this server and of course someone set it all up on the OS drive array.

So everything is on one drive or? Maybe outline your system setup for me i.e. hardware, # drives, internal external OS etc I would probably look at the event logs first to ensure all is well hardware Then ensure you have a good DR based backup in case things take a turn for the worse Then if the event logs are clear run a CHKDSK because I have seen hardware on the edge die and things get worse by doing a CHKDSK. Not saying that will happen, but since you inherited the system and things are in an unknown state I would check the health of the system all the way around, ensure you have a backup to roll back to etc Troy Werelius www.Lucid8.com

Yes everything is on one drive array. Older IBM rack server. RAID 1 drive array (internal). Windows Server 2003 Standard (up to date SP). The event logs do not indicate anything about hardware failures. I will pull a complete DR backup of the machine. Would a built-in Windows backup be good enough or do I need to use Backup Exec?

Whatever backup you feel most comfortable with, i.e. if the machine implodes what backup are you going to trust to recover from? I assume that you followed the other steps above, i.e. cleared out the destination directories of older logs, EDB etc Have you looked all all errors and warnings in the application log as well? Do you have lots of disk space on the LUN you are telling BE to recover to? Remember that it could be the database you are trying to recover is the problem as well Does the Database you are restoring to RSG still exist i.e. is it still in operation on this server or? Troy Werelius www.Lucid8.com

Hi Marty, Please run the Eseutil /mh and post the results here. Eseutil /M File Dump Mode http://technet.microsoft.com/en-us/library/aa997795(EXCHG.65).aspx If it is dirty shutdown, you may need to run the eseutil /p. Eseutil /P Repair Mode http://technet.microsoft.com/en-us/library/aa996773(EXCHG.65).aspx Frank Wang

Microsoft(R) Exchange Server Database Utilities Version 6.5 Copyright (C) Microsoft Corporation. All Rights Reserved. Initiating FILE DUMP mode... Database: c:\program files\exchsrvr\mdbdata\priv1.edb File Type: Database Format ulMagic: 0x89abcdef Engine ulMagic: 0x89abcdef Format ulVersion: 0x620,11 Engine ulVersion: 0x620,11 Created ulVersion: 0x620,9 DB Signature: Create time:04/11/2006 03:15:46 Rand:2126350 Computer: cbDbPage: 4096 dbtime: 938150156 (0x37eb090c) State: Dirty Shutdown Log Required: 54437-54440 (0xd4a5-0xd4a8) Streaming File: Yes Shadowed: Yes Last Objid: 156507 Scrub Dbtime: 0 (0x0) Scrub Date: 00/00/1900 00:00:00 Repair Count: 0 Repair Date: 00/00/1900 00:00:00 Old Repair Count: 0 Last Consistent: (0xD434,278B,1A5) 12/23/2009 08:45:20 Last Attach: (0xD434,2794,1DB) 12/23/2009 08:47:49 Last Detach: (0x0,0,0) 00/00/1900 00:00:00 Dbid: 1 Log Signature: Create time:04/11/2006 02:43:34 Rand:98257 Computer: OS Version: (5.2.3790 SP 2) Previous Full Backup: Log Gen: 54294-54298 (0xd416-0xd41a) - OSSnapshot Mark: (0xD41B,8,16) Mark: 12/22/2009 18:51:01 Previous Incremental Backup: Log Gen: 0-0 (0x0-0x0) Mark: (0x0,0,0) Mark: 00/00/1900 00:00:00 Current Full Backup: Log Gen: 0-0 (0x0-0x0) Mark: (0x0,0,0) Mark: 00/00/1900 00:00:00 Current Shadow copy backup: Log Gen: 0-0 (0x0-0x0) Mark: (0x0,0,0) Mark: 00/00/1900 00:00:00 cpgUpgrade55Format: 0 cpgUpgradeFreePages: 0 cpgUpgradeSpaceMapPages: 0 ECC Fix Success Count: none Old ECC Fix Success Count: none ECC Fix Error Count: none Old ECC Fix Error Count: none Bad Checksum Error Count: none Old bad Checksum Error Count: none Operation completed successfully in 3.0 seconds.

At this point I trust the Windows backup more than BE. All files in the MDBDATA folder are current. No warnings or errors in the application log besides the ones already mentioned. Plenty of disk space. Very good point. Yes the database does still exist. I have been restoring to it from old backups for the last couple of weeks and then exporting the mailboxes.

Since this was restored from online backup the database is in dirty shutdown mode. you need logs from Log Required: 54437-54440 1. what path did you specified for Restore.env prior to start restoring the database? by default it goes to c:\temp\folder if you find so then you need to follow certain steps to replay those logs to database. Let me know if you find that folderVinod |CCNA|MCSE 2003 +Messaging|MCTS|ITIL V3|

This DB is definitely in a dirty state and you may end up having to run a ESEUTIL /P as outlined in my original response step 4 and Step 5 of my last post HOWEVER ALL backups are in a dirty state when taken and upon restore they attempt to rollup and put them into a clean state, hence my concern over the 1022 which is an I/O based error. Even though you had previous backups that recovered fine to same destination this backup may be hitting a bad block upon restore or you developed a new HD issue at this time. This is reaching for sure but the 1022 I/O error is basically saying its having trouble accessing the data because of disk I/O related issues OR because the DB is really damaged. if you want to confirm if its system related or the backup of the files are corrupt Clean out the destination directory to ensure nothing is commingled from another backup upon restore Ensured there are no disk related errors i.e. bad blocks etc and for this you may want to run CHKDSK. Then restore the data again If you receive the same results then yes the DB itself contained corruption during the backup process or the backup process hiccuped at some point so I would then run a ESEUTIL /P against the database to see if its salvageable Troy Werelius www.Lucid8.com

I was restoring a handful of mailboxes, but not the entire store.

ok but in order to do that you had to restore the EDB and logs to the RSG. use the MS tolls to extract the specific mailboxes to PST or to merge into live exchange server Was this what you were doing or??? Troy Werelius www.Lucid8.com

When using backup exec it stages the entire database and restores mails/mailboxes back into the production store. /MartinExchange is a passion not just a collaboration software.

Honestly I am not sure of the actual process with this latest version of BackupExec I have been using. I was just using the Exchange agent to restore the mail for certain users from different tapes. I know in order for BE 11d to restore a single mailbox it has to load up the entire database first.

Well I want to ensure we are talking about the same thing here i.e. Are you using BE to do granular restores via their GRT technology Or are you using BE to restore the EDB, Logs etc to the RSG and then using MS tools to export to PST, recover etc? Troy Werelius www.Lucid8.com

Whatever BE 11d uses by default with the Exchange agent. Using Exmerge to export to .PST.

ok then yes my current comments stand i.e. to validate if this is a DB or DISK issue I would Clean out the destination directory to ensure nothing is commingled from another backup upon restore, i.e. remove all logs, EDBs etc from the RSG path Ensured there are no disk related errors i.e. bad blocks etc and for this you may want to run CHKDSK. Then restore the data again If you receive the same results then yes the DB itself contained corruption during the backup process or the backup process hiccuped at some point so I would then run a ESEUTIL /P against the database to see if its salvageable If you want to super validate this then repeat steps # 1 and 2 with a previously recovered database. If it works then its the database if not its the environment/disk etc Troy Werelius www.Lucid8.com

When I try to run eseutil /p it tells me that the database is not up to date. If I run eseutil /r to try and recover it and bring it up to date it tells me that the current log file is missing. I am thinking I should go back and run the restore process again. Sound logical?

Yes and follow steps 1-4 above as discussedTroy Werelius www.Lucid8.com

I tried that process and still end up with the same result. Then when I go to run ESEUTIL the repair option tells me the database is not up to date. The recovery option tells me that a log file is missing.

Yes BackupExec does use the c:\temp\ folder for making copies of the logs during the restore process. The log files numbers make no sense though. All of my log files start with E00...

Does it tell you which log file is missing? If the log is truly not there then you will need to run an eseutil /p Troy Werelius www.Lucid8.com

Here is the line about the log file: "Log Required: 54437-54440 (0xd4a5-0xd4a8)" But there are no log files named even close to that.

If the required logs are truly not there then you will need to run an eseutil /p Troy Werelius www.Lucid8.com

When I try to do that I get the following error: "Operation terminated with error -1022 <JET_errDiskIO, Disk IO error>

ok so this points to two possible issues; The disk that you are running this database on has some serious contention because of speed of the media or possibly the media is going bad which is causing an I/O issue. So what type of disk did you recover this DB onto? The database is seriously damaged and unrecoverable. To confirm the above I would try to move the files to another fast disk that you know is good and try again. You may even need to put this on another machine to do the repair since its possible the issue could also be within the controller but before we go there tell me about the disk this DB is on, i.e. type speed, history etc and also check your event logs to see if any Disk issues are being reported Troy Werelius www.Lucid8.com