DPM2012: DPM service on secondary DPM Server, crashing when backing up SS and BMR of primary DPM Server

Whenever i backup SS and BMR of Primary DPM Server with secondary DPM Server, dpm service crashes on secondary dpm server. When i check backup log on primary DPM server it says completed successfully. But when i check on secondary DPM Server i see there is data on disk, but it is not written when was backup taken. Of course it says backup failed.

Error Description:

The description for Event ID 999 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

An unexpected error caused a failure for process 'msdpm'.  Restart the DPM process 'msdpm'.

Problem Details:

<FatalServiceError><__System><ID>19</ID><Seq>2162</Seq><TimeCreated>18.1.2013 10:28:05</TimeCreated><Source>DpmThreadPool.cs</Source><Line>163</Line><HasError>True</HasError></__System><ExceptionType>FormatException</ExceptionType><ExceptionMessage>Input string was not in a correct format.</ExceptionMessage><ExceptionDetails>System.FormatException: Input string was not in a correct format.

   at System.Text.StringBuilder.AppendFormat(IFormatProvider provider, String format, Object[] args)

   at System.String.Format(IFormatProvider provider, String format, Object[] args)

   at Microsoft.Internal.EnterpriseStorage.Dls.Trace.TraceProvider.Trace(TraceFlag flag, String fileName, Int32 fileLine, Guid* taskId, Boolean taskIdSpecified, String formatString, Object[] args)

   at Microsoft.Internal.EnterpriseStorage.Dls.Trace.TraceProvider._TraceMessage(TraceFlag flag, String fileName, Int32 fileLine, String formatString, Object[] args)

   at Microsoft.Internal.EnterpriseStorage.Dls.WriterHelper.SystemStateWriterHelper.RenameBMRReplicaFolderIfNeeded(String roFileSpec)

   at Microsoft.Internal.EnterpriseStorage.Dls.WriterHelper.SystemStateWriterHelper.ValidateROListOnPreBackupSuccess(Message msg, RADataSourceStatusType raDatasourceStatus, Guid volumeBitmapId, List`1&amp; missingVolumesList, ReplicaDataset&amp; lastFullReplicaDataset, ROListType&amp; roList)

   at Microsoft.Internal.EnterpriseStorage.Dls.Prm.ReplicaPreBackupBlock.ValidateROList(Message msg, RADataSourceStatusType raDatasourceStatus, Guid datasetId)

   at Microsoft.Internal.EnterpriseStorage.Dls.Prm.ReplicaPreBackupBlock.RAPreBackupSuccess(Message msg)

   at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.Fsm.Transition.Execute(Message msg)

   at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.Fsm.Engine.ChangeState(Message msg)

   at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.TaskInstance.Process(Object dummy)

   at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.FsmThreadFunction.Function(Object taskThreadContextObj)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)

   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)</ExceptionDetails></FatalServiceError>

the message resource is present but the message is not found in the string/message table

January 18th, 2013 1:46pm

This is content of crash dump:

005C 183C 01/16 16:02:51.363 68 DpmThreadPool.cs(121)  F4E26FDA-0091-4957-B027-E615C2DE9E61 WARNING Caught unhandled exception : System.FormatException: Input string was not in a correct format.
005C 183C 01/16 16:02:51.363 68 DpmThreadPool.cs(121)  F4E26FDA-0091-4957-B027-E615C2DE9E61 WARNING    at System.Text.StringBuilder.AppendFormat(IFormatProvider provider, String format, Object[] args)
005C 183C 01/16 16:02:51.363 68 DpmThreadPool.cs(121)  F4E26FDA-0091-4957-B027-E615C2DE9E61 WARNING    at System.String.Format(IFormatProvider provider, String format, Object[] args)
005C 183C 01/16 16:02:51.363 68 DpmThreadPool.cs(121)  F4E26FDA-0091-4957-B027-E615C2DE9E61 WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.Trace.TraceProvider.Trace(TraceFlag flag, String fileName, Int32 fileLine, Guid* taskId, Boolean taskIdSpecified, String formatString, Object[] args)
005C 183C 01/16 16:02:51.363 68 DpmThreadPool.cs(121)  F4E26FDA-0091-4957-B027-E615C2DE9E61 WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.Trace.TraceProvider._TraceMessage(TraceFlag flag, String fileName, Int32 fileLine, String formatString, Object[] args)
005C 183C 01/16 16:02:51.363 68 DpmThreadPool.cs(121)  F4E26FDA-0091-4957-B027-E615C2DE9E61 WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.WriterHelper.SystemStateWriterHelper.RenameBMRReplicaFolderIfNeeded(String roFileSpec)
005C 183C 01/16 16:02:51.363 68 DpmThreadPool.cs(121)  F4E26FDA-0091-4957-B027-E615C2DE9E61 WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.WriterHelper.SystemStateWriterHelper.ValidateROListOnPreBackupSuccess(Message msg, RADataSourceStatusType raDatasourceStatus, Guid volumeBitmapId, List`1& missingVolumesList, ReplicaDataset& lastFullReplicaDataset, ROListType& roList)
005C 183C 01/16 16:02:51.363 68 DpmThreadPool.cs(121)  F4E26FDA-0091-4957-B027-E615C2DE9E61 WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.Prm.ReplicaPreBackupBlock.ValidateROList(Message msg, RADataSourceStatusType raDatasourceStatus, Guid datasetId)
005C 183C 01/16 16:02:51.363 68 DpmThreadPool.cs(121)  F4E26FDA-0091-4957-B027-E615C2DE9E61 WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.Prm.ReplicaPreBackupBlock.RAPreBackupSuccess(Message msg)
005C 183C 01/16 16:02:51.363 68 DpmThreadPool.cs(121)  F4E26FDA-0091-4957-B027-E615C2DE9E61 WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.Fsm.Transition.Execute(Message msg)
005C 183C 01/16 16:02:51.363 68 DpmThreadPool.cs(121)  F4E26FDA-0091-4957-B027-E615C2DE9E61 WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.Fsm.Engine.ChangeState(Message msg)
005C 183C 01/16 16:02:51.363 68 DpmThreadPool.cs(121)  F4E26FDA-0091-4957-B027-E615C2DE9E61 WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.TaskInstance.Process(Object dummy)
005C 183C 01/16 16:02:51.363 68 DpmThreadPool.cs(121)  F4E26FDA-0091-4957-B027-E615C2DE9E61 WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.FsmThreadFunction.Function(Object taskThreadContextObj)
005C 183C 01/16 16:02:51.363 68 DpmThreadPool.cs(121)  F4E26FDA-0091-4957-B027-E615C2DE9E61 WARNING    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
005C 183C 01/16 16:02:51.363 68 DpmThreadPool.cs(121)  F4E26FDA-0091-4957-B027-E615C2DE9E61 WARNING    at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
005C 183C 01/16 16:02:51.363 68 DpmThreadPool.cs(121)  F4E26FDA-0091-4957-B027-E615C2DE9E61 WARNING    at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)
005C 183C 01/16 16:02:51.379 68 DpmThreadPool.cs(149)  F4E26FDA-0091-4957-B027-E615C2DE9E61 WARNING Attempting to write error to the event log Input string was not in a correct format.
005C 183C 01/16 16:02:51.379 02 EventManager.cs(143)  F4E26FDA-0091-4957-B027-E615C2DE9E61 NORMAL Logging event to NT log from DpmThreadPool.cs(163): FatalServiceError
005C 183C 01/16 16:02:51.394 68 DpmThreadPool.cs(170)  F4E26FDA-0091-4957-B027-E615C2DE9E61 WARNING Attempting to invoke Watson
005C 183C 01/16 16:02:51.410 09 everettexception.cpp(761)  F4E26FDA-0091-4957-B027-E615C2DE9E61 CRITICAL Exception Message = Input string was not in a correct format. of type System.FormatException, process will terminate after generating dump

Free Windows Admin Tool Kit Click here and download it now
January 18th, 2013 4:13pm

Ditto...

Did you ever get this one resolved? It's been a few months with no response.

....

1)  023E339B-6482-447A-B243-D5334CE93250 WARNING    at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)
1E58 2060 03/13 01:31:09.475 68 DpmThreadPool.cs(149)  023E339B-6482-447A-B243-D5334CE93250 WARNING Attempting to write error to the event log Input string was not in a correct format.
1E58 2060 03/13 01:31:09.475 02 EventManager.cs(143)  023E339B-6482-447A-B243-D5334CE93250 NORMAL Logging event to NT log from DpmThreadPool.cs(163): FatalServiceError
1E58 2060 03/13 01:31:09.507 68 DpmThreadPool.cs(170)  023E339B-6482-447A-B243-D5334CE93250 WARNING Attempting to invoke Watson
1E58 2060 03/13 01:31:09.522 09 everettexception.cpp(761)  023E339B-6482-447A-B243-D5334CE93250 CRITICAL Exception Message = Input string was not in a correct format. of type System.FormatException, process will terminate after generating dump

I guess we'll wait until SP1 to get this one resolved... A client of mine has the habit of not isntalling nothing MS until is on SP1 at least. I'm starting to believe he's got a point.

Regards,

Fred

March 13th, 2013 5:01am

Unfotunatelly not, still looking for answer.
Free Windows Admin Tool Kit Click here and download it now
March 13th, 2013 7:19pm

Hello everybody,

I think we found what causes DPM service to crash. Look at the replica path for a problem server. You'll see something like C:\Program Files\Microsoft System Center 2012\DPM\DPM\Volumes\Replica\Non VSS Datasource Writer\vol_a5aadd1c-e19a-48e6-9745-92b6940b60f3\89bf78a5-9f99-47d4-aecb-6375a8cc8533\Full\C-Vol\

Remeber the "C" letter referring to volume here. Next move to protected server itself and find PSDataSourceConfig.xml file (Usually it resides under %ProgramFiles%\Microsoft Data Protection Manager\DPM\Datasources\). Inside that file find <FilesToProtect> tag, should looks like <FilesToProtect>D:\WindowsImageBackup\*</FilesToProtect>. Notice the volume here is "D". When volume letters in PSDataSourceConfig.xml and in the folder structure at DPM server differ, BMR backup will crash. Unfortunately, starting from DPM 2012, it also causes DPM service to crash.

You can safely change volume letter in PSDataSourceConfig to one used in folder name at DPM server and from now everything will run just fine.

You may ask, how is it possible for those to letters are different? Well, except manual changing, there is, actually, a chanse when it happens w/o anyone noticing:

You may have known that FilesToProtect tag useful for System State backups (please find more about it here: http://social.technet.microsoft.com/Forums/en-US/ba61b94c-f962-4cdb-99e9-caef17c27845/system-protection-system-cannot-find-path-jwindowsimagebackup-id-2033-0x80070003-cluster?forum=dpmssandbmrbackuphttp://technet.microsoft.com/en-us/library/bb809015.aspx,http://scug.be/scdpm/2009/06/30/scdpm-how-to-change-the-system-state-backup-location/http://blogs.technet.com/b/csstwplatform/archive/2011/03/24/dpm-backup-system-status-fail.aspx). When you install DPM agent, it populates this tag with volume which has maximum free space.

For example, lets assume we have a client with 50 GB free space at volume C: and with 100 GB free space at volume D:. We install DPM agent on that client. In that case, we will have following in PSDataSourceConfig.xml: <FilesToProtect>D:\WindowsImageBackup\*</FilesToProtect>. Then we add client into protection group at DPM server and create BMR backup. Replica path will look like \Full\D-Vol\. Later, we fill volume D: with files up to 20 GBs free left. So now we have 50 GB free at C: and 20 GB at D:. Then we uninstall DPM agent from this client and install it again. Since we have more free space at C: volume than at D:, <FilesToProtect> tag will point to volume C: now, but replica path at DPM server remain unchanged and contains D-Vol. After that, DPM server will crash every time we try to take BMR backup from this client.

January 11th, 2014 3:57pm

Thx for your help. I'll try it and let you know if that is solution.
Free Windows Admin Tool Kit Click here and download it now
January 12th, 2014 5:57am

Thank you!!! That was solution for me
January 12th, 2014 6:51am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics