0x80070003 error during tape backup

Gents, I tried to backup to tape Windows 2008 R2 System State/Bare Metal Recovery items and see the following error:

Triggering tape backup failed:

Error 2033: DPM encountered an error while performing an operation for D:\WindowsImageBackup\ on dpm-server

Error details: The system cannot find the path specified (0x80070003)

Recommended action: 1) Refer to the detailed error code in the description above. Retry this operation after the issue has been fixed.

2) If the error was due to insufficient resources, then it could be a transient failure and you should retry this operation after some time.

 

 I open R2 server in DPM 2010 console, choose System State/Bare Metal Recovery items on Recovery Tab -> (right click) Create recovery point -> Long term tape protection.

Consistency check is OK. I did it again, no errors.

I see some topics with the same error in this forum, but I don't see solution.

I checked remote R2 server, D:\WindowsImageBackup\ doesn't exist. But MTA share exists.

I see this issue with on R2 servers. These R2 servers were installed some time ago (1 month). Backup features were added after DPM agent installation.

Two another R2 servers don't have this issue and tape backup works fine for System State/Bare Metal Recovery. These servers were installed approx 6 moths ago.

 

 

Can anyone explain, what is wrong and how to fix that ?

 

October 12th, 2011 4:46pm

Hi,

Have you tried restoring that BMR backup to a network share to see if it restores OK ?    I suspect the problem might be that the link to the replica volume is missing - if you highlight the BMR data source in the PG, then click on "Click to view details"  link - is that dialog blank ?

Free Windows Admin Tool Kit Click here and download it now
October 12th, 2011 8:24pm

I tried to restore and see the following error:

DPM encountered an error while performing an operation for D:\WindowsImageBackup\ on dpm-server (ID 2033)

I can't restore any recovery point of these bad servers.

 

I opened 'View details', Replica Path is existing:

 

How to fix that ?

ps. There is only 1% of free space on DPM disks (3 x 1Gb).

 

October 13th, 2011 10:05am

Hi,

For some reason, DPM believes that the systemstate backup was taken from D:\WindowsImagebackup at one time.  Is this server a member of a cluster ?

<snip>
I checked remote R2 server, D:\WindowsImageBackup\ doesn't exist.
>snip<

Can you verify the present configuration of where DPM is storing the WindowsImageBackup by looking the configuration file as noted in this technet article.   If you make a change, please stop protection of the systemstate, delete replica, and reprotect to see if that clears up the problem.

http://technet.microsoft.com/en-us/library/bb809015.aspx

Free Windows Admin Tool Kit Click here and download it now
October 29th, 2011 4:28pm

No, this server is not a member of cluster.

I deleted these members, all data and recreated Protection groups. The issue is gone.

!!! All backups of these members were deleted, it's really nice. It's a nice product, which reported about good backups, but you will know about it when you will restore servers ;-/

 

October 31st, 2011 9:55am

Hi,

I think the issue is that BMR protection does not use PSDatasourceConfig.xml for backup, but it effects restores if it is wrong, IE: points to a drive that is not present.  I will get this issue documented.  If you needed to get the restore done, we could have worked with you to get the data off the DPM replica.

Periodic test restores should be a part of your backup strategy to help discover issues like this.

Sorry for the inconvenience that issue has caused.

Free Windows Admin Tool Kit Click here and download it now
October 31st, 2011 7:07pm

Mike, I've got the same issue on another R2 server. Monthly tape backup is failed and I can't restore System State of this server.

How to fix that ?

Thanks

November 7th, 2011 1:16pm

Gents, is there any news ?

 

Free Windows Admin Tool Kit Click here and download it now
November 15th, 2011 10:53am

Hi,

Is the root cause the same - the PSDatasourceConfig.xml points to a non-existent volume on the protected server ?

BMR volume on the DPM server is really locked down from a security standpoint, so to navigate to the replica volume so you can copy out the BMR backup use this procedure.

1)      Download psexec.exe from www.sysinternals.com

2)      Run PSEXEC -s cmd.exe   (this will switch window to system context)

3)      Afterwards - type WHOAMI it should return:

                  nt authority\system

4)  Now CD to the replica volume under program files\Microsoft DPM\DPM\Volumes\replica\...

5)  Copy the BMR backup to a network location and share it out to restore from Winre.

November 15th, 2011 5:28pm

Mike,

Thank for answer. I can copy backup to another folder, but I would like to fix this issue.

At the moment monthly tape backup is not done and I can't restore server for specific date.

I can delete this server from Protection Group and add again, but will lose all daily backups for ~20 days (short protection).

Is there way to fix it ?

Why this issue happens ?

 

Free Windows Admin Tool Kit Click here and download it now
November 16th, 2011 10:41am

Hi,

The only "known" cause for ths issue is that the PSDatasourceConfig.xml on the protected server points to an invalid drive.  If that is not the case, please open a support incident with us so can investigate the issue further.

November 16th, 2011 2:44pm

I see the following lines in PSDatasourceConfig.xml:

- <DatasourceConfig>   <WriterId>8c3d00f9-3ce9-4563-b373-19837bc2835e</WriterId>   <Version>01.00.0000.00</Version>   <VssWriterInvolved>false</VssWriterInvolved>   <LogicalPath>Computer</LogicalPath>   <ComponentName>System Protection</ComponentName>   <FilesToProtect>C:\WindowsImageBackup\*</FilesToProtect>   <Size>51200</Size>   <UseDRWithCC>true</UseDRWithCC> </DatasourceConfig>   C:\ is system and boot volume. But C:\WindowsImageBackup doesn't exist.   We use DPM by Gold Cert Partner program, not Volume Licensing. I can't open a support case.    
Free Windows Admin Tool Kit Click here and download it now
November 16th, 2011 3:31pm

Hi,

 

OK - then this is a condition we have not seen before, and needs additional troubleshooting that cannot be performed in a forum setting.

November 17th, 2011 3:44pm

If you need any help or more details, please let me know. I'd like to fix this issue as soon as possible.

I noticed that this issue happens when we forgotten to insert a free tape into tape library. We can do it with 1-3 days delay. If it can help...

Thanks

Free Windows Admin Tool Kit Click here and download it now
November 17th, 2011 4:38pm

dpm 2010 last patch, win 2008 R2 stand alone server,colocation enabled , 16 tapes autoloader, no problems of space like the error says, no cluster, same issue of Alternate

my tape backup failed after 2 days i continued it from "job waiting tape" , and seems always when we forgot to insert free tapes , for example on weekend but we have time to do that for registry changes with 1-5 delay. here are  3 email alerts, always these 3 email, same PG, the longer (500GB , about 8 tapes), please see also the time of 3 alerts:

 

The following alert occurred at 22/11/2011 13:07:23.

Status: Active
Description: Backup to tape failed.
DPM failed to access the volume \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy3825\ on

. This could be due to
1) Cluster failover during backup or

2) Inadequate disk space on the volume

For more information, open DPM Administrator Console and review the alert details in the Monitoring task area.

The following alert occurred at 22/11/2011 13:07:25.

Status: Active
Description: Backup to tape failed.
DPM encountered an error while performing an operation for \\?\Volume{5c405246-170b-11dd-a996-806e6f6e6963}\ on

For more information, open DPM Administrator Console and review the alert details in the Monitoring task area.

The following alert occurred at 22/11/2011 13:07:57.

Status: Active
Description: Backup to tape failed.
DPM encountered an error while performing an operation for \\?\Volume{f4074a14-c556-11dc-8773-000c29c7187e}\ on

For more information, open DPM Administrator Console and review the alert details in the Monitoring task area.

after that,  i see in managment menu,  expire tapes of last job beacuse tape job failed,  but if click on RUN JOB AGAIN i have always error that "no recovery point is present" , that is true, beacuse my RP can have only 1 week retention for the limit of vss data (64 very low  , 2012 fix this low limit?) , so ALL TAPE JOBS, long tape jobs of days is always losen, this is the real tape issue of dpm 2007 and 2010:

So the problem is the error , but the real problem is the NO tape job resume chance in colocation, i think (also i can't restart my dpm server with job waiting tape message, but this is less important)

bye and thanks

 

November 22nd, 2011 11:23pm

Lorenzo, can you please open a support case in support ?

Thanks

Free Windows Admin Tool Kit Click here and download it now
November 23rd, 2011 9:59am

OK i do that bye
November 24th, 2011 1:01am

sorry how can i open a support case for this case?

bye

Free Windows Admin Tool Kit Click here and download it now
November 24th, 2011 1:04am

If you bought DPM by Open Volume Subscription, you can open any number cases via web interface.

http://support.microsoft.com , see at the bottom "Contact a Support Professional by Email, Online, or Phone"

Software Assurance, enter Software Assurance code and email for that, choose 1 day - email.

 

November 24th, 2011 9:23am

Hi Mike,

I have exactly the problem your are describing, I get the same error message '0x80070003' tape jobs fails for system state.
When I look at the replica path I only see a blank dialog

How could I fix that?

 

Free Windows Admin Tool Kit Click here and download it now
December 1st, 2011 4:16pm

Hi

I think the issue is that BMR protection does not use PSDatasourceConfig.xml for backup, but it effects restores if it is wrong, IE: points to a drive that is not present. 

Check to see if the PSDatasourceConfig.xml is pointing to a non-existent drive letter, and if so, change it to a valid drive letter.

1. On the protected computer, open PSDatasourceConfig.xml in an XML or text editor. PSDatasourceConfig.xml is typically located at install path\Program Files\Microsoft Data Protection Manager\DPM\Datasources.

2. Under the System Protection section as shown below - Change the <FilesToProtect> value from present invalid drive letter IE: F: to the desired new locations drive letter IE: C: (do not change the folder name)


<ComponentName>System Protection</ComponentName>
<FilesToProtect>C:\WindowsImageBackup\*</FilesToProtect>


3. Save the file.

4. Stop BMR protection and delete the replica, then reprotect the servers BMR workload.

December 1st, 2011 5:51pm

Mike, Do you mean that DPM server should have the same drive letter as in the PSDatasourceConfig.xml or backup will be failed ?

 

I have issue with sql server. There is no enough free space on disk C:\ for full backup. I tried to change to C:, but backup was failed with error "No free space".

At the moment FilesToProtect is pointing to drive T:

<FilesToProtect>T:\WindowsImageBackup\*</FilesToProtect>

But DPM server doesn't have drive T:\.

 

Free Windows Admin Tool Kit Click here and download it now
December 2nd, 2011 9:46am

Hi,

I'm sorry, I thought your problem was for BMR like the others.  I have not seen this before for systemstate - please open a support case for complete investigation.

December 2nd, 2011 9:24pm

Installed DPM2012. I see the same issues as described in the first post.

Gents, WHEN it will be fixed ?

Free Windows Admin Tool Kit Click here and download it now
July 3rd, 2012 5:19pm

Hi,

As stated earlier in this post, we cannot troubleshoot this particular problem in a forum, please open a support incident so we can investigate your specific problem.

July 4th, 2012 12:24pm

Hello,

had the same issue and for me Mikes solution was helpful. Changed the nonexisting path to an exisiting , stopped protection, deleted replicas, reprotect, everything working.

Reason for the wrong path was the server being a part of an SQL Cluster, and initially it seems to happen that the largest drive is taken for the "filestoprotect" entry. If the cluster node is changed this drive is no longer accessible. The automatic choice of this path should better not include cluster volumes ...

Thank you.

Greetings, Frank

Free Windows Admin Tool Kit Click here and download it now
July 18th, 2012 2:05pm

Hi,

I had similar issue (Error details: The system cannot find the path specified (0x80070003)) during tape backup for BMR protection, and Mike solution did solve the problem. Thanks.

When I looked into PSDatasourceConfig.xml file, FileToProtec was pointing to an non-existing drive:<FilesToProtect>F:\WindowsImageBackup\*</FilesToProtect>

However, can someone explain me how does it come to have the FileToProtec  pointing suddenly to an non-existing drive letter, while tape backup was working fine before for BMR protection?

Why does it suddenly stop working?

Regards,

Evan

August 7th, 2014 9:34am

Hi,

This usually occurs when protecting a cluster node and the DPM agent is installed when it owns the clustered disks.  DPM agent install selects the drive with the most free space which in many cases is the clustered volume. later that volume is failed over to another node and suddenly the drive letter is not valid.

This issue was fixed in DPM 2012 UR3.

Issue 7

A bare metal recovery fails in certain situations.

Free Windows Admin Tool Kit Click here and download it now
August 7th, 2014 3:49pm

Hi,

I have the same issue on two machines in a PG that contains three, the other machine is fine. Every Saturday when the Backups to tape are run I get a Backup to tape failed for both Bare Metal Recovery and System State. Interestingly, the error details are slightly different for the two machines, what I get on one machine is "DPM encountered an error while performing an operation for D:\WindowsImageBackup\ on escalade.network.co.uk(ID 2033 Details: The system cannot find the path specified (0x80070003))"

However, the path that has been talked about is there in the PSDatasourceConfig.xml file <FilesToProtect>D:\WindowsImageBackup\*</FilesToProtect>, and that path on the physical drive.

The thing that's different on the other machine is the path is different ie C: and there is no C:\WindowsImageBackup on the machine at all but I get the same message, so they both can't be right.

What's really interesting is that on my machine that's working fine it also has a path C:\WindowsImageBackup in the PSDatasourceConfig.xml file, and that doesn't appear on the physical drive at all!!!

What I've generally done to get past the problem, is take SS and BMR out of the protection group and then add it back in and the problem goes away until the next week when it returns with the same two machines having the same problem. Clearly there's something awry, but I cannot fathom out what it is. Can anybody point me in the right direction?

Thanks

July 13th, 2015 11:28am

As I can see, this issue is repeated in some cases.

I opened a support case and MS support helped me fixing this issue.

I haven't seen this issue again with the latest version of DPM (keep fingers crossed ;)

I fixed this issue a lot of time ago and there is only a few details.

You can use that solution at your own risk.

Regarding the drive letter in the xml file:

DPM uses the drive with the most free space at the time the agent was installed, but does not ever change it after that.

The workaround below is broken into two part. Part A has workaround for making the newer BMR backups recoverable. The part B would make even the older BMR backups as recoverable.

Please create a backup about the DPMDB first!

http://technet.microsoft.com/en-us/library/ff399388.aspx


On the DPM server, open an elevated command prompt window, type cd <system drive>:\Program Files\Microsoft DPM\DPM\bin, and then type DPMBackup.exe -db and press ENTER.

Then follow these steps:

-- A.

-- 1. Run the below SQL query on the DPMDB database for the particular Protection Server with Server Name (e.g. below whms0135) and get the Physical Replica Id and DatasourceId.

Select D.PhysicalReplicaId, A.DataSourceId, C.DatasourceSequenceNumber, A.ProtectedGroupId, A.DataSourceName, A.ApplicationPath

from dbo.tbl_IM_DataSource A

join tbl_AM_Server B

      on A.ServerId = B.ServerId

join tbl_PRM_DatasourceConfigInfo C

      on A.DataSourceId = C.DatasourceId 

join tbl_PRM_LogicalReplica D

      on D.DatasourceId=A.DatasourceId

where B.ServerName like '%whms0135%'and

A.AppId = '8C3D00F9-3CE9-4563-B373-19837BC2835E'

-- 2. Open a Command window as Administrator and run the below command and get the volume drive letter based directory. For e.g. if the directory

--    contains D-Vol then note down the "D" drive. There should be only Drive letter based directory present. If there are

--    multiple then see which one has WindowsImageBackup folder underneath and note that drive letter and remove other directories. Only

--    one of them should be having WindowsImageBackup folder underneath. In case of doubt please contact PG before proceeding.

dir "%ProgramFiles%\Microsoft System Center 2012\DPM\DPM\Volumes\Replica\Non Vss Datasource Writer\vol_<PhysicalReplicaId>\<DatasourceId>\Full"

-- 3. Use that Drive letter noted above and put into the PSDatasourceConfig.xml on the corresponding PS Server under FilesToProtect node for System    

      Protection. Make sure the volume with the drive letter exists.

-- 4. Restart the DPMRA service

-- 5. Take an adhoc BMR backup.

-- 6. Try recovery from the latest BMR backup which should succeed.

-- B. For making the older BMR backup (which were failing during BMR recovery)to be also recoverable

-- 1. Run the following SQL query after substituting the values for DatasourceId

--    and the drive letter as fetched above in the step A.1.

update tbl_RM_RecoverableObjectFileSpec

Set Path = '<DriveLetter>:\WindowsImageBackup\'

where RecoverableObjectFileSpecid in

(

Select B.RecoverableObjectFileSpecId

from dbo.tbl_RM_RecoverableObject A

join dbo.tbl_RM_RecoverableObjectFileSpec B

ON B.RecoverableObjectId = A.RecoverableObjectId

join dbo.tbl_RM_RecoverySource C

ON B.DataSetId=C.DataSetId

where C.IsValid = 1 and

B.DatasetId  in

      (Select B.DataSetId

            from dbo.tbl_PRM_LogicalReplica A

            join dbo.tbl_RM_ReplicaDataset B

            on A.ReplicaId=B.ReplicaId

            where A.DataSourceId = '<DatasourceId>'

      )

)

-- 2. Restart the DPM UI and try out the BMR recovery from the older BMR backup that was not working and it should work now.

Free Windows Admin Tool Kit Click here and download it now
July 13th, 2015 12:02pm

Hi,

Thanks for the info. One thing you've said which got a bell ringing in my head. The two machines that I'm having issues with, at least one has been rebuilt from scratch (my colleague can't remember what he did with the other but it could have been). What I can say with cartainty, is the one that was rebuilt was rebuilt with the same name. Could this have implications?? If it does, would removing it from the protection Group, renaming the machine and then re-inserting in the protection group fix the issue? Just a thought.

July 20th, 2015 11:52am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics