Hi,

We've been using DPM for many years and have now upgraded to DPM 2012 SP1 Rollup 7. We can happily backup our Windows Server 2012 R2 deduped data volumes and it works fine.

But, when we try to then do end of Month backup on to a Tape, it suddenly fails. The error message we get is :

"The Operation failed since DPM was unable to access the file/folder \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy33533\4876e0d9-0fbc-4b13-b8cd-a76a054e0eb3\Full\HappyDays\SmartDoc3.doc on  DPM01.domain.local. This could be due to the file/folder being corrupt or the file system on the volume being corrupt. (ID 2036 Details: The file or directory is corrupted and unreadable (0x80070570))"

I can confirm that there are no errors on the file system or file corruption as reported by Chkdsk and also as we can open/write to the same files both, from original disk location and DPM disk backup.

Yet, we can backup the same source files that it complains are corrupt if we disable data de-duplication on that volume?

Anyone come across this issue or can help with suggestion?

Kind regards,

Luthfur

• Moved by Mike JacquetMicrosoft employee, Moderator Friday, October 03, 2014 8:21 PM

Hi,

FYI - DPM 2012 Sp1 does not officially support Windows 2012 R2 workloads, and DPM 2012 Sp1 is not supported on Windows server 2012 R2 operating system.

Most likely this is occurring because the DPM server is running on a Windows 2012 (Not R2) server and the Dedup data structures in Windows 2012 R2 are not able to be read by a Windows 2012 server.

Workaround 1:  Create a dummy folder on the protected Windows 2012 R2 volume.  Modify the DPM protection group and expand out the protected volume.  Deselect the dummy folder so it is not protected and save the change. That will force DPM to protect the volume in a non-dedup state and future tape backups and recoveries from NEW recovery points will succeed.  Down side is you need replica to be the same size as the non-dedup data size.

Workaround 2: Upgrade the DPM 2012 Sp1 server to DPM 2012 R2, then make a backup copy of the DPMDB.  Un-install DPM 2012 R2.  Perform a Windows 2012 to Windows R2 upgrade.  Re-install DPM 2012 R2, restore DPMDB, run dpmsync -sync - then ruin CC.  Now future tape backups and restores from all past RP's will

Hi,

We are running into the same problem.  It seems this is not a versioning issue.  We are running DPM 2012 R2 on Windows Server 2012 R2 and trying to backup a Server 2012 R2 deduped volume to tape as well and are getting the same error about corrupted files.  This causes the tape job to fail at that point which is a horrible process.  Tape is supposed to be long term storage.  Why you would fail a job and not continue the backup over 1 file? This is just plain bad.  This makes our backup solution unreliable at best.  You know what they call a network admin without a good backup? Unemployed! 

On to trying to find a resolution.  Is it possible to run chkdsk on the DPM volume in question?  Not sure if this is of any help but here is the snippet from the DPMRA log on the DPM server when the error is encountered.

11AC        1C18        11/21        16:14:27.114        18        filereader.cpp(142)        [00000000045763F0]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        WARNING        CFileReaderBase
Debug Information Start ----------------------------------------------------

11AC        1C18        11/21        16:14:27.114        18        filereader.cpp(143)        [00000000045763F0]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        WARNING        Failed
on file \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy753\0fb754fd-014f-4ebf-a3ae-c508b25efbc8\Full\Data\LPPR2014\mdppt2014.vdb3
(handle: 0xb90, OwnHandle: true, IsDir: false, FileSize: 10857472 bytes)

11AC        1C18        11/21        16:14:27.114        18        filereader.cpp(149)        [00000000045763F0]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        WARNING        File
properties (Attributes: 0x20, CreationTime: {0x6a5c4910,0x1cfcc30},
LastAccessTime: {0x6a5c4910,0x1cfcc30}, LastWriteTime: {0x3cedbc3e,0x1d00275},
FileSize: 10857472 bytes, ReparseTag:0x0)

11AC        1C18        11/21        16:14:27.114        18        filereader.cpp(151)        [00000000045763F0]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        WARNING        FileAccessFlags:
0x81020000, FileFlags: 0xa000000, FileDispositionFlags: 0x3,
FileShareModeForRead: 0x7, CaseSensitive: false, IsWinXP: false

11AC        1C18        11/21        16:14:27.114        18        filereader.cpp(157)        [00000000045763F0]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        WARNING        CFileReaderBase
Debug Information End ------------------------------------------------------

11AC        1C18        11/21        16:14:27.114        18        mtffile.cpp(1284)        [0000000004526A10]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        WARNING        Failed:
Hr: = [0x80070570] BackupRead failed on
file(\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy753\0fb754fd-014f-4ebf-a3ae-c508b25efbc8\Full\Data\LPPR2014\mdppt2014.vdb3)

11AC        1C18        11/21        16:14:27.122        18        dsmsendersubtaskbase.cpp(584)        [0000000000A32320]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        WARNING        Failed:
Hr: = [0x80070570] SendData() failed

11AC        1C18        11/21        16:14:27.122        18        dsmsubtaskbase.cpp(275)        [0000000000A32320]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        NORMAL        Hr:
= [0x80070570] CDsmSubTaskBase::ErrorCleanup: subtask state: 1

11AC        12AC        11/21        16:14:27.122        31        backupsubtask.cpp(650)        [0000000000A44F10]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        NORMAL        CBackupSubTask::CleanUp
[0000000000A44F10]

11AC        12AC        11/21        16:14:27.125        31        backupsubtask.cpp(204)        [0000000000A44F10]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        NORMAL        CBackupSubTask::GetFinalStatus
[0000000000A44F10]

11AC        12AC        11/21        16:14:27.127        31        aasubtask.cpp(906)        [0000000000A44F10]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        WARNING        <?xml
version="1.0"?>

11AC        12AC        11/21        16:14:27.127        31        aasubtask.cpp(906)        [0000000000A44F10]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        WARNING        <Status
xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd"
StatusCode="-2147023504" Reason="Error"
CommandID="RAReadDataset" CommandInstanceID="f7ba9b4a-5e74-45ea-bffa-7567b76e2d05"
GuidWorkItem="dfa25662-96f7-4d75-92ac-b19b73281c23"
TETaskInstanceID="50d635a0-cd50-4b1d-b867-6a7a36c9bbed"><ErrorInfo
xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd"
ErrorCode="998" DetailedCode="-2147023504"
DetailedSource="2"/><RAStatus><RAReadDataset
xmlns="http://schemas.microsoft.com/2003/dls/ArchiveAgent/StatusMessages.xsd"
BytesTransferred="17284595712"
NumberOfFilesTransferred="81512" NumberOfFilesFailed="0"
DataCorruptionDetected="false"/></RAStatus></Status>

11AC        0154        11/21        16:14:27.128        18        dsmsendersubtaskbase.cpp(400)        [0000000000A32320]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        NORMAL        CDsmSenderSubTaskBase
received session closed completion in CLEANUP state

11AC        1C18        11/21        16:14:27.128        05        fsmtransition.cpp(111)        [00000000009CF3C0]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        WARNING        Failed:
Hr: = [0x80072746] HasEventErrorCode: completion: 0x14, signature: 0xaabbcc00

11AC        1C18        11/21        16:14:27.128        29        mtabackupdatasetsubtask.cpp(1142)        [00000000009CECB0]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        WARNING        CMTABackupDatasetSubTask::ErrorOrCancelAtBackupLoop
=> Error At BackupLoop [00000000009CECB0]

11AC        1C18        11/21        16:14:27.128        29        mtasubtask.cpp(649)        [00000000009CECB0]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        WARNING        Failed:
Hr: = [0x80072746] CMTASubTask::UpdateStatus => hr

11AC        1C18        11/21        16:14:27.128        29        mtabackupdatasetsubtask.cpp(1219)        [00000000009CECB0]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        WARNING        CMTABackupDatasetSubTask::BackupFailed
=> [00000000009CECB0]

11AC        0154        11/21        16:14:27.129        05        fsmtransition.cpp(111)        [000000000451EAC0]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        WARNING        Failed:
Hr: = [0x80072746] HasEventErrorCode: completion: 0xb008, signature: 0xaabbcc00

11AC        0154        11/21        16:14:27.129        29        mtasubtask.cpp(381)        [0000000004524D40]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        WARNING        CMTASubTask::ErrorHandler
=> called [0000000004524D40]

11AC        0154        11/21        16:14:27.129        29        mtasubtask.cpp(649)        [0000000004524D40]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        WARNING        Failed:
Hr: = [0x80072746] CMTASubTask::UpdateStatus => hr

11AC        0154        11/21        16:14:27.130        29        mtasubtask.cpp(505)        [0000000004524D40]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        WARNING        <?xml
version="1.0"?>

11AC        0154        11/21        16:14:27.130        29        mtasubtask.cpp(505)        [0000000004524D40]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        WARNING        <Status
xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd"
StatusCode="-2147014842" Reason="Error"
CommandID="MTAPerformIO" CommandInstanceID="b260f67f-6345-456f-88b8-2da51146fa48"
GuidWorkItem="4f3d35f4-9111-4851-9667-ddd6a39c55d2"
TETaskInstanceID="50d635a0-cd50-4b1d-b867-6a7a36c9bbed"><ErrorInfo
xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="998"
DetailedCode="-2147014842"
DetailedSource="2"/></Status>

11AC        12AC        11/21        16:14:27.183        03        runtime.cpp(1376)        [00000000009646A0]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        FATAL        Subtask
failure, sending status response XML=[<?xml version="1.0"?>

11AC        12AC        11/21        16:14:27.183        03        runtime.cpp(1376)        [00000000009646A0]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        FATAL        <Status
xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd"
StatusCode="-2147023504" Reason="Error"
CommandID="RAReadDataset" CommandInstanceID="f7ba9b4a-5e74-45ea-bffa-7567b76e2d05"
GuidWorkItem="dfa25662-96f7-4d75-92ac-b19b73281c23"
TETaskInstanceID="50d635a0-cd50-4b1d-b867-6a7a36c9bbed"><ErrorInfo
xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd"
ErrorCode="2036" DetailedCode="-2147023504"
DetailedSource="2"><Parameter
Name="AgentTargetServer"
Value="DPM1.DOMAIN.LOCAL"/><Parameter Name="FileName"
Value="\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy753\0fb754fd-014f-4ebf-a3ae-c508b25efbc8\Full\Data\LPPR2014\mdppt2014.vdb3"/></ErrorInfo><RAStatus><RAReadDataset
xmlns="http://schemas.microsoft.com/2003/dls/ArchiveAgent/StatusMessages.xsd"
BytesTransferred="17284595712"
NumberOfFilesTransferred="81512" NumberOfFilesFailed="0"
DataCorruptionDetected="false"/></RAStatus></Status>

11AC        12AC        11/21        16:14:27.183        03        runtime.cpp(1376)        [00000000009646A0]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        FATAL        ]

11AC        0154        11/21        16:14:27.184        03        runtime.cpp(1376)        [00000000009646A0]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        FATAL        Subtask
failure, sending status response XML=[<?xml version="1.0"?>

11AC        0154        11/21        16:14:27.184        03        runtime.cpp(1376)        [00000000009646A0]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        FATAL        <Status
xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd"
StatusCode="-2147014842" Reason="Error"
CommandID="MTAPerformIO" CommandInstanceID="b260f67f-6345-456f-88b8-2da51146fa48"
GuidWorkItem="4f3d35f4-9111-4851-9667-ddd6a39c55d2"
TETaskInstanceID="50d635a0-cd50-4b1d-b867-6a7a36c9bbed"><ErrorInfo
xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="536872931"
DetailedCode="-2147014842"
DetailedSource="2"/></Status>

11AC        0154        11/21        16:14:27.184        03        runtime.cpp(1376)        [00000000009646A0]        50D635A0-CD50-4B1D-B867-6A7A36C9BBED        FATAL        ]

Hi,

Yes, you can safely run chkdsk against the replica volume, cancel any scheduled jobs for that data source before running.  We are reviewing our tape backup code to see if we can continue on failure like we do for disk to disk backups.

Hi,

Can you verify that there is no dedup corruption on the protected server.  DPM will synchronize corruption if present and that can lead to tape backup failures.

You can run Dedup Scrubbing Job with Full and ReadOnly flag for Dedup Volume on PS. Once this completes you can see the event log for any corruption reported. You can refer technet article for the same listed below.

Dedup Technet Article : http://blogs.technet.com/b/keithmayer/archive/2012/12/12/step-by-step-reduce-storage-costs-with-data-deduplication-in-windows-server-2012.aspx

Full Scrubbing Command : Start-DedupJob E: Type Scrubbing Full ReadOnly (Where E: is the Dedup Volume)

Mike,

We have been experiencing a similar issue since the last CU as best I can tell it was not an issue previously and has been an issue recently.  We however are running Windows 2012 not R2 on both the DPM and PS with dedupe enabled on the PS and have been backing up the deduped volumes on DPM 2012 R2 running on this same Windows 2012 non R2 DPM server for a long time without issue until recent CU's.

Thanks,

Dave

I forgot to add that I used PSEXEC console as system and accessed the Volumes within DPM's copy although on the PS there is no corruption as the folders, files, etc. that on the DPM copy does seem corrupt work fine on the PS.  So, it would appear that DPM is not copying corruption but potentially causing it on its own repository. 

One more thing I would like to bump the idea that a tape job should not fail on encountering any corrupt files only report some files are not backed up due to corruption as the disk to disk job does.

Thanks again,