DPM 2012 R2 backup Causes Redirected CSV IO on SOFS Cluster.

Hi, I have a Scale out Storage Spaces Server with 2 nodes, and a 10 node 2012 R2, Hyper-V cluster using this via SMB3.0

I also have installed a DPM2012 R2 backup server.

the DPM agent is installed on all nodes of all servers and I have followed the pre-requisite from Microsoft for setting up DPM backup of SMB Hyper-V machines.

The DPM backups all work fine. but occasionaly I get these errors on the SOFS cluster.

Cluster Shared Volume 'Volume3' ('Cluster Disk 4') has entered a paused state because of '(c0130021)'. All I/O will temporarily be queued until a path to the volume is reestablished.

I really thought this issue had been resolved in this revision, this doesn't seem to cause any issues with my VM's that I can notice. and all DPM backups are working fine, but it still causes me concern.

has anyone else seen this or have any suggestions what I can try to resolve.

Regards

Mark Green


December 18th, 2013 1:15pm

Hi Mark,

As you can figure out from the error message, it seems that the connection to the volume get's lost when you get this error.

Now this might be happening when you have backups in progress but the starting point will be to check the cluster network configuration as per me.

Regards,

Siddharth

Free Windows Admin Tool Kit Click here and download it now
December 31st, 2013 6:26am

I'm seeing this as well and I have 40GB InfiniBand between the Hyper-V Cluster and the SOFC.

Not seeing any issues related to it though yet.

Jas :)

December 31st, 2013 4:43pm

Hi Mark,

Have you looked at this article yet? Although this was written for Pre Windows 2012 Servers so redirected mode should not be a factor , however the configuration part for network would still apply.

http://support.microsoft.com/default.aspx?scid=kb;en-US;2473194

Regards,

Siddharth

Free Windows Admin Tool Kit Click here and download it now
January 2nd, 2014 6:07am

The issue I am getting is this one, as reported in DPM2012 SP1, which was resolved by installed the KB articles as mentions, but I am running 2012 R2 and I am seeing exactly the same issue.

http://blogs.technet.com/b/dpm/archive/2013/04/29/support-tip-hyper-v-hosts-fail-and-log-event-id-5120-when-being-backed-up.aspx

February 3rd, 2014 11:50am

apparently this has been fixed in UR1 for DPM2012 R2.  I will update and confirm.

regards

MArk

Free Windows Admin Tool Kit Click here and download it now
February 3rd, 2014 1:25pm

I've got UR1 for DPM2012 R2 and still having the same issue. Occasionally it's pausing so long that one or more VMs stops responding
February 21st, 2014 3:49pm

We also encounter this issue. We use Windows Server 2012 R2 and SCVMM 2012 R2 (with RU1). Be carefull with this issue, because it can cause serious issues. Btw, note that Windows Server 2012 R2 used Direct I/O instead of Redirected I/O.

If you can't find a full fix as we are in right now, there are two things that might offer a work-around for you:

  1. Disabled ODX (if your storage system does not support it):
    Deploy Windows Offloaded Data Transfers
    http://technet.microsoft.com/en-us/library/jj200627.aspx
     
  2. Serialize virtual machine backups per node
    Migrate to a hardware VSS provider
    http://technet.microsoft.com/en-us/library/hh758027.aspx

The second option works best, because this issue mostly occurs when you run a backup of many VMs at once. It it not a full fix and makes you backup windows much longer, but can avoid you other problems. Also keep a close eye on this link:

Recommended hotfixes and updates for Windows Server 2012 R2-based failover clusters
http://support.microsoft.com/kb/2920151

Free Windows Admin Tool Kit Click here and download it now
March 29th, 2014 12:51am

Hi apparently there are some hotfixes coming out today, that should resolve this issue on 2012 and 2012R2.

ill keep everyone posted.

regards

Mark

April 8th, 2014 5:09pm

Hi apparently there are some hotfixes coming out today, that should resolve this issue on 2012 and 2012R2.

ill keep everyone posted.

regards

Mark


What hotfixes? I asume KB2919355. Marked as answer; is it confirmed that this update solved th
Free Windows Admin Tool Kit Click here and download it now
April 15th, 2014 4:25pm

Hi,

I had the same issue with IBM Storage (DS3524 and V3700).

Problem in this case is, that by default ODX is enabled in the server. Then, when backup runs, Server is trying to use first ODX, and because the mentioned Storages does not support it, the Server runs into timeout and CSV fails.

Solution in my case:

disable ODX on all Hyper-V-Nodes, shhould be helpful in your case too, because the Storage Server has no ODX as a "receiver" built in, only as requestor.

Disable with:

Set-ItemProperty hklm:system\currentcontrolset\control\filesystem -Name "FilterSupportedFeaturesMode" -Value 1
Kind regards, David


April 17th, 2014 12:50pm

Hi,

I had the same issue with IBM Storage (DS3524 and V3700).

Problem in this case is, that by default ODX is enabled in the server. Then, when backup runs, Server is trying to use first ODX, and because the mentioned Storages does not support it, the Server runs into timeout and CSV fails.

Solution in my case:

disable ODX on all Hyper-V-Nodes, shhould be helpful in your case too, because the Storage Server has no ODX as a "receiver" built in, only as requestor.

Disable with:

Set-ItemProperty hklm:system\currentcontrolset\control\filesystem -Name "FilterSupportedFeaturesMode" -Value 1
Kind regards, David


David,

We also disabled ODX. It seems to help a bit, but in our case it does not solve the problem. The pause issue is still there. I agree it makes sense you should disable it if your storage vendor does not support it.

Free Windows Admin Tool Kit Click here and download it now
April 22nd, 2014 11:32am

Hi,

Sorry to hear that disabling ODX is not the solution. Did you had any success with any or a combination of the steps written above (ODX, Serialize, Updates)?

Kind regards, David

April 30th, 2014 6:17pm

We are also seeing CSVs going into a paused state during backups on our 2012 R2 cluster. Disabling ODX made no difference, and I can confirm update KB2919355 has not fixed the issue.

Frustrating as this was an issue in 2012 which was fixed. Although to be fair in 2012 it was taking the CSV completely offline for us. In R2 it doesn't seem to be having any obvious impact.

 
Free Windows Admin Tool Kit Click here and download it now
May 2nd, 2014 10:56am

I can confirm also that KB2919355 (and all other generally available updates as of 5/2/14) haven't fixed the issue. Still getting the error. ODX disabled as well.

May 2nd, 2014 6:19pm

hi,

what about the fix to this issue? i use windows server 2012 r2 cluster with veeam backup and i get the same error...i can say that the VM's does not enter into pause mode and they stay on the host but the CSV change host

THX


  • Edited by Avi G Saturday, May 10, 2014 6:17 PM
Free Windows Admin Tool Kit Click here and download it now
May 9th, 2014 10:52pm

Hi,

I have the same issue.

5node cluster with 2012r2 hyperv-v servers + iSCSI storage + DPM 2012R2 on VM into this cluster. 

Only error Event ID 5120 or 5217.

All VMs WinServers are OK, but sometimes I have trouble with disks of my FreeBSD VMs on this CSV.

My friend talk about it :  "this is property - not issue"

M.


  • Edited by MIMIozo Thursday, May 29, 2014 12:06 PM
May 29th, 2014 3:05pm

So, any confirmed fixes?

We'r still having same issue.

Free Windows Admin Tool Kit Click here and download it now
October 3rd, 2014 1:40pm

This is not my post. But we seem to have the same issue.

Finally, since Update Rollup 3 for SCDPM 2012 R2, the backup seems to operate much better.

October 4th, 2014 10:50pm

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics