I have a 2012 Cluster with a CSV. I have the Dell Equallogic HIT v 4.5 installed on both nodes, as well as the SCDPM 2012 SP1 Agent. When I add a replica to DPM to back up one (or any) of the virtual machines on the cluster the vss writer for Hyper-V is put in a bad state and DPM gives me the following error:

The VSS application writer or the VSS provider is in a bad state. Either it was already in a bad state or it entered a bad state during the current operation. (ID 30111 Details: VssError:A function call was made when the object was in an incorrect state
for that function
 (0x80042301))

The following VSS errors are logged in the Eventlog of the Cluster Node:

Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr = 0x80070005, Access is denied.
. This is often caused by incorrect security settings in either the writer or requestor process.

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Name: System Writer
   Writer Instance ID: {754c4bfd-91e7-47b5-94a2-565196237f2a}

Things I have done:

• Remove DPM SP1 Agent and reinstall on both nodes in cluster
• Remove SCVMM SP1 Agent and reinsall on both nodes in cluster
• Remove HIT and reinstall on both nodes in cluster
• Restart DPM server
• Restart both nodes in cluster

Before the DPM agent puts the Hyper-V VSS Writer into a bad state I can create SmartCopies from the HIT Auto-Snapshot Monitor software, but as soon as the VSS Writer is in a bad state ASM no longer works:

Started at 3:32:26 PM
Gathering Information...
Phase 1: Checking pre-requisites... (3:32:26 PM)
Phase 2: Initializing Smart Copy Operation (3:32:26 PM)
Retrieving writer information
Phase 3: Adding Components and Volumes (3:32:28 PM)
Adding components to the Smart Copy Set
Adding volumes to the Smart Copy Set
Phase 4: Creating Smart Copy (3:32:28 PM)
Creating Smart Copy Set

An error occurred:

An error occurred during phase: Creating Smart Copy
    
    Exception from HRESULT: 0x800423F9.
 
Creating Smart Copy Set

An error occurred:

An error occurred during phase: Creating Smart Copy
    
    Exception from HRESULT: 0x800423F9.
 

An error occurred:

Writer 'Microsoft Hyper-V VSS Writer' reported an error: 'VSS_WS_FAILED_AT_THAW'. Check the application component to verify it is in a valid state for the operation.

An error occurred:

One or more errors occurred during the operation.  Check the detailed progress updates for details.

An error occurred:

Smart Copy creation failed.

Source:  Creating Smart Copy Set

An error occurred:

An error occurred during phase: Creating Smart Copy
    
    Exception from HRESULT: 0x800423F9.
 

An error occurred:

Writer 'Microsoft Hyper-V VSS Writer' reported an error: 'VSS_WS_FAILED_AT_THAW'. Check the application component to verify it is in a valid state for the operation.

An error occurred:

One or more errors occurred during the operation.  Check the detailed progress updates for details.
Error:  Exception from HRESULT: 0x800423F9

I also have two more 2-node 2012 Clusters that have the exact same software installed, but do not exhibit this issue.

Thanks!

Hi,

lets eliminate the hardware provider from the picture.  DPM 2012 Sp1 protecting Windows 2012 Hyper-V no longer requires serialization or the use of a hardware provider.

To bypass the hardware provider Add the following new KEY on all of the nodes of the Hyper-V cluster.
 
Windows Registry Editor Version 5.00
[Software\Microsoft\Microsoft Data Protection Manager\Agent\UseSystemSoftwareProvider]

Then please ensure you are on DPM 2012 Sp1 UR2 (4.0.3408.0) which does include one fix for Windows 2012 Hyper-V protection.

2802159 Description of Update Rollup 2 for System Center 2012 Service Pack 1
http://support.microsoft.com/kb/2802159

Finally install the latest Windows cluster resiliency fix below.

Update improves cluster resiliency in Windows Server 2012
http://support.microsoft.com/kb/2848344

Note This update replaces a previous resiliency improvements update that is described in KB 2838669
There are no additional configuration changes required after installing this update.

Then see if the problems go away or symptoms change.

Update rollup 2 is installed. KB284344 is installed on both nodes.

I have added the KEY UseSystemSoftwareProvider to both nodes (and restarted them after migrating VMs), and am still getting the same errors in DPM. I also created a fresh protection group to try backing up the VMs again.

The VSS application writer or the VSS provider is in a bad state. Either it was already in a bad state or it entered a bad state during the current operation. (ID 30111 Details: VssError:A function call was made when the object was in an incorrect state
for that function
 (0x80042301))

I am going to try removing the hardware provider completely from the Nodes and I will report back. This is getting more than frustrating.

Hi,

It seems like there is a problem with the Hyper-v writer, and DPM is the victim.  You want to open a support case and let the Windows Hyper-V support folks take some VSS traces. 

I have removed the hardware provider and still no luck. You are correct that the Hyper-V Vss writer seems to be the culprit. Here is my output from vssadmin list wrtiers:

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2012 Microsoft Corp.

Writer name: 'Task Scheduler Writer'
   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   State: [1] Stable
   Last error: No error

Writer name: 'VSS Metadata Store Writer'
   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   State: [1] Stable
   Last error: No error

Writer name: 'Performance Counters Writer'
   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   State: [1] Stable
   Last error: No error

Writer name: 'Microsoft Hyper-V VSS Writer'
   Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
   Writer Instance Id: {72b4583b-b5d4-4195-90b9-92cc34bb2644}
   State: [7] Failed
   Last error: Timed out

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {4ad02755-85cd-471e-8360-45545a58d802}
   State: [1] Stable
   Last error: No error

Writer name: 'Cluster Shared Volume VSS Writer'
   Writer Id: {1072ae1c-e5a7-4ea1-9e4a-6f7964656570}
   Writer Instance Id: {d9c7650f-24e8-4248-aed3-728abbce6929}
   State: [1] Stable
   Last error: No error

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {d1b4d6f2-40a8-48cd-bc30-6b15a60e4004}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {87d73261-c662-4db8-a5c6-6d9cbbc48e9a}
   State: [1] Stable
   Last error: No error

Writer name: 'BITS Writer'
   Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
   Writer Instance Id: {0113ad9f-591f-4aec-81c6-81d0d4082cea}
   State: [1] Stable
   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {fabf8bd5-3c2b-4bc3-836f-760501180f3d}
   State: [1] Stable
   Last error: No error

Writer name: 'Cluster Database'
   Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
   Writer Instance Id: {136a3871-dd86-4013-9d25-cde67045d6e8}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {779a54c2-8f89-4afb-aaea-7977b035650e}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {b4e65ba3-e75c-4b49-9fc5-1caf8945247d}
   State: [1] Stable
   Last error: No error

 

 I will log a support case. If I am provided with a resolution I'll post it back here.

We are also having the same issue, I was curious to see if you had resolved this yet?

We also have this problem, did you solve it?

Can you tell us how you solved it.

Having the same issue too, eager to get a solution! 

We solved the problem by disable the Qlogic driver of or fiber card that we don't use anymore and fiber is disconnected.

We also have this problem, did you solve it?

Please