Hi,

we have a big problem with Snapshots, which are done by DPM 2012r2 over the Hyper-V Manager.

Our System Center Data Protection Manager backs up our whole system. Some Virtual Machines only via a Cluster Network Snapshot over the Hyper-V Manager.

Now we have the problem, that some VMs still have Checkpoints/Snapshots from the DPM-Backup and we could not delete it. But it's necessary to delete it! If we click right on the Snapshopt/Checkpoint the options that are shown: Settings, Export, Rename and Help. There is no chance to click delete or remove!

In the eventlog is no information about that. In the moment it is very bad to reboot the Hosts or restart some cluster services! The VMs have to run and there is no time for downtime now.

Is it an error of the Hyper-v Manager? Or anyone a solution for this problem?

I hope that somebody could help me!

best regards

Huffelpaff

Hi,

Are you running DP 2012 R2 + UR3 and is your Hyper-v cluster running Windows 2012 R2 ?

There are some circumstances where DPM and Agent loose communication that can leave a snapshot behind.  We're working on fixing that issue in a future DPM update.  As a workaround - If you stop the DPMRA service on the node where the VM snapshot is stuck - that should delete the backup snapshot. 

Hi Mike! Any chance this fix gets into UR4?

Hi,

We just discovered this issue recently.  DPM 2012 R2 UR4 is code complete and is in final testing, so the fix won't make it in time for UR4.  I think it best to figure out why communications between DPM server and DPMRA seem to get lost and try to eliminate that to avoid more "backup in progress: hangs.

Any node or DPM sever stability issues lately that might effect / interrupt backups while in progress ?

I tried starting and stopping the DPMRA services on one of the hosts that has a VM with this issue and that workaround did NOT fix the leftover checkpoints.

So Huffelpaff please let me know if that solves your issue, as we are also having the same issue!

• Edited by Striker169 Friday, October 17, 2014 1:51 PM add more detail

Hi Mike,

thank you for thos awnser! I've been looking for DPMRA services are running. But on the hyper-v host and on the vms there is no DPMRA service running at the moment. All Operating Systems are on the lates patchlevel. so there ist no backup progress in the moment.

but I have another problem. there are snapshots of vms that can't be deleted! and this is very bad.

Today is a good change to restart the servers and the vms, I hope that would be helpfull.

I tried starting and stopping the DPMRA services on one of the hosts that has a VM with this issue and that workaround did NOT fix the leftover checkpoints.

So Huffelpaff please let me know if that solves your issue, as we are also having the same issue!

• Edited by Striker169 Friday, October 17, 2014 1:51 PM add more detail

Hi,

If the vm is not stuck showing backup on progress, then the condition is something different from what I originally thought.  DPM is not directly responsible for hyper-v snapshot deletion after backup is complete so don't know why they remain. 

I tried starting and stopping the DPMRA services on one of the hosts that has a VM with this issue and that workaround did NOT fix the leftover checkpoints.

So Huffelpaff please let me know if that solves your issue, as we are also having the same issue!

• Edited by Striker169 Friday, October 17, 2014 1:51 PM add more detail

I tried starting and stopping the DPMRA services on one of the hosts that has a VM with this issue and that workaround did NOT fix the leftover checkpoints.

So Huffelpaff please let me know if that solves your issue, as we are also having the same issue!

• Edited by Striker169 Friday, October 17, 2014 1:51 PM add more detail

Having the same issue with 2012R2 U4 of DPM.  Restarting the DPMRA service makes no difference and snapshots remain in place.

Snapshots are not shown in VMM and new snapshots can be taken in VMM but don't appear as checkpoints in VMM, only in Hyper-V manager (newly created snaps can be deleted/or reverted to but not the one the DPM )

Do you know if this is going to be addressed in U5 and if so when U5 may be released?

Hi,

Please check the DPMRACURR.ERRLOG after a backup completes.  You should see entries like the following to release the snapshot.  Note the times are in GMT time in the DPM logs.  Once the snapshot is released DPM is done and the merge and delete of the snapshots are performed by Hyper-V.

22E4	18A4	02/09	23:26:38.999	31	freesnapshotsubtask.cpp(46)	[00000000037E3390]	A8EFADFB-6660-45CB-B3D4-707E144EE710	NORMAL	CFreeSnapshotSubTask: constructor [00000000037E3390]
22E4	18A4	02/09	23:26:38.999	31	freesnapshotsubtask.cpp(123)	[00000000037E3390]	A8EFADFB-6660-45CB-B3D4-707E144EE710	NORMAL	CFreeSnapshotSubTask::TriggerDone [00000000037E3390]
22E4	18A4	02/09	23:26:39.421	31	freesnapshotsubtask.cpp(562)	[00000000037E3390]	A8EFADFB-6660-45CB-B3D4-707E144EE710	NORMAL	CFreeSnapshotSubTask: StartBackupComplete [00000000037E3390]
22E4	18A4	02/09	23:26:39.421	31	vsssnapshotrequestor.cpp(1033)	[0000000001502A10]	A8EFADFB-6660-45CB-B3D4-707E144EE710	NORMAL	CVssSnapshotRequestor::SetBackupCompletion [0000000001502A10]
22E4	18A4	02/09	23:26:39.437	31	freesnapshotsubtask.cpp(625)	[00000000037E3390]	A8EFADFB-6660-45CB-B3D4-707E144EE710	NORMAL	CFreeSnapshotSubTask: StartBackupComplete [00000000037E3390], pSnapshotRequestor [0000000001502A10]
22E4	18A4	02/09	23:26:39.437	31	vsssnapshotrequestor.cpp(1442)	[0000000001502A10]	A8EFADFB-6660-45CB-B3D4-707E144EE710	NORMAL	CVssSnapshotRequestor::StartBackupComplete [0000000001502A10]
22E4	18A4	02/09	23:26:44.468	31	vssbaserequestor.cpp(943)	[0000000001502A10]		NORMAL	QueryStatus returned 0x4230a, Releasing VssAsync [0000000001631450]
22E4	18A4	02/09	23:26:44.468	31	vssbaserequestor.cpp(1067)	[0000000001502A10]		NORMAL	CVssBaseRequestor::StartGatherWriterStatus [0000000001502A10]
22E4	18A4	02/09	23:26:45.468	31	vssbaserequestor.cpp(943)	[0000000001502A10]		NORMAL	QueryStatus returned 0x4230a, Releasing VssAsync [0000000001631950]
22E4	18A4	02/09	23:26:45.468	31	freesnapshotsubtask.cpp(675)	[00000000037E3390]		NORMAL	CFreeSnapshotSubTask: ReleaseSnapshot [00000000037E3390]
22E4	18A4	02/09	23:26:45.468	31	vssbaserequestor.cpp(1100)	[0000000001502A10]		NORMAL	CVssBaseRequestor::CheckWriterStatus [0000000001502A10]
22E4	18A4	02/09	23:26:45.468	31	vssbaserequestor.cpp(1131)	[0000000001502A10]		NORMAL	Checking Writer status for writerid: {66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE}                                       writerName: Microsoft Hyper-V VSS Writer
22E4	18A4	02/09	23:26:45.468	31	vsssnapshotrequestor.cpp(1472)	[0000000001502A10]		NORMAL	CVssSnapshotRequestor::ReleaseSnaphot [0000000001502A10]
22E4	18A4	02/09	23:26:45.468	31	vsssnapshotrequestor.cpp(2148)	[0000000001502A10]		NORMAL	CVssSnapshotRequestor::CleanUp [0000000001502A10]
22E4	18A4	02/09	23:26:45.468	31	vsssnapshotrequestor.cpp(1734)	[0000000001502A10]		NORMAL	CVssSnapshotRequestor::ReleaseVolumesForSnapshot [0000000001502A10]
22E4	0F9C	02/09	23:26:45.671	31	freesnapshotsubtask.cpp(181)	[00000000037E3390]		NORMAL	CFreeSnapshotSubTask::GetFinalStatus [00000000037E3390]
22E4	0F9C	02/09	23:26:46.077	31	freesnapshotsubtask.cpp(63)	[00000000037E3390]	A8EFADFB-6660-45CB-B3D4-707E144EE710	NORMAL	CFreeSnapshotSubTask: destructor [00000000037E3390]
22E4	0F9C	02/09	23:26:46.515	31	vsssnapshotrequestor.cpp(117)	[0000000001502A10]	A8EFADFB-6660-45CB-B3D4-707E144EE710	NORMAL	CVssSnapshotRequestor::~CVssSnapshotRequestor [0000000001502A10]
22E4	0F9C	02/09	23:26:46.515	31	vsssnapshotrequestor.cpp(2148)	[0000000001502A10]	A8EFADFB-6660-45CB-B3D4-707E144EE710	NORMAL	CVssSnapshotRequestor::CleanUp [0000000001502A10]
22E4	0F9C	02/09	23:26:46.515	31	vsssnapshotrequestor.cpp(1734)	[0000000001502A10]	A8EFADFB-6660-45CB-B3D4-707E144EE710	NORMAL	CVssSnapshotRequestor::ReleaseVolumesForSnapshot [0000000001502A10]
22E4	0F9C	02/09	23:26:46.515	31	vssbaserequestor.cpp(69)	[0000000001502A10]	A8EFADFB-6660-45CB-B3D4-707E144EE710	NORMAL	CVssBaseRequestor: destructor [0000000001502A10]

At this time, you should see two events in the Hyper-V-VMMS - Admin log showing the merge operation.

Log Name:      Microsoft-Windows-Hyper-V-VMMS-Admin
Source:        Microsoft-Windows-Hyper-V-VMMS
Date:          2/9/2015 5:26:44 PM
Event ID:      19070
Task Category: None
Level:         Information
Keywords:     
User:          SYSTEM
Computer:     Nodename.contoso.com
Description:
'WINB-Server.' background disk merge has been started. (Virtual machine ID 941B3C5C-FBD9-415C-BB28-CD2C98E0D2CC)

Log Name:      Microsoft-Windows-Hyper-V-VMMS-Admin
Source:        Microsoft-Windows-Hyper-V-VMMS
Date:          2/9/2015 5:26:50 PM
Event ID:      19080
Task Category: None
Level:         Information
Keywords:     
User:          SYSTEM
Computer:     Nodename.contoso.com
Description:
'WINB-Server.' background disk merge has been finished successfully. (Virtual machine ID 941B3C5C-FBD9-415C-BB28-CD2C98E0D2CC)

Mike, I've just had this situation occur in my environment. I was doing some maintenance on a VM, so the VMM job (host-level backup) failed and the VM was put into a inconsistent replica state. I noticed there was a .avhdx left behind for the system disk.

We added another disk to the VM, started it, then ran synchronization job in DPM. That went well, but now there are 2 snapshots/checkpoints left behind - how do we get rid of them? As previously stated, they're only visible from Hyper-V manager and there's no option to delete them.

Btw, we're on DPM 2012 R2 UR6, so the problem hasn't been solved yet I presume.

I'd expect DPM would be able to handle such situation where there might be changes happening to a VM during a backup window...

I've tried deleting the snapshots using Remove-VMSnapshot (failed with catastrophic failure - failed to generate VHD tree), also tried migrating the VM to different host (Live Migration is failing now too, had to turn of the VM).

• Edited by MarkosP 9 hours 24 minutes ago

Hi,

It's is the responsibility of Windows Hyper-v and vss infrastructure to merge .avhd files after a successful or failed backup. The backup requestor (DPM in this case) does not have any control over that process.  It's unfortunate that Hyper-V does not expose those snapshots in the GUI for you to perform a manual merge if they are orphaned after a backup failure.

Please see this helps.

Manually Merge .avhd to .vhd in Hyper-V

Mike, I've just had this situation occur in my environment. I was doing some maintenance on a VM, so the VMM job (host-level backup) failed and the VM was put into a inconsistent replica state. I noticed there was a .avhdx left behind for the system disk.

We added another disk to the VM, started it, then ran synchronization job in DPM. That went well, but now there are 2 snapshots/checkpoints left behind - how do we get rid of them? As previously stated, they're only visible from Hyper-V manager and there's no option to delete them.

Btw, we're on DPM 2012 R2 UR6, so the problem hasn't been solved yet I presume.

I'd expect DPM would be able to handle such situation where there might be changes happening to a VM during a backup window...

I've tried deleting the snapshots using Remove-VMSnapshot (failed with catastrophic failure - failed to generate VHD tree), also tried migrating the VM to different host (Live Migration is failing now too, had to turn of the VM).

• Edited by MarkosP Tuesday, June 23, 2015 10:16 PM

Mike, I've just had this situation occur in my environment. I was doing some maintenance on a VM, so the VMM job (host-level backup) failed and the VM was put into a inconsistent replica state. I noticed there was a .avhdx left behind for the system disk.

We added another disk to the VM, started it, then ran synchronization job in DPM. That went well, but now there are 2 snapshots/checkpoints left behind - how do we get rid of them? As previously stated, they're only visible from Hyper-V manager and there's no option to delete them.

Btw, we're on DPM 2012 R2 UR6, so the problem hasn't been solved yet I presume.

I'd expect DPM would be able to handle such situation where there might be changes happening to a VM during a backup window...

I've tried deleting the snapshots using Remove-VMSnapshot (failed with catastrophic failure - failed to generate VHD tree), also tried migrating the VM to different host (Live Migration is failing now too, had to turn of the VM).

• Edited by MarkosP Tuesday, June 23, 2015 10:16 PM

Mike, I've just had this situation occur in my environment. I was doing some maintenance on a VM, so the VMM job (host-level backup) failed and the VM was put into a inconsistent replica state. I noticed there was a .avhdx left behind for the system disk.

We added another disk to the VM, started it, then ran synchronization job in DPM. That went well, but now there are 2 snapshots/checkpoints left behind - how do we get rid of them? As previously stated, they're only visible from Hyper-V manager and there's no option to delete them.

Btw, we're on DPM 2012 R2 UR6, so the problem hasn't been solved yet I presume.

I'd expect DPM would be able to handle such situation where there might be changes happening to a VM during a backup window...

I've tried deleting the snapshots using Remove-VMSnapshot (failed with catastrophic failure - failed to generate VHD tree), also tried migrating the VM to different host (Live Migration is failing now too, had to turn of the VM).

• Edited by MarkosP Tuesday, June 23, 2015 10:16 PM

Mike, thanks, I was able to merge the disks using that procedure. However the snapshots were still there.

I was able to get rid of the snapshots by removing the VM from the cluster and Hyper-V manager and then importing the VM back. This went well, however VMM didn't quite like this, so in the end, I was forced to completely remove the VM and recreate it as a new one (from VMM).