DPM backups using hardware VSS provider creating leftover snapshots as iSCSI targets

Hello all,

This question was originally posted in the Windows Server File Services and Storage forum here; the personnel monitoring that forum have asked me to pose the question/situation here, so here we go...

I have a Windows Server 2012 Failover cluster running a number of virtual machines. Storage for the cluster is provided by a Windows Storage Server 2012 instance and mapped to the nodes via iSCSI. In addition, I use DPM 2012 SP1 to protect the clustered VMs. The snapshot provider configuration was set up in accordance with the instructions and guidance at http://blogs.technet.com/b/filecab/archive/2012/10/08/iscsi-target-storage-vds-vss-provider.aspx . I had the same setup with Server 2008 R2, but recently upgraded to take advantage of the hardware snapshot provider and performance improvements claimed by Microsoft.

Since configuring the protection group for the clustered virtual machines, the group shows that all of the members are in an ideal state and there are no errors in the cluster event log. However, there are some issues in the event logs for the cluster nodes around the time the backups are being taken. This isn't the reason for this post, however, though it may be related.

While having a look at the iSCSI configuration on the storage server yesterday, I noticed the following...

Get-iSCSIVirtualDisk

ClusterGroupName   :
ComputerName       : myserver
Description        : Cluster Shared Volume
DiskType           : Fixed
HostVolumeId       : {67F71B49-BD09-457E-8E33-34607970BE1C}
LocalMountDeviceId :
OriginalPath       :
ParentPath         :
Path               : e:\csv.vhd
SerialNumber       : C4A45A4D-E7E0-4AEE-B800-403DD32E8A61
Size               : 8796093022208
SnapshotIds        : {{19C35EF1-C02D-49E4-9C38-B3AE500E2524}, {82FA960F-529E-4A41-A0B7-4F4981846051},
                     {E59C4532-5280-4465-B2EB-9FD31175F665}, {ED2F4661-2882-4C11-9B27-C946D6695219}...}
Status             : Connected
VirtualDiskIndex   : 1777681073

ClusterGroupName   :
ComputerName       : myserver
Description        : Snapshot of virtual disk 1777681073 taken at 12/8/2013 12:04:38 AM
DiskType           : ReadOnlySnapshot
HostVolumeId       : {67F71B49-BD09-457E-8E33-34607970BE1C}
LocalMountDeviceId :
OriginalPath       : e:\csv.vhd
ParentPath         :
Path               : \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy{2F56E973-560E-11E3-93F6-00259067F05B}\csv.vhd
SerialNumber       : C4A45A4D-E7E0-4AEE-B800-403DD32E8A61
Size               : 8796093022208
SnapshotIds        :
Status             : NotConnected
VirtualDiskIndex   : 1977840284

MANY others listed here...

ClusterGroupName   :
ComputerName       : myserver
Description        : Snapshot of virtual disk 1777681073 taken at 12/18/2013 1:03:52 AM
DiskType           : ReadOnlySnapshot
HostVolumeId       : {67F71B49-BD09-457E-8E33-34607970BE1C}
LocalMountDeviceId :
OriginalPath       : e:\csv.vhd
ParentPath         :
Path               : \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy{F1BD4A63-60EC-11E3-93F6-00259067F05B}\csv.vhd
SerialNumber       : C4A45A4D-E7E0-4AEE-B800-403DD32E8A61
Size               : 8796093022208
SnapshotIds        :
Status             : NotConnected
VirtualDiskIndex   : 2007241302

My searches for an explanation for this has, so far, turned up nothing.

When I do a "vssadmin list shadows" a whole bunch of crap is spit out as well:

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2012 Microsoft Corp.

Contents of shadow copy set ID: {2020c103-c81f-4c9f-a3e9-43ab496ba5b4}
   Contained 1 shadow copies at creation time: 11/29/2013 5:14:45 PM
      Shadow Copy ID: {09b6cd4f-28d2-4498-a132-3f9d6c716580}
         Original Volume: (E:)\\?\Volume{67f71b49-bd09-457e-8e33-34607970be1c}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1
         Originating Machine: myserver
         Service Machine: myserver
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: DataVolumeRollback
         Attributes: Persistent, No auto release, No writers, Differential, Auto recovered

....

Contents of shadow copy set ID: {b3650eb5-ae1c-4ff1-92f0-90a33d0f82b1}
   Contained 1 shadow copies at creation time: 12/18/2013 12:34:08 AM
      Shadow Copy ID: {757b8c99-265a-4206-8a16-31b28597cf2f}
         Original Volume: (E:)\\?\Volume{67f71b49-bd09-457e-8e33-34607970be1c}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy40
         Originating Machine: myserver
         Service Machine: myserver
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: DataVolumeRollback
         Attributes: Persistent, No auto release, No writers, Differential, Auto recovered

Contents of shadow copy set ID: {4f58d81d-ebe1-48fe-a88b-17ff0b20757f}
   Contained 1 shadow copies at creation time: 12/18/2013 1:03:52 AM
      Shadow Copy ID: {318a9fb3-1476-4a41-98eb-00a60718cb3f}
         Original Volume: (E:)\\?\Volume{67f71b49-bd09-457e-8e33-34607970be1c}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy41
         Originating Machine: myserver
         Service Machine: myserver
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: DataVolumeRollback
         Attributes: Persistent, No auto release, No writers, Differential, Auto recovered

I can't find anything related in the system event log, but the timing of the majority of the creation times coincides somewhat with my backup schedule. There are 3 virtual machines in the protection group, not sure if this is coincidence or not, but every morning there are 3 more items in the iSCSI target list.

I was wondering if anyone could possibly enlighten me as to what exactly is going on here? Why are these snapshots appearing as iSCSI virtual disks? This looks like a potential cause for a major disaster...

Thanks for reading.

Greg

December 30th, 2013 11:54pm

Hope everyone's had a great holiday season!

Here's a BUMP and a hope that someone in the Microsoft community can respond to this issue, it would be nice to not have to create a premier support case about this.

Cheers,
Greg

Free Windows Admin Tool Kit Click here and download it now
January 4th, 2014 1:17am

Hi,

Install this Windows fix and if that does not help, please open a support case with the Windows group.

2878635 - Update is available that improves the resiliency of the cloud service provider in Windows Server 2012: December 2013
http://support.microsoft.com/kb/2878635

January 4th, 2014 1:27am

Thanks for the response Mike. I've installed the update, which sadly prompted for a reboot.

Since this server provides the storage for all of the virtual machines which our company relies on for basic operations, this isn't a trivial thing, so I've scheduled a reboot for this weekend after hours (Friday January 10).

A few questions:

-Do you think it is safe to remove the iSCSI target artifacts before the reboot? Or should I wait until after the reboot to do this?
-Are there any recommended steps or best practices I should follow in order to safely remove the shadow copies being left behind (which now total 101)?

Essentially, I'm concerned that all of these artifacts and shadow copies could compromise the integrity of my cluster shared volume, which would be absolutely catastrophic to our environment and my sanity. What will happen to these artifacts when the system is rebooted?...

Thanks for any guidance or suggestions.

Greg


  • Edited by gr0x0rd Tuesday, January 07, 2014 7:39 PM
Free Windows Admin Tool Kit Click here and download it now
January 7th, 2014 10:23pm

Hi,

I'm not on the Windows team, so don't know about the inner workings of the fix and what to expect.  However, you can try using diskshadow.exe to delete all the snapshots for the given volume and see if that clears them up.

DISKSHADOW> delete shadows /?

DELETE SHADOWS { ALL | VOLUME <volume> | OLDEST <volume> | SET <setID> | ID <shadowID> | EXPOSED <drive letter, mountPoint or share> }

        Delete shadow copies, both persistent and non-persistent.

        ALL                     All shadow copies.
        VOLUME <volume>         Delete all shadow copies of the given volume.
        OLDEST <volume>         Delete the oldest shadow copy of the given volume.
        SET <setID>             Delete the shadow copies in the shadow copy set specified by the setId parameter.
        ID <shadowID>           Delete the shadow copy specified by the shadowId parameter.
        EXPOSED <exposeName>    Delete the shadow copy that is exposed at the specified drive letter, mount point or share.

        Examples: DELETE SHADOWS ALL
                  DELETE SHADOWS EXPOSED p:
                  DELETE SHADOWS EXPOSED ShareName

So something like:  DELETE SHADOWS VOLUME E:   or  DELETE SHADOWS ALL

January 7th, 2014 11:00pm

Thanks Mike.

A few moments ago I deleted the shadows and the corresponding iSCSI target artifacts. My heart was in my throat for part of the process, but at the moment, everything appears to be stable.

I will post again after I bring the cluster down and reboot the storage server on Friday afternoon.

Again, thanks for the help.

Greg

Free Windows Admin Tool Kit Click here and download it now
January 9th, 2014 12:21am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics