DPM 2012 R2 and Bare Metal Recovery/System State System Protection Hyper-V Generation 2 VM Support

I have a physical Server 2012 R2 vhost running 2 Server 2012 R2 VMs setup as Generation 2 based. My DPM 2012 R2 server will perform the backup of the vhost just fine, but will consistently fail on the two Generation 2 based VMs. After further researching, I noticed that the physical Server 2012 R2 does not contain a EFI System Partition but the two VM's do (99 MB right in front of the C: boot partition).

When I try to get a backup/sync change through DPM, it fails I'm guessing because of the EFI partition. The event log on the VM server indicates Event ID 521 with error code 0x80780119. Launching Windows Server Backup, the failed status gives: There is not enough disk space to create the volume shadow copy (because I'm guessing it is trying to store those on the EFI partition) The two VMs were loaded from ISO based installation. I guess my question is:

Does DPM 2012 R2 support generation 2 VMs Bare Metal Recovery backups due to the EFI partition?

December 19th, 2013 7:19pm

Hi,

As you have already discovered, this is not a DPM issue as DPM just leverages Windows backup.   Can you supply a screenshot of disk management inside one of the effected guests so I can try to duplicate the configuration.

Free Windows Admin Tool Kit Click here and download it now
December 19th, 2013 8:41pm

December 19th, 2013 8:59pm

Thanks - setting this up now (in my spare time) and will see if I can find a workaround.  Stay Tuned..

UPDATE

I was able to duplicate your error:

C:\Windows\system32>wbadmin start backup -allcritical -backuptarget:e:
wbadmin 1.0 - Backup command-line tool
(C) Copyright 2013 Microsoft Corporation. All rights reserved.

Retrieving volume information...
This will back up (EFI System Partition),(C:),Recovery (300.00 MB) to e:.
Do you want to start the backup operation?
[Y] Yes [N] No y

The backup storage location that you selected is on the same physical hard
disk as your operating system. If the hard disk fails, you may lose both
your system data and backup data.

The backup operation to E: is starting.
Creating a shadow copy of the volumes specified for backup...
Summary of the backup operation:
------------------

The backup operation stopped before completing.
The backup operation stopped before completing.
Detailed error: ERROR - A Volume Shadow Copy Service operation error has
occurred: (0x8004231f)
Insufficient storage available to create either the shadow copy storage file or
other shadow copy data.

Log of files successfully backed up:
C:\Windows\Logs\WindowsServerBackup\Backup-19-12-2013_18-39-24.log

Log of files for which backup failed:
C:\Windows\Logs\WindowsServerBackup\Backup_Error-19-12-2013_18-39-24.log

There is not enough disk space to create the volume shadow copy on the storage l
ocation. Make sure that, for all volumes to be backup up, the minimum required d
isk space for shadow copy creation is available. This applies to both the backup
 storage destination and volumes included in the backup.
Minimum requirement: For volumes less than 500 megabytes, the minimum is 50 mega
bytes of free space. For volumes more than 500 megabytes, the minimum is 320 meg
abytes of free space.
Recommended: At least 1 gigabyte of free disk space on each volume if volume siz
e is more than 1 gigabyte.
ERROR - A Volume Shadow Copy Service operation error has
occurred: (0x8004231f)
Insufficient storage available to create either the shadow copy storage file or
other shadow copy data.

CAUSE

The problem is that the 300MB Recovery Partition only has about 31MB free space and VSS requires minimum of 50MB to create a snapshot.

You can see this by assigning a drive letter to the volume and doing a DIR /A


C:\Windows\system32>mountvol
Creates, deletes, or lists a volume mount point.

MOUNTVOL [drive:]path VolumeName
MOUNTVOL [drive:]path /D
MOUNTVOL [drive:]path /L
MOUNTVOL [drive:]path /P
MOUNTVOL /R
MOUNTVOL /N
MOUNTVOL /E
MOUNTVOL drive: /S

    path        Specifies the existing NTFS directory where the mount
                point will reside.
    VolumeName  Specifies the volume name that is the target of the mount
                point.
    /D          Removes the volume mount point from the specified directory.
    /L          Lists the mounted volume name for the specified directory.
    /P          Removes the volume mount point from the specified directory,
                dismounts the volume, and makes the volume not mountable.
                You can make the volume mountable again by creating a volume
                mount point.
    /R          Removes volume mount point directories and registry settings
                for volumes that are no longer in the system.
    /N          Disables automatic mounting of new volumes.
    /E          Re-enables automatic mounting of new volumes.
    /S          Mount the EFI System Partition on the given drive.

Possible values for VolumeName along with current mount points are:

    \\?\Volume{80001a12-3033-4169-8985-5ce8dacf9b07}\
        C:\

    \\?\Volume{fa2cb1b2-93bf-4ce5-ad31-07581a9e887a}\
        E:\

    \\?\Volume{152eadd9-f001-4562-bb9c-dc0cc50c638c}\
        *** NO MOUNT POINTS ***

    \\?\Volume{889daeac-68ea-11e3-80b4-806e6f6e6963}\
        D:\

C:\Windows\system32>mountvol x: \\?\Volume{152eadd9-f001-4562-bb9c-dc0cc50c638c}\

C:\Windows\system32>mountvol
Creates, deletes, or lists a volume mount point.

MOUNTVOL [drive:]path VolumeName
MOUNTVOL [drive:]path /D
MOUNTVOL [drive:]path /L
MOUNTVOL [drive:]path /P
MOUNTVOL /R
MOUNTVOL /N
MOUNTVOL /E
MOUNTVOL drive: /S

    path        Specifies the existing NTFS directory where the mount
                point will reside.
    VolumeName  Specifies the volume name that is the target of the mount
                point.
    /D          Removes the volume mount point from the specified directory.
    /L          Lists the mounted volume name for the specified directory.
    /P          Removes the volume mount point from the specified directory,
                dismounts the volume, and makes the volume not mountable.
                You can make the volume mountable again by creating a volume
                mount point.
    /R          Removes volume mount point directories and registry settings
                for volumes that are no longer in the system.
    /N          Disables automatic mounting of new volumes.
    /E          Re-enables automatic mounting of new volumes.
    /S          Mount the EFI System Partition on the given drive.

Possible values for VolumeName along with current mount points are:

    \\?\Volume{80001a12-3033-4169-8985-5ce8dacf9b07}\
        C:\

    \\?\Volume{fa2cb1b2-93bf-4ce5-ad31-07581a9e887a}\
        E:\

    \\?\Volume{152eadd9-f001-4562-bb9c-dc0cc50c638c}\
        X:\

    \\?\Volume{889daeac-68ea-11e3-80b4-806e6f6e6963}\
        D:\

X:\>dir /a /s
 Volume in drive X is Recovery
 Volume Serial Number is BA3C-AF14

 Directory of X:\

12/19/2013  10:18 AM    <DIR>          Recovery
12/19/2013  11:05 AM    <DIR>          System Volume Information
               0 File(s)              0 bytes

 Directory of X:\Recovery

12/19/2013  10:18 AM    <DIR>          .
12/19/2013  10:18 AM    <DIR>          ..
12/19/2013  10:57 AM    <DIR>          WindowsRE
               0 File(s)              0 bytes

 Directory of X:\Recovery\WindowsRE

12/19/2013  10:57 AM    <DIR>          .
12/19/2013  10:57 AM    <DIR>          ..
06/18/2013  07:08 AM         3,170,304 boot.sdi
12/19/2013  10:19 AM             1,013 ReAgent.xml
09/30/2013  01:21 PM       249,785,287 Winre.wim
               3 File(s)    252,956,604 bytes

     Total Files Listed:
               3 File(s)    252,956,604 bytes
               7 Dir(s)      31,383,552 bytes free

SOLUTION

You can move the shadowcopy storage for the recovery partition volume to the C: volume.

C:\Windows\system32>vssadmin add shadowstorage /for=\\?\Volume{152eadd9-f001-4562-bb9c-dc0cc50c638c}\ /on=c: /maxsize=500MB
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Successfully added the shadow copy storage association

C:\Windows\System32>vssadmin list shadowstorage
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Shadow Copy Storage association
   For volume: (C:)\\?\Volume{80001a12-3033-4169-8985-5ce8dacf9b07}\
   Shadow Copy Storage volume: (C:)\\?\Volume{80001a12-3033-4169-8985-5ce8dacf9b07}\
   Used Shadow Copy Storage space: 0 bytes (0%)
   Allocated Shadow Copy Storage space: 0 bytes (0%)
   Maximum Shadow Copy Storage space: 2.85 GB (10%)

Shadow Copy Storage association
   For volume: (E:)\\?\Volume{fa2cb1b2-93bf-4ce5-ad31-07581a9e887a}\
   Shadow Copy Storage volume: (E:)\\?\Volume{fa2cb1b2-93bf-4ce5-ad31-07581a9e887a}\
   Used Shadow Copy Storage space: 0 bytes (0%)
   Allocated Shadow Copy Storage space: 0 bytes (0%)
   Maximum Shadow Copy Storage space: 29.4 GB (30%)

Shadow Copy Storage association
   For volume: (\\?\Volume{152eadd9-f001-4562-bb9c-dc0cc50c638c}\)\\?\Volume{152eadd9-f001-4562-bb9c-dc0cc50c638c}\
   Shadow Copy Storage volume: (C:)\\?\Volume{80001a12-3033-4169-8985-5ce8dacf9b07}\
   Used Shadow Copy Storage space: 0 bytes (0%)
   Allocated Shadow Copy Storage space: 0 bytes (0%)
   Maximum Shadow Copy Storage space: 500 MB (1%)

RETEST WBADMIN

C:\>wbadmin start backup -allcritical -backuptarget:e:
wbadmin 1.0 - Backup command-line tool
(C) Copyright 2013 Microsoft Corporation. All rights reserved.

Retrieving volume information...
This will back up (EFI System Partition),(C:),Recovery(X:) to e:.
Do you want to start the backup operation?
[Y] Yes [N] No y

The backup storage location that you selected is on the same physical hard
disk as your operating system. If the hard disk fails, you may lose both
your system data and backup data.

The backup operation to E: is starting.
Creating a shadow copy of the volumes specified for backup...
Creating a shadow copy of the volumes specified for backup...
Creating a backup of volume (EFI System Partition) (99.00 MB), copied (0%).
Compacting the virtual hard disk for volume (EFI System Partition) (99.00 MB), completed (0%).
The backup of volume (EFI System Partition) (99.00 MB) completed successfully.
Creating a backup of volume (C:), copied (0%).
Creating a backup of volume (C:), copied (0%).
Creating a backup of volume (C:), copied (0%).

Free Windows Admin Tool Kit Click here and download it now
December 19th, 2013 9:12pm

Mike,

Not sure I would call that a solution. This is a work around at best, and one that certainly doesn't scale very well. Also, from what is being reported here...

http://social.technet.microsoft.com/Forums/windowsserver/en-US/7373a7b8-01c8-4e2b-aaaa-513b7dad56f4/windows-server-2012-r2-vm-back-up-fails-with-insufficient-storage-available-to-create-either-the

"...the backups work however after recovering the new VM image the Add ShadowStorage Map is lost and has to be recreated for backups to work again."

...it's not a work around that can be relied on in a production environment. So how long will we have to wait for a hotfix? Realizing without one backups of Gen2 VMs are broken.

January 4th, 2014 9:01pm

Hi,

As mentioned earlier, this is not a problem in DPM code. The windows backup group would need to make windows server backup exclude that small 300mb Partition / volume from shadow copy list and just back it up natively.

Please work with the Windows support group to report the problem and ask for a better solution

Free Windows Admin Tool Kit Click here and download it now
January 4th, 2014 11:20pm

Thanks Mike, that's pure semantics though because it's through DPM that my backups are failing...DPM code, or not. Of course it is much easier for you to say "this is not a problem in DPM code" and deflect taking any responsibility to help resolve the problem.
January 4th, 2014 11:49pm

Hi,

From a DPM perspective, I provided a workaround to allow DPM recovery points to succeed which solves the problem as reported. There are no DPM code changes that can fix or influence Windows server BMR backup behavior.

UPDATE - FYI - I have pinged the Windows product group to make them aware of the reported problem so they can review internally.

Free Windows Admin Tool Kit Click here and download it now
January 5th, 2014 1:41am

UPDATE-2

The Windows product group has determined that this is a code defect and are working on a fix.  In the meantime, you can use one of the following workarounds so DPM initiated BMR backups succeed.

Solution 1) Permanently disable WINRE so a snapshot is not attempted on the recovery partition when Windows image backup (BMR) is taken.

a) open an administrative command prompt.
b) Run: reagentc /disable
c) BMR backups will now succeed. To enable WINRE again run: reagentc /enable

Solution 2) Use a DPM pre-backup and post-backup script to disable WINRE before the BMR backup is taken and Re-enable WINRE after the BMR backup completes.

Using pre-backup and post-backup scripts

a) On the Windows 2012 R2 protected server (hyper-V guest) replace the ScriptingConfig.xml file with the new one below.


LOCATION:  C:\Program Files\Microsoft Data Protection Manager\DPM\Scripting\ScriptingConfig.xml

CONTENT OF NEW ScriptingConfig.xml.  Note the trailing space in "System Protection " <--must be trailing space.


<?xml version="1.0" encoding="utf-8"?>
<ScriptConfiguration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns="http://schemas.microsoft.com/2003/dls/ScriptingConfig.xsd">
<DatasourceScriptConfig DataSourceName="System Protection ">
<PreBackupScript>C:\temp\Disable-WINRE.cmd</PreBackupScript>
<PostBackupScript>C:\temp\Enable-WINRE.cmd</PostBackupScript>
<TimeOut>5</TimeOut>
</DatasourceScriptConfig>
</ScriptConfiguration>


b) Make the following two batch files (disable-winre.cmd and enable-winre.cmd) and save them in c:\temp folder on the protected server.

Disable-WINRE.cmd content.

reagentc /disable


Enable-WINRE.cmd content.

reagentc /enable

January 7th, 2014 9:57pm

Thank you for your assistance Mike in the notification to the Windows product group to make them aware of this issue. We have opted to not run any generation 2 based VMs until this problem has been resolved. 
Free Windows Admin Tool Kit Click here and download it now
January 9th, 2014 1:16am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics