I have a customer with two Windows 2008 R2 Hyper-V host servers running a wbadmin (Windows Backup).  The one server backs up just fine.  However, the other server consistently fails with the following error:

Detailed error: ERROR - The shared restore point operation failed with error (0x81000101) The creation of a shadow copy has timed out. Try this operation again.

   In the Event Viewer (Application log), I see the following error:

The backup operation that started at '‎2011‎-‎04‎-‎27T06:15:23.955000000Z' has failed because the Volume Shadow Copy Service operation to create a shadow copy of the volumes being backed up failed with following error code '2155348001'. Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.

   However I can't find any further error messages (in either the guest VM's event viewer or in the Hyper-V host's event viewer).  

   A few additional notes:

• Server has SP1 installed.  The problem with the backup predates the SP1 installation.
• Reboots don't fix the backup problem.
• After a failed backup, both the Hyper-V VSS writer and the System Writer have the following state in "vssadmin list writers": State: [5] Waiting for completion 
• I can't do a manual volume or file/folder backup of the drive which has VM's on it using wbadmin because the Volume Shadow fails.
• A "DELETE SHADOWS ALL" from DISKSHADOW didn't fix the problem.
• I was wondering if the problem was due to KB982210  (http://garysgambit.blogspot.com/2010/05/windows-2008-hyper-v-vss-backup-bug.html and http://support.microsoft.com/kb/982210).  The HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\STORAGE\Volume on this system does have a a lot of entries, and the initial logon to the host after a reboot does take a while.  But my Vhdmp.sys has a newer version than this hotfix so I'm assuming that the SP1 for Windows 2008 R2 replaced this file and my guess is that this hotfix won't work on Windows 2008 R2 SP1 servers.  

   Anyone with any good ideas?  [At this point I may even accept bad ones...]

Thanks! dbaddorf

Hi,

 

This problem can occur due to the following reasons

 

l  The VHDs are not formatted as NTFS.

l  We do not have enough space inside the virtual machine's VHD.

l  We do not have Integration services installed.

l  The OS is unable to auto-mount the snapshotted VHDs during the backup process.

 

Please follow the steps mentioned Below

 

l  Make sure that the VHDs are NTFS formatted.

l  We have enough free space inside the virtual machine's VHD.

l  The latest Integration services is installed.

l  While backing an online Virtual Machine NTBackup mounts the snapshotted VHDs, so that the it can revert them back to their proper state once backup completes.

 

Tim Quan

• Marked as answer by Tim QuanModerator Monday, May 02, 2011 2:45 AM
• Unmarked as answer by Tim QuanModerator Tuesday, May 03, 2011 2:21 AM

Tim,

   Thanks so much for getting back to me.

• VHD's (for Windows hosts) are NTFS:  There are 4 Windows VM's, each with multiple volumes - all of the volumes are formatted with NTFS.  There is one Linux (CentOS) VM, which is obviously not formatted with NTFS, but the backups had been failing before this VM was added.  Host volume is also NTFS.
• Space:  Each of the .vhd files for the VM's are dynamically expanding .vhd files with plenty of free space.  The host drive has 278GB of free space.
• Integration Services:  Each of the Windows VM's have integration services installed.  The Linux VM doesn't, but again, the backups have been failing from before the linux VM was added.
• I'm not sure that I totally understand what you are referring to for this point.  There are no Hyper-V snapshots for any of the VM's.  It seems like the VSS service times out when trying to make a shadow copy of the Hyper-V host volume.  (At least while the VM's are running).  This is the problem which I'm trying to fix but I don't know how...

   I'd certainly be glad for any further input!  Thanks again!

dbaddorf

A further note on this... I was able to get a backup when all the VM's where shut down. So it certainly seems like a VSS writer problem snapshotting the volume when the Hyper-V VM's were running. dbaddorf

Any luck resolving this issue? I am having the same issue. I have Windows 2008 R2 Hyper-V backups failing with the same error. All VM's are NTFS, latest integration services installed, plenty of space for all hosts. I'm rooting around everywhere trying to find a solid answer.

Not yet.  I've narrowed it down to two VM's which are causing problems (one is a DC and one is running Exchange).  My other VM's are fine to stay up and running for the backup - but these two cause the backup to fail if they are not shutdown for the backup (at least when the backup first starts).  I am going to open a case with Microsoft Support and will update this forum entry with any resolution...

I am also having the same issue with a server.  Did you get this resolved with Microsoft?

LynnCY,

   No I did not get this problem resolved.  MS Support has worked on this problem for about 4 weeks and have done some extensive diagnosing but have not determined what the problem is.  In the meantime I have just scripted my host backup to shutdown the offending VM's (one is a DC and one is running Exchange 2010) before the backup.  Once the backup starts then I can restart the VM's and the backup still works fine.  

dbaddorf

Sadly, I am having the same problem. If the VMs are in "stop" state the backup works fine. And MS Support has been sitting on this for a month with no response other than to ask me to run diagnostics. It actually backed up correctly during one of the tests but not there-after.

Resolution:

- Run regedit.exe and navigate to "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\SPP"

- Create a new Registry value of type DWORD with name "CreateTimeout"

- Change value to 12000000(2*10*60*1000 = 20 mins) in decimal

http://blogs.technet.com/b/asiasupp/archive/2011/08/01/windows-server-backup-failed-to-backup-with-error-0x81000101.aspx

• Proposed as answer by danma_ Monday, August 01, 2011 7:52 AM
• Marked as answer by dbaddorf Wednesday, August 03, 2011 11:38 AM

YES! This seems to have fixed it. But ya gotta be kidding! We've been fighting this for months with Microsoft "Support".

Thanks so much!!! This fixed it for me, too!  And I've had a case open w/ MS support for months as well!

dbaddorf

Thanks so much for this resolution. I've been fighting VSS failures for months. I've applied the registry timeout fix and we'll see if that helps.

One thing I noticed is you said you are runninn a virtual DC. Be careful with snapshots. I understand DCs should never be restored to point-in-time as is may cause serious problems with replication.

Actually by the time I put both "problem" VM's back on the Hyper-V host I had to change the timeout to 18000000.  But the backup from the Hyper-V works without having to shut any of the VM's down.  

Dave, thanks for the reminder about using snapshots w/ DC's.  Microsoft doesn't recommend using snapshots in any production environment.

Can someone clarify the reg key path?  Does the SPP  in HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\SPP  stand for Software Protection Platform or does the folder SPP need to be created at that same level.    I just want to put the CreateTimeout dword in the correct folder.  Thank you for your help.

Jeff

Jeff,

   I'm not sure what SPP stands for, but the key HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\SPP should already be there.  You just need to add the registry value "CreateTimeout"...

Dave

Worked perfectly for me. Thanks for the tip!

Rob,

   I'm glad that it helped!

Dave

I am having the same issue with the same error backing up the system state using the wbadmin. I made the change in the registry but I guess I need to reboot the server for that to take effect since I ran the wbadmin command to backup the system state after the change without booting the server and it did not work. Can you confirm if you had to reboot?

If I recall correctly, I didn't need to reboot.  Maybe you have to try a longer timeout...

dave

The issue (I found) was simply a timeout problem. It should not take a reboot to solve.

hth

Hi

I do not have any registry entry called SPP, I however have SoftwareProtectionPlatform. Are these the same? Should I create the DWORD "CreateTimeout" inside this key?

Thanks

The SPP  Key should be below that of "SoftwareProtectionPlatform" and above "Svchost"   it has two subkeys  "clients" and "leases" but the DWORD "CreateTimeout" should be in the root of the SPP Key.

This solved my problem as well. Out of the blue my HyperV Host stopped backing up data, stopping at a volume that does not exist on the host normally. Turns out Windows Server Backup mounts the volumes of the guests inside the host as shadow copies. So its no wonder I could not find the Volume GUID on the HyperV host - its only available during backup! This was very missleading, since I guessed it tried to back up a nonexistent volume.

Anyways: the creation of the timeout registry value fixed my problem. Thanks!

  Thanks danma_ for the solution! This worked for me! Sometimes backups did work, another time they failed. It started when I added an additional VM in Hyper-V. I changed the value and I saw that It took a bit over 10 minutes after VSS was ready and could continue to backup, I guess it sometimes completed below 10 minutes and then the backup did work. A big "thank you" for the tip!

Sorry for necroposting (althoug the issues is still valid for many), but you have one extra zero in 12000000, which is 200 minutes, not 20.

Hi

I do not have any registry entry called SPP, I however have SoftwareProtectionPlatform. Are these the same? Should I create the DWORD "CreateTimeout" inside this key?

Thanks

I do not have the 'SPP' key either, and I have the SoftwareProtectionPlatform as well.  I wonder if this is due to server versions (I'm on Server 2012)?  If so what is the solution for this, if anyone has an idea.

Thanks.

Resolution:

- Run regedit.exe and navigate to "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\SPP"

- Create a new Registry value of type DWORD with name "CreateTimeout"

- Change value to 12000000(2*10*60*1000 = 20 mins) in decimal

http://blogs.technet.com/b/asiasupp/archive/2011/08/01/windows-server-backup-failed-to-backup-with-error-0x81000101.aspx

2*10*60*1000 = 1200000 and not 12000000, you got a 0 too much!