Windows 2008 R2 Hyper-V backup fails: VSS error 2155348001
I have a customer with two Windows 2008 R2 Hyper-V host servers running a wbadmin (Windows Backup). The one server backs up just fine. However, the other server consistently fails with the following error:
Detailed error: ERROR - The shared restore point operation failed with error (0x81000101) The creation of a shadow copy has timed out. Try this operation again.
In the Event Viewer (Application log), I see the following error:
The backup operation that started at '2011-04-27T06:15:23.955000000Z' has failed because the Volume Shadow Copy Service operation to create a shadow copy of the volumes being backed up failed with following error code '2155348001'.
Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.
However I can't find any further error messages (in either the guest VM's event viewer or in the Hyper-V host's event viewer).
A few additional notes:
- Server has SP1 installed. The problem with the backup predates the SP1 installation.
- Reboots don't fix the backup problem.
- After a failed backup, both the Hyper-V VSS writer and the System Writer have the following state in "vssadmin list writers": State: [5] Waiting for completion
- I can't do a manual volume or file/folder backup of the drive which has VM's on it using wbadmin because the Volume Shadow fails.
- A "DELETE SHADOWS ALL" from DISKSHADOW didn't fix the problem.
- I was wondering if the problem was due to KB982210 (http://garysgambit.blogspot.com/2010/05/windows-2008-hyper-v-vss-backup-bug.html and http://support.microsoft.com/kb/982210). The HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\STORAGE\Volume
on this system does have a a lot of entries, and the initial logon to the host after a reboot does take a while. But my Vhdmp.sys has a newer version than this hotfix so I'm assuming that the SP1 for Windows 2008 R2 replaced this file and my
guess is that this hotfix won't work on Windows 2008 R2 SP1 servers.
Anyone with any good ideas? [At this point I may even accept bad ones...]
Thanks! dbaddorf
April 28th, 2011 6:20pm
Hi,
This problem can occur due to the following reasons
l
The VHDs are not formatted as NTFS.
l
We do not have enough space inside the virtual machine's VHD.
l
We do not have Integration services installed.
l
The OS is unable to auto-mount the snapshotted VHDs during the backup process.
Please follow the steps mentioned Below
l
Make sure that the VHDs are NTFS formatted.
l
We have enough free space inside the virtual machine's VHD.
l
The latest Integration services is installed.
l
While backing an online Virtual Machine NTBackup mounts the snapshotted VHDs, so that the it can revert them back to their proper state once backup completes.
Tim Quan
- Marked as answer by
Tim QuanModerator
Monday, May 02, 2011 2:45 AM
- Unmarked as answer by
Tim QuanModerator
Tuesday, May 03, 2011 2:21 AM
April 29th, 2011 9:05am
Tim,
Thanks so much for getting back to me.
- VHD's (for Windows hosts) are NTFS: There are 4 Windows VM's, each with multiple volumes - all of the volumes are formatted with NTFS. There is one Linux (CentOS) VM, which is obviously not formatted with NTFS, but the backups had been failing
before this VM was added. Host volume is also NTFS.
- Space: Each of the .vhd files for the VM's are dynamically expanding .vhd files with plenty of free space. The host drive has 278GB of free space.
- Integration Services: Each of the Windows VM's have integration services installed. The Linux VM doesn't, but again, the backups have been failing from before the linux VM was added.
- I'm not sure that I totally understand what you are referring to for this point. There are no Hyper-V snapshots for any of the VM's. It seems like the VSS service times out when trying to make a shadow copy of the Hyper-V host volume. (At
least while the VM's are running). This is the problem which I'm trying to fix but I don't know how...
I'd certainly be glad for any further input! Thanks again!
dbaddorf
May 2nd, 2011 3:11pm
A further note on this... I was able to get a backup when all the VM's where shut down. So it certainly seems like a VSS writer problem snapshotting the volume when the Hyper-V VM's were running. dbaddorf
May 4th, 2011 2:22pm
Any luck resolving this issue? I am having the same issue. I have Windows 2008 R2 Hyper-V backups failing with the same error. All VM's are NTFS, latest integration services installed, plenty of space for all hosts. I'm rooting around everywhere trying
to find a solid answer.
May 27th, 2011 5:39pm
Not yet. I've narrowed it down to two VM's which are causing problems (one is a DC and one is running Exchange). My other VM's are fine to stay up and running for the backup - but these two cause the backup to fail if they are not shutdown
for the backup (at least when the backup first starts). I am going to open a case with Microsoft Support and will update this forum entry with any resolution...
May 31st, 2011 2:24pm
I am also having the same issue with a server. Did you get this resolved with Microsoft?
June 30th, 2011 9:42pm
LynnCY,
No I did not get this problem resolved. MS Support has worked on this problem for about 4 weeks and have done some extensive diagnosing but have not determined what the problem is. In the meantime I have just scripted my host backup
to shutdown the offending VM's (one is a DC and one is running Exchange 2010) before the backup. Once the backup starts then I can restart the VM's and the backup still works fine.
dbaddorf
July 5th, 2011 4:51pm
Sadly, I am having the same problem. If the VMs are in "stop" state the backup works fine. And MS Support has been sitting on this for a month with no response other than to ask me to run diagnostics. It actually backed up correctly during one of the tests
but not there-after.
July 31st, 2011 11:19pm
YES! This seems to have fixed it. But ya gotta be kidding! We've been fighting this for months with Microsoft "Support".
August 1st, 2011 8:32pm
Thanks so much!!! This fixed it for me, too! And I've had a case open w/ MS support for months as well!
dbaddorf
August 3rd, 2011 2:38pm
Thanks so much for this resolution. I've been fighting VSS failures for months. I've applied the registry timeout fix and we'll see if that helps.
One thing I noticed is you said you are runninn a virtual DC. Be careful with snapshots. I understand DCs should never be restored to point-in-time as is may cause serious problems with replication.
August 15th, 2011 5:22am
Actually by the time I put both "problem" VM's back on the Hyper-V host I had to change the timeout to 18000000. But the backup from the Hyper-V works without having to shut any of the VM's down.
Dave, thanks for the reminder about using snapshots w/ DC's. Microsoft doesn't recommend using snapshots in any production environment.
August 15th, 2011 2:57pm
Can someone clarify the reg key path? Does the SPP in HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\SPP stand for Software Protection Platform or does the folder SPP need to be created at that same level.
I just want to put the CreateTimeout dword in the correct folder. Thank you for your help.
Jeff
December 1st, 2011 3:13am
Jeff,
I'm not sure what SPP stands for, but the key HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\SPP should already be there. You just need to add the registry value "CreateTimeout"...
Dave
December 1st, 2011 3:27pm
Worked perfectly for me. Thanks for the tip!
January 22nd, 2012 8:24am
Rob,
I'm glad that it helped!
Dave
January 23rd, 2012 3:53pm
I am having the same issue with the same error backing up the system state using the wbadmin. I made the change in the registry but I guess I need to reboot the server for that to take effect since I ran the wbadmin command to backup the system state after
the change without booting the server and it did not work. Can you confirm if you had to reboot?
February 23rd, 2012 9:51pm
If I recall correctly, I didn't need to reboot. Maybe you have to try a longer timeout...
dave
February 23rd, 2012 10:12pm
The issue (I found) was simply a timeout problem. It should not take a reboot to solve.
hth
February 23rd, 2012 10:31pm
Hi
I do not have any registry entry called SPP, I however have SoftwareProtectionPlatform. Are these the same? Should I create the DWORD "CreateTimeout" inside this key?
Thanks
April 2nd, 2012 9:11am
The SPP Key should be below that of "SoftwareProtectionPlatform" and above "Svchost" it has two subkeys "clients" and "leases" but the DWORD "CreateTimeout" should be in the root of the SPP Key.
May 9th, 2012 1:14am
This solved my problem as well. Out of the blue my HyperV Host stopped backing up data, stopping at a volume that does not exist on the host normally. Turns out Windows Server Backup mounts the volumes of the guests inside the host as shadow copies. So its
no wonder I could not find the Volume GUID on the HyperV host - its only available during backup! This was very missleading, since I guessed it tried to back up a nonexistent volume.
Anyways: the creation of the timeout registry value fixed my problem. Thanks!
May 18th, 2012 5:01pm
Thanks danma_ for the solution! This worked for me! Sometimes backups did work, another time they failed. It started when I added an additional VM in Hyper-V. I changed the value and I saw that It took a bit over 10 minutes after VSS was ready and could
continue to backup, I guess it sometimes completed below 10 minutes and then the backup did work. A big "thank you" for the tip!
November 15th, 2012 10:01pm
Sorry for necroposting (althoug the issues is still valid for many), but
you have one extra zero in 12000000,
which is 200 minutes, not 20.
May 6th, 2013 4:49pm
Hi
I do not have any registry entry called SPP, I however have SoftwareProtectionPlatform. Are these the same? Should I create the DWORD "CreateTimeout" inside this key?
Thanks
I do not have the 'SPP' key either, and I have the SoftwareProtectionPlatform as well. I wonder if this is due to server versions (I'm on Server 2012)? If so what is the solution for this, if anyone has an idea.
Thanks.
November 7th, 2013 9:55pm