Volsnap Event ID: 25 and Loss of Previous Backups
I have a Windows Small Business Server 2008 machine being backed up using the Windows Server Backup utility.  The backups are being made to two external USB attached 1TB drives.  Each drive is rotated out weekly, meaning that one drive is attached one week, then the other drive is swapped in the following week, ad infinitum for the best part of a year.  Two backups were being made each day to the attached drive, one at 12pm and one at 9pm.

About two weeks ago it appears that one of the two drives reached capacity and began shedding the oldest backups, as confirmed by the VOLSNAP Event ID: 33 entries in the System Event log.  On the second day, however, ALL shadow copies on the attached backup driver (volume) were deleted, as confirmed by a single VOLSNAP Event ID: 25.  (See below for both Event ID message contents.)  Unfortuneately, neither events were noted until the drive was swapped out with the other drive and some how the problem was propagated to the second drive, resulting in the deletion of all shadow copies on the second drive, too. 

The end result... BOTH drives lost ALL previous backups leaving only the most recent full backup and no clue as to why it happened or how to prevent it from happening again.  Virtually no older backup file versions could be recovered.

I was under the impression that, by design, the WSB utility would only start deleting the oldest backups (on a full drive) in order to make room for the newer backups.  Granted, had I been keeping attention, I would have actually swapped out the full drives before the actually got full, so that NO data was deleted.

If I'm to continue using the Windows Server Backup utility, I need to know what happened and how to absolutely prevent it from happening again.  I never expected to loose BOTH backups at the same time, and due to budget restraints, two rotating backup drives seemed a logical solution.

Here is text from the two VOLSNAP Event ID messages found in the System Log:

Event ID: 25
Source:  volsnap
Level: Error
Description:  The shadow copies of volume \\?\Volume{93896cfc-d904-11dd-8cb5-001ec9ef572e} were deleted because the shadow copy storage could not grow in time.  Consider reducing the IO load on the system or choose a shadow copy storage volume that is not being shadow copied.

Event ID: 33
Source: volsnap
Level: Information
Description: The oldest shadow copy of volume \\?\Volume{93896cfc-d904-11dd-8cb5-001ec9ef572e} was deleted to keep disk space usage for shadow copies of volume \\?\Volume{93896cfc-d904-11dd-8cb5-001ec9ef572e} below the user defined limit.


NOTE:  Though I'm not sure if this has any bearing on the situation, the server is configured to make two snapshots a day of two volumes on the server, however, when I looked at the Shadow Copy settings I couldn't quite be sure if the VSS wasn't somehow also making snapshots on the two external drives, even though I haven't specified VSS to do so.  None show up on the two replacement drives I'm now using, so I'd say not.


Information on how to prevent this from happening again would greatly be appreciated.  (I'll be making sure no backup drive gets even close to being full from now on, that's for sure!)


Thanks,



Joseph.
February 11th, 2010 8:37pm

Hello Joseph,
    The EventId 25 from volsnap deleting all the snapshots is not an expected behaviour. I need to follow up with some other teams to find out
    why such an event happened. I have not seen this issue earlier.

    The windows backup explicitly reclaims space for the current backup by deleting older snapshots OR volsnap will automatically delete older
    versions in case of Copy On Write (COW) which is exactly what happened with the EventId: 33 and this is an expected behaviour.

    Can you please share with us some informations:

    1. What is the free/used space of the source volumes?
    2. What is the free/used space of the target volumes?
    3. The VSS shadow copy space on the target volulme. Use the following command from an elevated window. Assuming D: to be the target volume.
        vssadmin list shadowstorage /for=D:

Thanks,

Bikash Agrawala [MSFT]

--------------------------------------------------------------------------------- This posting is provided "AS IS" with no warranties, and confers no rights

Free Windows Admin Tool Kit Click here and download it now
February 12th, 2010 1:22pm

^ T:\ external 1.5TB USB2 drive is used for backup correct.

 

That T:\ drive is EXCLUSIVELY used for that, no other process is accessing that drive except BITLOCKER for the encryption

 

Let me know if you need more details for troubleshooting, thanks


  • Edited by Kilimats Monday, November 21, 2011 10:09 AM
November 21st, 2011 10:09am

I'm no guru with VSS but to my understanding, the built in backup use VOLUME shadow copy to store incremental backups. The shadow copy tab is about FILE shadow copy for versioning, different stuff i think. Here is a good explanation: http://www.wbadmin.info/articles/how-does-windows-server-2008-backup-work.html

How do i escalate this a year and a half old issue to Microsoft ?


  • Edited by Kilimats Monday, November 21, 2011 11:40 AM
Free Windows Admin Tool Kit Click here and download it now
November 21st, 2011 11:39am

Has any one tried applying this fix

http://support.microsoft.com/kb/833167


That hot fix is for Win2003

  • Edited by Robert Ebers Wednesday, October 10, 2012 8:43 PM
October 10th, 2012 8:43pm

I've tried to fix a similar problem with Microsoft Support.

During normal Windows 2008 R2 file server server operation (writing files to the NTFS volume) Event 25 appears sporadicly and all snapshots are lost.
There is only the "Microsoft Software Shodow Copy provioder 1.0" active, the System ist only used as file server for backups, ist has a 40 TByte RAID Volume with a sequential read/write performace of 950 GByte per Second. Cluster size of the NTFS volume is 16k and the MinDiffAreaFileSize was set to the maximum recommended valzue of 3000 MByte.

Unfortunately, Microsoft has no intention to fix that problem, because Microsoft does not want to enter into competition with vendors of professional storage systems. After some time Microsoft will tell the customers, that the product ist out of mainstream support. Probably also applies to Windows Server 2012.

I have a similar problem on a Windows 7 laptop system.  VSS has already corrupted my HDD a couple of times costing me untold hours (actually days) of work in restoring backups, recovering data and reinstalling applications.  I already tried increasing the VSS storage (from 10GB to 48GB) and increasing MinDiffAreaFileSize to the recommended 5% of restore point storage allocated.  This seemed to help for a couple of days and then the problem recurred.  I had high hopes of fixing the problem by reformatting the main partition to use 16K, as suggested in this KB article, but your experience above has dashed my hopes.  In fact, I'm not even sure that the above-mentioned KB is relevant.

This appears to be a fairly common problem, going back for MORE THAN 6 YEARS!  This KB article refers to a hotfix for Server 2003, but if the problem was already fixed in 2003 SP1, why is it still occurring in later OSs like Windows 7 and Server 2008?  Has it been fixed in any of the newer OSs, e.g. Windows 8?

Some articles blame 3d party VSS writers for this behaviour, and suggest:

vssadmin list writers

to identify them.  On my system, this yields a list of 11, but no clue as to which are (not) from Microsoft.

How can I make this distinction?

What good is a backup service, which is not only unreliable, but even corrupts HD storage?

The easiest solution would seem to be to turn off VSS, but when I look at the services on my laptop, it isn't running anyway?!  How can this be?

This KB article suggests using a different HDD (not physically possible with my laptop) or at least another volume for either the VSS storage or the pagefile.  Can anyone suggest which of these options would be better and why?

Is there some other workaround for this?

P.S. To answer my own question about whether to move VSS storage or pagefile, MS has made the decision for me, by crippling vssadmin under Windows 7.  See this article.

  • Edited by rsbrux Monday, June 24, 2013 6:42 AM added information
Free Windows Admin Tool Kit Click here and download it now
June 23rd, 2013 11:16am

I have a Windows Server 2008 R2 SP1 on Dell Server PE R210 with two 2TB drives using RAID 1 on an SAS 6/iR Adapter. The drives are about half full. It is used as a file server for a 50 person network and also has two hyper-v's. One hyper is a Barracuda Spam and Virus Filter and the other is another 2008 R2 server running one specific application. The server is stable, except that from time to time I do have trouble with opening the WSB console. It will work fine for months then all of a sudden take hours. Disk Management also will run into similar problems around the same time. I attributed this to possibly problems with the USB backup drive or corruption in the WSB catalogs. One time deleting and restoring the catalogs from the backup disk fixed the WSB console issues. Recently I have been having problems opening WSB and disk management again. I rebooted the server and I noticed that WSB opened quickly but my backups were not accessible anymore from Recover..., only the one it did previously. I tried the restore catalog function but that didn't work. I dug deeper and ran vssadmin list shadows and there was only one entry. There were about 100 copies\backups previously. I noticed the event 25 volsnap and wound up here searching for an answer. I expect to be able to go back to any point in time in the last few months and be able to pull up a missing file. WSB sure gives the impression it should do that. Inexplicably, all the backup copies appear to be gone. I understand overwiting the oldest copy to make room, which I am familiar with. In fact I was getting ready to replace this drive with another as it was almost full. It had about 150 GB left on a 2TB USB drive. It still says 150GB free, even though the shadow copies appear to be gone. The vhd from the backup is around 800GB. This leads me to believe that they might still be there since it didn't seem to free up space, or at least not in proportion to what it should have if it deleted all the copies. I should be looking at a TB or so free, if the shadow copies were truly deleted from the disk. Are they corrupted, so they appear missing, but still take up space? I don't know. Any help would be appreciated. Definetly going to be moving away from WSB after this. I had my issues with it before, but was always able to make it work. This is just inexcusable and I'm very pissed as a loyal Microsoft customer that their product takes liberty to just delete files when things get a little hairy.

  • Edited by AZ2006 Friday, August 30, 2013 7:03 PM
August 30th, 2013 7:00pm

Same  Problem here Server is a Storage Server 2012 Cluster (HP EasyStore 5530)

Fully Patched. 
Free Windows Admin Tool Kit Click here and download it now
February 24th, 2014 9:57am

I think this problem is related to memory consumption problems. When you are using slow drive (such as USB one) Windows cashes disk writes using system RAM. The root of the problem is: there is no default limit of memory Windows x64 can use as cache. So every time Windows writes large data to relatively slow drive, you may see something like

  • Available memory is almost exhausted.
  • The system file cache consumes most of the physical RAM.

So when you write your backup data, and the VSS needs some more memory and asks system for more, it may get an error, and delete all the snapshots.

http://support.microsoft.com/kb/976618/en-us

Maybe this tool will help "Microsoft Windows Dynamic Cache Service"

Or maybe setting LowMemoryThreshold will be helpfull.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management

LowMemoryThreshold                (REG_DWORD) This value is in Mbytes







March 1st, 2014 11:21pm

It's happening again - Volsnap 25, deleting all previous backups...

This seems to be the case since the introduction of VSS in Windows 2003 (if you search vor volsnap 25 events). Microsoft - no one listening?

Free Windows Admin Tool Kit Click here and download it now
June 12th, 2014 8:07am

Still today (on 3 Feb 2015), I have the same problem.

On SBS2008, as well as on WS2012 Essential, Previous existing backup are completely erased from external drive and I found a volsnap EventID 25 in the System log...

So, it seems that since 11 Feb 2010 (date of the 1st message in this huge thread), Microsoft provides NO resolution for this crucial problem : Deleting ALL existing backup on a drive due to an unkown and unpredictable VSS behavior ??? That's just incredible !!!

Is there someone from MS support team that read and continue to update this thread?

February 3rd, 2015 1:49pm

I started a new question because I hadn't come across this thread.

I have exactly the same problem on a fully patched Windows 2012 server and lost about 200 backups.

Can anyone confirm that the suggestion above works (vssadmin resize shadowstorage /for=Volume_Letter: /on=Volume_Letter: /maxsize=80%) or is this still unresolved?

Thank you

Free Windows Admin Tool Kit Click here and download it now
May 12th, 2015 4:07pm

I had this problem (volsnap event id 25) too. All previous VSS copies removed.

vssadmin List ShadowStorage
and
vssadmin List Shadows
both returned empty. Grr...

They were still visible in the Device Manager with env setting devmgr_show_nonpresent_devices=1 and the Show Hidden Devices option on. They were greyed out, so I had to delete these by hand.

Anyway, I think the description of the event is correct. Too much disk IO at startup is the cause for this problem. I my case it was HKLM\SYSTEM\CurrentControlSet\Control\CrashControl "DedicatedDumpFile"="C:\\CrashDumpFile.dmp"
This creates a dump file at boot time with the size of your RAM. Very Disk IO intensive!
After I renamed value DedicatedDumpFile to DedicatedDumpFile.OFF VSS copies survived a reboot!

On my workstations I create manual/scheduled VSS snapshots using wmic.
Wmic.exe shadowcopy call create ClientAccessible,"C:\"
since vssadmin doesn't provide that on workstations ;)

So for a solution you might want to check what files are written during the boot. It will help.

Cheers

May 15th, 2015 6:18pm

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics