Hello,

We're trying to enable VSS backups through Windows Server Backup, but still having issues.
We're running a few VMs on CentOS 6.6 x64, fully updated / patched, hyperv modules installed, cpanel installed. 
However, while trying to backup those VMs, *some* of them are crashing / loads going up like crazy.

We do have a few VMs running Ubuntu 14.04 LTS, fully updated / patched as well, but we're still able to backup those VMs without any issues.

We've been able to isolate the issue and it turns out that the VMs that get backed up are the ones that have a VHDX file of <50GB.
VMs that are having issues are the ones having a bigger VHDX file (>100GB).

Both type of VMs are running the exact same OS, patches, modules and services.
So it really appears to be the size of the disk that is causing issues.

Some snippet of our logfiles when a backup is working:
Jun  9 08:39:02 dev Hyper-V VSS: VSS: freeze of /boot: Success
Jun  9 08:39:02 dev Hyper-V VSS: VSS: freeze of /: Success
Jun  9 08:39:02 dev Hyper-V VSS: VSS: thaw of /boot: Success
Jun  9 08:39:02 dev Hyper-V VSS: VSS: thaw of /: Success


And when backup isn't working:
Jun  9 12:18:04 dev Hyper-V VSS: VSS: freeze of /boot: Success
[...] and it hangs / load spikes like crazy.

Does it sound familiar to you guys ?
Any fixes available ?

This is quite crucial as we need to get those backups up and running.

Thanks,
- Luc

• Changed type ltellier Tuesday, June 09, 2015 4:41 PM

Hello,

It is still not working.

Here is what we've done so far.

Removing the LIS built into CentOS6.6:

root@dev [~]# rpm -qa |grep hyperv
hyperv-daemons-0-0.15.20130826git.el6.x86_64
hypervkvpd-0-0.15.20130826git.el6.x86_64
hypervfcopyd-0-0.15.20130826git.el6.x86_64
hypervvssd-0-0.15.20130826git.el6.x86_64
hyperv-daemons-license-0-0.15.20130826git.el6.noarch

root@dev [~]# rpm -e hyperv-daemons hypervkvpd hypervfcopyd hypervvssd hyperv-daemons-license

root@dev [~]# rpm -qa |grep hyperv
root@dev [~]# 

Installing the latest LIS:

root@dev [~]# cd /mnt/RHEL66/
root@dev [/mnt/RHEL66]# ./install.sh 
Installing the Linux Integration Services for Microsoft Hyper-V...
Preparing...                ########################################### [100%]
   1:kmod-microsoft-hyper-v ########################################### [100%]
Preparing...                ########################################### [100%]
   1:microsoft-hyper-v      ########################################### [100%]
Saving old initramfs
Installing new initramfs
Adding KVP Daemon to Chkconfig....
Starting KVP Daemon....
Adding VSS Daemon to Chkconfig....
Starting VSS Daemon....
Adding FCOPY Daemon to Chkconfig....
Starting FCOPY Daemon....
 Linux Integration Services for Hyper-V has been installed. Please reboot your system.

We then rebooted the VM, started a backup and unfortunately, didn't work at all.

Processes are getting hung all around and the only way to make the VM reachable is to reboot it.

Unfortunately, we're not seeing any errors within the VM log files.

Please let me know if you need anything else.

Thanks,

• Edited by ltellier Thursday, July 02, 2015 6:47 PM

Hello Joshua,

Here you go:

root@dev [~]# ps aux | grep vss
root      1256  0.0  0.0   4056   468 ?        Ss   13:55   0:00 /usr/sbin/hv_vss_daemon
root      1905  0.0  0.0 103248   876 pts/0    S+   14:08   0:00 grep vss

root@dev [~]# lsof -p 1256
COMMAND    PID USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
hv_vss_da 1256 root  cwd    DIR              253,0     4096       2 /
hv_vss_da 1256 root  rtd    DIR              253,0     4096       2 /
hv_vss_da 1256 root  txt    REG              253,0     9744 1835719 /usr/sbin/hv_vss_daemon
hv_vss_da 1256 root  mem    REG              253,0  1921176  392461 /lib64/libc-2.12.so
hv_vss_da 1256 root  mem    REG              253,0   154624  392821 /lib64/ld-2.12.so
hv_vss_da 1256 root    0u   CHR                1,3      0t0    4165 /dev/null
hv_vss_da 1256 root    1u   CHR                1,3      0t0    4165 /dev/null
hv_vss_da 1256 root    2u   CHR                1,3      0t0    4165 /dev/null
hv_vss_da 1256 root    3u  unix 0xffff8801012fea40      0t0   11580 socket
hv_vss_da 1256 root    4u  sock                0,6      0t0   11581 can't identify protocol

As for the VM specs:

GEN: Version 1
CPU: 1x

RAM: 4GB (static)

HDD: 100GB (Configured on IDE Controller 0:0)
vNIC: 1x (Configured as non-Legacy, VMQ enabled, IPSec offloading enabled, Max Number offloaded 512, Static MAC address specified, Protected neetwork enabled)

Please let me know if you need further information, I will be happy to help!

- Luc

• Edited by ltellier Friday, July 10, 2015 6:18 PM

Just FYI, I've updated to LIS 4.0.11 and still got the same issue; the VM just hangs and becomes unusable.

/var/log/messages:

Aug 15 21:43:14 dev HV_FCOPY: HV_FCOPY starting; pid is:1261
Aug 15 21:43:14 dev HV_FCOPY: open /dev/vmbus/hv_fcopy failed; error: 2 No such file or directory
Aug 15 21:43:14 dev KVP: KVP starting; pid is:1273
Aug 15 21:43:14 dev Hyper-V VSS: VSS starting; pid is:1285
Aug 15 21:43:14 dev kernel: hv_utils: VSS daemon registered
Aug 15 21:43:14 dev KVP: KVP LIC Version: 4.0.11
Aug 15 21:43:14 dev kernel: hv_utils: KVP: user-mode registering done.

dmesg:

hv_utils: VSS daemon registered
hv_utils: KVP: user-mode registering done.
hv_utils: VSS: timeout waiting for daemon to reply
hv_utils: VSS: timeout waiting for daemon to reply
[...]

RPMs:

# rpm -qa |grep hyper
kmod-microsoft-hyper-v-4.0.11-20150728.x86_64
microsoft-hyper-v-4.0.11-20150728.x86_64

Please let me know if you guys need more information.

Thanks,

- Luc

• Edited by ltellier Sunday, August 16, 2015 1:54 AM

Not such if useful.

Looks like your hv_vss_daemon is not running. Query status and try to start if not running

/etc/init.d/hv_vss_daemon status
/etc/init.d/hv_vss_daemon stop
/etc/init.d/hv_vss_daemon start

Not such if useful.

Looks like your hv_vss_daemon is not running. Query status and try to start if not running

/etc/init.d/hv_vss_daemon status
/etc/init.d/hv_vss_daemon stop
/etc/init.d/hv_vss_daemon start

Thanks Mike, unfortunately, it is still not working.

VSS is still running but for some reason, it times out:

hv_utils: VSS daemon registered
hv_utils: KVP: user-mode registering done.
hv_utils: VSS: timeout waiting for daemon to reply
hv_utils: VSS: timeout waiting for daemon to reply