Packet Loss - Server 2012 & HyperV
We have setup and installed a new HP ML350p Gen8 server (E5-2620, 64GB Ram, 8 x 10K SAS Raid 10). The server has Windows Server 2012 installed as a host and two virtual servers running.
Nic1 - Server2012 (Host)
Nic2 - SBS2011 (Guest)
Nic3 - Server2008R2 (Guest)
Nic4 - Future Server
The problem is we are expiriencing random packet loss on all network cards at different times, overall its about 2-8%. As well as the occasional missing ping we are seeing blocks of 5-6 Request Time Out but otherwise its always <1ms. Pinging from the same
computer to the old ML350 G6 Server with Server2003 and other hardware is fine. I have also experienced the issue with just NIC1 and all VM's turned off.
So far we have tried tweeking the nic advanced properties with no success...
ethernet@wirespeed - Disabled
Large Send Offload - Disabled
Speed&Duplex - 1000mps & Full
PowerMangment - Off
Both the driver and firmware are the latest from HP and Server2012 has all updates installed.
We installed an Intel NIC to rule out any issues with the HP/Broadcom NICs but experienced the same issues.
Any suggestions as to what could be causing this?
December 27th, 2012 5:23pm
Also see the attached screenshot...
Ping from an XP desktop to virtual server (NIC3) had a block of timeouts but at the same time virtual (NIC1) to virtual (NIC3) was fine
December 27th, 2012 5:27pm
Anothre screen shot...
December 27th, 2012 6:08pm
After further testing overnight we are still seeing around 2-5% packet loss however it is inbound only. A ping from the new server back to the old had 0 packets lost compared to almost 1800 from old to new.
Is this normal?
MY only concern is that RDP freezes during packet loss so if we migrate our SQL database across it may cause issues with out applications.
December 28th, 2012 12:06pm
I have been running MS Network monitor and managed to capture a 6 packet drop in pings. The 16 second drops starts with a DHCPV6 SOLICIT and then during the drop the only traffic is DHCPV6 SOLICIT and ARP Requests.
Im going to turn off IPV6 on the nic and see what happens.
December 28th, 2012 4:44pm
Turning off ipv6 appears to have fixed the issue on the host however im not sure what im going to do about SBS2011.
Also i will test to see if givng the network card a fixed ipv6 address fixes the issue.
December 28th, 2012 5:54pm
Did you turn off IP 6 via the checkbox in network adapter properties or disable IPv6 completely via the registry (and was there a reboot involved)?
January 3rd, 2013 1:19am
Hi,
As a test you could try do disable all tcp offloading on the nics
i have had similar issues on my home hyper-v lab and in my case it was due to tcp offloading.
January 3rd, 2013 12:49pm
Did you turn off IP 6 via the checkbox in network adapter properties or disable IPv6 completely via the registry (and was there a reboot involved)?
We turned it off by unticking the checkbox and then rebooted. Since then there have been 0 dropped packets on all cards with it disabled.
January 3rd, 2013 8:32pm
Hi,
As a test you could try do disable all tcp offloading on the nics
i have had similar issues on my home hyper-v lab and in my case it was due to tcp offloading.
We disabled everything on the NIC advanced properties, on the Virtual NICS properties and at OS level but nothing made a difference only ipv6.
January 3rd, 2013 8:33pm
We've done the same - at the moment we're still busted but going to try some more stuff and see what happens. We're on Gen8 blades running 2012 DC. All 16 blades (plus 4 more blades in another chassis) affected as well as their VMs. Pretty
annoying!
January 3rd, 2013 9:20pm
We've done the same - at the moment we're still busted but going to try some more stuff and see what happens. We're on Gen8 blades running 2012 DC. All 16 blades (plus 4 more blades in another chassis) affected as well as their VMs.
Pretty annoying!
I dont think its related to Server 2012 or Gen8 Servers, since spotting the problem we have also found it occurs on Windows 7 and Server 2008R2. I need to investigate further and my ipv6 knowledge isnt great but it seems the card sends out a ipv6 DHCP address
and then blocks certain ipv4 traffic during the process.
January 4th, 2013 6:22pm
It sounds like you might have something in your network infrastructure that is causing this issue in regards to IPV6. I doubt very much that it is an issue with Windows IPV6 itself, as that is running in thousands of shops with no issues.
BTW, when it comes to 'disabling IPV6', clicking the checkbox is does not completely disable it. And, if you ever want to turn it back on, you will have to manually go through the environment to find those systems you disabled. Microsoft published
an article (http://support.microsoft.com/kb/929852) on the correct way to disable it.
January 4th, 2013 6:30pm
It sounds like you might have something in your network infrastructure that is causing this issue in regards to IPV6. I doubt very much that it is an issue with Windows IPV6 itself, as that is running in thousands of shops with no issues.
BTW, when it comes to 'disabling IPV6', clicking the checkbox is does not completely disable it. And, if you ever want to turn it back on, you will have to manually go through the environment to find those systems you disabled. Microsoft published
an article (http://support.microsoft.com/kb/929852) on the correct way to disab
January 4th, 2013 6:43pm
From the above capture there is a 30 seconds gap between the first ping and last ping. Every capture we have done the drop always starts with a DHCPV6 SOLICIT message.
January 4th, 2013 6:47pm
Sorry, despite who my employer is, I am by no means a networking expert. I do seem to recall, though, that Microsoft first does try an IPV6 request before an IPV4 request because it is more efficient. But after whatever 'handshaking' is done
and IPV4 is selected, IPV6 gets out of the way entirely. Without getting someone in there to evaluate the IPV6 settings on your network, it's really anybody's guess as to why partially disabling IPV6 'fixes' your issue. I would assume that you
have similar networking gear in multiple sites and that it is configured similarly, hence the similar results.
January 5th, 2013 2:39am
In my enviroment, I migrated a cluster from 2008 R2 to 2012, the hosts are HP Blade bl460c G6. After that migration some VM started to lost packets.
I solved the problem adding a new network card to VM and changing the IP direcction to new network card.
March 21st, 2013 6:57pm
Just a short note to share the fact that we've been troubleshooting an issue with lost pings on a WS2012 Hyper-V cluster on the HP c-Class BL685g7. After reviewing the wireshark network traces, troubleshooting lead us to look at the WS2012 teaming
(LBFO). As you may know, there are multiple ways to implement OS-provided teaming in WS2012. Changing the load balancing from switch independent / address hash to switch independent / Hyper-V resolved the issue, which before this change manifested
itself in a 2-5% ping loss on a ping -t.
Further reference for your reading pleasure:
Windows Server 2012 NIC Teaming (LBFO) Deployment and Management --> search google or the microsoft site.
July 25th, 2013 3:31am