Packet Loss - Server 2012 & HyperV

We have setup and installed a new HP ML350p Gen8 server (E5-2620, 64GB Ram, 8 x 10K SAS Raid 10). The server has Windows Server 2012 installed as a host and two virtual servers running.

Nic1 - Server2012 (Host)
Nic2 - SBS2011 (Guest)
Nic3 - Server2008R2 (Guest)
Nic4 - Future Server

The problem is we are expiriencing random packet loss on all network cards at different times, overall its about 2-8%. As well as the occasional missing ping we are seeing blocks of 5-6 Request Time Out but otherwise its always <1ms. Pinging from the same computer to the old ML350 G6 Server with Server2003 and other hardware is fine. I have also experienced the issue with just NIC1 and all VM's turned off.

So far we have tried tweeking the nic advanced properties with no success...

ethernet@wirespeed - Disabled
Large Send Offload - Disabled
Speed&Duplex - 1000mps & Full
PowerMangment - Off

Both the driver and firmware are the latest from HP and Server2012 has all updates installed.

We installed an Intel NIC to rule out any issues with the HP/Broadcom NICs but experienced the same issues.

Any suggestions as to what could be causing this?

December 27th, 2012 5:23pm

Also see the attached screenshot...

Ping from an XP desktop to virtual server (NIC3) had a block of timeouts but at the same time virtual (NIC1) to virtual (NIC3) was fine

Free Windows Admin Tool Kit Click here and download it now
December 27th, 2012 5:27pm

Anothre screen shot...
December 27th, 2012 6:08pm

After further testing overnight we are still seeing around 2-5% packet loss however it is inbound only. A ping from the new server back to the old had 0 packets lost compared to almost 1800 from old to new.

Is this normal?

MY only concern is that RDP freezes during packet loss so if we migrate our SQL database across it may cause issues with out applications.

Free Windows Admin Tool Kit Click here and download it now
December 28th, 2012 12:06pm

I have been running MS Network monitor and managed to capture a 6 packet drop in pings. The 16 second drops starts with a DHCPV6 SOLICIT and then during the drop the only traffic is DHCPV6 SOLICIT and ARP Requests.

Im going to turn off IPV6 on the nic and see what happens.

December 28th, 2012 4:44pm

Turning off ipv6 appears to have fixed the issue on the host however im not sure what im going to do about SBS2011.

Also i will test to see if givng the network card a fixed ipv6 address fixes the issue.

Free Windows Admin Tool Kit Click here and download it now
December 28th, 2012 5:54pm

Did you turn off IP 6 via the checkbox in network adapter properties or disable IPv6 completely via the registry (and was there a reboot involved)?
January 3rd, 2013 1:19am

Hi,

As a test you could try do disable all tcp offloading on the nics

i have had similar issues on my home hyper-v lab and in my case it was due to tcp offloading.

Free Windows Admin Tool Kit Click here and download it now
January 3rd, 2013 12:49pm

Did you turn off IP 6 via the checkbox in network adapter properties or disable IPv6 completely via the registry (and was there a reboot involved)?

We turned it off by unticking the checkbox and then rebooted. Since then there have been 0 dropped packets on all cards with it disabled.

January 3rd, 2013 8:32pm

Hi,

As a test you could try do disable all tcp offloading on the nics

i have had similar issues on my home hyper-v lab and in my case it was due to tcp offloading.


We disabled everything on the NIC advanced properties, on the Virtual NICS properties and at OS level but nothing made a difference only ipv6.
Free Windows Admin Tool Kit Click here and download it now
January 3rd, 2013 8:33pm

We've done the same - at the moment we're still busted but going to try some more stuff and see what happens.  We're on Gen8 blades running 2012 DC.  All 16 blades (plus 4 more blades in another chassis) affected as well as their VMs.  Pretty annoying!
January 3rd, 2013 9:20pm

We've done the same - at the moment we're still busted but going to try some more stuff and see what happens.  We're on Gen8 blades running 2012 DC.  All 16 blades (plus 4 more blades in another chassis) affected as well as their VMs.  Pretty annoying!
I dont think its related to Server 2012 or Gen8 Servers, since spotting the problem we have also found it occurs on Windows 7 and Server 2008R2. I need to investigate further and my ipv6 knowledge isnt great but it seems the card sends out a ipv6 DHCP address and then blocks certain ipv4 traffic during the process.
Free Windows Admin Tool Kit Click here and download it now
January 4th, 2013 6:22pm

It sounds like you might have something in your network infrastructure that is causing this issue in regards to IPV6.   I doubt very much that it is an issue with Windows IPV6 itself, as that is running in thousands of shops with no issues.

BTW, when it comes to 'disabling IPV6', clicking the checkbox is does not completely disable it.  And, if you ever want to turn it back on, you will have to manually go through the environment to find those systems you disabled.  Microsoft published an article (http://support.microsoft.com/kb/929852) on the correct way to disable it.

January 4th, 2013 6:30pm

It sounds like you might have something in your network infrastructure that is causing this issue in regards to IPV6.   I doubt very much that it is an issue with Windows IPV6 itself, as that is running in thousands of shops with no issues.

BTW, when it comes to 'disabling IPV6', clicking the checkbox is does not completely disable it.  And, if you ever want to turn it back on, you will have to manually go through the environment to find those systems you disabled.  Microsoft published an article (http://support.microsoft.com/kb/929852) on the correct way to disab

Free Windows Admin Tool Kit Click here and download it now
January 4th, 2013 6:43pm

From the above capture there is a 30 seconds gap between the first ping and last ping. Every capture we have done the drop always starts with a DHCPV6 SOLICIT message.

January 4th, 2013 6:47pm

Sorry, despite who my employer is, I am by no means a networking expert.  I do seem to recall, though, that Microsoft first does try an IPV6 request before an IPV4 request because it is more efficient.  But after whatever 'handshaking' is done and IPV4 is selected, IPV6 gets out of the way entirely.  Without getting someone in there to evaluate the IPV6 settings on your network, it's really anybody's guess as to why partially disabling IPV6 'fixes' your issue.  I would assume that you have similar networking gear in multiple sites and that it is configured similarly, hence the similar results.
Free Windows Admin Tool Kit Click here and download it now
January 5th, 2013 2:39am

In my enviroment, I migrated a cluster from 2008 R2 to 2012, the hosts are HP Blade bl460c G6. After that migration some VM started to lost packets.

I solved the problem adding a new network card to VM and changing the IP direcction to new network card.

March 21st, 2013 6:57pm

Just a short note to share the fact that we've been troubleshooting an issue with lost pings on a WS2012 Hyper-V cluster on the HP c-Class BL685g7.  After reviewing the wireshark network traces, troubleshooting lead us to look at the WS2012 teaming (LBFO).  As you may know, there are multiple ways to implement OS-provided teaming in WS2012.  Changing the load balancing from switch independent / address hash to switch independent / Hyper-V resolved the issue, which before this change manifested itself in a 2-5% ping loss on a ping -t.

Further reference for your reading pleasure:
Windows Server 2012 NIC Teaming (LBFO) Deployment and Management --> search google or the microsoft site.

Free Windows Admin Tool Kit Click here and download it now
July 25th, 2013 3:31am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics