Dear all,

I have a big problem with my hyper-v server and network card. I've found a lot of topic around this question but none of them helped me.

I have a Win 2008 R2 Server with only Hyper-V installed. I have 3 NIC , one is reserved to the host management and the others two are reserved for virtual machine .

All NIC are Intel, one is integrated on the Intel motherboard and the others are Intel Gigabit ET Dual port .

Before installing the role of Hyper-V all NIC worked perfectly with speed around 1Gb (file transfer average 90 MB/Sec, of course I have a Gigabit LAN) .

After Hyper-V install, the host system  is going very slow and the network file transfer using the dedicated NIC is about ten time slower (9-10MB /sec) . Same problem with all VM off and also if I remove all VM !

I tried all of suggestion found in old post (disabling TCP checksum / offload ...) but nothing happens.

What is really strange is that by removing all network card all the Host operating system seems to improve the speed... but, of course, I can't use  my LAN :(

I need help because every night I must make a full backup of Virtual hard disk from the host to a backup server in my lan.

My configuration:

- Server Win 2008 R2 SP1 with Hyper-V and no others role

- 2 Virtual Machine (one Win 2008 R2 and one Win 7 Ultimate)

- NAS Linux (access via SMB with no problem)

- Various Windows 7 Pro client

 

Thanks in advance to everybody,

Luca M.

What network adapter are you using?  Have you got the latest network drivers from the manufacturer?  Is it possible to try a different network adapter?

Ben and Luca, Thanks for posting. 

We are having almost the exact same issue. We also need help.  High urgency.

We have tried all of the suggestions above. Mutiple clean installs with latest drivers, BIOS, Etc.
Without Hyper-V 90MB/s network copy speed.  With Hyper-V on the Host 9MB/s
Windows 2008 R2 SP1 Enterprise, MotherBoard Intel S5520HCR, 48GB RAM,  Dual CPU

Thanks - John Webster
john.webster@netsupport.com

Dear Ben,

I'm using Intel Gigabit ET Dual port network adapter (http://www.intel.com/products/server/adapters/gb-ET-Dual-Port/gb-ET-Dual-Port-overview.htm)

I've tried with another Intel NIC too but I have the same problem.

I'm using the latest driver, bios etc.

Any idea?

Thanks for any help with this!

 

Luca M.

Were you seeing this problem before installing SP1?

I've installed hyper-v after SP1 , so I can't know if this problem is linked to the service pack .

But now I'm 100% sure that this issue is caused by Hyper-V , and it not depend by the nic... : if I disable the role in the host (bcdedit /set hypervisorlaunchtype off ) , after reboot my file trasfer are:

- in local (copy/paste a big file in my phisical local disk) : 338 MB / sec

- in LAN (from my NAS to the Host OS running Hyper V) : 116 MB / sec

When I re-enable Hyper-V, the speeds become:

- in local: 98 MB/sec

- in LAN: 10,5 MB/sec

The problem disappear also if I boot in safe mode with networking .

I have a RAID 6 with 4 SSD and a megaraid 8161 sata II controller, I can reach sometimes 400 MB /sec so I think the problem is not related to hard drive.

My CPU is an Intel I7 980X .

I noticed that when hyper-v is enabled and I start a file transfer my cpu usage increase a lot, sometimes it reaches 80-90% !

Any idea with this new information?

Really thanks to your support.

Luca M.

 

EDIT:

I confirm that the problem isn't related to the network adapter. Now if I try by removing all NIC the internal trasfer speed (copy/past from physical disk) still remains slow (near 100 MB / sec) .

I'm thinking about some BIOS setting wrong... but virtulization functions are all enabled...

Please help me!

Ben Luca, thanks for the posts. 

We  are still having the issue as well.

We have tried 5 separate installs of the Server OS. 
Three different people have done the installs.
We have used 2008 R2 SP1 Ent, 2008 R2  SP1 Std,  2008 R2 Ent, 2008 R2 Ent with SP1 installed later via update.
Is does not seem to be SP1 specific.  We have tried with and without the additional Intel 4 Port NICs.
We have done BOIS updates, used default drivers, and the most current ones off the intel site.
We have tried the changes to the TCP offload settings. 

The only constant is that once we enable the Hyper-V role the copy slows way down.
Also, the server slows down just opening windows.  
There are brief periods of time that we thought we had it fixed, but the problem reappears.

We have boxed the server up and sending back to the shop today for more detailed testing.
Please let me know how we can work together to resolve this urgent issue?

Thanks - John Webster
john.webster@netsupport.com 

 

 

Dear John,

can you write your hardware configuration ?

Maybe we can find something that link our problem...

Thanks a lot,

Luca M.

One more note, before we discovered the issue we tested the server with JamSoft Heavy-load for 36 hours and had no issues.
However looking back, I now realize the IO was all local and we did not test the NIC's IO except for Web Browsing.

Thanks - John Webster

This is a long shot but have you tried to disable IPv6 on all NIC cards?

Something to test:

Can you test the performance of the network adapters when Hyper-V is installed - but the network adapters are not configured for a virtual network? (this will help me to figure out where we should be looking)

BIOS virtualization options sounds to be the cause of the issue.

 

It's not only NIC as you see CPU overload and IO going down (local copy).

 

Could be a good test to switch off BIOS options see the results...

 

Downgrade to a previous BIOS version (stable and working with Hyper-V) test again.

 

Then if you find the cause, get in touch with the constructor.

 

Good luck!

 

Regards, Yannick.

Specialy if you confirm that the issue start even with VM off.

 

 

Dear Ben and Yannic,

the issue start even with Vm off and even without any VM .

I've tried various combination of bios settings, but performances still remain slow.

Tried different BIOS version, too.

Two hours ago I've installed VMWARE ESXI and it runs perfectly at full speed... so now I think it's not a BIOS-related problem.

What to say... I will use VWMARE again until hyper-v performance will be acceptable !

But in someone has some new idea or something I'll try it !

Thanks to everybody.

Luca M.

Luca,

I may not be able to get the detailed hardaware configuation.
But, I have summary below.
We have sent the server back to the vendor to be diagnosed.
They may just provide us a replacement.  I will post any results that they find.

Thanks - John Webster

Server Info:
=========
CPU Intel  E5620 x 2
48GB RAM
Server Board:  S5520HCR
http://ark.intel.com/Product.aspx?id=36599  
 Disk Controler Adaptec 5405  ( 600GB  x 4  15K SAS RAID 10)
http://www.adaptec.com/en-us/support/raid/sas_raid/sas-5405/
OnBoard RAID (2TB x 2 WD RE4 SATA RAID1)
Intel 4 Port Server NIC (I do not have part number right now)

Hi Crivella,

Even I am facing this kind of issue but I am using Hyper-V with Clustering on SAN.
Please clarify if u can
1) R u using any clustering for VM's on Hyper-V.
2) How do you access NAS box on Hyper-V?

Thanks & Regards
Anand S

 

He said that just adding the role Hyper-v server start the problem.

 

No VM, no Cluster, no NAS... Just having the service.

 

Yannick

Did you ever figure out the solution to this problem? (windows 2008 r2 host server slows down dramatically when hyper-v role is installed, even with no VMs running)

I'm experiencing the same thing, even with a "virgin" install of win2008r2 (no non-MS drivers, no patches).

tried with and without patches, tried disabling tcp offload.

as soon as i remove hyper-v role performance goes back to normal.

 

p.s. also experiencing very slow performance on host with hyper-v 2008r2-sp1 server (the free command-line only version).

 

We have the same problem in our environment (Windows 2008R2 Enterprise, intel 5520 server, 4x intel nic). We can reproduce the "slowness" only after creating a virtual network. only installing the hyper-v role gives no problems, only after creating a virtual network the "slowness" comes.

 

windows 2008r2 enterprise sp1, intel driver 16.3 or microsoft driver (same issue)

 

regards

Armin Sattler

Don't know if this thread is dead, but I am having a simular issue. My CPU is fine but but network is SLOOOOOW. Did the TCPIP offload work around with no change. Win 2008R2 SP1 new VM on top of CORE host. On a Lenovo RD240

Have you installed KB2263829 on your Hyper-V?

I would like to throw my hat in the ring for this issue as well.  I see a reocurring theme, it appears most of us are using Intel Motherboards and Intel NIC's in combination with Hyper-V.

The symptom is the servers disk IO drops to 25% and the system slows to a crawl after installing the Hyper-V role.  Uninstalling the Role results in the system going back to fast speeds.

I've installed the latest NIC Drivers for intel (currently 16.5) and updating bios and all drivers, chipset, etc.

I have replicated this on 3 servers, all Intel Chassis, Intel Mobo's, Adaptec RAID controllers, and Intel On-board NIC's.

Is there anyone out there that can help?

Yes, TCP Offloading is off on all NIC's

No there is no Graphics card

Yes KB2263829 is installed

Have you tried disabling TCP offloading? That's a common issue with network throughput

 

Explicit Intel NIC mention: http://social.technet.microsoft.com/Forums/en-US/winserverhyperv/thread/bdc40358-45c8-4c4b-883b-a695f382e01a

Other offloading/slow network threads

http://social.technet.microsoft.com/Forums/en-US/winserverhyperv/thread/133547e6-95c4-4596-bff9-3099000c2078

http://social.technet.microsoft.com/Forums/en-US/winservercore/thread/d0c55df9-a27c-4876-bc5a-8ac7f1b46462

http://msdn.microsoft.com/en-us/library/ff571012.aspx

 

 

Yes, I specifically turned off EVERYTHING to do with offloading on all the NIC's, this makes no difference.  I've attached images so you know I'm telling the truth, for reference sake there are no VM's yet, this is a fresh load of Windows, and there is NO Virtual Network.

If I run HD Tune before installing the Hyper-V Role, I get about 350MB/s on my Adaptec RAID 5 Array with 6 Drives.

After Installing Hyper-V Role I get 60MB/s on the same RAID, and the system just slows to a crawl.  I can replicate it.

 

I've had this same problem with a PowerEdge R910. As soon as Hyper-V is installed, disk copy performance drops to 10-20% of what it was prior to the installation.

I've found a workaround that resolves the problem for me. Our R910 has 4 x Intel e7-8867l processor. If I disable the C states option within the processor settings of the BIOS, the copy speed returns to pre-Hyper-V installation speeds.

I have a support incident raised with Microsoft and will be requesting a fix for this issue that doesn't include disabling the C States functionality.

I faced the same problem and it was solved by disabling Ethernet@WireSpeed option for the physical NIC mapped to this VM

• Proposed as answer by lepach8 Friday, February 01, 2013 5:31 PM

Add me to this un-resolved issue. Though, my issue is only with the VMs themselves. The Host gets 92 MB/s (gigabit network) through its file shares. Latest BIOS, nic drivers, two Intel nics. W2K8R2 SP1 One NIC is dedicated to HOST only. The other NIC is dedicated to the Hyper-V Virtual Network. Per this thread, I've attempted to disable the TCP Checksum values. THat had no affect with the xfers from the VMs. I also read in this thread where I could have a 2nd virtual "private" network between the host and VMs for internal transfers from host to VMs. I'll do that during the next maintenance window, as that is a good idea. But, that doesn't resolve the issue at hand here.

I faced the same problem and it was solved by disabling Ethernet@WireSpeed option for the physical NIC mapped to this VM

That seems to be a Broadcom-only setting (per Bing searching) as I don't see anything like that with my I

Hi,  I fixed the issue, today, by updating to Intel's 15th Dec 2011 version of the Intel NIC drivers.  Host was a Dell R610 with Broadcom (updated) and Intel Gigabit ET NICs.  The host ran like a pig and it took me hours just to get remote management working let alone anything else.  I digress.  So if you must use Hyper-V and have these Intel NICs make sure you use these new drivers -

 

File name: PROWinx64.exe	Version: 16.8		Download
Date: 12/15/2011	Status: Latest
Size: 30.78 MB	Language: English
Operating Systems: Windows Web Server 2008 R2*, Windows Server 2008 R2 Enterprise*, Windows Server 2008 R2 for Datacenter*, Windows Server 2008 R2 Foundation*, Windows Server 2008 R2 Standard*

• Edited by AlexPT Sunday, January 15, 2012 7:56 PM wrong date entered initially
• Proposed as answer by AlexPT Sunday, January 15, 2012 7:56 PM

Clearing the checksum fixed it for me

Same experience for me too. I just applied those registry settings to my recent hyper-v core and now I'm seeing better connections too.

I used this article here and it seems much better now. crossing fingers.

http://gavinmckay.wordpress.com/2011/03/12/windows-server-2008-r2-hyper-v-guests-lose-network-connectivity-temporarily/

Dear all,

I have a big problem with my hyper-v server and network card. I've found a lot of topic around this question but none of them helped me.

I have a Win 2008 R2 Server with only Hyper-V installed. I have 3 NIC , one is reserved to the host management and the others two are reserved for virtual machine .

All NIC are Intel, one is integrated on the Intel motherboard and the others are Intel Gigabit ET Dual port .

Before installing the role of Hyper-V all NIC worked perfectly with speed around 1Gb (file transfer average 90 MB/Sec, of course I have a Gigabit LAN) .

After Hyper-V install, the host system  is going very slow and the network file transfer using the dedicated NIC is about ten time slower (9-10MB /sec) . Same problem with all VM off and also if I remove all VM !

I tried all of suggestion found in old post (disabling TCP checksum / offload ...) but nothing happens.

What is really strange is that by removing all network card all the Host operating system seems to improve the speed... but, of course, I can't use  my LAN :(

I need help because every night I must make a full backup of Virtual hard disk from the host to a backup server in my lan.

My configuration:

- Server Win 2008 R2 SP1 with Hyper-V and no others role

- 2 Virtual Machine (one Win 2008 R2 and one Win 7 Ultimate)

- NAS Linux (access via SMB with no problem)

- Various Windows 7 Pro client

Thanks in advance to everybody,

Luca M.

Hello,

maybe I have the solution for this problem. I Have the following configuration:

Intel 5520HC Mainboard

Windows Server 2008 R2 Enterprise with HyperV Role SP1

>32 GB Memory

I have extremly slow performance (VGA, LAN, HD(!!!))

Then I found the following article:

http://www.thomas-krenn.com/de/wiki/Schlechte_LAN_und_I/O_Performance_mit_Microsoft_Hyper-V_auf_Intel_Systemen_mit_mehr_als_32_GByte_RAM

This solve my problem complete!

Mfg

Bjrn

This issue still seems to be plaguing as we are having the same issue on multiple client sites.

Try http://support.microsoft.com/kb/2517329 , disabling C states, or both.

in Dell PE810 disable these solve above problem

BIOS/Processor Settings
C1E Disable
C States Disable

I had situation (which is why I found this thread) where I would get reasonable speeds in all directions and between hosts, except in one case.  From an i5-2500 workstation to an AMD FX-6100 hyper-v host local share the transfer rate was a ridiculous 30KB/s.  From hyper-v to the workstation it was about 70MB/s.  Humorously, or so it seemed, I had a Windows 8 VMware VM on the same i5 workstation and its transfer rate to the hyper-v was in a reasonable MB/s range.  

In short, I isolated the issue to IPv6 on the workstation.  Once I disabled IPv6, the transfer speed to hyper-v went up to the MB/s. 

No other server (2003 or 2008 or various NAS) had an issue with the workstation (nor with hyper-v, either way), only hyper-v had an issue with the workstation and only when transferring to a hyper-v share (which made vm conversions I was doing from esxi to hyper-v impossibly slow).  Why disabling IPv6 makes a difference I've not figured out, but for now I can continue with my evaluation of hyper-v.

Hope this helps someone else.

Bruce

I had this problem with a Dell PE R320 and the Broadcom Gigabit nic's.  I found that by having one for Hyper V Virtual Network only, no management, and the other dedicated to the host, that the host would work fine and that the VM would still be slow.  I disabled ALL the offloading and that still had no effect.  I jacked around with all the settings for the adapter on the host then and found additionally that disabling Virtual Machine Queues on the adapter on the host finally solved my problem.

TR Technologies, Inc. - IT Service Provider

• Proposed as answer by Brian Plumb Tuesday, January 29, 2013 2:07 AM
• Unproposed as answer by Brian Plumb Tuesday, January 29, 2013 2:07 AM
• Proposed as answer by MerlinWV Tuesday, July 23, 2013 7:05 PM
• Edited by juicejug1016 Monday, September 30, 2013 8:11 PM add my signature for being helpful

Thank you JuiceJug1016! I'm an IT consultant and Dell reseller and have two customers using Dell PE T320 servers and I couldn't figure out why the one customer was having such weird ping times. The first customer with the same configured unit was fine. The second, getting ready for the production environment was suffering terribly with slow network ping times and file transfer. After checking the network configuration I discovered the "Virtual Machine Queues" was turned on for the second customer. Soon as I turned it off everything was cool.

• Edited by Brian Plumb Tuesday, January 29, 2013 2:04 AM
• Proposed as answer by smjain Friday, March 22, 2013 10:32 PM

Thanks JuiceJug1016! exactly the same issue host machine windows 2012 PE T320 host network browsing speeds fine any guest machines has 315KB sec copy speeds spent weeks playing with virtual switch setup nic teaming etc and as soon as I disabled Virtual Machine Queues on the host adapter my vm's were using my speed returned to normal speed.

 Thanks juicejug1016! I have a PE420, this tip was a life safer... 8-)

Thanks juicejug1016.  I'm running Windows Server 2012 Hyper-V on a Poweredge R720 (A Dell build on it too!) and found that disabling just the "Virtual Machine Queues" on the host's Broadcom cards fixed the slow networking on the guest OS's immediately.  

Many thanks

Genius.  I wish I had found your post 5 days ago.

Amazing answer. I spent a huge amount of grief trying to fix this same problem and stumbled upon this answer. This fix worked and I am grateful.

I seem to be having very similar performance issues with a freshly built Dell R720xd running the Hyper-V 2012 server. The hypervisor only. That being the case, I can't seem to find my may through the RSAT to find the NIC properties pages shown above to disable the "virtual machine queues". Any ideas?

Sheldon

I had this problem with a Dell PE R320 and the Broadcom Gigabit nic's.  I found that by having one for Hyper V Virtual Network only, no management, and the other dedicated to the host, that the host would work fine and that the VM would still be slow.  I disabled ALL the offloading and that still had no effect.  I jacked around with all the settings for the adapter on the host then and found additionally that disabling Virtual Machine Queues on the adapter on the host finally solved my problem.

How would I go about locating these settings in the Hypervisor 2012 only install using RSAT? Thoughts?

I had this problem with a Dell PE R320 and the Broadcom Gigabit nic's.  I found that by having one for Hyper V Virtual Network only, no management, and the other dedicated to the host, that the host would work fine and that the VM would still be slow.  I disabled ALL the offloading and that still had no effect.  I jacked around with all the settings for the adapter on the host then and found additionally that disabling Virtual Machine Queues on the adapter on the host finally solved my problem.

How would I go about locating these settings in the Hypervisor 2012 only install using RSAT? Thoughts?

With Get-NetAdapterAdvancedProperty. Simple usage:

Get-NetAdapterAdvancedProperty -Name "Ethernet 2"

All physical adapters:

Get-NetAdapter -Physical | Get-NetAdapterAdvancedProperty

The issue is that not all vendors name the property the same way, and some NICs don't support it at all. You can sometimes pull things like:

Get-NetAdapter -Physical | Get-NetAdapterAdvancedProperty | Where-Object "DisplayName" -Like "*queue*"

To change it:

Get-NetAdapter "Ethernet 3" | Get-NetAdapterAdvancedProperty -DisplayName "Virtual Machine Queues" | Set-NetAdapterAdvancedProperty -DisplayValue "Disabled"

Make sure you use the display name for your adapter and the same sort of language for DisplayValue that it uses. Instead of DisplayValue, you can also use -RegistryValue 0.

You can also open up regedit on your Hyper-V Server and dig through the subkeys of HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4d36e972-e325-11ce-bfc1-08002be10318} and change them that way. That's all the above PowerShell commands do. You can find the allowable settings under the adapter's subkey \Ndi\params\<fieldname>\Enum.

I've also found that if your adapters don't support VMQ at all and you're using teaming in 2012 to host a virtual switch, it sometimes help if you disable VMQ on the virtual adapter that hosts your virtual s

Yes! Yes! Yes!

You are my lifesafer!

My T320 with Server 2008 R2 very Slow with HyperV and now it ist like a MIG-29

thx Ben

• Proposed as answer by schrippe Friday, August 23, 2013 2:51 PM

Thank you, your solution helped me.

Thanks juicejug1016 work for me as well.

Thanks!!! My PE720 had this issue and your tip worked like a charm!

B.r

Joakim

South Brains AB
www.southbrains.se

For Dell R320 it really works. THX!

im facing a similar issue, Dell R420, Win2012R2+Hyper-V and the latency is above 200ms and losing 5% of the packets and latency above 100ms on a regular 6ms link

It worth a try the article?

http://support.microsoft.com/kb/2902166/en-us

I disabled the VMQ on the configuration of the HV physical machine, a LACP with 2 NICs, so i disabled on both.

Before, i tried to disable  VMQ on the Multiplex adapter (connected to the vswitch and dindt worked

The ping times drop form 200ms to 6ms, normal in my environment

Before and after:

PS C:\Users\Administrator> Get-NetAdapter -Physical | Get-NetAdapterAdvancedProperty | Where-Object "DisplayName" -Like "*queue*"

Name                      DisplayName                    DisplayValue                   RegistryKeyword RegistryValue
----                      -----------                    ------------                   --------------- -------------
SLOT 1 Port 2             Maximum Number of RSS Queues   RSS 4 Queues                   *NumRssQueues   {4}
SLOT 1 Port 2             Virtual Machine Queues         Enabled                        *VMQ            {1}
SLOT 1 Port 4             Maximum Number of RSS Queues   RSS 4 Queues                   *NumRssQueues   {4}
SLOT 1 Port 4             Virtual Machine Queues         Enabled                        *VMQ            {1}
SLOT 1 Port 1             Maximum Number of RSS Queues   RSS 4 Queues                   *NumRssQueues   {4}
SLOT 1 Port 1             Virtual Machine Queues         Enabled                        *VMQ            {1}
SLOT 1 Port 3             Maximum Number of RSS Queues   RSS 4 Queues                   *NumRssQueues   {4}
SLOT 1 Port 3             Virtual Machine Queues         Enabled                        *VMQ            {1}


PS C:\Users\Administrator> Get-NetAdapter -Physical | Get-NetAdapterAdvancedProperty | Where-Object "DisplayName" -Like "*queue*"

Name                      DisplayName                    DisplayValue                   RegistryKeyword RegistryValue
----                      -----------                    ------------                   --------------- -------------
SLOT 1 Port 2             Maximum Number of RSS Queues   RSS 4 Queues                   *NumRssQueues   {4}
SLOT 1 Port 2             Virtual Machine Queues         Disabled                       *VMQ            {0}
SLOT 1 Port 1             Maximum Number of RSS Queues   RSS 4 Queues                   *NumRssQueues   {4}
SLOT 1 Port 1             Virtual Machine Queues         Enabled                        *VMQ            {1}
SLOT 1 Port 3             Maximum Number of RSS Queues   RSS 4 Queues                   *NumRssQueues   {4}
SLOT 1 Port 3             Virtual Machine Queues         Enabled                        *VMQ            {1}
SLOT 1 Port 4             Maximum Number of RSS Queues   RSS 4 Queues                   *NumRssQueues   {4}
SLOT 1 Port 4             Virtual Machine Queues         Disabled                       *VMQ            {0}

PS C:\Users\Administrator> Get-NetLbfoTeam


Name                   : TeamMGMT
Members                : {SLOT 1 Port 1, SLOT 1 Port 3}
TeamNics               : TeamMGMT
TeamingMode            : Lacp
LoadBalancingAlgorithm : TransportPorts
Status                 : Up

Name                   : TeamVM
Members                : {SLOT 1 Port 4, SLOT 1 Port 2}
TeamNics               : TeamVM
TeamingMode            : Lacp
LoadBalancingAlgorithm : HyperVPort
Status                 : Up

I suggest that changing the defaults for a NIC is not generally going to fix performance problems. Same for IP6 etc.

I suggest testing performance with a fresh install of Windows or Server or Hyper-V can then run some checks.

Windows 8.1 has more advanced monitoring from the taskmanager which may point to a problem. Might be worth it to use a rig with 8.1 to check the network. 

You're a star.  This immediately beefed up performance many times. Thanks

Thank You Eric Siron,

I ran Enable-pssesion -computername name -credentials $cred (saved earlier) and ran the get-adapter command about and my network performance was fixed. It was the virtual machine queue that was killing performance. I was having issues from One VM to another on the same Hyper-V host.

Ian