Hyper-V guests on AMD-based Dell servers - perceived peformance is poor but benchmarks are fine

We have a Hyper-V cluster with Windows Server 2012 (it had been Server 2008 R2 before with the same issue). The hosts have 4 12-core Opteron CPUs and 320 GB RAM.

Now, the perceived performance on the virtual machines is quite poor. This is, certain actions in certain applications take way longer than on a physical machine (while other actions in other applications are working fine).

I ran several benchmarks in multiple virtual and physical machines, but could not find a notable difference. CPU, disk, cache performance are perfect. RAM performance is rather poor compared to Intel systems (ca. 1900 MB/s while my i5 desktop shows 7500 MB/s), but I found that it's similarly poor on physical AMD-based servers (around 2100 MB/s, so it seems to be related to the CPUs or chipsets). (These physical AMD-based servers perform well, so the issue does not seem to be related to that RAM performance.)

Does anyone have an idea what could cause this? Are there any other things I could test or performance counters I could watch?


  • Edited by svhelden Tuesday, April 02, 2013 5:14 AM
April 2nd, 2013 5:14am

Hiya,

1: Which disk subsystem are you running on?

2: Are you running benchmarks on live environment?

3: How many virtual guests are on your host?

4: The definition is that per server in the cluster?

Free Windows Admin Tool Kit Click here and download it now
April 2nd, 2013 7:22pm

Take a look to following Kb
http://support.microsoft.com/kb/2770440
Not sure if it apply to your case
April 2nd, 2013 9:29pm

1: Which disk subsystem are you running on?

A mirrored SAN based on DataCore SANSymphony-V, connected by 8 Gbit FibreChannel. Data is stored on RAID 5 arrays of 15k SAS disks.

Read/Write/Seek performance in virtual machines and on Hypervisor is similar to other physical servers, according to benchmarks.

2: Are you running benchmarks on live environment?

Yes.

3: How many virtual guests are on your host?

A total of 60 guests is distributed across 3 (currently 2) hosts. For testing, I put 59 guests on 1 host and only 1 guest on the other, and then ran benchmarks in the single guest of host 1 and in one guest of host 2, but results were similar. The single guest was only few percent faster.

4: The definition is that per server in the cluster?

I'm not sure what you are referring to. The configuration (4 CPUs, 320 GB RAM) is per host. Currently the cluster is only running on 2 hosts (the 3rd host has still Hyper-V 2008 R2), so the Server 2012 cluster has 8 CPUs / 96 cores and 640 GB RAM in total.


  • Edited by svhelden Wednesday, April 03, 2013 5:11 AM
Free Windows Admin Tool Kit Click here and download it now
April 3rd, 2013 5:07am


http://support.microsoft.com/kb/2770440
Does not apply to us. Thanks anyway.
April 3rd, 2013 5:11am

Hi,

How you configured memory for this virtual machine, fixed memory or dynamic memory? How many memory do you assigned for the VM?

If possible give us a screen capture of the virtual machine memory configuration.

Whats OS running in the Hyper-V guest VM? Have you installed latest Hyper-V integration services?

Check that, install latest integration service and give us feedback for further troubleshooting. For more information please refer to following MS articles:

Hyper-V Dynamic Memory Configuration Guide
http://technet.microsoft.com/en-us/library/ff817651(v=WS.10).aspx
Optimizing Performance on Hyper-V
http://technet.microsoft.com/en-US/library/cc768529(v=BTS.10).aspx

Hope this helps!

TechNet Subscriber Support

If you are TechNet Subscription user and have any feedback on our support quality, please send your feedback here.

Free Windows Admin Tool Kit Click here and download it now
April 3rd, 2013 6:25am

Hi, mosts guests use Dynamic Memory with something like "between 4 and 16 GB". The VMs I tested with had 4 GB RAM and 4 to 12 virtual cores. 

Most VMs run Windows Server 2008 R2 with latest (6.2) integration services. There are few VMs with Server 2012 which show similar behavior.

April 3rd, 2013 6:27am

Hi, Please check the below things 1) Check Storage IOPS of VHD's location where they stored. 2) NIC cards of the Host machines. 3) disable Ethernet@WireSpeed option for the physical NIC mapped to this VM. 4) check the link http://www.thomas-krenn.com/de/wiki/Schlechte_LAN_und_I/O_Performance_mit_Microsoft_Hyper-V_auf_Intel_Systemen_mit_mehr_als_32_GByte_RAM
Free Windows Admin Tool Kit Click here and download it now
April 3rd, 2013 7:20am

Hi,

As this thread has been quiet for a while, we assume that the issue has been resolved. At this time, we will mark it as 'Answered' as the previous steps should be helpful for many similar scenarios.

If the issue still persists and you want to return to this question, please reply this post directly so we will be notified to follow it up. You can also choose to unmark the answer as you wish.

In addition, we'd love to hear your feedback about the solution. By sharing your experience you can help other community members facing similar problems.

Thanks!

April 9th, 2013 2:34am

Hi, the issue is not resolved yet. All the mentioned steps do either not apply to our system or have already been checked.

Right now I'm playing around with Server Performance Advisor to gather some more reliable performance data.

In the meantime, I noticed that the hosts perform ca. 100,000 context switches per second. I read somewhere that 1,000 for a single CPU would be normal, so might that be an indicator of something wrong? The "% Total Run Time" on Logical and Virtual processors are rather low, so that should not be the reason.

Free Windows Admin Tool Kit Click here and download it now
April 9th, 2013 4:52am

Hi!

It's interesting that some applications run just fine, while some doesn't. From the above discussion, it appears to me that the Hyper-V host operates at optimal performance, but some of the virtual instances won't play along as they should. This is not a new or strange occurance, it's been known since virtualization became a de facto standard in most corporations a couple of years ago, that not all applications are suitable candidates for virtualization.

Could you provide us with some examples of the applications that doesn't perform well in virtual Machines?

April 9th, 2013 5:48am

We finally replaced our Hyper-V hosts with Intel-based ones. Performance problems are gone. It seems that these AMD-based hosts simply had poor memory performance, and that affected certain applications more than others.
  • Marked as answer by svhelden 1 hour 34 minutes ago
Free Windows Admin Tool Kit Click here and download it now
August 24th, 2015 1:29am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics