I have a client using a specific application that communicates with a MySql server that is installed on the same 2008 R2 virtual machine.

When using this application it seems to take a long time when querying the MySql database. The program will start not responding and could take up to 1-2 minutes to display in certain areas where it should take less than 10 seconds.

Here is what I know:

• The long pauses seem to be when it is querying the database.
• The long pause only occurs when it is running on a virtual machine in our environment
• CPU is not a problem. Take manager in the VM shows less than 10% CPU and VMM shows 0% CPU Average
• Memory is not the problem (tried dynamic and static) and shows less than 35% of the total amount of memory (8GB total currently)
• Disk does not show any performance issues and show about what I should see with 6Gbps SAS disks (500+)
• I've tried multiple hosts and even had it running on a host with no other VMs
• My hosts are BL685c G7 with 4x AMD 16-Core 6278 2.4Ghz processors and 512GB of DDR3-10600R RAM. Thats 64 total cores.
• My SAN is a HP 3PAR 7200 running 4GB fiber (four paths)
• I've tried fiber channel passthrough to the VM as well as a vhdx file for the database drive
• I've tried RAID5 64KiB, RAID1 64KiB, and RAID1 256KiB for the RAID
• My 3PAR reports show less than 80 IOPs per drive in my enclosure for READ/WRITE
• My 3PAR is using 380GB of adaptive flash cache

I have taken this VM and duplicated (system image) it on other hardware. When it is virtual the problem exist and when it is not virtual the performance is fine. 

The problem is related to being virtualized but I just can't figure out what is causing it! Has anyone had issues with applications communicating with MySql on Hyper-V? I really think this is something Hyper-V related but just can't pinpoint it.

Anyone have any ideas?

Anyone have any ideas?

Kind of a long shot, but try disabling C states and C1E. A few "that doesn't make any sense" performance issues have been traced to C1E. This is a BIOS setting and dependent upon your hardware.

Kind of a long shot, but try disabling C states and C1E. A few "that doesn't make any sense" performance issues have been traced to C1E. This is a BIOS setting and dependent upon your hardware.

I did think about that too and turned the Minimum Processor Idle Power State to "No C-states". It still is an issue.

I just can't figure it out. I do have some SSD in my 3PAR that I tried over virtual fiber and there is a 15 second difference. So local storage and fiber SSD takes 50-54 seconds and when I put it on my fiber SAS (same 3PAR SAN, just different drives) it takes 34-36 seconds.

So each time there is almost exactly a 15 seconds difference. I was thinking my 4GB fiber at one point but if that was the case then the SSD in my 3PAR should be showing 50-54 seconds for this test.

So its almost like it HAS to be the 3PAR but I haven't tried a physical server connected via fiber to the SAN yet so I can't 100% rule out Hyper-V doing something stupid.

I did think about that too and turned the Minimum Processor Idle Power State to "No C-states". It still is an issue.

I just can't figure it out. I do have some SSD in my 3PAR that I tried over virtual fiber and there is a 15 second difference. So local storage and fiber SSD takes 50-54 seconds and when I put it on my fiber SAS (same 3PAR SAN, just different drives) it takes 34-36 seconds.

So each time there is almost exactly a 15 seconds difference. I was thinking my 4GB fiber at one point but if that was the case then the SSD in my 3PAR should be showing 50-54 seconds for this test.

So its almost like it HAS to be the 3PAR but I haven't tried a physical server connected via fiber to the SAN yet so I can't 100% rule out Hyper-V doing something stupid.

Your hardware doesn't separate C-states from C1E? I only ask because all of the ones that I've worked on treat them separately. C1E seems like the bigger culprit.

Some slow down is to be expected as shared disk access is a tricky issue in any situation and Hyper-V needs to be certain it's treating all VMs equally. This seems like it might be that balancing act getting overly aggressive -- and not the first time I've seen something like that. I always forget, but there are a handful of registry entries you can tinker with. YMMV: https://msdn.microsoft.com/en-us/library/windows/hardware/dn567657.aspx#storageio.

While we're taking long shots, you might also see if disabling ODX has any effect: https://technet.microsoft.com/en-us/library/jj200627.aspx. I'm starting to think that storage vendors do an equally bad job of implementing ODX as networking vendors do with VMQ.

Your hardware doesn't separate C-states from C1E? I only ask because all of the ones that I've worked on treat them separately. C1E seems like the bigger culprit.

Some slow down is to be expected as shared disk access is a tricky issue in any situation and Hyper-V needs to be certain it's treating all VMs equally. This seems like it might be that balancing act getting overly aggressive -- and not the first time I've seen something like that. I always forget, but there are a handful of registry entries you can tinker with. YMMV: https://msdn.microsoft.com/en-us/library/windows/hardware/dn567657.aspx#storageio.

While we're taking long shots, you might also see if disabling ODX has any effect: https://technet.microsoft.com/en-us/library/jj200627.aspx. I'm starting to think that storage vendors do an equally bad job of implementing ODX as networking vendors do with VMQ.

I am not seeing anything about C1E in the BIOS. Its a HP BL685c G7. Maybe i'm just missing it and will look further.

I am testing disabling ODX as we speak.

In my situation there isn't really a shared disk. I guess the way that 3PAR does the virtual raids across all the disk in a disk group could be considered shared disk, but all my tests have been with fiber passthrough to LUNs that are dedicated to this test environment.

I am not seeing anything about C1E in the BIOS. Its a HP BL685c G7. Maybe i'm just missing it and will look further.

I am testing disabling ODX as we speak.

In my situation there isn't really a shared disk. I guess the way that 3PAR does the virtual raids across all the disk in a disk group could be considered shared disk, but all my tests have been with fiber passthrough to LUNs that are dedicated to this test environment.

I don't mean "shared disk", I mean "shared disk access", as in multiple threads accessing the same storage subsystem simultaneously. It's a painful process in a supervisor. It has to be excruciating in a hypervisor.

I don't mean "shared disk", I mean "shared disk access", as in multiple threads accessing the same storage subsystem simultaneously. It's a painful process in a supervisor. It has to be excruciating in a hypervisor.

I gotcha.

I did find the C1E State. The options I have for Minimum Processor Idle Power State is:

No C-states
C1E State (AMD C1 Clock Ramping)
Core C6 (CC6) State

I gotcha.

I did find the C1E State. The options I have for Minimum Processor Idle Power State is:

No C-states
C1E State (AMD C1 Clock Ramping)
Core C6 (CC6) State

Ok I disabled ODX and the results seem to be the same.. 52 seconds for the test using the SAS drives on my 3PAR.

I'm really thinking its something with the 3PAR which i'm waiting for HP (taking forever) to work with me on.

Ok I disabled ODX and the results seem to be the same.. 52 seconds for the test using the SAS drives on my 3PAR.

I'm really thinking its something with the 3PAR which i'm waiting for HP (taking forever) to work with me on.

Eric,

I found out what the issue is I think. The AMD 6276 processors i'm using seem to be not very good with single threads.

I have a host with only one virtual machine. I have performance monitor open for Hyper-V Logical Processors and watching % Total Run Time of all cores. I also have performance monitor up with Hyper-V Hypervisor Virtual Processors and watching % total Run time.

When the query to mysql (on same virtual machine) works I can see a single core on both the virtual and the host max completely out. When I watch this when running the same from local storage (C: drive on virtual machine) and not over fiber it is a little better but maxes out. I'm guessing fiber adds a little overhead?

Is there anything I can adjust for this so the processors function better or is this possibly a hyper-v issue? MySql should be using more than 1 core but in this case from what i'm seeing it isn't.

Here is a picture during the query: (top is virtual processor and bottom is host logical processor):

Eric,

I found out what the issue is I think. The AMD 6276 processors i'm using seem to be not very good with single threads.

I have a host with only one virtual machine. I have performance monitor open for Hyper-V Logical Processors and watching % Total Run Time of all cores. I also have performance monitor up with Hyper-V Hypervisor Virtual Processors and watching % total Run time.

When the query to mysql (on same virtual machine) works I can see a single core on both the virtual and the host max completely out. When I watch this when running the same from local storage (C: drive on virtual machine) and not over fiber it is a little better but maxes out. I'm guessing fiber adds a little overhead?

Is there anything I can adjust for this so the processors function better or is this possibly a hyper-v issue? MySql should be using more than 1 core but in this case from what i'm seeing it isn't.

Here is a picture during the query: (top is virtual processor and bottom is host logical processor):

Hyper-V typically does a very good job dealing with threads; no worse than Windows in my experience. I did a brief looking for MySQL, and it seems that it will process one query per thread, so one query could make a performance graph like that. Hyper-V doesn't emulate the CPU. All it does is find an available core and send the thread to it as-is.

Uhm. Yeah we are also experiencing with another client on this same system that is using Microsoft SQL Server. The report takes 4-6 hours to run when normally it takes 1 hour to run when not virtualized.

Still looking into that. I've read some bad things about AMD 6276 processors and single threads. I think i'm going to try to order form AMD 6328 3.2GHz processors for testing.

Uhm. Yeah we are also experiencing with another client on this same system that is using Microsoft SQL Server. The report takes 4-6 hours to run when normally it takes 1 hour to run when not virtualized.

Still looking into that. I've read some bad things about AMD 6276 processors and single threads. I think i'm going to try to order form AMD 6328 3.2GHz processors for testing.