Dynamic Memory is not working all the time

We are in the process off moving our 2008R2 VM's from the 2008R2 HyperV servers to new Server 2012R2 Hosts.

We shut down the VM's copy the files and VHD's to the new CSV's en import the VM in the Hyperv Manager. Then we make them high available in the Failover Cluster Manager (Configure role - Virtual machine). We mount the integration tools and update the VM to version 6.3.9600.16384

For a specific type of VM (mostly RDS Host servers) we always had Dynamic Memory configured (when they were hosted on de 2008R2 platform), so we are using the same settings on the 2012r2 platform. The memory settings were;

Startup memory: 1024 MB

Minimum memory: 1024 MB

Maximum memory: 12288 MB

These VM's reboot every morning, this is done for specific reasons. But now once in a while (once per week/2 weeks) we notice that the VM's are not using more memory then 1024 MB while the demand is much higher. Rebooting the server helps most of the times, live migrating to another host also helps. In the VM we see that memory usage in the taskmanager is 99-100%, and after the move it immediately starts using more than the minimum configured amount.

Until the failover the memory usage was 1024 MB and it did not get any higher.

This happened several times. Last week we changed the Memory configuration to:

Startup memory : 2048 MB

Minimum memory: 2048 MB

Maximum memory: 12288 MB

But this morning we had a call about the performance of one of the VM's, We saw that it was only using 2 GB memory while the demand was much higher. After live migrating it to another host it started using more memory immediately.

The 2012R2 hosts are not overcommited, there is a lot of memory still available for the VM's. Those VM's never had this problem on the 2008R2 Hyperv platform.

Any idea why this happens?

Peter Camps


  • Edited by Peter Camps Monday, November 24, 2014 12:13 PM
November 24th, 2014 11:59am

As you have shifted these machines over to Server 2012, have you been updating the Integration Services? It almost seems to me the server never comes out of the start up state... Also for another test lets try and do start-up memory of 2048 and minimum memory of 1024. I'd be interested to see if the memory falls back to 1024 after the initial start-up.

In the Hyper-V Application logs on the host in the windows event viewer do you see any errors with dynamic memory?

In the guest go to perfmon and view these performance counters if available.

  • Hyper-V Dynamic Memory > Guest Visible Memory
  • Hyper-V Dynamic Memory > Physical Memory

Also, in the hyper-v console does it show memory status as OK and what is the assigned memory vs memory demand.

Free Windows Admin Tool Kit Click here and download it now
November 24th, 2014 10:17pm

Hi,

It has taken quite a while, but this morning we had one RDS server that was only assigned 2 GB RAM (the startup memory amount), while it was configured to a maximum of 10 GB RAM it did not use more than 2.

At that moment it said in my hyperv manager the following;

Memory Status and Demand were blank. The memory usage in the VM was 100% for more than one hour, i started Perfmon on that Host and viewed the performance counters you mentioned before.

As you can see, all values are 2 GB, the amount that was assigned to the VM. There were no events in the eventviewer on the Host and in the VM itself. I live-migrated the VM to another Host in that cluster and immediately the values changed as you can see down here (the screenshots were made 30 minutes after the live migration).

This cluster is completely up to date with all Windows Updates en HP SPP (firmware and drivers). I am unable to reproduces the issue, but it still happens once in a while.

Any further ideas are appreciated.

Grtx

Peter Camps

  • Edited by Peter Camps Monday, January 05, 2015 11:19 AM
January 5th, 2015 10:46am

Hello,

We have seen it happen to several different VM's (2008R2 OS in those VM's) and on several different Hosts. Even on different Clusters (all Server 2012R2 Hosts).

Edit (19-05-2015) **** We have also seen the problem happen in 2012R2 VM's ****

Because of this issue we have updated all our clusters, but did does not seem to have resolved it. Some of the VM's we migrated from our old 2008R2 clusters and some were newly installed on the 2012R2 cluster.

Most of the hosts have 2 to 3 VM's that reboot in the morning, Some reboot at 04:00 AM, others a bit later. The hosts normally have 25 to 50 GB RAM available when all the VM's are running, so there is plenty off avaliable RAM.

We haven't profiled the servers to see what the server requires at startup. Mainly because we have never seen this issue on our 2008R2 cluster (with the same settings), and it only happens occasionally.


Free Windows Admin Tool Kit Click here and download it now
January 6th, 2015 1:18pm

Hello,

We have opened a case with Microsoft concerning this issue. We have also discovered that it also happens with 2012R2 VM's, at first we thought it only happened with 2008R2 VM's but this is not true.

At this moment we are collecting information, performance logging, we have done several MSDT inventory on VM's and hosts. So far there is not yet a solution but i hope to have some positive news the next few weeks.

As soon as i have a solution or important facts about this issue i will post it here. But at this moment i think this will take a while.

If somebody as has any news or information, please share it with us. It has been a problem for quite a while now, and we would like to have this resolved.

Grtx

Peter Camps



May 19th, 2015 7:14am

Hi,

Yesterday we got a call from Microsoft that a fix was available;

https://support.microsoft.com/en-us/kb/3050265

It is a (optional) client fix (in the VM) and at the moment it is only available for Windows 7 and Server 2008R2, the fix for Server 2012R2 will be released a bit later.


  • Edited by Peter Camps Thursday, June 04, 2015 9:40 AM
Free Windows Admin Tool Kit Click here and download it now
June 4th, 2015 6:46am

WE HAVE A WINNER!!!

Product Group have managed to reproduce the issue after we supplied the VMs, hardware specifications and scripts to auto reboot VMs until one isn't assigned Dynamic Memory...

Response from our TAM:
"PG has the repro and a fix has been developed, should be out soon once they are done with the regression testing. Will post more once its available to me."

This means they have accepted it as a BUG and have created a hotfix for it. they are currently testing the hotfix then they will throw it over to me to test in our environment. If it looks good it will be released as a hotfix, i'll post as soon as I have a release date. I'll ask them for an RCA also, not sure if they will supply one though.


Peter, you have my Case ID don't you? ask your engineer to contact Kartikay for an update.
July 17th, 2015 12:44pm

Hi Jim,

That's great news !!!

Really hope that they have a definite solution. I immediately mailed our MS contact with your case ID and asked him to look into it.

Keeping my fingers crossed.

Free Windows Admin Tool Kit Click here and download it now
July 20th, 2015 2:59pm

Hey guys,

I also have a case open with Microsoft for this issue. So far, the tech has recommended installing the hotfix KB3046826 (seems unrelated to dynamic memory..)

And he also recommended disabling NUMA spanning on the Hyper-V host (not really willing to do this to all of my hosts as it could impact some large VMs)

If Microsoft provides you with a working hotfix please let us know!

July 26th, 2015 10:21pm

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics