Scenario

I have a 3 Windows 2012 R2 Hyper-V hosts which is participating in a cluster, all of the virtual machines are hosted on CSV's. We also use Dynamic Memory for most of the servers

We do regular reboots every week for our virtual machines, but we've found that sometimes, a virtual guest will suddenly stop requesting additional memory.  At the time of writing, I have identified 10 machines - on different hosts, and even different clusters which are exhibiting this behaviour.  They're also running OS's ranging from Windows 7 to Windows 2008 R2 and Windows 2012.

I have selected one to demonstrate below.

Here, you can see the virtual machine is configured to use Dynamic Memory

and here, you can see that it's not actually demanding more than the start memory, despite the fact that I've overloaded the servers virtual memory by using MSPAINT.

Workaround

Rebooting the virtual machine does not resolve the issue, but live migrating the virtual machine from the host it is on to another host suddenly allows it to demand more memory.  One can then live-migrate it back to the original host and the problem is no longer there.

Troubleshooting

We have performed the following troubleshooting steps:

• Verified that the Integration Services are up-to-date (currently running 6.3.9600.16384)
• Verified that the virtual server has up-to-date Windows Updates
• Checked the Hyper-V-Worker\Admin event log on the current host - no errors pertaining to this VM.
• Verified that the Hyper-V host has sufficient RAM (currently only using 542 out of 768 GB)
• Checked the System event log on the virtual machine - nothing out of the ordinary to report.
• Verified that all services which start with the word "Hyper-V" on the guest machine is running.

Using the following command, I can easily identify which machines are affected (this command applies to SCVMM only):

Get-SCVirtualMachine | where {$_.DynamicMemoryEnabled -eq $true -and $_.DynamicMemoryDemandMB -eq $_.Memory} | ft Name,HostName,OperatingSystem,Memory,DynamicMemoryDemandMB,DynamicMemoryMinimumMB,DynamicMemoryMaximumMB

I do not know any diagnostic program which can give me more information regarding the memory status of a server - specifically one which can tell me more information about Memory demand, and why it's not working. 

Considering that the problem is resolved when I live migrate the affected machine, it would suggest that the issue is with the host, but as live migrating it back no longer exhibits the issue - that seems to negate that theory.  Additionally, the other 103 virtual machines which run on this host seem to not show this issue.

It's almost as if the machine is being ran with static memory, and Hyper-V doesn't realise it's dynamic, but the moment it's live migrated, it suddenly realises this and enables dynamic memory.

Any thoughts?

• Edited by LesterClayton Tuesday, May 05, 2015 1:10 PM Code Block a Line

Hiya,

next time you see this behavior, could you try and run "Update-ClusterVirtualMachineConfiguration" on the affected server and see if that does change anything.

https://technet.microsoft.com/en-us/library/ee461058.aspx

• Proposed as answer by Elton_JiMicrosoft contingent staff, Moderator Wednesday, May 20, 2015 2:43 PM
• Marked as answer by Elton_JiMicrosoft contingent staff, Moderator Thursday, May 21, 2015 7:36 AM
• Unmarked as answer by LesterClayton Wednesday, June 17, 2015 3:01 PM
• Unproposed as answer by LesterClayton Wednesday, August 05, 2015 10:45 AM

Why on earth is this thread marked as answered????

Go speak to PSS or PG...

It's marked as Answered because I didn't respond to the suggestion made.  I have unfortunately been so busy that I've not had the opportunity to look into this since I posted my original question.

That being said, I am now able to provide feedback.  Thanks for the nudge, it reminded me about it.

At the time of writing, I have 3 virtual machines which is experiencing this issue, where it's not demanding any more than the startup RAM.  I have updated the configuration on one of them using the powershell command, and have attempted to demand more memory with no success.

Any further input is welcome, but I suspect I am going to have to raise a support case for this.

There have been a few cases raised about this recently... See this thread which has a couple of CaseIDs in it. One mine, one a guy called Peter. Mine has gone to the PG and they are currently building a repro environment. by all means raise the case and use ours as reference.

https://social.technet.microsoft.com/Forums/windowsserver/en-US/538b8898-5c29-4408-9fdd-531eff4241c5/dynamic-memory-is-not-working-all-the-time?forum=winserverhyperv

Is it intermittent in your environment?
Couple of "work arounds":
1 reboot the machine or shut it down and start it back up!
2 sometimes a live migration will kick the balloon driver into life!

WE HAVE A WINNER!!!

Product Group have managed to reproduce the issue after we supplied the VMs, hardware specifications and scripts to auto reboot VMs until one isn't assigned Dynamic Memory...

Response from our TAM:
"PG has the repro and a fix has been developed, should be out soon once they are done with the regression testing. Will post more once its available to me."

This means they have accepted it as a BUG and have created a hotfix for it. they are currently testing the hotfix then they will throw it over to me to test in our environment. If it looks good it will be released as a hotfix, i'll post as soon as I have a release date. I'll ask them for an RCA also, not sure if they will supply one though.

Peter, you have my Case ID don't you? ask your engineer to contact Kartikay for an update.

• Proposed as answer by Jesper Arnecke Thursday, August 27, 2015 9:00 AM

Thank you very much for updating my thread as well as the other one :)

No worries, I know there are about 4 threads for the same subject knocking around... Keep them all up to date :-)

Hotfix for this issue will be released on 13th October 2015. I'm pushing for a private hotfix and will update when I know more.

Thanks for keeping us up to date :)

Just been informed I will be getting a test hotfix on August 10th. The problem is host side not guest side so we should only have to patch hosts rather than guests.

Any news on this ?
We also have the issue.

I am currently testing a private hotfix, which seems to have fixed the issue. It is currently aptly name KB9999999!

As far as i'm aware 13th October is the official release date.

I have asked them for the KB number this will be published under

Thank you for the reply Jim,

Just got it with a third VM, 2008 R1. Live Migration does not help this time!