Dynamic memory allocation failure

Dear reader,

We sometimes have a problem where our windows 2012 r2 RDS virtual servers, that reside on windows 2012r2 hyper-v hosts, loose their dynamic memory and only have their startup memory left to work with. Users start complaining that things are very slow etc.

If I check several screens (RDS Broker load balancing, hyper-v manager, cluster manager and the vm's task manager) it's clear that the vm only has its startup memory allocated. I'm not sure if this happens instantly or immidiatly after the nightly reboot. To resolve the problem we have to call all users on the vm where it happens and ask them to logoff (if they are even able to), and then we reboot the machine.

I have checked the logs from the machine where the VM resides on and the logs from the vm itself. But I cannot find anything. We also have alot of windows 2008r2 vm's with dynamivc memory, but none of those have ever had this problem.

Searched the internet, but so far it seems we are only.

Can anyone give me a lead to troubleshoot this?

Best regards,

Ruud B

October 27th, 2014 10:03am

Some workloads just does not work well with dynamic memory. SQL is one of them. RDS should work OK. On the host, check the current amount of memory available to virtual machines in Performance Monitor counter: Hyper-V Dynamic Memory Balancer Available Memory.
Free Windows Admin Tool Kit Click here and download it now
October 27th, 2014 10:36am

So, what your saying is that a third party app running on the RDS could be responsible for this error?
October 27th, 2014 12:59pm

No, I'm talking about RDS itself.


Free Windows Admin Tool Kit Click here and download it now
October 27th, 2014 1:30pm

Hiya,

A few bullets from the top of my head:

1: RDS itself does not have any problems with dynamic memory.

2: The 2008 R2 servers needs an update to support dynamic memory.

3: SQL server will, by nature, consume all memory available, unless min/max memory setting is  configured. Even in newer versions of SQL, this is still a consideration.

4: Not all applications support dynamic memory. Most applications utilizing a memory cache will suffer from this. So yes, this could indeed be a third party application causing this.

October 27th, 2014 2:25pm

We worked around the issue by fixing all rds to 25Gb, and disable dynamic mem. In fact we do not actually need dynamic memory, becaue we do not have an environment that justifies it.

But offcourse the issue exists, and should be resolved by MS. Maybe someone in the future stumbles over this post and has a solution. If one does be so kind to let us know.Thanks in advance.

Best regards.

Ruud Boersma

Free Windows Admin Tool Kit Click here and download it now
October 28th, 2014 8:51am

We worked around the issue by fixing all rds to 25Gb, and disable dynamic mem. In fact we do not actually need dynamic memory, becaue we do not have an environment that justifies it.

But offcourse the issue exists, and should be resolved by MS. Maybe someone in the future stumbles over this post and has a solution. If one does be so kind to let us know.Thanks in advance.

Best regards.

Ruud B

October 28th, 2014 9:13am

Hi all,

I'm going to be "one of those people" who revives dead posts for something that is relevant but obviously not fixed... sorry in advance!

We have the exact same situation, a bunch of RDSH guests with Dynamic memory turned on (60+ of them). every day between 1-8 of them will fail to allocate Dynamic memory and will be stuck on their startup RAM amount. This really hurts our users at peak times.

I have engaged our TAM and have raised a case with PSS, Usual story "your the only one with this problem". Which obviously isn't true.

So, we have tons of free RAM on the hosts, 600GB+ on most of them, the issue affects RDSH hosts at random, across multiple hosts and clusters.

The screen shots attached are of one of our hosts from this morning. it has 8GB startup, 8GB minimum, 32GB Maximum RAM configured, with a 23% buffer. the host has 752GB RAM FREE. Notice how the perf counter "Hyper-V Dynamic Memory Integration Service" is reporting "0", it should be reporting "32768". also under task manager on the VM we are missing "Maximum memory" which should be just below "Hardware reserved" in the bottom right hand corner.

Looks like the balloon driver is being delayed at boot time, we are going to XPerf all the servers in the hope that we can catch the critter. It's an unusual problem.

We only have Acrobat PDF viewer, word viewer, excel viewer and two custom dot.NET applications installed on the guests. Some of the servers are also just dumb RDSH hosts, with not connection broker configured, using an F5 loadbalancer for load distribution and session management. All guests are 2012R2 patched up to March 2015, integration Services are installed and up to date (its an enlightened OS remember).

Free Windows Admin Tool Kit Click here and download it now
May 1st, 2015 8:23am

We have been having this same issue, it happens after a reboot and it's as if a VM doesn't know it is set to use dynamic memory - it gets "stuck" at whatever it's startup memory is.  It happens a lot with Chitrix boxes but I've had it happen with other VMs too.  Usually a reboot fixes it but sometimes I've had to shut down the VM and restart it.

It started happening a couple of months ago and it hasn't been a huge deal so far but something is definitely going wrong and it's not just "something specific" to the OP's environment.

-edit-

This post seems to refer to the same issue.

https://social.technet.microsoft.com/Forums/en-US/cc4b4672-473b-4ebf-994f-93034dfdff42/some-windows-guests-not-demanding-more-memory?forum=winserverhyperv


  • Edited by Matt Br Monday, May 11, 2015 5:36 PM
May 11th, 2015 1:47pm

We have been having this same issue, it happens after a reboot and it's as if a VM doesn't know it is set to use dynamic memory - it gets "stuck" at whatever it's startup memory is.  It happens a lot with Chitrix boxes but I've had it happen with other VMs too.  Usually a reboot fixes it but sometimes I've had to shut down the VM and restart it.

It started happening a couple of months ago and it hasn't been a huge deal so far but something is definitely going wrong and it's not just "something specific" to the OP's environment.

-edit-

This post seems to refer to the same issue.

https://social.technet.microsoft.com/Forums/en-US/cc4b4672-473b-4ebf-994f-93034dfdff42/some-windows-guests-not-demanding-more-memory?forum=winserverhyperv


  • Edited by Matt Br Monday, May 11, 2015 5:36 PM
Free Windows Admin Tool Kit Click here and download it now
May 11th, 2015 1:47pm

Our case is progressing nicely MS have admitted that it looks like something in dmvsc.sys which is the dynamic memory driver in 2012 R2. Product team is currently investigating...

A live migration usually kicks dynamic memory back into life (forces a reset in the driver stack for dynamic memory) so currently we are just live migrating machines around (a lot!) to get around the issue. Hopefully an update will follow very soon, i'm due a call with our TAM for a catchup on the issue as it is raised as SEV B.

May 13th, 2015 3:00pm

Thanks for the tip about the live migration, I'll try that next time it happens instead of a reboot!  In the meantime I hope they get it sorted so we can get a hotfix.
Free Windows Admin Tool Kit Click here and download it now
May 13th, 2015 6:29pm

Incidentally this is what we are seeing on the "memory" tab in the Hyper-V manager when one of the VMs is having the dynamic memory issue.


May 14th, 2015 4:26pm

I have the exact same issue. I hope MS will get a fix soon. 
Free Windows Admin Tool Kit Click here and download it now
May 15th, 2015 2:42am

@ Matt Br,

We see the exact same behaviour in Hyper-V Manager...

May 15th, 2015 7:59am

Had the issue on a VM this morning and tried the Live Migration suggestion but unfortunately it didn't solve the problem.  

From what I can remember I've only seen this on 2008 R2 and older VMs - has anyone had the issue with 2012+?

Free Windows Admin Tool Kit Click here and download it now
May 15th, 2015 8:52am

We are seeing this on several 2012R2 Hyper-v clusters with 2012 R2 RDSH guests.

Just chased our TAM, next stage is live debugging and diag of one of our production hosts and guests! :-S

May 15th, 2015 11:04am

Matt,

What workload are you running in your VMs? Ours are mainly RDSH (only seen these VMs affected).
What underlying storage and NICs are you using on the hyper-V hosts? Are they clustered?

FYI we are running QLogic 8262 CNA's for storage and networking and we have EMC Powerpath installed. That's all we have on the hosts (apart from SCEP \ SCOM \ SCCM clients).

The guests have:
Acrobat Reader,
SCEP Client
SCOM Client
Excel Viewer
Word Viewer
Classic Start Menu
Couple of non installed custom dot.net applications written in house

Free Windows Admin Tool Kit Click here and download it now
May 15th, 2015 11:08am

I'm afraid it doesn't seem to be be confined to any one particular type of workload and it has cropped up on both clustered and standalone hosts.
May 15th, 2015 3:49pm

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics