Exchange 2013 CU7 - Frequent Outlook 2013 Disconnections and Delays

We currently have the following environment:

Windows 2012 VM on Hyper-V with 24GB of RAM and 4 virtual procs (1 socket)
Cisco UCS, B200 M3 Blades
Exchange 2013 CU7
Exchange 2007 is in our environment, but public folders have been removed and only a few stale, unused accounts exist on it. 
Two dual-role CAS/Mailbox servers
Mailbox servers are in a DAG
MAPI/HTTP is our connection protocol
.NET Framework 4.5.2
Clients are Outlook 2013 SP1 on Windows 7 and 8.1
(Currently round robin DNS, but we have a hardware load balancer we've temporarily taken out of the mix to solve this problem)

What we're seeing is frequent disconnections from Exchange, regardless of the CAS server. That is, both CAS servers will disconnect clients, but not at the same time.  When we look at the Exchange boxes, we notice that the CPU is 99% - 100% each time the disconnections occur.  When we hunt down the IISWorker process causing the sudden CPU run, we see it comes back to the following app pools:

MSExchangeMAPIFrontEndAppPool
MSExchangeMAPIMailboxAppPool

(We were running RPC/HTTP, but switched to MAPI/HTTP to resolve the problem.  We had the same issue with equivalent RPC app pools at the time.)

Eventually, within a minute or two, the CPU load will decrease and the clients connected via that CAS will regain connection.  As you can imagine, the pain is felt more frequently by uncached clients--although everyone, regardless of caching, will see the disconnections.

Basically, the problem sounds exactly as shown in this KB:  http://support.microsoft.com/kb/2995145  Unfortunately, we already had .NET 4.5.2 installed, and have changed the environment variable and registry as shown in the article.  We're in contact with Microsoft Support, but thus far they're scratching their heads.

I'm clearing up all small errors to decrease the noise in the application error logs, but the most persistent one is Event ID 106 (MSExchange Common):

"Performance counter updating error. Counter name is Time in Resource per second, category name is MSExchange Activity Context Resources. Optional code: 2. Exception: The exception thrown is : System.InvalidOperationException: Instance 'ad-powershell-defaultdomain' already exists with a lifetime of Process.  It cannot be recreated or reused until it has been removed or until the process using it has exited."

Loading/reloading and recreation of the performance counters does not fix the problem, by Microsoft Support or us.  Of course, this error existed before we started having issues; I'd just like to clean it up just in case it's a contributing factor.

Has anyone seen anything similar? 


  • Edited by MAMP2 Monday, March 09, 2015 10:39 PM Misspelling/Added Link
March 9th, 2015 10:27pm

Hi,

From your description, "the CPU is 99% - 100% each time the disconnections occur". I recommend you check which program consumes excessive CPU resources on Exchange servers at first.

Hope this can be helpful to you.

Best regards,

Free Windows Admin Tool Kit Click here and download it now
March 10th, 2015 4:52am

Amy,

Thanks for the reply, but as mentioned above it's the following two app pools causing the high CPU use:

MSExchangeMAPIFrontEndAppPool
MSExchangeMAPIMailboxAppPool

March 10th, 2015 6:35am

I would open a ticket with Microsoft Support.
Free Windows Admin Tool Kit Click here and download it now
March 10th, 2015 7:20am

Thanks for replying, Andy, but as mentioned above we have a ticket open with them; unfortunately, they're scratching their heads. I was hoping that someone else might have experienced something similar.
March 10th, 2015 7:53am

Thanks for replying, Andy, but as mentioned above we have a ticket open with them; unfortunately, they're scratching their heads. I was hoping that someone else might have experienced something similar.

Ah ok. I see that now. Escalate! ( I know easier said than done). But surely they can get traces from you during those high CPU spikes.

BTW, for your user count and profile, did you size the VM guests using the Mailbox Calculator?

Free Windows Admin Tool Kit Click here and download it now
March 10th, 2015 8:17am

We had the same problem (Exchange CU7 100% CPU) with the same conditions. In fact your original post was the same day we started seeing the problem. Also had an MS support case with the same steps with no result. Thought maybe it was too many mailboxes per database (1200) until I started talking to others and they are running 10,000+ mailboxes per database. 

Currently we have isolated it to the load balancer which is an F5 BigIP. Removing the load balancer and using round robin DNS has been the band-aid. Next step is to open a case with F5 to determine why it is not closing connections suddenly after CU7.

April 3rd, 2015 6:56pm

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics