Event logs repeating event ID 5011 WAS and 6002 ping of mdb

Hi,

We're working on a new Exchange 2013 CU2 migration on Windows Server 2012 from Exchange 2007 SP3 on Windows 2008.  We have two Exchange 2013 physical servers running CAS and a single DAG (2 servers 1 DB) with a 2008 R2 witness server.  Everything seems to be working fine but we are getting the following events every few minutes in the event logs on both servers.  I've searched high and low and can't seem to find a resolution.  Any suggestions?

Event ID 5011 from the Source WAS with this info:

A process serving application pool 'MSExchangeSyncAppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '21380'. The data field contains the error number.

Event ID 6002 from the source MSExchange Mid-Tier Storage

Ping of mdb '5d3c32ed-5c53-44b7-abe4-34fcdf08f848' timed out after '00:00:00' minutes.  Last successful ping was at '8/8/2013 5:46:37 PM' UTC.

Event ID 1006 source MSExchangeDiagnostics

The performance counter '\\SERVER-EX1\LogicalDisk(HarddiskVolume1)\Free Megabytes' sustained a value of '108.00', for the '15' minute(s) interval starting at '8/8/2013 5:34:00 PM'. Additional information: None. Trigger Name:DatabaseDriveSpaceTrigger. Instance:harddiskvolume1

Event ID 4999 source MSExchange Common

Watson report about to be sent for process id: 11124, with parameters: E12, c-RTL-AMD64, 15.00.0712.022, unknown, M.E.Data.Directory, M.E.D.D.ConnectionPoolManager.BlockImpersonatedCallers, M.E.Common.FailFastException, e5e6, 15.00.0712.016.
ErrorReportingEnabled: False



August 8th, 2013 9:47pm

Hi,

Exchange 2013 CU2 is available now. You can firstly install it and see whether it covers these issues.

http://www.microsoft.com/en-us/download/default.aspx

Thanks,

If you have feedback for TechNet Subscriber Support, contact tnsfl@microsoft.com

Free Windows Admin Tool Kit Click here and download it now
August 9th, 2013 7:48pm

Hi Simon,

Thank you for your help.

I was mistaken when I stated that we were using Exchange 2013 CU1.  We are actually using Exchange 2013 CU2.  I've changed the original post.  The CU2 media was used to install the server from the start.

Now that you know that CU2 is installed.  Do you have any suggestions?

August 12th, 2013 12:43pm

I'm having the same issue you are... I started getting a warning 4998 after installing CU1, CU2 did nothing to resolve.

The closest KB I've seen so-far is http://support.microsoft.com/kb/2639150 which is useless.

My environment is similar but I only have two servers 2007 SP3 CU10, 2013 CU2... I wish I had helpful information but I'm still yet to figure this out (also, my phones will stop syncing every few days and I'll have to run an iisreset to fix things temporarily and I know it because it stopped logging these errors for that time period).

Free Windows Admin Tool Kit Click here and download it now
August 13th, 2013 10:17pm

Well I'm glad I'm not the only one having this issue Brandon.

So far we haven't noticed any actual problems due to these events.  We only have a couple mailboxes migrated over to 2013 right now though.  The majority of our mailboxes are still on 2007.  What about you?

We've submitted an MS support ticket and have been working with them on these.  They have said that the 5011 and 4999 are related to each other.  They said that we should continue with our mailbox migration and to follow up if we encounter any problems.  Their theory is that these errors will go away when coexistence is complete and 2007 has been removed from our environment.  I really don't want to risk production connectivity issues but I'm not sure what more I can do.  I've searched everywhere for a solution.  To me it looks like a bug in 2010 made it back into the code of 2013 and MS needs to release a patch.

For the even ID's 6002 and 1006 the MS engineer stated that these were known ID's with CU2 and they don't indicate a problem.  It would be nice if MS would clean up some of these issues.

August 14th, 2013 12:46am

Yeah we've got 10 (of 1000) migrated so-far... I'll be opening a ticket next week with Microsoft unless they can fix it for you or I can find that hidden fix.

Last night I installed 2007 SP3 Rollup 11 with the latest security update for 2013 CU2, rebooted both and it did nothing to resolve.

I stopped recycling my exchange sync app pool and hopefully it stops crashing. My biggest problem is when the errors cease I get a false sense of 'it fixed itself' (they're about 8min apart) then phones stop syncing and I recycle the app pool and the errors come back but at least the phones sync.

I also get the 6002 and 1006 id's and found somewhere that you can disable that performance monitor but I'm not so concerned about that at the moment too (search for Orbdot on this page http://blogs.technet.com/b/exchange/archive/2013/07/09/released-exchange-server-2013-rtm-cumulative-update-2.aspx#3587722 ).



Free Windows Admin Tool Kit Click here and download it now
August 14th, 2013 8:48pm

You can add me to the 'I got the same issue' list. I'm running 2013 CU2 in coexistence mode with 2007 SP3 Rollup 10. My main concern is 5011 MSExchangeSyncAppPool error. 
August 15th, 2013 1:26pm

Thanks for the info rjensen.

I have 50 mailboxes migrated over and phones lost connectivity today.  It happened just like Brandon described.  I checked the logs and the WAS error stopped 45 minutes before the user reported that their phone couldn't connect.  I did an iisreset, the error came back, and everything started working.

After that, I escalated my ticket with Microsoft Support.  I worked with an engineer for a few hours after it happened.  He checked the app pool and some other configuration settings and didn't find any configuration issues.  He's doing more research and will get back to me tomorrow.

Do you guys think that finishing the migration and removing EX 2007 will stop this issue?  According to KB 2639150, this issue occurs because the Exchange ActiveSync ProxyHandler does not close anonymous connections between the Exchange 2010 Client Access server and the Exchange Server 2003 servers.  If it is the same problem, just 2013 and 2007, I have to guess that removing 2007 will remove the open anonymous connections.  Thoughts?

Free Windows Admin Tool Kit Click here and download it now
August 15th, 2013 8:11pm

M1P, I first was using iisreset (wasn't so good since we're using rpc over http and it disconnects outlook), but if you need to fix it for the phones quickly the most effective way is to right click and manually recycle this Application Pool in IIS: MSExchangeSyncAppPool (it'll take a minute and then everything will work again. I just today put mine on a 3hour recycle cycle for the sake of stability and keeping the trust of my end users... hopefully it works correctly).

I am really hoping that finishing the migration will solve the problem, but like the problems I'm hearing about with Post-Public Folder migrations I can't help but feel a little skeptical (although it should cease all connections back to 2007). I'm also worried about more issues I might have to face in post-migration but I've gotta get moving on this soon (users are getting anxious for the fancy cross browser owa and I'm now behind schedule).

Let us know what you find out. Thanks for the updates.

August 15th, 2013 11:35pm

Thanks for the info Brandon.  Where are the Microsoft Moderators here?  I was hoping someone would chime in and say that this is a known issue or something.  Come on Microsoft, can we have some help with these issues?

As far as recycling the app pool every 3 hours, does that cause any user connection interruptions that you're aware of?  I know it will cause a slight delay as the app pool re-caches.  I'm wondering if the users will get an error.  I'm considering doing a app pool recycle nightly.  Please reply to this post and let us know if your configuration stabilizes the issue.

I guess I have to do my research on the Post-Public Folder migration issues you mentioned.  Now I'm nervous of what lies ahead. :-(

My users are also very anxious.  That's the main reason we're pushing ahead.  I'll keep you updated.

Free Windows Admin Tool Kit Click here and download it now
August 16th, 2013 12:52pm

So-far so good with the 3hr recycle (I haven't been checking logs for connection interruptions). We usually don't set our users up with push and put them on a 15min check (because we've had so many battery complaints and push seems to drain the battery fast). Some have push and they haven't complained (or noticed) yet and I don't think any errors have popped up on the phones. It has only been a day, but it has been more stable and I'm hopeful. Also in our Org, we could be fine with a 1 to 2 or even 5 minute delay in most email scenarios (I usually get calls at 15mins).

I agree, where are the Microsoft Mods... not even an attempt to troubleshoot.

At this point, I'm going to plan our deployment to each department next week and just push forward in the migration and hopefully when I can finally drop the 2007 box our event logs will have less red and yellow.

August 16th, 2013 4:40pm

Well I set mine up to recycle outside of business hours.  It didn't seem to help.  Mobile devices just disconnected again.  I manually recycled the app pool and tried to sync again and it didn't work.  I had to do an IIS reset for mobile connectivity.  This sucks.
Free Windows Admin Tool Kit Click here and download it now
August 17th, 2013 11:12am

Yeah, I noticed as well. The auto-recycle doesn't appear to do anything. When it stopped I just manually recycle it and it luckily came back.

I'm going to assume the most stability we're going to get is with a nightly iisreset...

August 19th, 2013 11:44am

I've turned off the recycling.  We had several failures over the weekend that required IIS restarts.  I even tried stopping and starting the app pool.  The app pool would stop but it wouldn't start back up without an IIS reset.

We're looking into putting IIS ARR in front of our CAS servers and then using the microsoft-server-activesync/healthcheck.htm to determine availability.  Right now we're waiting for the problem to happen again to make sure the healthcheck.htm actually reports that there's a problem with activesync.  If it doesn't, then we're going to have to figure out a way to determine failure and IIS reset with our monitoring server. http://blogs.technet.com/b/exchange/archive/2013/08/02/part-2-reverse-proxy-for-exchange-server-2013-using-iis-arr.aspx

A moderator moved my thread but still no valid responses from Microsoft on this.  All the engineers on my support ticket are still "researching" this issue.  I've been waiting since Thursday for a call back from the escalation team.  I've found more people with this same problem online.

Free Windows Admin Tool Kit Click here and download it now
August 19th, 2013 1:56pm

It looks like the microsoft-server-activesync/healthcheck.htm page reports when there is a problem.  If you have a monitoring server you can use that to indicate the issue and automate the IIS reset.  It's not the most elegant work around but it's better than having users without connectivity.

I'm still waiting for Microsoft to diagnose this bug.

August 20th, 2013 5:43pm

Nice find! I can replace that scheduled reset now.

I'll get my monitoring server to verify the text on the page https://server.domain.com/Microsoft-Server-ActiveSync/HealthCheck.htm is "200 OK<br/>server.domain.local"

Free Windows Admin Tool Kit Click here and download it now
August 20th, 2013 6:43pm

I migrated our last mailbox off of Exchange 2007 yesterday and both the 5011 and 4999 events are gone from the logs.  Still no input whatsoever from Microsoft on this post.  

Also, no input whatsoever from Microsoft on the support ticket I've submitted.  They've just done a bunch of information gathering and that's it.  Every time I press them to ask a developer to look at the code to confirm that Exchange ActiveSync ProxyHandler does not close anonymous connections, they tell me that is an Exchange 2010 problem and not 2013.  Based on my findings it's obvious that the bug made it into Exchange 2013.

What a joke.  Microsoft has failed us on every level with this release.

We're still monitoring Activesync and I'll let you know if it crashes now that all mailboxes are on 2013.  I'll be migrating the public folders this weekend.  We'll see how that goes.

August 23rd, 2013 1:19pm

I'm glad to hear that the problem clears up after all the mailboxes have been moved from 2007. To be honest I am a little upset that you were not able to find a solution from Microsoft. I updated my DNS to point to my new installation but backed them off when I saw all the errors in the Event Logs. Since nothing was resolved it looks like I will have to do a mad dash mailbox move.

Thanks for the update and let us know if you have any new problems. 

Free Windows Admin Tool Kit Click here and download it now
August 26th, 2013 6:04pm

So far so good.  We haven't had one Activsync disconnection event since last week.  We were having them almost daily through the migration.

I migrated the public folders this weekend and had an issue where they would appear under a folder called IPM_SUBTREE instead of at the root.  I rolled back that migration and am researching the issue.  It looks like that might be a client side issue.

August 26th, 2013 7:05pm

Here's Microsoft's final result on my support incident.  Good luck to the rest of you on these errors.

After analyzing the Event logs and the Active Monitoring logs, there is not much information which could say what is causing the issue. We are continuously monitoring the cases which we get which are of similar kind. Once we gather more information related to the same issue, we will come up with a fix in the later update release of Exchange 2013.


Free Windows Admin Tool Kit Click here and download it now
August 27th, 2013 5:47pm

Hi,

i have the same issue and Systems and i am concerned because i cant complete the migration at this time.

Every 10 Minutes Systemlog WAS 5011 and a new Prozess for MSExchangeSyncAppPool

The coexistence with E2K7 must be in use for an old Blackberry System until .....?.

Its time for MS to make a suggestion to resolve!

Looking forward CU3..

filuee

September 11th, 2013 5:23am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics