Hi There,

we have 4 HTS/CAS Exchange 2010 SP3 servers in our network, all are load balanced via F5 HLB.

Issue Reported: we were reported that users were unable to login in OWA and outlook / mobile devices were affected(they were not able to send/receive emails, mails were stored in outbox).

Resolution Given: Please refer the events I captured in the bottom of this topic, this event occurred only on one of the HTS/CAS Server, and issue has been resolved after rebooting that affected server. this event also stopped after rebooting that server.

Question to you all: anyone is aware of any relevant article or experienced this issue so far, so I can go back to customer with a detailed root cause analysis? I'm already having set of actions to be performed for detailed analysis, but interested to know others opinion as well.

Log Name:      Application
Source:        MSExchange Autodiscover
Date:          7/10/2015 12:55:39 PM upto 10/07/2015 01:34 PM
Event ID:      1
Task Category: Web
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      servername.domain.net
Description:
Unhandled Exception "Could not find any available Global Catalog in forest csmglobal.net."
Stack trace: Microsoft.Exchange.Data.Directory.ADTransientException: Could not find any available Global Catalog in forest csmglobal.net.
   at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, ADObjectId domain, String serverName, Int32 port, NetworkCredential credential)
   at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType)
   at Microsoft.Exchange.Data.Directory.ADSession.GetConnection(String preferredServer, Boolean isWriteOperation, Boolean isNotifyOperation, String optionalBaseDN, ADObjectId& rootId, ADScope scope)
   at Microsoft.Exchange.Data.Directory.ADSession.GetReadConnection(String preferredServer, String optionalBaseDN, ADObjectId& rootId, ADRawEntry scopeDeteriminingObject)
   at Microsoft.Exchange.Data.Directory.ADSession.Find(ADObjectId rootId, String optionalBaseDN, ADObjectId readId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties, CreateObjectDelegate objectCreator, CreateObjectsDelegate arrayCreator, Boolean includeDeletedObjects)
   at Microsoft.Exchange.Data.Directory.ADSession.Find(ADObjectId rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties, CreateObjectDelegate objectCtor, CreateObjectsDelegate arrayCtor)
   at Microsoft.Exchange.Data.Directory.ADSession.Find[TResult](ADObjectId rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties)
   at Microsoft.Exchange.Data.Directory.Recipient.ADRecipientSession.FindBySid(SecurityIdentifier sId)
   at Microsoft.Exchange.Autodiscover.WCF.GetUserSettingsRequestMessage.<>c__DisplayClass1.<ExecuteCommand>b__0()
   at Microsoft.Exchange.Autodiscover.RequestDetailsLogger.TrackLatency(LogField logField, Action method)
   at Microsoft.Exchange.Autodiscover.WCF.GetUserSettingsRequestMessage.ExecuteCommand(IIdentity callerIdentity, CallContext callContext)
   at Microsoft.Exchange.Autodiscover.WCF.GetUserSettingsRequestMessage.Execute()
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MSExchange Autodiscover" />
    <EventID Qualifiers="49152">1</EventID>
    <Level>2</Level>
    <Task>2</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-07-10T10:55:39.000000000Z" />
    <EventRecordID>3545460</EventRecordID>
    <Channel>Application</Channel>
    <Computer>NLADMSXGL001.csmglobal.net</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Could not find any available Global Catalog in forest domain.com.</Data>
    <Data>Microsoft.Exchange.Data.Directory.ADTransientException: Could not find any available Global Catalog in forest csmglobal.net.
   at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, ADObjectId domain, String serverName, Int32 port, NetworkCredential credential)
   at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType)
   at Microsoft.Exchange.Data.Directory.ADSession.GetConnection(String preferredServer, Boolean isWriteOperation, Boolean isNotifyOperation, String optionalBaseDN, ADObjectId&amp; rootId, ADScope scope)
   at Microsoft.Exchange.Data.Directory.ADSession.GetReadConnection(String preferredServer, String optionalBaseDN, ADObjectId&amp; rootId, ADRawEntry scopeDeteriminingObject)
   at Microsoft.Exchange.Data.Directory.ADSession.Find(ADObjectId rootId, String optionalBaseDN, ADObjectId readId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties, CreateObjectDelegate objectCreator, CreateObjectsDelegate arrayCreator, Boolean includeDeletedObjects)
   at Microsoft.Exchange.Data.Directory.ADSession.Find(ADObjectId rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties, CreateObjectDelegate objectCtor, CreateObjectsDelegate arrayCtor)
   at Microsoft.Exchange.Data.Directory.ADSession.Find[TResult](ADObjectId rootId, QueryScope scope, QueryFilter filter, SortBy sortBy, Int32 maxResults, IEnumerable`1 properties)
   at Microsoft.Exchange.Data.Directory.Recipient.ADRecipientSession.FindBySid(SecurityIdentifier sId)
   at Microsoft.Exchange.Autodiscover.WCF.GetUserSettingsRequestMessage.&lt;&gt;c__DisplayClass1.&lt;ExecuteCommand&gt;b__0()
   at Microsoft.Exchange.Autodiscover.RequestDetailsLogger.TrackLatency(LogField logField, Action method)
   at Microsoft.Exchange.Autodiscover.WCF.GetUserSettingsRequestMessage.ExecuteCommand(IIdentity callerIdentity, CallContext callContext)
   at Microsoft.Exchange.Autodiscover.WCF.GetUserSettingsRequestMessage.Execute()</Data>
  </EventData>
</Event>


Log Name:      Application
Source:        MSExchange ADAccess
Date:          7/10/2015 12:55:38 PM upto: 01:34PM
Event ID:      2084
Task Category: Topology
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      Servername.domain.com
Description:
Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1424). No Domain Controller server is up in the local site 'NLADM01'. Exchange Active Directory Provider will use the following out of site Domain Controller servers: 
list of our GC's were listed here.
  
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MSExchange ADAccess" />
    <EventID Qualifiers="16388">2084</EventID>
    <Level>4</Level>
    <Task>3</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-07-10T10:55:38.000000000Z" />
    <EventRecordID>3545439</EventRecordID>
    <Channel>Application</Channel>
    <Computer>NLADMSXGL001.csmglobal.net</Computer>
    <Security />
  </System>
  <EventData>
    <Data>MSEXCHANGEADTOPOLOGYSERVICE.EXE</Data>
    <Data>1424</Data>
    <Data>NLADM01</Data>
    <Data>SGSRESC100.csmglobal.net
NLGCMSC100.csmglobal.net
THRYGSC100.csmglobal.net
NLDMNSCT100.csmglobal.net
</Data>
  </EventData>
</Event>

• Edited by Magudeeswaran M Tuesday, July 14, 2015 4:41 AM

I'm not sure what you're asking, but this looks like you had a network event that broke connectivity to domain controllers, or you don't have one in the same site as the Exchange server, or that one experienced an outage.  You should always have redundant DC/GCs in the site where you have an Exchange server.

Hello Ed,

There are multiple reasons I could say to why I had to post this questions in Technet.

1. Yes we have 5 GCs in our site where the Exchange servers are hosted, infact the GCs are listed in the EMC as well.

2. If it happened due to network connectivity, I believe it should also have been experienced by other HTS/CAS servers at the same time, as I said earlier it happens with only a single node each time.

3. Out of 4 HTS/CAS servers, this issue has happened on 3 servers, each time it was a different server, so i'm totally out of ideas, infact can't justify myself that NIC on the affected machine/network/switch can't be an issue either.

At the moment, I have an idea moving out the affected machine out of HLB and increase the logging level for Exchange AD topology service and look for a most reasonable event and move on from there. but this approach is only possible when I experience this issue next time.

If you can suggest me any other valid approach, I can try it out.

In your HLB configuration do the HT+CAS servers point to the HLB as the default gateway?  Your idea might isolate this as a LB issue.

yes msstd:FQDN is relayed via HLB.

if this is an issue with HLB, I shouldn't have those critical events on Exchange servers, isn't?

I have an evident that during the outage time,though OWA logon page has appeared but the authentication was failing, error said "the target mailbox is not longer available". which means I feel like the CAS/HTS was reachable in the network but this server was unable to relay the authentication to GCs, evident is our above critical events.

also the affected server HTS/CAS was reachable to us via RDP, this means I feel like the authentication was alone affected. what you think?

MS has suggested to implement the below parameters in my environment, we are yet to implement this but not sure whether this will really fix the issue permanently or not.

MS has proposed to run the below commands on each Exchange and also on each GCs.

netsh interface tcp set global rss=disabled

netsh interface tcp set global chimney=disabled

netsh interface tcp set global autotuninglevel=disabled

 

reboot each server right after executing the above commands.

 

They have also proposed to create a registry on all the Exchange servers to disable NetDMA, below the steps:

Click Start, click Run, type regedit, and then click OK.

Locate the following registry subkey, and then click it: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

Double-click the EnableTCPA registry entry. Note If this registry entry does not exist, right-click Parameters, point to New, click DWORD Value, type EnableTCPA, and then press ENTER.

To disable NetDMA, type 0 in the Value data box, and then click OK.

Reboot the server.