All Global Catalog Servers in forest DC=,DC=,DC= are not responding
New exchange 2007 Enterprise -sp1 on Windows 2008 64x environment. Fresh new forest one 2008 AD site and one subnet. -Two DC/GC/DNS servers SP2 -Two Hub Transport/CAS servers SP1 -Two nodes A/P Single Copy Cluster have mailbox store SP1 -16 SG and 16 DBs -iSCSI Equalogic SAN is used -Problem: Every 12 hours event 2103 error is reported in the application log of mailbox server and many followed by events 1022. At this time new users cannot logon to their mailboxes using OWA and Outlook but can logon to windows fine. No issues what so ever for already logged in users. Mail flow is fine at that time. The problem last for 5 to 7 minutes and disappears by itself. During the daylight saving time we used experience it around 1:00PM and 1:00AM daily. Now we went back to standard time the issue is shifted to 12:00PM and 12:00AM. During the problem I can ping both DCs from the exchange mailbox server and can access all shares fine on DCs. I have checked the Task Scheduler on all servers, I could not find any task is scheduled at those times or runs every 12 hours. I disabled Symantec AV and rebooted all servers and used AutoRuns to make sure AV is not started automatically, still no luck I have opened a ticket with Microsoft. Now it has been two months with resolution. The case has been escalated but no luck. The issue stats with this event: Log Name: Application Source: MSExchange ADAccess Date: 11/13/2009 12:16:04 PM Event ID: 2103 Task Category: Topology Level: Error Keywords: Classic User: N/A Computer: DOMAINMBXN02.Domain.C OM Description: Process MAD.EXE (PID=5948). All Global Catalog Servers in forest DC=,DC=,DC=ca are not responding: DC1.Domain.COM DC2.Domain.COM Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="MSExchange ADAccess" /> <EventID Qualifiers="49156">2103</EventID> <Level>2</Level> <Task>3</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2009-11-13T17:16:04.000Z" /> <EventRecordID>34358</EventRecordID> <Channel>Application</Channel> <Computer>MBX2.Domain.COM</Computer> <Security /> </System> <EventData> <Data>MAD.EXE</Data> <Data>5948</Data> <Data>DC=,DC=,DC=</Data> <Data>DC1.Domain.COM DC2.Domain.com </Data> </EventData> </Event> Followed by many of the following: Log Name: Application Source: MSExchangeIS Mailbox Store Date: 11/13/2009 12:19:38 PM Event ID: 1022 Task Category: Logons Level: Error Keywords: Classic User: N/A Computer: MBX2.Domain.com Description: Logon Failure on database "SG14\DB2" - Windows account Domain\HUB02$; mailbox /o=/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=user name. Error: 2414 Client Machine: HUB2 Client Process: edgetransport.exe Client ProcessId: 0 Client ApplicationId: Client=Hub Transport Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="MSExchangeIS Mailbox Store" /> <EventID Qualifiers="49168">1022</EventID> <Level>2</Level> <Task>16</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2009-11-13T17:19:38.000Z" /> <EventRecordID>34360</EventRecordID> <Channel>Application</Channel> <Computer>MBX2.Domain.com</Computer> <Security /> </System> <EventData> <Data>DOMAIN\HUB2$</Data> <Data>/o=DOMAIN/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=user name</Data> <Data>2414</Data> <Data>SG14\DB2</Data> <Data>HUB2</Data> <Data>edgetransport.exe</Data> <Data>0</Data> <Data>Client=Hub Transport</Data> <Binary>36A1CCFAABEE83419833359EF3CD09C05B444941475F4354585D00000E020000FFB81500000000000002000200003A67F01FFE000000A41D40100F010480A41940100F0104808569201000000000234620100000000065732010000000004D4E2010000000001B5F2010000000001B5F2010000000001B5F2010000000008569201000000000234620100000000065732010000000004D4E2010000000001B5F2010000000001B5F2010000000001B5F201000000000FB49201000000000FB4A2010000000008569201000000000234620100000000065732010000000004D4E2010000000001B5F2010000000001B5F2010000000001B5F2010000000008569201000000000234620100000000065732010000000004D4E2010000000001B5F2010000000001B5F2010000000001B5F201000000000FB49201000000000FB4A201000000000FB49201000000000FB4A201000000000FB49201000000000FB4A2010000000008569201000000000234620100000000065732010000000004D4E2010000000001B5F2010000000001B5F2010000000001B5F201000000000FB49201000000000FB4A2010000000008569201000000000234620100000000065732010000000004D4E2010000000001B5F2010000000001B5F2010000000001B5F201000000000FB49201000000000FB4A201000000000FB60201000000000BD53201000000000BD44201000000000CD64201000000000E82240106E090000E83740106E090000970840106E090000</Binary> </EventData> </Event> Any idea is really appreciated.Abe
November 13th, 2009 11:48pm

Run Exbpa and look at the event logs for AD related issues. also run DCDIAG on both the DCs. Look for event logs on both the GC's.Also disbale the windows FW on both the DC's and the Exchange server. disable the AV based FW. Raj
Free Windows Admin Tool Kit Click here and download it now
November 14th, 2009 12:55am

I wouldnt second guess Microsoft support, but any events in the DCs logs when this is happening? Since you know this happens every 12 hours, I would ask the network guys to put a sniffer on the network switchesthese servers are connected to and see if you can identify any high utilization when this happens. It could be backups or scans or scheduled file transfers interering with the Exchange servers ability to talk to the DCs.
November 14th, 2009 1:36am

Thnks for the reply. The exchange engineer from MS ran Exbpa during the problem nothing is unsuall shown.FW and AV were disabled and rebooted the servers.MPS report was ran on both servers and was analyzed by MS DS engineer and found nothing related to DCs.During the problem DC functionality are fine. Users can logon on the domain at that time fine.Abe
Free Windows Admin Tool Kit Click here and download it now
November 14th, 2009 2:47am

Thanks for your reply. Found no logs during that period. I even increased DS loging but nothing was found realted to this issue. We ran Netmon on both domain servers and exchange. We found that exchange stops sending requests to DC\GC servers for that period of time.We also ran performance monitor and found problem or any lack of resources.All servers are connected to one switch. Our network team monitored the switch and collected some data during the problem but again found nothing. Switch utilization is fine at time.No backup at that or any scheduled tasks at that time. This is newlly installed environment.I have been working on this issue for two months now.Any idea guys?Abe
November 14th, 2009 2:58am

Abe, get the updated firmware and driver where ever possible on the server. I am almost sure its coming from the HW. Run any kind of diagnostic tool that you have for the HW and look for issue.Raj
Free Windows Admin Tool Kit Click here and download it now
November 14th, 2009 10:09am

By any chance your Exchange server is pointing to external DNS server in network settings?Vinod |CCNA|MCSE 2003 +Messaging|MCTS|ITIL V3|
November 16th, 2009 2:39pm

Thanks for the reply Vinod. I have checked and all exchange server network setting, they all pointing to internal DNS.Abe
Free Windows Admin Tool Kit Click here and download it now
November 16th, 2009 4:41pm

Hi Rajnish, All herdware update are recent,Dell technician cam and installed and configured all serversfor us.I have applied Exchage SP2 over the weekend hoping this issue goes away. I will know at 12:00 Pm today.Thanks again for the reply.Abe
November 16th, 2009 4:44pm

Hi Any update on this issue? If this post is helpful, please mark it as such ________________________________________ Alessandro Cardoso MVP | Microsoft Heroes | MCT blog: http://virtualisationandmanagement.wordpress.com http://virtualizacaoegerenciamento.wordpress.com
Free Windows Admin Tool Kit Click here and download it now
December 21st, 2010 12:24am

After doing all these troubleshooting I'd upgrade the Mailbox Server from SP1 to SP2.
December 21st, 2010 4:13pm

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics