Problems with NTFRS replication

I have a client site with three DC's.

PIC-DC1 is a Windows Server 2008

PIC-DC2 is a Windows Server 2003

PIC-DC3 is a Windows Server 2012R2 which I just promoted to DC.

I noticed that when DC2 goes down which has been happening frequently that network logins time out.  DHCP clients are set to use both servers to resolve DNS.

I have also noticed on DC1 and DC3 that I have NTFRS errors and that the SYSVOL and NETLOGON share is missing.  Error messages in the log are EVENT 13508

In the history of this client, DC1 crashed.  I seized roles to DC2, cleaned up the meta data and re-added DC1 as a DC after it was successfully rebuilt.  I have a feeling that it was at this point that we have been having this issue.

DNS and active directory seems to be replicating just fine.  As well, replication entries seem to be in place.  DNS resolution seems normal.  I will post DCDIAG from DC1 and DC2 here as well other diagnostic information for review.

FROM DC1

Directory Server Diagnosis

Performing initial setup:
   Trying to find home server...
   Home Server = PIC-DC1
   * Identified AD Forest.
   Done gathering initial info.

Doing initial required tests

   Testing server: Default-First-Site-Name\PIC-DC1
      Starting test: Connectivity
         ......................... PIC-DC1 passed test Connectivity

Doing primary tests

   Testing server: Default-First-Site-Name\PIC-DC1
      Starting test: Advertising
         Warning: DsGetDcName returned information for \\PIC-DC2.pic.local,
         when we were trying to reach PIC-DC1.
         SERVER IS NOT RESPONDING or IS NOT CONSIDERED SUITABLE.
         ......................... PIC-DC1 failed test Advertising
      Starting test: FrsEvent
         There are warning or error events within the last 24 hours after the
         SYSVOL has been shared.  Failing SYSVOL replication problems may cause
         Group Policy problems.
         ......................... PIC-DC1 passed test FrsEvent
      Starting test: DFSREvent
         ......................... PIC-DC1 passed test DFSREvent
      Starting test: SysVolCheck
         ......................... PIC-DC1 passed test SysVolCheck
      Starting test: KccEvent
         ......................... PIC-DC1 passed test KccEvent
      Starting test: KnowsOfRoleHolders
         ......................... PIC-DC1 passed test KnowsOfRoleHolders
      Starting test: MachineAccount
         ......................... PIC-DC1 passed test MachineAccount
      Starting test: NCSecDesc
         ......................... PIC-DC1 passed test NCSecDesc
      Starting test: NetLogons
         Unable to connect to the NETLOGON share! (\\PIC-DC1\netlogon)
         [PIC-DC1] An net use or LsaPolicy operation failed with error 67,
         The network name cannot be found..
         ......................... PIC-DC1 failed test NetLogons
      Starting test: ObjectsReplicated
         ......................... PIC-DC1 passed test ObjectsReplicated
      Starting test: Replications
         ......................... PIC-DC1 passed test Replications
      Starting test: RidManager
         ......................... PIC-DC1 passed test RidManager
      Starting test: Services
         ......................... PIC-DC1 passed test Services
      Starting test: SystemLog
         An Warning Event occurred.  EventID: 0x800007DD
            Time Generated: 06/26/2015   08:41:08
            Event String:
            The S: disk is at or near capacity.  You may need to delete some fi
es.
         An Error Event occurred.  EventID: 0xC0060019
            Time Generated: 06/26/2015   08:43:28
            Event String:
            The shadow copies of volume D: were deleted because the shadow copy
storage could not grow in time.  Consider reducing the IO load on the system or
choose a shadow copy storage volume that is not being shadow copied.
         ......................... PIC-DC1 failed test SystemLog
      Starting test: VerifyReferences
         ......................... PIC-DC1 passed test VerifyReferences


   Running partition tests on : ForestDnsZones
      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test
         CrossRefValidation

   Running partition tests on : DomainDnsZones
      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test
         CrossRefValidation

   Running partition tests on : Schema
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation

   Running partition tests on : Configuration
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation

   Running partition tests on : pic
      Starting test: CheckSDRefDom
         ......................... pic passed test CheckSDRefDom
      Starting test: CrossRefValidation
         ......................... pic passed test CrossRefValidation

   Running enterprise tests on : pic.local
      Starting test: LocatorCheck
         ......................... pic.local passed test LocatorCheck
      Starting test: Intersite
         ......................... pic.local passed test Intersite

FROM DC2

Domain Controller Diagnosis

Performing initial setup:
   Done gathering initial info.

Doing initial required tests

   Testing server: Default-First-Site-Name\PIC-DC2
      Starting test: Connectivity
         ......................... PIC-DC2 passed test Connectivity

Doing primary tests

   Testing server: Default-First-Site-Name\PIC-DC2
      Starting test: Replications
         ......................... PIC-DC2 passed test Replications
      Starting test: NCSecDesc
         ......................... PIC-DC2 passed test NCSecDesc
      Starting test: NetLogons
         ......................... PIC-DC2 passed test NetLogons
      Starting test: Advertising
         ......................... PIC-DC2 passed test Advertising
      Starting test: KnowsOfRoleHolders
         ......................... PIC-DC2 passed test KnowsOfRoleHolders
      Starting test: RidManager
         ......................... PIC-DC2 passed test RidManager
      Starting test: MachineAccount
         ......................... PIC-DC2 passed test MachineAccount
      Starting test: Services
         ......................... PIC-DC2 passed test Services
      Starting test: ObjectsReplicated
         ......................... PIC-DC2 passed test ObjectsReplicated
      Starting test: frssysvol
         ......................... PIC-DC2 passed test frssysvol
      Starting test: frsevent
         There are warning or error events within the last 24 hours after the
         SYSVOL has been shared.  Failing SYSVOL replication problems may cause
         Group Policy problems.
         ......................... PIC-DC2 failed test frsevent
      Starting test: kccevent
         ......................... PIC-DC2 passed test kccevent
      Starting test: systemlog
         ......................... PIC-DC2 passed test systemlog
      Starting test: VerifyReferences
         ......................... PIC-DC2 passed test VerifyReferences

   Running partition tests on : ForestDnsZones
      Starting test: CrossRefValidation
         ......................... ForestDnsZones passed test CrossRefValidatio

      Starting test: CheckSDRefDom
         ......................... ForestDnsZones passed test CheckSDRefDom

   Running partition tests on : DomainDnsZones
      Starting test: CrossRefValidation
         ......................... DomainDnsZones passed test CrossRefValidatio

      Starting test: CheckSDRefDom
         ......................... DomainDnsZones passed test CheckSDRefDom

   Running partition tests on : Schema
      Starting test: CrossRefValidation
         ......................... Schema passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom

   Running partition tests on : Configuration
      Starting test: CrossRefValidation
         ......................... Configuration passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom

   Running partition tests on : pic
      Starting test: CrossRefValidation
         ......................... pic passed test CrossRefValidation
      Starting test: CheckSDRefDom
         ......................... pic passed test CheckSDRefDom

   Running enterprise tests on : pic.local
      Starting test: Intersite
         ......................... pic.local passed test Intersite
      Starting test: FsmoCheck
         ......................... pic.local passed test FsmoCheck

C:\Documents and Settings\james>


June 26th, 2015 1:47pm

AD Replication Status Tool shows no errors

REPADMIN from DC1


Default-First-Site-Name\PIC-DC1
DSA Options : IS_GC 
objectGuid  : 22a95a40-4bfd-46c8-9e1b-8afa8f5ee193
invocationID: 7c3a6b31-c366-48ae-943f-b9d7a3c3ec88

==== INBOUND NEIGHBORS ======================================

DC=pic,DC=local
    Default-First-Site-Name\PIC-DC3 via RPC
        objectGuid: 9e03e76d-5a38-498b-b448-8c9c6be97663
        Last attempt @ 2015-06-26 09:56.32 was successful.
    Default-First-Site-Name\PIC-DC2 via RPC
        objectGuid: 2c79447a-e22b-4d7f-8e31-6ff708e078ab
        Last attempt @ 2015-06-26 09:56.32 was successful.

CN=Configuration,DC=pic,DC=local
    Default-First-Site-Name\PIC-DC3 via RPC
        objectGuid: 9e03e76d-5a38-498b-b448-8c9c6be97663
        Last attempt @ 2015-06-26 09:56.32 was successful.
    Default-First-Site-Name\PIC-DC2 via RPC
        objectGuid: 2c79447a-e22b-4d7f-8e31-6ff708e078ab
        Last attempt @ 2015-06-26 09:56.32 was successful.

CN=Schema,CN=Configuration,DC=pic,DC=local
    Default-First-Site-Name\PIC-DC3 via RPC
        objectGuid: 9e03e76d-5a38-498b-b448-8c9c6be97663
        Last attempt @ 2015-06-26 09:56.32 was successful.
    Default-First-Site-Name\PIC-DC2 via RPC
        objectGuid: 2c79447a-e22b-4d7f-8e31-6ff708e078ab
        Last attempt @ 2015-06-26 09:56.32 was successful.

DC=DomainDnsZones,DC=pic,DC=local
    Default-First-Site-Name\PIC-DC3 via RPC
        objectGuid: 9e03e76d-5a38-498b-b448-8c9c6be97663
        Last attempt @ 2015-06-26 09:56.32 was successful.
    Default-First-Site-Name\PIC-DC2 via RPC
        objectGuid: 2c79447a-e22b-4d7f-8e31-6ff708e078ab
        Last attempt @ 2015-06-26 09:56.32 was successful.

DC=ForestDnsZones,DC=pic,DC=local
    Default-First-Site-Name\PIC-DC3 via RPC
        objectGuid: 9e03e76d-5a38-498b-b448-8c9c6be97663
        Last attempt @ 2015-06-26 09:56.32 was successful.
    Default-First-Site-Name\PIC-DC2 via RPC
        objectGuid: 2c79447a-e22b-4d7f-8e31-6ff708e078ab
        Last attempt @ 2015-06-26 09:56.32 was successful.

==== OUTBOUND NEIGHBORS FOR CHANGE NOTIFICATIONS ============

DC=pic,DC=local
    Default-First-Site-Name\PIC-DC3 via RPC
        objectGuid: 9e03e76d-5a38-498b-b448-8c9c6be97663
    Default-First-Site-Name\PIC-DC2 via RPC
        objectGuid: 2c79447a-e22b-4d7f-8e31-6ff708e078ab

CN=Configuration,DC=pic,DC=local
    Default-First-Site-Name\PIC-DC3 via RPC
        objectGuid: 9e03e76d-5a38-498b-b448-8c9c6be97663
    Default-First-Site-Name\PIC-DC2 via RPC
        objectGuid: 2c79447a-e22b-4d7f-8e31-6ff708e078ab

CN=Schema,CN=Configuration,DC=pic,DC=local
    Default-First-Site-Name\PIC-DC3 via RPC
        objectGuid: 9e03e76d-5a38-498b-b448-8c9c6be97663
    Default-First-Site-Name\PIC-DC2 via RPC
        objectGuid: 2c79447a-e22b-4d7f-8e31-6ff708e078ab

DC=DomainDnsZones,DC=pic,DC=local
    Default-First-Site-Name\PIC-DC2 via RPC
        objectGuid: 2c79447a-e22b-4d7f-8e31-6ff708e078ab
    Default-First-Site-Name\PIC-DC3 via RPC
        objectGuid: 9e03e76d-5a38-498b-b448-8c9c6be97663

DC=ForestDnsZones,DC=pic,DC=local
    Default-First-Site-Name\PIC-DC2 via RPC
        objectGuid: 2c79447a-e22b-4d7f-8e31-6ff708e078ab
    Default-First-Site-Name\PIC-DC3 via RPC
        objectGuid: 9e03e76d-5a38-498b-b448-8c9c6be97663

Free Windows Admin Tool Kit Click here and download it now
June 26th, 2015 1:58pm

------------------------------------------------------------
FRSDiag v1.7 on 6/26/2015 9:56:37 AM
.\PIC-DC1 on 2015-06-26 at 9.56.37 AM
------------------------------------------------------------

Checking for minimum FRS version requirement ... passed
Checking for errors/warnings in ntfrsutl ds ... passed
Checking for Replica Set configuration triggers... passed
Checking for suspicious file Backlog size... passed
Checking Overall Disk Space and SYSVOL structure (note: integrity is not checked)... passed
Checking for suspicious inlog entries ... passed
Checking for suspicious outlog entries ... passed
Checking for appropriate staging area size ... passed
Checking for errors in debug logs ...
ERROR on NtFrs_0005.log : "ERROR_ACCESS_DENIED" : <SndCsMain:                     7880:   904: S0: 09:30:02> :SR: Cmd 066752a8, CxtG bb52c01d, WS ERROR_ACCESS_DENIED, To   PIC-DC2.pic.local Len:  (368) [SndFail - Send Penalty]
ERROR on NtFrs_0005.log : "ERROR_ACCESS_DENIED" : <SndCsMain:                     5808:   877: S0: 09:45:02> :SR: Cmd 0117de30, CxtG bb52c01d, WS ERROR_ACCESS_DENIED, To   PIC-DC2.pic.local Len:  (368) [SndFail - rpc call]
ERROR on NtFrs_0005.log : "ERROR_ACCESS_DENIED" : <SndCsMain:                     5808:   904: S0: 09:45:02> :SR: Cmd 0117de30, CxtG bb52c01d, WS ERROR_ACCESS_DENIED, To   PIC-DC2.pic.local Len:  (368) [SndFail - Send Penalty]

Found 5534 ERROR_ACCESS_DENIED error(s)! Latest ones (up to 3) listed above

 ......... failed with 5534 error entries
Checking NtFrs Service (and dependent services) state...
ERROR : Cannot access SYSVOL share on PIC-DC1
ERROR : Cannot access NETLOGON share on PIC-DC1
 ......... failed 2
Checking NtFrs related Registry Keys for possible problems...
SYSTEM\CurrentControlSet\Services\Netlogon\Parameters\SysvolReady = 0 :: ERROR: SysvolReady is not set to 1 :: SYSVOL is likely not Sharing! This key should NOT be changed manually but this should be addressed! See article KB.327781 (How to Troubleshoot Missing SYSVOL and NETLOGON Shares on Windows Server) for further information!
failed with 1 error(s) and 0 warning(s)

Checking Repadmin Showreps for errors...passed


Final Result = failed with 5537 error(s)
June 26th, 2015 2:02pm

Hi,

Can you please provide the following Data.

IPCONFIG /All from all DC's without editing. Also run the following command and upload the output to common location. Check all required ports are open using the portQuery Tool

Repadmin /Replsum /Errorsonly

dcdiag /v /c /d /e /s:FQDN

portqry -n IP address of working DC -e 3269 -p both

3268 389 53 137 135 88 123 etc.


Free Windows Admin Tool Kit Click here and download it now
June 26th, 2015 2:03pm

Windows IP Configuration

   Host Name . . . . . . . . . . . . : PIC-DC1
   Primary Dns Suffix  . . . . . . . : pic.local
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : pic.local

Ethernet adapter Local Area Connection 8:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Broadcom BCM5708C NetXtreme II GigE (NDIS
 VBD Client) #8
   Physical Address. . . . . . . . . : 84-2B-2B-09-90-EF
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes

Ethernet adapter Local Area Connection 7:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Broadcom BCM5708C NetXtreme II GigE (NDIS
 VBD Client) #7
   Physical Address. . . . . . . . . : 84-2B-2B-09-90-ED
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::25a5:468f:d536:f6f7%16(Preferred)
   IPv4 Address. . . . . . . . . . . : 192.168.1.5(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.100
   DHCPv6 IAID . . . . . . . . . . . : 495201067
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-13-B6-9A-2D-00-1E-C9-E3-37-29

   DNS Servers . . . . . . . . . . . : 192.168.1.6
                                       192.168.1.5
   NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter Local Area Connection* 8:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : isatap.{CC781E30-581E-4948-9DCA-818CA511E
774}
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter Local Area Connection* 9:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : isatap.{789ED374-948A-42F1-A907-07E9B675E
B89}
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Source DSA          largest delta    fails/total %%   error
 PIC-DC1                   46m:48s    0 /  10    0
 PIC-DC2                   46m:48s    0 /  10    0
 PIC-DC3                   43m:16s    0 /  10    0


Destination DSA     largest delta    fails/total %%   error
 PIC-DC1                   26m:28s    0 /  10    0
 PIC-DC2                   43m:17s    0 /  10    0
 PIC-DC3                   46m:49s    0 /  10    0



Windows IP Configuration



   Host Name . . . . . . . . . . . . : PIC-DC2

   Primary Dns Suffix  . . . . . . . : pic.local

   Node Type . . . . . . . . . . . . : Hybrid

   IP Routing Enabled. . . . . . . . : No

   WINS Proxy Enabled. . . . . . . . : No

   DNS Suffix Search List. . . . . . : pic.local



Ethernet adapter Local Area Connection:



   Connection-specific DNS Suffix  . : 

   Description . . . . . . . . . . . : Broadcom NetXtreme Gigabit Ethernet

   Physical Address. . . . . . . . . : 00-15-C5-F6-22-2A

   DHCP Enabled. . . . . . . . . . . : Yes

   Autoconfiguration Enabled . . . . : Yes

   Autoconfiguration IP Address. . . : 169.254.140.131

   Subnet Mask . . . . . . . . . . . : 255.255.0.0

   Default Gateway . . . . . . . . . : 

   Primary WINS Server . . . . . . . : 192.168.1.10



Ethernet adapter Local Area Connection 2:



   Connection-specific DNS Suffix  . : 

   Description . . . . . . . . . . . : Intel(R) PRO/1000 GT Desktop Adapter

   Physical Address. . . . . . . . . : 00-1B-21-21-42-DC

   DHCP Enabled. . . . . . . . . . . : No

   IP Address. . . . . . . . . . . . : 192.168.1.6

   Subnet Mask . . . . . . . . . . . : 255.255.255.0

   Default Gateway . . . . . . . . . : 192.168.1.100

   DNS Servers . . . . . . . . . . . : 192.168.1.5

                                       192.168.1.6
Replication Summary Start Time: 2015-06-29 10:49:16



Beginning data collection for replication summary, this may take awhile:

  ......





Source DC           largest delta  fails/total  %%  error

 PIC-DC1                   06m:33s    0 /  10    0  

 PIC-DC2                   06m:33s    0 /  10    0  

 PIC-DC3                   04m:46s    0 /  10    0  





Destination DC    largest delta    fails/total  %%  error

 PIC-DC1                   06m:34s    0 /  10    0  

 PIC-DC2                   06m:33s    0 /  10    0  

 PIC-DC3                   04m:46s    0 /  10    0  




June 29th, 2015 3:02pm

The rest of the data I have in log files.

https://drive.google.com/a/dcs-support.com/file/d/0B2ElvcACDfsPckJXZzJtYTB0TDg/view?usp=sharing

https://drive.google.com/a/dcs-support.com/file/d/0B2ElvcACDfsPSFJ4RWszcFN1Vlk/view?usp=sharing

https://drive.google.com/a/dcs-support.com/file/d/0B2ElvcACDfsPVVZGd0lNX3p6VVE/view?usp=sharing

https://drive.google.com/a/dcs-support.com/file/d/0B2ElvcACDfsPTllCRXRlblVORkU/view?usp=sharing

Free Windows Admin Tool Kit Click here and download it now
June 29th, 2015 3:08pm

Hi,

Please see the Host Name . . . . . . . . . . . . : PIC-DC2having the wrong TCP/IP settings. Also check in PIC-DC1 DNS ip address should be the local DNS server IP-Address unless you have separate DNS server. I will go through the logs you have uploaded further and provide with my findings later.  There are two network adaptor on DC. Never recommended.

Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix  . : 

   Description . . . . . . . . . . . : Broadcom NetXtreme Gigabit Ethernet

   Physical Address. . . . . . . . . : 00-15-C5-F6-22-2A

   DHCP Enabled. . . . . . . . . . . : Yes

   Autoconfiguration Enabled . . . . : Yes

   Autoconfiguration IP Address. . . : 169.254.140.131

   Subnet Mask . . . . . . . . . . . : 255.255.0.0

   Default Gateway . . . . . . . . . : 

   Primary WINS Server . . . . . . . : 192.168.1.10


June 29th, 2015 4:27pm

That adapter is not being used.  I just disabled it.

Free Windows Admin Tool Kit Click here and download it now
June 29th, 2015 9:23pm

New Output

Windows IP Configuration

   Host Name . . . . . . . . . . . . : PIC-DC2
   Primary Dns Suffix  . . . . . . . : pic.local
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : pic.local

Ethernet adapter Local Area Connection 2:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 GT Desktop Adapter
   Physical Address. . . . . . . . . : 00-1B-21-21-42-DC
   DHCP Enabled. . . . . . . . . . . : No
   IP Address. . . . . . . . . . . . : 192.168.1.6
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.100
   DNS Servers . . . . . . . . . . . : 192.168.1.5
                                       192.168.1.6
------------------------------------------------------------------------------------
   Host Name . . . . . . . . . . . . : PIC-DC1
   Primary Dns Suffix  . . . . . . . : pic.local
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : pic.local

Ethernet adapter Local Area Connection 7:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Broadcom BCM5708C NetXtreme II GigE (NDI
 VBD Client) #7
   Physical Address. . . . . . . . . : 84-2B-2B-09-90-ED
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::25a5:468f:d536:f6f7%16(Preferred)
   IPv4 Address. . . . . . . . . . . : 192.168.1.5(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.100
   DHCPv6 IAID . . . . . . . . . . . : 495201067
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-13-B6-9A-2D-00-1E-C9-E3-37-2

   DNS Servers . . . . . . . . . . . : 192.168.1.6
                                       192.168.1.5
   NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter Local Area Connection* 8:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : isatap.{CC781E30-581E-4948-9DCA-818CA511
774}
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter Local Area Connection* 9:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : isatap.{789ED374-948A-42F1-A907-07E9B675
B89}
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

June 29th, 2015 9:28pm

Depending on how long the issue has been going on (a week or more) then your only option to fix it maybe to point everything to the working DCs and demote the DCs having the issue and then re-promote it. But before you do have you used a port query tool to make sure all the required ports are open? The link below states the required mores needed to be open.

https://technet.microsoft.com/en-us/library/dd772723%28v=ws.10%29.aspx?f=255&MSPPError=-2147217396

Free Windows Admin Tool Kit Click here and download it now
June 30th, 2015 11:15am

I believe this has been happening for years and we are just stumbling on it now.  Were you not able to read the port query results logs I uploaded?
June 30th, 2015 12:06pm

I know this seems tedious but as part of a domain controller or windows admin is port checking, the script below is a powershell script which you can run on the DC having issues, it will ask you for a IP to go to and a port, use the IP of your domain/ forest role holder. Check all the ports if any of them are filtered or blocked the script will tell you.

#Script allows easy port access check.
#

$Ipaddress= Read-Host "Enter the IP address:"
#$UdporTcp = Read-Host "Enter 1 for TCP or 2 for UDP:"
$Port= Read-host "Enter the port number to access:"
#if($UdporTcp -eq 1){
$t = New-Object Net.Sockets.TcpClient
#}
#else{
#$t = New-Object Net.Sockets.udpClient
#}
$t.Connect($Ipaddress,$Port)
    if($t.Connected)
    {
        "Port $Port is operational"
    }
    else
    {
        "Port $Port is closed, You may need to contact your IT team to open it. "
    }


Free Windows Admin Tool Kit Click here and download it now
June 30th, 2015 3:27pm

Use the powershell script in support of the following commands you should have already ran on the DC.

DCDIAG (Any issues/ errors?)

Net share (are the sysvol/ netlogon/ etc. shared out?)

NSLookup and point it to the troubled server (is it resolving DNS?)

gpupdate /force (does group policy get applied without fault?)

If anything fails use the script to check the corresponding port, an example: if NSLookup fails then check port 53

June 30th, 2015 4:24pm

I know this seems tedious but as part of a domain controller or windows admin is port checking, the script below is a powershell script which you can run on the DC having issues, it will ask you for a IP to go to and a port, use the IP of your domain/ forest role holder. Check all the ports if any of them are filtered or blocked the script will tell you.

#Script allows easy port access check.
#

$Ipaddress= Read-Host "Enter the IP address:"
#$UdporTcp = Read-Host "Enter 1 for TCP or 2 for UDP:"
$Port= Read-host "Enter the port number to access:"
#if($UdporTcp -eq 1){
$t = New-Object Net.Sockets.TcpClient
#}
#else{
#$t = New-Object Net.Sockets.udpClient
#}
$t.Connect($Ipaddress,$Port)
    if($t.Connected)
    {
        "Port $Port is operational"
    }
    else
    {
        "Port $Port is closed, You may need to contact your IT team to open it. "
    }


Free Windows Admin Tool Kit Click here and download it now
June 30th, 2015 7:24pm

Port Query results from DC1 to DC2 all ports open and listening.

=============================================

 Starting portqry.exe -n pic-dc2 -e 135 -p TCP ...


Querying target system called:

 pic-dc2

Attempting to resolve name to IP address...

Name resolved to 192.168.1.6

querying...

TCP port 135 (epmap service): LISTENING

Using ephemeral source port
Querying Endpoint Mapper Database...
Server's response:

UUID: f5cc59b4-4264-101a-8c59-08002b2f8426 NtFrs Service
ncacn_ip_tcp:pic-dc2[2138]

UUID: d049b186-814f-11d1-9a3c-00c04fc9b232 NtFrs API
ncacn_ip_tcp:pic-dc2[2138]

UUID: a00c021c-2be2-11d2-b678-0000f87a8f8e PERFMON SERVICE
ncacn_ip_tcp:pic-dc2[2138]

UUID: 2f5f6521-cb55-1059-b446-00df0bce31db Unimodem LRPC Endpoint
ncacn_np:pic-dc2[\\pipe\\tapsrv]

UUID: 3473dd4d-2e88-4006-9cba-22570909dd10 WinHttp Auto-Proxy Service
ncacn_np:pic-dc2[\\PIPE\\W32TIME_ALT]

UUID: 3d267954-eeb7-11d1-b94e-00c04fa3080d 
ncacn_np:pic-dc2[\\pipe\\HydraLsPipe]

UUID: 3d267954-eeb7-11d1-b94e-00c04fa3080d 
ncacn_ip_tcp:pic-dc2[1081]

UUID: 12d4b7c8-77d5-11d1-8c24-00c04fa3080d 
ncacn_np:pic-dc2[\\pipe\\HydraLsPipe]

UUID: 12d4b7c8-77d5-11d1-8c24-00c04fa3080d 
ncacn_ip_tcp:pic-dc2[1081]

UUID: 493c451c-155c-11d3-a314-00c04fb16103 
ncacn_np:pic-dc2[\\pipe\\HydraLsPipe]

UUID: 493c451c-155c-11d3-a314-00c04fb16103 
ncacn_ip_tcp:pic-dc2[1081]

UUID: 45f52c28-7f9f-101a-b52b-08002b2efabe 
ncacn_ip_tcp:pic-dc2[1080]

UUID: 45f52c28-7f9f-101a-b52b-08002b2efabe 
ncacn_np:pic-dc2[\\pipe\\WinsPipe]

UUID: 811109bf-a4e1-11d1-ab54-00a0c91e9b45 
ncacn_ip_tcp:pic-dc2[1080]

UUID: 811109bf-a4e1-11d1-ab54-00a0c91e9b45 
ncacn_np:pic-dc2[\\pipe\\WinsPipe]

UUID: 50abc2a4-574d-40b3-9d66-ee4fd5fba076 
ncacn_ip_tcp:pic-dc2[1045]

UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
ncacn_np:pic-dc2[\\PIPE\\lsass]

UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
ncacn_np:pic-dc2[\\PIPE\\protected_storage]

UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
ncacn_ip_tcp:pic-dc2[1026]

UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
ncacn_http:pic-dc2[1027]

UUID: 12345778-1234-abcd-ef00-0123456789ab 
ncacn_np:pic-dc2[\\PIPE\\lsass]

UUID: 12345778-1234-abcd-ef00-0123456789ab 
ncacn_np:pic-dc2[\\PIPE\\protected_storage]

UUID: 12345778-1234-abcd-ef00-0123456789ab 
ncacn_ip_tcp:pic-dc2[1026]

UUID: 12345778-1234-abcd-ef00-0123456789ab 
ncacn_http:pic-dc2[1027]

UUID: 12345778-1234-abcd-ef00-0123456789ac 
ncacn_np:pic-dc2[\\PIPE\\lsass]

UUID: 12345778-1234-abcd-ef00-0123456789ac 
ncacn_np:pic-dc2[\\PIPE\\protected_storage]

UUID: 12345778-1234-abcd-ef00-0123456789ac 
ncacn_ip_tcp:pic-dc2[1026]

UUID: 12345778-1234-abcd-ef00-0123456789ac 
ncacn_http:pic-dc2[1027]

UUID: ecec0d70-a603-11d0-96b1-00a0c91ece30 NTDS Backup Interface
ncacn_np:pic-dc2[\\PIPE\\lsass]

UUID: ecec0d70-a603-11d0-96b1-00a0c91ece30 NTDS Backup Interface
ncacn_np:pic-dc2[\\PIPE\\protected_storage]

UUID: ecec0d70-a603-11d0-96b1-00a0c91ece30 NTDS Backup Interface
ncacn_ip_tcp:pic-dc2[1026]

UUID: ecec0d70-a603-11d0-96b1-00a0c91ece30 NTDS Backup Interface
ncacn_http:pic-dc2[1027]

UUID: 16e0cf3a-a604-11d0-96b1-00a0c91ece30 NTDS Restore Interface
ncacn_np:pic-dc2[\\PIPE\\lsass]

UUID: 16e0cf3a-a604-11d0-96b1-00a0c91ece30 NTDS Restore Interface
ncacn_np:pic-dc2[\\PIPE\\protected_storage]

UUID: 16e0cf3a-a604-11d0-96b1-00a0c91ece30 NTDS Restore Interface
ncacn_ip_tcp:pic-dc2[1026]

UUID: 16e0cf3a-a604-11d0-96b1-00a0c91ece30 NTDS Restore Interface
ncacn_http:pic-dc2[1027]

UUID: 12345678-1234-abcd-ef00-01234567cffb 
ncacn_np:pic-dc2[\\PIPE\\lsass]

UUID: 12345678-1234-abcd-ef00-01234567cffb 
ncacn_np:pic-dc2[\\PIPE\\protected_storage]

UUID: 12345678-1234-abcd-ef00-01234567cffb 
ncacn_ip_tcp:pic-dc2[1026]

UUID: 12345678-1234-abcd-ef00-01234567cffb 
ncacn_http:pic-dc2[1027]

UUID: 12345678-1234-abcd-ef00-0123456789ab IPSec Policy agent endpoint
ncacn_np:pic-dc2[\\PIPE\\lsass]

UUID: 12345678-1234-abcd-ef00-0123456789ab IPSec Policy agent endpoint
ncacn_np:pic-dc2[\\PIPE\\protected_storage]

UUID: 12345678-1234-abcd-ef00-0123456789ab IPSec Policy agent endpoint
ncacn_ip_tcp:pic-dc2[1026]

UUID: 12345678-1234-abcd-ef00-0123456789ab IPSec Policy agent endpoint
ncacn_http:pic-dc2[1027]

UUID: 1ff70682-0a51-30e8-076d-740be8cee98b 
ncacn_np:pic-dc2[\\PIPE\\atsvc]

UUID: 378e52b0-c0a9-11cf-822d-00aa0051e40f 
ncacn_np:pic-dc2[\\PIPE\\atsvc]

UUID: 0a74ef1c-41a4-4e06-83ae-dc74fb1cdd53 
ncacn_np:pic-dc2[\\PIPE\\atsvc]

UUID: d674a233-5829-49dd-90f0-60cf9ceb7129 ICF+ FW API
ncacn_np:pic-dc2[\\PIPE\\atsvc]

UUID: d674a233-5829-49dd-90f0-60cf9ceb7129 ICF+ FW API
ncacn_np:pic-dc2[\\PIPE\\wkssvc]

UUID: d674a233-5829-49dd-90f0-60cf9ceb7129 ICF+ FW API
ncacn_np:pic-dc2[\\PIPE\\srvsvc]

UUID: d674a233-5829-49dd-90f0-60cf9ceb7129 ICF+ FW API
ncacn_np:pic-dc2[\\pipe\\keysvc]

UUID: d674a233-5829-49dd-90f0-60cf9ceb7129 ICF+ FW API
ncacn_np:pic-dc2[\\PIPE\\browser]

Total endpoints found: 52



==== End of RPC Endpoint Mapper query response ====
portqry.exe -n pic-dc2 -e 135 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n pic-dc2 -e 389 -p BOTH ...


Querying target system called:

 pic-dc2

Attempting to resolve name to IP address...

Name resolved to 192.168.1.6

querying...

TCP port 389 (ldap service): LISTENING

Using ephemeral source port
Sending LDAP query to TCP port 389...

LDAP query response:


currentdate: 06/29/2015 14:59:22 (unadjusted GMT)
subschemaSubentry: CN=Aggregate,CN=Schema,CN=Configuration,DC=pic,DC=local
dsServiceName: CN=NTDS Settings,CN=PIC-DC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=pic,DC=local
namingContexts: DC=pic,DC=local
defaultNamingContext: DC=pic,DC=local
schemaNamingContext: CN=Schema,CN=Configuration,DC=pic,DC=local
configurationNamingContext: CN=Configuration,DC=pic,DC=local
rootDomainNamingContext: DC=pic,DC=local
supportedControl: 1.2.840.113556.1.4.319
supportedLDAPVersion: 3
supportedLDAPPolicies: MaxPoolThreads
highestCommittedUSN: 4617402
supportedSASLMechanisms: GSSAPI
dnsHostName: PIC-DC2.pic.local
ldapServiceName: pic.local:pic-dc2$@PIC.LOCAL
serverName: CN=PIC-DC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=pic,DC=local
supportedCapabilities: 1.2.840.113556.1.4.800
isSynchronized: TRUE
isGlobalCatalogReady: TRUE
domainFunctionality: 2
forestFunctionality: 2
domainControllerFunctionality: 2


======== End of LDAP query response ========

UDP port 389 (unknown service): LISTENING or FILTERED

Using ephemeral source port
Sending LDAP query to UDP port 389...

LDAP query response:


currentdate: 06/29/2015 14:59:26 (unadjusted GMT)
subschemaSubentry: CN=Aggregate,CN=Schema,CN=Configuration,DC=pic,DC=local
dsServiceName: CN=NTDS Settings,CN=PIC-DC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=pic,DC=local
namingContexts: DC=pic,DC=local
defaultNamingContext: DC=pic,DC=local
schemaNamingContext: CN=Schema,CN=Configuration,DC=pic,DC=local
configurationNamingContext: CN=Configuration,DC=pic,DC=local
rootDomainNamingContext: DC=pic,DC=local
supportedControl: 1.2.840.113556.1.4.319
supportedLDAPVersion: 3
supportedLDAPPolicies: MaxPoolThreads
highestCommittedUSN: 4617402
supportedSASLMechanisms: GSSAPI
dnsHostName: PIC-DC2.pic.local
ldapServiceName: pic.local:pic-dc2$@PIC.LOCAL
serverName: CN=PIC-DC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=pic,DC=local
supportedCapabilities: 1.2.840.113556.1.4.800
isSynchronized: TRUE
isGlobalCatalogReady: TRUE
domainFunctionality: 2
forestFunctionality: 2
domainControllerFunctionality: 2


======== End of LDAP query response ========

UDP port 389 is LISTENING

portqry.exe -n pic-dc2 -e 389 -p BOTH exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n pic-dc2 -e 636 -p TCP ...


Querying target system called:

 pic-dc2

Attempting to resolve name to IP address...

Name resolved to 192.168.1.6

querying...

TCP port 636 (ldaps service): LISTENING
portqry.exe -n pic-dc2 -e 636 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n pic-dc2 -e 3268 -p TCP ...


Querying target system called:

 pic-dc2

Attempting to resolve name to IP address...

Name resolved to 192.168.1.6

querying...

TCP port 3268 (msft-gc service): LISTENING

Using ephemeral source port
Sending LDAP query to TCP port 3268...

LDAP query response:


currentdate: 06/29/2015 14:59:26 (unadjusted GMT)
subschemaSubentry: CN=Aggregate,CN=Schema,CN=Configuration,DC=pic,DC=local
dsServiceName: CN=NTDS Settings,CN=PIC-DC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=pic,DC=local
namingContexts: DC=pic,DC=local
defaultNamingContext: DC=pic,DC=local
schemaNamingContext: CN=Schema,CN=Configuration,DC=pic,DC=local
configurationNamingContext: CN=Configuration,DC=pic,DC=local
rootDomainNamingContext: DC=pic,DC=local
supportedControl: 1.2.840.113556.1.4.319
supportedLDAPVersion: 3
supportedLDAPPolicies: MaxPoolThreads
highestCommittedUSN: 4617402
supportedSASLMechanisms: GSSAPI
dnsHostName: PIC-DC2.pic.local
ldapServiceName: pic.local:pic-dc2$@PIC.LOCAL
serverName: CN=PIC-DC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=pic,DC=local
supportedCapabilities: 1.2.840.113556.1.4.800
isSynchronized: TRUE
isGlobalCatalogReady: TRUE
domainFunctionality: 2
forestFunctionality: 2
domainControllerFunctionality: 2


======== End of LDAP query response ========
portqry.exe -n pic-dc2 -e 3268 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n pic-dc2 -e 3269 -p TCP ...


Querying target system called:

 pic-dc2

Attempting to resolve name to IP address...

Name resolved to 192.168.1.6

querying...

TCP port 3269 (msft-gc-ssl service): LISTENING
portqry.exe -n pic-dc2 -e 3269 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n pic-dc2 -e 53 -p BOTH ...


Querying target system called:

 pic-dc2

Attempting to resolve name to IP address...

Name resolved to 192.168.1.6

querying...

TCP port 53 (domain service): LISTENING

UDP port 53 (domain service): LISTENING
portqry.exe -n pic-dc2 -e 53 -p BOTH exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n pic-dc2 -e 88 -p BOTH ...


Querying target system called:

 pic-dc2

Attempting to resolve name to IP address...

Name resolved to 192.168.1.6

querying...

TCP port 88 (kerberos service): LISTENING

UDP port 88 (kerberos service): LISTENING or FILTERED
portqry.exe -n pic-dc2 -e 88 -p BOTH exits with return code 0x00000002.
=============================================

 Starting portqry.exe -n pic-dc2 -e 445 -p TCP ...


Querying target system called:

 pic-dc2

Attempting to resolve name to IP address...

Name resolved to 192.168.1.6

querying...

TCP port 445 (microsoft-ds service): LISTENING
portqry.exe -n pic-dc2 -e 445 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n pic-dc2 -e 137 -p UDP ...


Querying target system called:

 pic-dc2

Attempting to resolve name to IP address...


Name resolved to 192.168.1.6

querying...

UDP port 137 (netbios-ns service): LISTENING or FILTERED

Using ephemeral source port
Attempting NETBIOS adapter status query to UDP port 137...

Server's response: MAC address 001b212142dc
UDP port: LISTENING
portqry.exe -n pic-dc2 -e 137 -p UDP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n pic-dc2 -e 138 -p UDP ...


Querying target system called:

 pic-dc2

Attempting to resolve name to IP address...


Name resolved to 192.168.1.6

querying...

UDP port 138 (netbios-dgm service): LISTENING or FILTERED
portqry.exe -n pic-dc2 -e 138 -p UDP exits with return code 0x00000002.
=============================================

 Starting portqry.exe -n pic-dc2 -e 139 -p TCP ...


Querying target system called:

 pic-dc2

Attempting to resolve name to IP address...

Name resolved to 192.168.1.6

querying...

TCP port 139 (netbios-ssn service): LISTENING
portqry.exe -n pic-dc2 -e 139 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n pic-dc2 -e 42 -p TCP ...


Querying target system called:

 pic-dc2

Attempting to resolve name to IP address...

Name resolved to 192.168.1.6

querying...

TCP port 42 (nameserver service): LISTENING
portqry.exe -n pic-dc2 -e 42 -p TCP exits with return code 0x00000000.

June 30th, 2015 10:12pm

Hi,

dcdiag /v /c /d /e /s:FQDN

I cannot get the output you shared. Can you make it on onedrive and provide me with URL.

Free Windows Admin Tool Kit Click here and download it now
June 30th, 2015 11:39pm

Is port TCP 5722 open? I assume DCDIAG and NET SHARE commands came back as passing for DC3, if so did you compare the registry keys with DC1? Did you use the following links below for troubleshooting?

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NtFrs\

https://technet.microsoft.com/en-us/library/cc816833%28v=ws.10%29.aspx?f=255&MSPPError=-2147217396

https://support.microsoft.com/en-us/kb/257338

June 30th, 2015 11:54pm

Here you go

https://onedrive.live.com/redir?resid=A750400A0F35241B%21143

Free Windows Admin Tool Kit Click here and download it now
July 1st, 2015 10:17am

5722 is not listening on any DC"s

DC3 is a brand new DC and it is also not replicating properly.  Same errors as DC1.

July 1st, 2015 10:26am

TCP port 5722 is the port AD uses to replicate the sysvol, if this is blocked then yes you have no sysvol replication. If you have a Network/ WAN team send them the link below, tell them these ports need to be open between the AD domain controllers in order to make them communicate properly.

https://technet.microsoft.com/en-us/library/dd772723%28v=ws.10%29.aspx?f=255&MSPPError=-2147217396

 
Free Windows Admin Tool Kit Click here and download it now
July 1st, 2015 10:35am

There's no communication problem between the DC's.  The port is simply not listening.

I performed a netstat and did not see the port.

July 1st, 2015 10:40am

Not listening means its blocked, it needs to listen
... 5722 is the port for sysvol, you can still do repadmin and so forth and not get issues but if 5722 is blocked/ not listening then you will get sysvol issues.
Free Windows Admin Tool Kit Click here and download it now
July 1st, 2015 10:42am

July 1st, 2015 10:45am

As you can see, the port is not open.
Free Windows Admin Tool Kit Click here and download it now
July 1st, 2015 10:45am

If you want a properly functioning and self efficient DC/ GC environment then the following ports need to be open/ listening between the DC's. Whoever does your switch and router configuration needs to setup the routes and vlans to allow this for the DC's/ GC's

TCP and UDP 389
TCP 3268
TCP 636
TCP 3269
TCP and UDP 53
TCP and UDP 88
TCP 135
TCP Dynamic (port is 49152, and the new default end port is 65535 for vista/2008 and up)  port range is 1025 through 5000 for 2003 and below.
TCP 5722
UDP 123
TCP and UDP 464
UDP 138
UDP Dynamic
UDP 137
TCP 139

July 1st, 2015 10:52am

All DC's are are on the same physical network in the same building.  There are no routers, firewalls and other switches between them

If the port is not open then there is something wrong on the DC itself.


Free Windows Admin Tool Kit Click here and download it now
July 1st, 2015 10:56am

There is a router and a switch... If they are all physical devices then go into the data center where they reside and trace the cable from the back of the DC to the next device it plugs into, that would be the switch, from the switch I bet the router would be next and is often the IP for the Gateway listed in the NIC settings. If it is virtual then go into the hypervisor and look at the virtual switch settings and go from there.
July 1st, 2015 11:04am

I'm not sure what you are not understanding.

There is NO process on the DC that has that port open or listening.

NONE. ZERO. ZILCH.

Free Windows Admin Tool Kit Click here and download it now
July 1st, 2015 11:08am

this is outside the DC past the NICs down the cabling.... it is a device the governs IP Routing often a Cisco device maybe a netgear device, these devices have the ability to explicitly block IP ranges and ports/ protocols... It is one of these devices that is blocking your DC communication.

July 1st, 2015 11:14am

This is a really simple environment.  All's DC's connected to the same unmanaged switch.

I have not found evidence of this port listening in any other client sites with multiple domain controllers.

This problem started after DC1 had a problem and we had to rebuild it.  The DC was cleaned up form the metadata and re-promoted as a DC.

Free Windows Admin Tool Kit Click here and download it now
July 1st, 2015 11:36am

Hi,

Can you please upload the output of following to common location onedrive.

dcdiag /v /c /d /e /s:FQDN

Also possible can you Modify the TCPIP property of your both the DC settings as DNS IP should be the IP of local server first then as alternate DNS you can specify the second Dc IP address.Windows IP Configuration
    Host Name . . . . . . . . . . . . : PIC-DC2
    IP Address. . . . . . . . . . . . : 192.168.1.6
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.100
   DNS Servers . . . . . . . . . . . : 192.168.1.6 & 192.168.1.5
                                        
------------------------------------------------------------------------------------
   Host Name . . . . . . . . . . . . : PIC-DC1
    
    IPv4 Address. . . . . . . . . . . : 192.168.1.5(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.100
    DNS Servers . . . . . . . . . . . : 192.168.1.5 & then 192.168.1.6
    NetBIOS over Tcpip. . . . . . . . : Enabled

July 1st, 2015 11:46am

If all windows firewalls are disabled and if all DC's use the same gateway "192.168.1.100" and the gateway is 100% not the problem and as such the servers and network devices are not the problems then.....

Do you have a HIPS installed on the DC's?

An IPSEC policy applied somewhere?

When you took down DC1 and rebuilt it did you have to do a metadata cleanup? Was it holding the FSMO roles for your domain at the time? If so did you sieze the roles from another DC before tearing it down? In ADUC under the following path:

Domain/System/File Replication Service/Domain System Volume (SYSVOL share)

Do you see all your DC's?

Does your net share command look like the picture below?

Free Windows Admin Tool Kit Click here and download it now
July 1st, 2015 11:49am

I switched the DNS servers around yesterday to see if it would make any kind of difference.  I will put them back, but it will not help.  The DCDIAG results are in the one drive folder.
July 1st, 2015 11:52am

Yes, seized the roles, performed a clean up, and repromoted.

Transferred the roles back to dc1.  I think at this point the sysvol replication never happened.

The SYSVOL and NETLOGON shares are only present on DC2.

Free Windows Admin Tool Kit Click here and download it now
July 1st, 2015 12:00pm

Hi,

Can you provide me the link for DCDIAG Log files.

July 1st, 2015 12:09pm

http://1drv.ms/1LDm4ZJ

http://1drv.ms/1T6fOMg

Free Windows Admin Tool Kit Click here and download it now
July 1st, 2015 12:12pm

Hi, Can you check and see FRS service is configured with local system Account and not any other service account.
July 1st, 2015 12:48pm

All DC's set to local
Free Windows Admin Tool Kit Click here and download it now
July 1st, 2015 12:52pm

Also you can try this

https://support.microsoft.com/en-us/kb/315457

On each domain controller in the domain, follow these steps: 

  1. Click Start, click Run, type cmd, and then click OK.
  2. Type net start ntfrs to start the File Replication service.
  3. Type ntfrsutl ds |findstr /i "root stage", and then press ENTER. The NTFRSUTIL command returns the current root directory for the SYSVOL replica set that is referred to as the replica set root and the staging folder. For example, this command returns:    

Root: C:\WINNT\SYSVOL\domain
Stage: C:\WINNT\SYSVOL\staging\domain

4. Type Linkd %systemroot%\SYSVOL\SYSVOL\ DNS Domain name, and then press ENTER. The LINKD command returns the following:    

Source DNS Domain Name is linked to %systemroot%\SYSVOL\domain

5. Type linkd "%systemroot%\SYSVOL\staging areas\DNS Domain Name", and then press ENTER. This command returns the following:    

Source DNS Domain Name is linked to %systemroot%\SYSVOL\Staging\domain
  

July 1st, 2015 12:53pm

So it seems that data is missing from SYSVOL folders on DC2

Under domain\ this directory is empty

Under sysvol\sysvol the pic.local looks like a folder not a shortcut

Also on DC1 I'm getting this error

The Netlogon service could not create server share C:\Windows\SYSVOL\sysvol\pic.local\SCRIPTS.  The following error occurred: 
The system cannot find the file specified.

Free Windows Admin Tool Kit Click here and download it now
July 1st, 2015 3:46pm

Once I moved the correct data into the sysvol\domain directory and restored the junction point in sysvol\sysvol I restarted FRS and NETLOGON services and replication resumed as normal.

I had not paid attention to this error on DC2

 

Event Type: Error
Event Source: NtFrs
Event Category: None
Event ID: 13559
Date: 6/30/2015
Time: 4:54:31 PM
User: N/A
Computer: PIC-DC2
Description:
The File Replication Service has detected that the replica root path has changed from "c:\windows\sysvol\domain" to "c:\windows\sysvol\domain". If this is an intentional move then a file with the name NTFRS_CMD_FILE_MOVE_ROOT needs to be created under the new root path. 
This was detected for the following replica set: 
    "DOMAIN SYSTEM VOLUME (SYSVOL SHARE)" 

  Also this article helped:

http://searchwindowsserver.techtarget.com/tip/How-to-rebuild-the-SYSVOL-tree-when-none-exists-in-Active-Directory

Thank you for leading me in the right direction Purvesh



July 1st, 2015 4:32pm

Hi,

Thanks for your updates and feedback. I am glade that your issue is resolved. Please mark this as Answered so that others can refer it in case of similar issue.

Free Windows Admin Tool Kit Click here and download it now
July 2nd, 2015 12:17am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics