Problem with MSDTC and duplicate CID values - which aren't.
Hi, I have a problem with msdtc and CID values. We use MSDTC to run distributed queries across SQL servers.
Occaisionally we come accross a problem when setting up new servers wherewhen trying to run a distributed transaction we get the response back 'server x.x.x.x is not configured for rpc'. When we check it is and when digging further with DTCPING we
get the error that the CID values are the same.
This is usually caused by servers being built from images and is usually rectified by either manually altering the CID values in the registry or following the process of removing MSDTC, rebooting and re-installing MSDTC again.
However we have been increasingly getting this problem where imaging has not taken place and the problem is not rectified by removal of MSDTC or changing CID values in the registry.
So although the CID values involved are clearly not the same MSDTC reports them as being the same.
The setup is as follows, a 'live' pair of SQL servers are clustered using Steeleyes data replication and SQL protection suite, a 'reporting' SQL server pulls information from the 'live' member of the pair using MSDTC.
the logs of DTCPING are as follows:
reporting server that initiates the DT:
Platform:Windows 2003
Security settings for DTC under Windows 2003
NetworkDtcAccess :true
NetworkDtcAccessAdmin :true
NetworkDtcAccessClients :true
NetworkDtcAccessTransactions:true
NetworkDtcAccessTip :false
XaTransactions :false
DomainControllerState :false
AccountName :NT Authority\NetworkService
TurnOffRpcSecurity :true
IP Configure Information
Host Name . . . . . . . . . : digitalk_sr_485
DNS Servers . . . . . . . . :
Node Type . . . . . . . . . :
NetBIOS Scope ID. . . . . . :
IP Routing Enabled. . . . . : no
WINS Proxy Enabled. . . . . : no
NetBIOS Resolution Uses DNS : no
Ethernet adapter {2A6C2B9D-5BF4-4E67-A12A-FC9BE10F94E5}:
Description . . . . . . . . : HP Network Team #1
Physical Address. . . . . . : 00-21-5A-5D-F4-64
DHCP Enabled. . . . . . . . : no
IP Address. . . . . . . . . : 10.40.110.160
Subnet Mask . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . : 10.40.110.1
DHCP Server . . . . . . . . : 255.255.255.255
Primary WINS Server . . . . : 0.0.0.0
Secondary WINS Server . . . : 0.0.0.0
Lease Obtained. . . . . . . : Thu Jan 01 00:00:00 1970
Lease Expires . . . . . . . : Thu Jan 01 00:00:00 1970
++++++++++++lmhosts.sam++++++++++++
++++++++++++hosts ++++++++++++
127.0.0.1 localhost
10.40.110.151 digitalk-sr-483
10.40.110.152 digitalk-sr-484
10.40.110.101 digitalk-sr-407
10.40.110.102 digitalk-sr-408
++++++++++++++++++++++++++++++++++++++++++++++
DTCping 1.8 Report for
++++++++++++++++++++++++++++++++++++++++++++++
RPC server is ready
++++++++++++Validating Remote Computer Name++++++++++++
10-07, 12:25:43.171-->Start DTC connection test
Name Resolution:
digitalk-sr-407-->10.40.110.101-->digitalk-sr-407
10-07, 12:25:43.187-->Start RPC test (-->digitalk-sr-407)
RPC test is successful
Partner's CID:71DEBDAB-30EB-4B0B-AA98-53DD110AFE07
++++++++++++RPC test completed+++++++++++++++
++++++++++++Start DTC Binding Test +++++++++++++
Trying Bind to digitalk-sr-407
10-07, 12:25:43.281--> Initiating DTC Binding Test....
Test Guid:94EF4A05-E187-4534-9EEF-C96D7A7924B4
Binding success: -->digitalk-sr-407
++++++++++++DTC Binding Test END+++++++++++++
'Live' server that returns the data to the reporting server:
Platform:Windows 2003
Security settings for DTC under Windows 2003
NetworkDtcAccess :true
NetworkDtcAccessAdmin :true
NetworkDtcAccessClients :true
NetworkDtcAccessTransactions:true
NetworkDtcAccessTip :false
XaTransactions :false
DomainControllerState :false
AccountName :NT Authority\NetworkService
TurnOffRpcSecurity :true
IP Configure Information
Host Name . . . . . . . . . : digitalk-sr-407
DNS Servers . . . . . . . . : 208.67.222.222
208.67.220.220
Node Type . . . . . . . . . :
NetBIOS Scope ID. . . . . . :
IP Routing Enabled. . . . . : no
WINS Proxy Enabled. . . . . : no
NetBIOS Resolution Uses DNS : no
Ethernet adapter {73C0FD75-8264-4376-A0FB-BA95983ADB24}:
Description . . . . . . . . : HP Network Team #1
Physical Address. . . . . . : 00-18-FE-FB-DE-38
DHCP Enabled. . . . . . . . : no
IP Address. . . . . . . . . : 10.40.110.100
Subnet Mask . . . . . . . . : 255.255.255.0
IP Address. . . . . . . . . : 10.40.110.101
Subnet Mask . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . : 10.40.110.1
DHCP Server . . . . . . . . : 255.255.255.255
Primary WINS Server . . . . : 0.0.0.0
Secondary WINS Server . . . : 0.0.0.0
Lease Obtained. . . . . . . : Thu Jan 01 00:00:00 1970
Lease Expires . . . . . . . : Thu Jan 01 00:00:00 1970
Ethernet adapter {0E8BF709-586A-4FEC-9091-AABAA46440E2}:
Description . . . . . . . . : HP NC7771 Gigabit Server Adapter #2
Physical Address. . . . . . : 00-17-08-2B-EA-5B
DHCP Enabled. . . . . . . . : no
IP Address. . . . . . . . . : 100.100.100.1
Subnet Mask . . . . . . . . : 255.0.0.0
Default Gateway . . . . . . :
DHCP Server . . . . . . . . : 255.255.255.255
Primary WINS Server . . . . : 0.0.0.0
Secondary WINS Server . . . : 0.0.0.0
Lease Obtained. . . . . . . : Thu Jan 01 00:00:00 1970
Lease Expires . . . . . . . : Thu Jan 01 00:00:00 1970
++++++++++++lmhosts.sam++++++++++++
++++++++++++hosts ++++++++++++
127.0.0.1 localhost
10.40.110.101 digitalk-sr-407
10.40.110.102 digitalk-sr-408
10.40.110.160 digitalk_sr_485
++++++++++++++++++++++++++++++++++++++++++++++
DTCping 1.8 Report for
++++++++++++++++++++++++++++++++++++++++++++++
RPC server is ready
10-07, 12:25:17.485-->RPC server: received following information:
Network Name: digitalk-sr-407
Source Port: 1442
Partner LOG: 3780.log
Partner CID: 94EF4A05-E187-4534-9EEF-C96D7A7924B4
++++++++++++Start Reverse Bind Test+++++++++++++
Received Bind call from
Network Name: digitalk-sr-407
Source Port: 1442
Hosting Machine:
10-07, 12:25:17.673-->Trying to Reverse Bind to ...
Test Guid:94EF4A05-E187-4534-9EEF-C96D7A7924B4
Name Resolution:
-->10.40.110.101-->digitalk-sr-407
Received reverse bind call from
Network Name: digitalk-sr-407
Source Port: 1442
Hosting Machine:
Reverse Binding success: -->
++++++++++++Reverse Bind Test ENDED++++++++++
10-07, 12:25:17.673-->Called POKE from Partner:
Network Name: digitalk-sr-407
Source Port: 1442
Hosting Machine:
++++++++++++Validating Remote Computer Name++++++++++++
10-07, 12:25:24.782-->Start DTC connection test
Name Resolution:
digitalk_sr_485-->10.40.110.160-->digitalk_sr_485
10-07, 12:25:24.782-->Start RPC test (-->digitalk_sr_485)
WARNING:the CID values for both test machines are the same
while this problem won't stop DTCping test, MSDTC will fail for this
10-07, 12:25:24.782-->RPC server: received following information:
Network Name: digitalk-sr-407
Source Port: 1442
Partner LOG: 3404.log
Partner CID: 71DEBDAB-30EB-4B0B-AA98-53DD110AFE07
RPC test is successful
Partner's CID:71DEBDAB-30EB-4B0B-AA98-53DD110AFE07
++++++++++++RPC test completed+++++++++++++++
++++++++++++Start DTC Binding Test +++++++++++++
Trying Bind to
10-07, 12:25:24.782--> Initiating DTC Binding Test....
Test Guid:71DEBDAB-30EB-4B0B-AA98-53DD110AFE07
++++++++++++Start Reverse Bind Test+++++++++++++
Received Bind call from
Network Name: digitalk-sr-407
Source Port: 1442
Hosting Machine:
10-07, 12:25:24.782-->Trying to Reverse Bind to ...
Test Guid:71DEBDAB-30EB-4B0B-AA98-53DD110AFE07
Name Resolution:
-->10.40.110.101-->digitalk-sr-407
Received reverse bind call from
Network Name: digitalk-sr-407
Source Port: 1442
Hosting Machine:
Reverse Binding success: -->
++++++++++++Reverse Bind Test ENDED++++++++++
Binding success: -->
++++++++++++DTC Binding Test END+++++++++++++
As you can see the 'Live' server reports that the CID is the same on both machines and so MSDTC will fail, we have completed the following steps on both servers and the result is the same.
1)Manually changed the CID concerned on both servers and rebooted both servers afterwards
2)Manually removed network DTC, uninstalled MSDTC (MSDTC -uninstall), deleted the related registry keys (hklm/software/microsoft/msdtc, HKLM/system/currentcontrolset/services/MSDTC and hkey_classes_root\CID
(when present after the uninstall))
Rebooted the server, re-installed MSDTC, re-added network DTC, reconfigured dcomcnfg.exe to allow DTC and restarted MSDTC.
And still we get the same error message when using DTCPING and distributed transactions cannot take place.
As you can see from the DTCPING logs the CID values are not the same on both servers, which is very puzzling is there another issue that maybe causing the 'live' server to get its CID value mixed up?
Thanks
James
October 7th, 2010 9:12am
Just to add the 'Live' cluster pair have two NIC cards, one acts as dedicated link between the two 'Live' servers for clustering the volume, the other NIC is for all other network activity.
Is it possible that it mirrors back the response to DTCPING to itself and so gets confused by seeing its own CID again?
Free Windows Admin Tool Kit Click here and download it now
October 7th, 2010 9:17am