Sql server failover clustering can not failover to another node suddenly (Network Steve Forum)

Sql server failover clustering can not failover to another node suddenly

I have set up two ACTIVE-ACTIVE sql server failover clusters using two physical win2008R2 servers (four sql instances, both servers have an active instance, and also another standby instance).

It's online for at least two years.

It all went well until suddenly someday one cluster failover to another server, that means two sql server instances are running on one box.
when i try to failover back manually, it always ended up failed. i try many times, no luck.

The windows event log during my manual failover operation which tell the operation failed.

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

- <System>
<Provider Name="Microsoft-Windows-FailoverClustering" Guid="{BAF908EA-3421-4CA9-9B84-6689B8C6F85F}" />
<EventID>1069</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>3</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime="2013-08-05T15:29:38.026463100Z" />
<EventRecordID>124624</EventRecordID>
<Correlation />
<Execution ProcessID="3004" ThreadID="9112" />
<Channel>System</Channel>
<Computer>DB-CLUSTER1-1.mycompany.com</Computer>
<Security UserID="S-1-5-18" />
</System>
- <EventData>
<Data Name="ResourceName">SQL Server (SQLCLUSTER02)</Data>
<Data Name="ResourceGroup">SQL Server (SQLCLUSTER02)</Data>
</EventData>
</Event>

i dont know how to troubleshooting this failover issue. Any help would be appreciated~

updated:

here 's the event log when all this started:

Could not register Service Control Handler. Operating system error = 2310 (Shared resource not exists)

August 5th, 2013 6:36pm

Hi,

Do you have a cluster log? Try looking in C:\Windows\Cluster. If you don't have one, in DOS run "cluster log /gen", which will create one. Then when you next failover, check the log for errors.

Anything in the SQL Server errorlog?

Free Windows Admin Tool Kit Click here and download it now

August 5th, 2013 6:52pm

Hi andrew, thanks for your quick reply.

Here's part of cluster.log (from the time i start the test just now):

It seems that might be some disk problem, but i dont know for sure.

also, i update my original post, add the event log when this happened in 20 June, 2013.

000009fc.00002bb0::2013/08/05-15:23:50.213 INFO [NM] Received request from client address fe80::3d48:f03d:c782:380b.
000009fc.000030d8::2013/08/05-15:23:50.275 INFO [NM] Received request from client address fe80::3d48:f03d:c782:380b.
000009fc.000030d8::2013/08/05-15:23:50.547 INFO [NM] Received request from client address fe80::3d48:f03d:c782:380b.
000009fc.00002f3c::2013/08/05-15:23:50.663 INFO [API] s_ApiGetQuorumResource final status 0.
00001438.00003ce0::2013/08/05-15:23:50.812 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
000009fc.00003980::2013/08/05-15:23:50.812 WARN [RCM] Failed to load restype 'MSMQ': error 21.
000009fc.00003980::2013/08/05-15:23:50.812 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00001438.00003ce0::2013/08/05-15:23:50.815 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
000009fc.00002f3c::2013/08/05-15:23:50.816 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00001438.00003ce0::2013/08/05-15:23:50.922 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
000009fc.00003980::2013/08/05-15:23:50.922 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
000009fc.00003980::2013/08/05-15:23:50.922 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00001438.00003ce0::2013/08/05-15:23:50.924 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
000009fc.00002f3c::2013/08/05-15:23:50.924 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00001438.00003ce0::2013/08/05-15:23:51.208 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
000009fc.00003980::2013/08/05-15:23:51.208 WARN [RCM] Failed to load restype 'MSMQ': error 21.
000009fc.00003980::2013/08/05-15:23:51.208 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00001438.00003ce0::2013/08/05-15:23:51.210 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
000009fc.00002f3c::2013/08/05-15:23:51.210 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00001438.00003ce0::2013/08/05-15:23:51.214 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
000009fc.00003980::2013/08/05-15:23:51.214 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
000009fc.00003980::2013/08/05-15:23:51.214 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00001438.00003ce0::2013/08/05-15:23:51.216 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
000009fc.00002f3c::2013/08/05-15:23:51.216 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00001438.00003ce0::2013/08/05-15:23:55.765 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
000009fc.00005398::2013/08/05-15:23:55.765 WARN [RCM] Failed to load restype 'MSMQ': error 21.
000009fc.00005398::2013/08/05-15:23:55.765 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00001438.00003ce0::2013/08/05-15:23:55.767 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
000009fc.00003980::2013/08/05-15:23:55.767 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00001438.00003ce0::2013/08/05-15:23:55.772 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
000009fc.00005398::2013/08/05-15:23:55.772 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
000009fc.00005398::2013/08/05-15:23:55.772 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00001438.00003ce0::2013/08/05-15:23:55.774 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
000009fc.00003980::2013/08/05-15:23:55.774 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
000009fc.00003980::2013/08/05-15:24:29.535 INFO [RCM] rcm::RcmApi::MoveGroup: (SQL Server (SQLCLUSTER02), 2)
000009fc.000030d8::2013/08/05-15:24:29.614 INFO [NM] Received request from client address fe80::3d48:f03d:c782:380b.
000009fc.000030d8::2013/08/05-15:28:35.254 INFO [NM] Received request from client address fe80::3d48:f03d:c782:380b.
000009fc.000030d8::2013/08/05-15:30:12.510 INFO [NM] Received request from client address fe80::3d48:f03d:c782:380b.
000009fc.000030d8::2013/08/05-15:31:54.179 INFO [NM] Received request from client address fe80::3d48:f03d:c782:380b.
000009fc.000030d8::2013/08/05-15:32:39.333 INFO [NM] Received request from client address fe80::3d48:f03d:c782:380b.
000009fc.000030d8::2013/08/05-15:34:50.371 INFO [NM] Received request from client address fe80::3d48:f03d:c782:380b.
000009fc.00000b90::2013/08/05-15:34:50.381 INFO [RCM] rcm::RcmGum::GroupMoveOperation(SQL Server (SQLCLUSTER02),2)
000009fc.000030d8::2013/08/05-15:34:50.389 INFO [RCM] TransitionToState(SqlClusterDriver2) Offline-->OnlineCallIssued.
000009fc.000030d8::2013/08/05-15:34:50.397 INFO [RCM] rcm::RcmGroup::UpdateStateIfChanged: (SQL Server (SQLCLUSTER02), BringingToPersistentState --> Pending)
000009fc.000030d8::2013/08/05-15:34:50.398 INFO [RCM] rcm::RcmResource::Online: bringing SQL Network Name (SqlCluster02)'s provider resource 'SQL IP Address 1 (SqlCluster02)' online.
000009fc.000030d8::2013/08/05-15:34:50.398 INFO [RCM] TransitionToState(SQL IP Address 1 (SqlCluster02)) Offline-->OnlineCallIssued.
000009fc.000030d8::2013/08/05-15:34:50.398 INFO [RCM] TransitionToState(SQL Network Name (SqlCluster02)) Offline-->WaitingToComeOnline.
000009fc.000030d8::2013/08/05-15:34:50.398 INFO [RCM] rcm::RcmResource::Online: FileServer-(SqlCluster02)(SqlClusterDriver2)'s provider 'SQL Network Name (SqlCluster02)' is already coming online (state is WaitingToComeOnline).
000009fc.000030d8::2013/08/05-15:34:50.398 INFO [RCM] rcm::RcmResource::Online: FileServer-(SqlCluster02)(SqlClusterDriver2)'s provider 'SQL IP Address 1 (SqlCluster02)' is already coming online (state is OnlineCallIssued).
000009fc.000030d8::2013/08/05-15:34:50.398 INFO [RCM] rcm::RcmResource::Online: FileServer-(SqlCluster02)(SqlClusterDriver2)'s provider 'SqlClusterDriver2' is already coming online (state is OnlineCallIssued).
000009fc.000030d8::2013/08/05-15:34:50.398 INFO [RCM] TransitionToState(FileServer-(SqlCluster02)(SqlClusterDriver2)) Offline-->WaitingToComeOnline.
000009fc.000030d8::2013/08/05-15:34:50.399 INFO [RCM] rcm::RcmResource::Online: bringing SQL Server Agent (SQLCLUSTER02)'s provider resource 'SQL Server (SQLCLUSTER02)' online.
000009fc.000030d8::2013/08/05-15:34:50.399 INFO [RCM] rcm::RcmResource::Online: SQL Server Agent (SQLCLUSTER02)'s provider 'SQL Network Name (SqlCluster02)' is already coming online (state is WaitingToComeOnline).
000009fc.000030d8::2013/08/05-15:34:50.399 INFO [RCM] rcm::RcmResource::Online: SQL Server Agent (SQLCLUSTER02)'s provider 'SQL IP Address 1 (SqlCluster02)' is already coming online (state is OnlineCallIssued).
000009fc.000030d8::2013/08/05-15:34:50.399 INFO [RCM] rcm::RcmResource::Online: SQL Server Agent (SQLCLUSTER02)'s provider 'SqlClusterDriver2' is already coming online (state is OnlineCallIssued).
000009fc.000030d8::2013/08/05-15:34:50.399 INFO [RCM] TransitionToState(SQL Server (SQLCLUSTER02)) Offline-->WaitingToComeOnline.
000009fc.000030d8::2013/08/05-15:34:50.399 INFO [RCM] TransitionToState(SQL Server Agent (SQLCLUSTER02)) OfflineDueToProvider-->WaitingToComeOnline.
00001468.00000c20::2013/08/05-15:34:50.399 INFO [RES] IP Address <SQL IP Address 1 (SqlCluster02)>: Bringing resource online...
00001468.00002a90::2013/08/05-15:34:50.399 INFO [RES] Physical Disk <SqlClusterDriver2>: Online request.
00001468.00005988::2013/08/05-15:34:50.399 INFO [RES] IP Address <SQL IP Address 1 (SqlCluster02)>: Online thread running.
000009fc.000030d8::2013/08/05-15:34:50.399 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'SqlClusterDriver2', gen(0) result 997.
000009fc.000030d8::2013/08/05-15:34:50.399 INFO [RCM] TransitionToState(SqlClusterDriver2) OnlineCallIssued-->OnlinePending.
000009fc.00000788::2013/08/05-15:34:50.399 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL IP Address 1 (SqlCluster02)', gen(0) result 997.
000009fc.00000788::2013/08/05-15:34:50.399 INFO [RCM] TransitionToState(SQL IP Address 1 (SqlCluster02)) OnlineCallIssued-->OnlinePending.
00001468.00003798::2013/08/05-15:34:50.407 INFO [RES] Physical Disk <SqlClusterDriver2>: Arbitrate request: FastPath 0
00001468.00005988::2013/08/05-15:34:50.408 INFO [RES] IP Address <SQL IP Address 1 (SqlCluster02)>: Checking for network match: network masks 00FFFFFF=00FFFFFF and addresses 36C8C80A^00C8C80A, role 3.
00001468.00003798::2013/08/05-15:34:50.409 INFO [RES] Physical Disk: Enter EnumerateDevices: EnumDevice 0
00001468.00005988::2013/08/05-15:34:50.410 INFO [RES] IP Address <SQL IP Address 1 (SqlCluster02)>: Online: Opened object handle for netinterface 20a7b195-48d4-49ec-99b4-7b75b8ab6b25.
00001468.00005988::2013/08/05-15:34:50.410 INFO [RES] IP Address <SQL IP Address 1 (SqlCluster02)>: Online: Registered notification for netinterface 20a7b195-48d4-49ec-99b4-7b75b8ab6b25.
00001468.00003798::2013/08/05-15:34:50.413 INFO [RES] Physical Disk: Exit EnumerateDevices: status 0
00001468.00005988::2013/08/05-15:34:50.417 INFO [ClNet] Adapter * 2 RFC2863 operational status = 1.
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] Created adapter: DeviceGuid: ED3DA250-4D37-402F-8DE6-8D7FD2ADF6A0
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] DeviceName: Microsoft Failover Cluster Virtual Adapter
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] ConnectoidName: * 2
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] Netbios/TCP: 1
00001468.00005988::2013/08/05-15:34:50.417 INFO [ClNet] Adapter Team 1 RFC2863 operational status = 1.
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] Created adapter: DeviceGuid: 6A92A5D0-8210-481D-8E29-4B39F5E64259
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] DeviceName: BASP Virtual Adapter
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] ConnectoidName: Team 1
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] Netbios/TCP: 1
00001468.00005988::2013/08/05-15:34:50.417 INFO [ClNet] Adapter 3 RFC2863 operational status = 1.
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] Created adapter: DeviceGuid: CAE5B831-79BF-4B07-8B83-CC875CA5159F
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] DeviceName: IBM USB Remote NDIS Network Device
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] ConnectoidName: 3
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] Netbios/TCP: 1
00001468.00005988::2013/08/05-15:34:50.417 INFO [ClNet] Adapter isatap.{6A92A5D0-8210-481D-8E29-4B39F5E64259} RFC2863 operational status = 2.
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] Created adapter: DeviceGuid: 9637BD41-35CE-4D1B-AB16-B75C686B3358
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] DeviceName: Microsoft ISATAP Adapter
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] ConnectoidName: isatap.{6A92A5D0-8210-481D-8E29-4B39F5E64259}
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] Netbios/TCP: 0
00001468.00005988::2013/08/05-15:34:50.417 INFO [ClNet] Adapter isatap.{CAE5B831-79BF-4B07-8B83-CC875CA5159F} RFC2863 operational status = 2.
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] Created adapter: DeviceGuid: 6E2EBE4C-E50F-4138-8495-74B407C92E45
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] DeviceName: Microsoft ISATAP Adapter #2
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] ConnectoidName: isatap.{CAE5B831-79BF-4B07-8B83-CC875CA5159F}
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] Netbios/TCP: 0
00001468.00005988::2013/08/05-15:34:50.417 INFO [ClNet] Adapter isatap.{ED3DA250-4D37-402F-8DE6-8D7FD2ADF6A0} RFC2863 operational status = 2.
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] Created adapter: DeviceGuid: 828C5BD8-A0DA-45CA-8611-C911E55F0C67
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] DeviceName: Microsoft ISATAP Adapter #3
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] ConnectoidName: isatap.{ED3DA250-4D37-402F-8DE6-8D7FD2ADF6A0}
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] Netbios/TCP: 0
00001468.00005988::2013/08/05-15:34:50.417 INFO [ClNet] Adapter Teredo Tunneling Pseudo-Interface RFC2863 operational status = 2.
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] Created adapter: DeviceGuid: A4595AD2-CE71-4DD2-8C34-E89D0E1B269A
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] DeviceName: Teredo Tunneling Pseudo-Interface
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] ConnectoidName: Teredo Tunneling Pseudo-Interface
00001468.00005988::2013/08/05-15:34:50.417 DBG [ClNet] Netbios/TCP: 0
000009fc.00000788::2013/08/05-15:34:50.417 INFO [GUM] Node 2: Processing RequestLock 2:329329
000009fc.000030d8::2013/08/05-15:34:50.417 DBG [NETFTAPI] 10.200.200.54 NsiAddInstance
000009fc.00000834::2013/08/05-15:34:50.424 INFO [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 2250372)
00001468.00005988::2013/08/05-15:34:50.473 INFO [RES] IP Address <SQL IP Address 1 (SqlCluster02)>: Created IP interface 36C8C80A.
00001468.00003798::2013/08/05-15:34:51.002 INFO [RES] Physical Disk: Successful reserve, disk is unowned, key 4ca87466734d
00001468.00003798::2013/08/05-15:34:51.002 INFO [RES] Physical Disk: Update disk props returns 0
00001468.00003798::2013/08/05-15:34:51.007 INFO [RES] Physical Disk <SqlClusterDriver2>: Disk is offline
00001468.00002be8::2013/08/05-15:34:51.025 INFO [RES] Physical Disk <SqlClusterDriver2>: Completed HardDiskpUpdateRegistryWorker (start time 15:34:51.007)
00001468.00003798::2013/08/05-15:34:51.051 WARN [RES] Physical Disk <SqlClusterDriver2>: OnlineThread: Failed to get volume guid for device \\?\GLOBALROOT\Device\Harddisk10\Partition2\. Error 3
00001468.00001488::2013/08/05-15:34:51.862 INFO [RES] Physical Disk: PNP: Adding volume \\?\STORAGE#Volume#{2074c4d7-9726-11e0-af43-e61f13a81f8b}#0000000008100000#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
00001468.00001488::2013/08/05-15:34:51.863 INFO [RES] Physical Disk: PNP: Add Volume exit, status 0
000009fc.00000d38::2013/08/05-15:34:51.876 INFO [DCM] filter.Event ->CfsVolumeStateArrived FromPNP for ctx=0 status 00000000
000009fc.00000d38::2013/08/05-15:34:51.876 INFO [DCM] Pnp Volume arrived: \\?\Volume{4d50c727-8af3-432d-94f5-ba2669299680}
000009fc.00000d38::2013/08/05-15:34:51.877 INFO [DCM] PnP OnVolumeArrived(Disk 10, Part 2)
000009fc.00000d38::2013/08/05-15:34:51.877 INFO [DCM] PnP event for a disk which is no longer managed by CSV
000009fc.00000788::2013/08/05-15:34:53.489 DBG [NETFTAPI] 10.200.200.54 (IpDadStatePreferred ) NsiParameterNotification
000009fc.00000788::2013/08/05-15:34:53.491 DBG [NETFTAPI] 10.200.200.54 NetftLocalAdd
000009fc.00000788::2013/08/05-15:34:53.491 DBG [NETFTAPI] 10.200.200.54 NetftLocalConnect
000009fc.00000bdc::2013/08/05-15:34:53.509 INFO [IM] got event: Local endpoint 10.200.200.54:~0~ connected
00001468.00005988::2013/08/05-15:34:53.575 INFO [RHS] Resource SQL IP Address 1 (SqlCluster02) has come online. RHS is about to report status change to RCM
00001468.00005988::2013/08/05-15:34:53.575 INFO [RES] IP Address <SQL IP Address 1 (SqlCluster02)>: IP Address 10.200.200.54 on adapter BASP Virtual Adapter online
000009fc.000038b8::2013/08/05-15:34:53.575 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL IP Address 1 (SqlCluster02)', gen(0) result 0.
000009fc.000038b8::2013/08/05-15:34:53.575 INFO [RCM] TransitionToState(SQL IP Address 1 (SqlCluster02)) OnlinePending-->Online.
000009fc.00005de4::2013/08/05-15:34:53.578 INFO [RCM] TransitionToState(SQL Network Name (SqlCluster02)) WaitingToComeOnline-->OnlineCallIssued.
000009fc.0000535c::2013/08/05-15:34:53.579 INFO [CM] mscs::RegCheckpoint::RestoreCheckpoint: Restoring checkpoint SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10_50.SQLCLUSTER02\Cluster for resource SQL Network Name (SqlCluster02).
000009fc.0000535c::2013/08/05-15:34:53.590 INFO [CM] mscs::RegCheckpoint::RestoreCheckpoint: Restoring checkpoint SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10_50.SQLCLUSTER02\MSSQLServer for resource SQL Network Name (SqlCluster02).
000009fc.0000535c::2013/08/05-15:34:53.728 INFO [CM] mscs::RegCheckpoint::RestoreCheckpoint: Restoring checkpoint SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10_50.SQLCLUSTER02\Replication for resource SQL Network Name (SqlCluster02).
000009fc.0000535c::2013/08/05-15:34:53.733 INFO [CM] mscs::RegCheckpoint::RestoreCheckpoint: Restoring checkpoint SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10_50.SQLCLUSTER02\Providers for resource SQL Network Name (SqlCluster02).
000009fc.0000535c::2013/08/05-15:34:53.749 INFO [CM] mscs::RegCheckpoint::RestoreCheckpoint: Restoring checkpoint SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10_50.SQLCLUSTER02\SQLServerSCP for resource SQL Network Name (SqlCluster02).
000009fc.0000535c::2013/08/05-15:34:53.750 INFO [CM] mscs::RegCheckpoint::RestoreCheckpoint: Restoring checkpoint SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10_50.SQLCLUSTER02\CPE for resource SQL Network Name (SqlCluster02).
000009fc.0000535c::2013/08/05-15:34:53.755 INFO [CM] mscs::RegCheckpoint::RestoreCheckpoint: Restoring checkpoint SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10_50.SQLCLUSTER02\SQLServerAgent for resource SQL Network Name (SqlCluster02).
00001468.0000407c::2013/08/05-15:34:53.797 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Bringing resource online...
000009fc.00000788::2013/08/05-15:34:53.797 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL Network Name (SqlCluster02)', gen(0) result 997.
000009fc.00000788::2013/08/05-15:34:53.797 INFO [RCM] TransitionToState(SQL Network Name (SqlCluster02)) OnlineCallIssued-->OnlinePending.
00001468.0000407c::2013/08/05-15:34:53.797 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: TimerQueueTimer rescheduled to fire after 600 secs
00001468.0000407c::2013/08/05-15:34:53.803 INFO [ClNet] Adapter * 2 RFC2863 operational status = 1.
00001468.0000407c::2013/08/05-15:34:53.803 DBG [ClNet] Created adapter: DeviceGuid: ED3DA250-4D37-402F-8DE6-8D7FD2ADF6A0
00001468.0000407c::2013/08/05-15:34:53.803 DBG [ClNet] DeviceName: Microsoft Failover Cluster Virtual Adapter
00001468.0000407c::2013/08/05-15:34:53.803 DBG [ClNet] ConnectoidName: * 2
00001468.0000407c::2013/08/05-15:34:53.803 DBG [ClNet] Netbios/TCP: 1
00001468.0000407c::2013/08/05-15:34:53.803 DBG [ClNet] DNS Suffix:
00001468.0000407c::2013/08/05-15:34:53.803 DBG [ClNet] DnsServer: fec0:0:0:ffff::1%1
00001468.0000407c::2013/08/05-15:34:53.803 DBG [ClNet] DnsServer: fec0:0:0:ffff::2%1
00001468.0000407c::2013/08/05-15:34:53.803 DBG [ClNet] DnsServer: fec0:0:0:ffff::3%1
00001468.0000407c::2013/08/05-15:34:53.804 INFO [ClNet] Adapter Team 1 RFC2863 operational status = 1.
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] Created adapter: DeviceGuid: 6A92A5D0-8210-481D-8E29-4B39F5E64259
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] DeviceName: BASP Virtual Adapter
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] ConnectoidName: Team 1
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] Netbios/TCP: 1
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] DNS Suffix:
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] DnsServer: 10.200.200.104
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] DnsServer: 10.200.200.105
00001468.0000407c::2013/08/05-15:34:53.804 INFO [ClNet] Adapter 3 RFC2863 operational status = 1.
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] Created adapter: DeviceGuid: CAE5B831-79BF-4B07-8B83-CC875CA5159F
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] DeviceName: IBM USB Remote NDIS Network Device
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] ConnectoidName: 3
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] Netbios/TCP: 1
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] DNS Suffix:
00001468.0000407c::2013/08/05-15:34:53.804 INFO [ClNet] Adapter isatap.{6A92A5D0-8210-481D-8E29-4B39F5E64259} RFC2863 operational status = 2.
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] Created adapter: DeviceGuid: 9637BD41-35CE-4D1B-AB16-B75C686B3358
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] DeviceName: Microsoft ISATAP Adapter
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] ConnectoidName: isatap.{6A92A5D0-8210-481D-8E29-4B39F5E64259}
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] Netbios/TCP: 0
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] DNS Suffix:
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] DnsServer: 10.200.200.104
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] DnsServer: 10.200.200.105
00001468.0000407c::2013/08/05-15:34:53.804 INFO [ClNet] Adapter isatap.{CAE5B831-79BF-4B07-8B83-CC875CA5159F} RFC2863 operational status = 2.
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] Created adapter: DeviceGuid: 6E2EBE4C-E50F-4138-8495-74B407C92E45
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] DeviceName: Microsoft ISATAP Adapter #2
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] ConnectoidName: isatap.{CAE5B831-79BF-4B07-8B83-CC875CA5159F}
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] Netbios/TCP: 0
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] DNS Suffix:
00001468.0000407c::2013/08/05-15:34:53.804 INFO [ClNet] Adapter isatap.{ED3DA250-4D37-402F-8DE6-8D7FD2ADF6A0} RFC2863 operational status = 2.
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] Created adapter: DeviceGuid: 828C5BD8-A0DA-45CA-8611-C911E55F0C67
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] DeviceName: Microsoft ISATAP Adapter #3
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] ConnectoidName: isatap.{ED3DA250-4D37-402F-8DE6-8D7FD2ADF6A0}
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] Netbios/TCP: 0
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] DNS Suffix:
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] DnsServer: fec0:0:0:ffff::1%1
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] DnsServer: fec0:0:0:ffff::2%1
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] DnsServer: fec0:0:0:ffff::3%1
00001468.0000407c::2013/08/05-15:34:53.804 INFO [ClNet] Adapter Teredo Tunneling Pseudo-Interface RFC2863 operational status = 2.
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] Created adapter: DeviceGuid: A4595AD2-CE71-4DD2-8C34-E89D0E1B269A
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] DeviceName: Teredo Tunneling Pseudo-Interface
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] ConnectoidName: Teredo Tunneling Pseudo-Interface
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] Netbios/TCP: 0
00001468.0000407c::2013/08/05-15:34:53.804 DBG [ClNet] DNS Suffix:
00001468.0000407c::2013/08/05-15:34:53.806 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: adapter Team 1 (2)
00001468.0000407c::2013/08/05-15:34:53.817 WARN [RES] Network Name <SQL Network Name (SqlCluster02)>: Trying to remove credentials for LocalSystem returned status C0000225, STATUS_NOT_FOUND is a non-critical failure for a remove operation
00001468.0000407c::2013/08/05-15:34:53.877 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Initiating the Network Name operation : ' SqlCluster02 '
00001468.0000407c::2013/08/05-15:34:53.881 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Trying to find computer account SQLCLUSTER02 object GUID(bcd515b739c13a42b86ee7732e2b0165) on any available domain controller.
00001468.0000407c::2013/08/05-15:34:54.014 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Found computer account SQLCLUSTER02 on domain controller \\web-2.mycompany.com.
00001468.0000407c::2013/08/05-15:34:54.014 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Trying to obtain the VSToken for Core Cluster Name resource
00001468.0000407c::2013/08/05-15:34:54.130 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: GetCoreNetnameObject_VSToken returning status 0
00001468.0000407c::2013/08/05-15:34:54.131 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Obtained the security token for cluster name account.
00001468.0000407c::2013/08/05-15:34:54.244 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Got new Logon Session.
00001468.0000407c::2013/08/05-15:34:54.244 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Checking necessary attributes for computer account SQLCLUSTER02
00001468.0000407c::2013/08/05-15:34:54.246 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Successfully validated the attributes for computer account SQLCLUSTER02
00001468.0000407c::2013/08/05-15:34:54.246 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Initiating the Network Name operation : ' SqlCluster02 '
00001468.0000407c::2013/08/05-15:34:54.246 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Updating LSA credentials for Network Name resource SQLCLUSTER02
00001468.0000407c::2013/08/05-15:34:54.246 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Re-registering DNS records time period (3999765 secs ) between last registration and now is greater than 198646532493696
00001468.0000407c::2013/08/05-15:34:54.247 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Initiating the Network Name operation : ' SqlCluster02 DNS '
00001468.0000407c::2013/08/05-15:34:54.247 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Trying to obtain the VSToken for Core Cluster Name resource
00001468.0000407c::2013/08/05-15:34:54.247 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: GetCoreNetnameObject_VSToken returning status 0
00001468.0000407c::2013/08/05-15:34:54.277 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: DNS name SqlCluster02 flushed from DNS resolver Cache
00001468.0000407c::2013/08/05-15:34:54.279 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Replaced DNS name SqlCluster02.mycompany.com with IP Address 10.200.200.54 over adapter 'Team 1'.
00001468.0000407c::2013/08/05-15:34:54.279 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: DNS name SqlCluster02.mycompany.com Registration with LSA was successful
00001468.0000407c::2013/08/05-15:34:54.287 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Modified DNS name SqlCluster02.mycompany.com with IP Address 10.200.200.54 over adapter 'Team 1'.
00001468.0000407c::2013/08/05-15:34:54.287 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: DNS name SqlCluster02.mycompany.com Registration with LSA was successful
00001468.0000407c::2013/08/05-15:34:54.299 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Initiating the Network Name operation : ' SqlCluster02 NetBT '
00001468.0000407c::2013/08/05-15:34:54.303 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Registered server name SQLCLUSTER02 on transport \Device\NetbiosSmb.
00001468.0000407c::2013/08/05-15:34:54.303 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Initiating the Network Name operation : ' SQLCLUSTER02( \Device\NetBt_If2 )'
00001468.00003798::2013/08/05-15:34:55.052 WARN [RES] Physical Disk <SqlClusterDriver2>: OnlineThread: Failed to set volguid \??\Volume{4d50c727-8af3-432d-94f5-ba2669299680}. Error: 183.
00001468.00003798::2013/08/05-15:34:55.052 INFO [RES] Physical Disk <SqlClusterDriver2>: Found 2 mount points for device \Device\Harddisk10\Partition2
00001468.00003798::2013/08/05-15:34:55.052 INFO [RES] Physical Disk <SqlClusterDriver2>: VolumeIsNtfs: Volume \\?\GLOBALROOT\Device\Harddisk10\Partition2\ has FS type NTFS
00001468.00003798::2013/08/05-15:34:55.052 INFO [RES] Physical Disk: Volume \\?\GLOBALROOT\Device\Harddisk10\Partition2\ has FS type NTFS
00001468.00003798::2013/08/05-15:34:55.052 INFO [RES] Physical Disk: MountPoint I:\ points to volume \\?\Volume{4d50c727-8af3-432d-94f5-ba2669299680}\
00001468.00003798::2013/08/05-15:34:55.052 DBG [ClRtl] CaptureShareInfo: Share 0000000000000050, Server 0000000000000000, Path 0000000000000056 SD 0000000000000000 size 262
00001468.00003798::2013/08/05-15:34:55.052 WARN [ClRtl] SsCoreShareAdd(): status = 2118 share = I$ server = (null)
00001468.00003798::2013/08/05-15:34:55.052 DBG [ClRtl] CaptureShareInfo: Share 0000000000000050, Server 0000000000000000, Path 0000000000000056 SD 0000000000000000 size 262
00001468.00003798::2013/08/05-15:34:55.052 INFO [RHS] Resource SqlClusterDriver2 has come online. RHS is about to report status change to RCM
000009fc.00000788::2013/08/05-15:34:55.053 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'SqlClusterDriver2', gen(0) result 0.
000009fc.00000788::2013/08/05-15:34:55.053 INFO [RCM] TransitionToState(SqlClusterDriver2) OnlinePending-->Online.
00001468.00003798::2013/08/05-15:34:55.053 INFO [RES] Physical Disk: DriveLetter mask: 0x100
00001468.00003798::2013/08/05-15:34:55.057 INFO [RES] Physical Disk: HardDiskpScopeShareCallback: Enter resourceName SQL Network Name (SqlCluster02)
00001468.00003798::2013/08/05-15:34:55.058 DBG [ClRtl] CaptureShareInfo: Share 0000000000000050, Server 0000000000000056, Path 0000000000000070 SD 0000000000000000 size 288
00001468.00003798::2013/08/05-15:34:55.058 INFO [RES] Physical Disk: Attempt to ADD admin share I$ to/from netname SQLCLUSTER02, returned 0
00001468.0000407c::2013/08/05-15:34:57.305 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Registered server name SQLCLUSTER02 on transport \Device\NetBt_If2.
00001468.0000407c::2013/08/05-15:35:00.305 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Registered workstation name SQLCLUSTER02 on transport \Device\NetBt_If2.
00001468.0000407c::2013/08/05-15:35:00.305 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Initiating the Network Name operation : '(LSA)'
00001468.0000407c::2013/08/05-15:35:00.305 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: DNS name SqlCluster02 Registration with LSA was successful
00001468.0000407c::2013/08/05-15:35:00.306 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: NetBIOS name SQLCLUSTER02 Registration with LSA was successful
00001468.0000407c::2013/08/05-15:35:00.306 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: FQDN name SqlCluster02.mycompany.com Registration with LSA was successful
00001468.0000407c::2013/08/05-15:35:00.306 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Initiating the Network Name operation : ' SqlCluster02 '
00001468.0000407c::2013/08/05-15:35:00.316 DBG [ClRtl] CaptureShareInfo: Share 0000000000000050, Server 0000000000000056, Path 0000000000000070 SD 0000000000000000 size 288
00001468.0000407c::2013/08/05-15:35:00.316 WARN [ClRtl] SsCoreShareAdd(): status = 2118 share = I$ server = SQLCLUSTER02
00001468.0000407c::2013/08/05-15:35:00.316 DBG [ClRtl] CaptureShareInfo: Share 0000000000000050, Server 0000000000000056, Path 0000000000000070 SD 0000000000000000 size 288
00001468.0000407c::2013/08/05-15:35:00.320 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Network Name SQLCLUSTER02 is now online
00001468.0000407c::2013/08/05-15:35:00.320 INFO [RHS] Resource SQL Network Name (SqlCluster02) has come online. RHS is about to report status change to RCM
000009fc.00000788::2013/08/05-15:35:00.320 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL Network Name (SqlCluster02)', gen(0) result 0.
000009fc.00000788::2013/08/05-15:35:00.320 INFO [RCM] TransitionToState(SQL Network Name (SqlCluster02)) OnlinePending-->Online.
000009fc.0000535c::2013/08/05-15:35:00.320 INFO [RCM] TransitionToState(FileServer-(SqlCluster02)(SqlClusterDriver2)) WaitingToComeOnline-->OnlineCallIssued.
000009fc.0000535c::2013/08/05-15:35:00.320 INFO [RCM] TransitionToState(SQL Server (SQLCLUSTER02)) WaitingToComeOnline-->OnlineCallIssued.
000009fc.000038b8::2013/08/05-15:35:00.321 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'FileServer-(SqlCluster02)(SqlClusterDriver2)', gen(2) result 997.
000009fc.000038b8::2013/08/05-15:35:00.321 INFO [RCM] TransitionToState(FileServer-(SqlCluster02)(SqlClusterDriver2)) OnlineCallIssued-->OnlinePending.
000009fc.0000535c::2013/08/05-15:35:00.321 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL Server (SQLCLUSTER02)', gen(28) result 997.
000009fc.0000535c::2013/08/05-15:35:00.321 INFO [RCM] TransitionToState(SQL Server (SQLCLUSTER02)) OnlineCallIssued-->OnlinePending.
00001468.00002884::2013/08/05-15:35:00.326 INFO [ClNet] Adapter * 2 RFC2863 operational status = 1.
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] Created adapter: DeviceGuid: ED3DA250-4D37-402F-8DE6-8D7FD2ADF6A0
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] DeviceName: Microsoft Failover Cluster Virtual Adapter
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] ConnectoidName: * 2
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] Netbios/TCP: 1
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] DNS Suffix:
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] DnsServer: fec0:0:0:ffff::1%1
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] DnsServer: fec0:0:0:ffff::2%1
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] DnsServer: fec0:0:0:ffff::3%1
00001468.00002884::2013/08/05-15:35:00.326 INFO [ClNet] Adapter Team 1 RFC2863 operational status = 1.
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] Created adapter: DeviceGuid: 6A92A5D0-8210-481D-8E29-4B39F5E64259
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] DeviceName: BASP Virtual Adapter
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] ConnectoidName: Team 1
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] Netbios/TCP: 1
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] DNS Suffix:
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] DnsServer: 10.200.200.104
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] DnsServer: 10.200.200.105
00001468.00002884::2013/08/05-15:35:00.326 INFO [ClNet] Adapter 3 RFC2863 operational status = 1.
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] Created adapter: DeviceGuid: CAE5B831-79BF-4B07-8B83-CC875CA5159F
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] DeviceName: IBM USB Remote NDIS Network Device
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] ConnectoidName: 3
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] Netbios/TCP: 1
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] DNS Suffix:
00001468.00002884::2013/08/05-15:35:00.326 INFO [ClNet] Adapter isatap.{6A92A5D0-8210-481D-8E29-4B39F5E64259} RFC2863 operational status = 2.
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] Created adapter: DeviceGuid: 9637BD41-35CE-4D1B-AB16-B75C686B3358
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] DeviceName: Microsoft ISATAP Adapter
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] ConnectoidName: isatap.{6A92A5D0-8210-481D-8E29-4B39F5E64259}
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] Netbios/TCP: 0
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] DNS Suffix:
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] DnsServer: 10.200.200.104
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] DnsServer: 10.200.200.105
00001468.00002884::2013/08/05-15:35:00.326 INFO [ClNet] Adapter isatap.{CAE5B831-79BF-4B07-8B83-CC875CA5159F} RFC2863 operational status = 2.
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] Created adapter: DeviceGuid: 6E2EBE4C-E50F-4138-8495-74B407C92E45
00001468.00002884::2013/08/05-15:35:00.326 DBG [ClNet] DeviceName: Microsoft ISATAP Adapter #2
00001468.00002884::2013/08/05-15:35:00.327 DBG [ClNet] ConnectoidName: isatap.{CAE5B831-79BF-4B07-8B83-CC875CA5159F}
00001468.00002884::2013/08/05-15:35:00.327 DBG [ClNet] Netbios/TCP: 0
00001468.00002884::2013/08/05-15:35:00.327 DBG [ClNet] DNS Suffix:
00001468.00002884::2013/08/05-15:35:00.327 INFO [ClNet] Adapter isatap.{ED3DA250-4D37-402F-8DE6-8D7FD2ADF6A0} RFC2863 operational status = 2.
00001468.00002884::2013/08/05-15:35:00.327 DBG [ClNet] Created adapter: DeviceGuid: 828C5BD8-A0DA-45CA-8611-C911E55F0C67
00001468.00002884::2013/08/05-15:35:00.327 DBG [ClNet] DeviceName: Microsoft ISATAP Adapter #3
00001468.00002884::2013/08/05-15:35:00.327 DBG [ClNet] ConnectoidName: isatap.{ED3DA250-4D37-402F-8DE6-8D7FD2ADF6A0}
00001468.00002884::2013/08/05-15:35:00.327 DBG [ClNet] Netbios/TCP: 0
00001468.00002884::2013/08/05-15:35:00.327 DBG [ClNet] DNS Suffix:
00001468.00002884::2013/08/05-15:35:00.327 DBG [ClNet] DnsServer: fec0:0:0:ffff::1%1
00001468.00002884::2013/08/05-15:35:00.327 DBG [ClNet] DnsServer: fec0:0:0:ffff::2%1
00001468.00002884::2013/08/05-15:35:00.327 DBG [ClNet] DnsServer: fec0:0:0:ffff::3%1
00001468.00002884::2013/08/05-15:35:00.327 INFO [ClNet] Adapter Teredo Tunneling Pseudo-Interface RFC2863 operational status = 2.
00001468.00002884::2013/08/05-15:35:00.327 DBG [ClNet] Created adapter: DeviceGuid: A4595AD2-CE71-4DD2-8C34-E89D0E1B269A
00001468.00002884::2013/08/05-15:35:00.327 DBG [ClNet] DeviceName: Teredo Tunneling Pseudo-Interface
00001468.00002884::2013/08/05-15:35:00.327 DBG [ClNet] ConnectoidName: Teredo Tunneling Pseudo-Interface
00001468.00002884::2013/08/05-15:35:00.327 DBG [ClNet] Netbios/TCP: 0
00001468.00002884::2013/08/05-15:35:00.327 DBG [ClNet] DNS Suffix:
00001468.00002884::2013/08/05-15:35:00.336 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: adapter Team 1 (2)
00001468.00002884::2013/08/05-15:35:00.339 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Setting resource specific message to <>.
00001468.00002884::2013/08/05-15:35:00.339 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Endpoint is up.
00001468.0000548c::2013/08/05-15:35:00.340 INFO [RES] File Server <FileServer-(SqlCluster02)(SqlClusterDriver2)>: Shares 'are being scoped to virtual name SQLCLUSTER02
00001468.0000548c::2013/08/05-15:35:00.342 INFO [RES] File Server <FileServer-(SqlCluster02)(SqlClusterDriver2)>: Successfully added Share backup with Path I:\backup on server SQLCLUSTER02
00001468.0000548c::2013/08/05-15:35:00.381 INFO [RHS] Resource FileServer-(SqlCluster02)(SqlClusterDriver2) has come online. RHS is about to report status change to RCM
000009fc.000055f0::2013/08/05-15:35:00.381 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'FileServer-(SqlCluster02)(SqlClusterDriver2)', gen(2) result 0.
000009fc.000055f0::2013/08/05-15:35:00.381 INFO [RCM] TransitionToState(FileServer-(SqlCluster02)(SqlClusterDriver2)) OnlinePending-->Online.
00001468.0000548c::2013/08/05-15:35:00.381 INFO [RES] File Server <FileServer-(SqlCluster02)(SqlClusterDriver2)>: FileServerpMigrateSubdirShares: Exit status 0
00001614.00004ed8::2013/08/05-15:35:01.585 ERR [RES] SQL Server <SQL Server (SQLCLUSTER02)>: [sqsrvres] StartResourceService: Failed to start MSSQL$SQLCLUSTER02 service. CurrentState: 1
00001614.00004ed8::2013/08/05-15:35:01.585 ERR [RES] SQL Server <SQL Server (SQLCLUSTER02)>: [sqsrvres] OnlineThread: ResUtilsStartResourceService failed (status 435)
00001614.00004ed8::2013/08/05-15:35:01.585 ERR [RES] SQL Server <SQL Server (SQLCLUSTER02)>: [sqsrvres] OnlineThread: Error 435 bringing resource online.
00001614.00004ed8::2013/08/05-15:35:01.585 ERR [RHS] Online for resource SQL Server (SQLCLUSTER02) failed.
000009fc.00005de4::2013/08/05-15:35:01.586 WARN [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL Server (SQLCLUSTER02)', gen(28) result 5018.
000009fc.00005de4::2013/08/05-15:35:01.586 INFO [RCM] TransitionToState(SQL Server (SQLCLUSTER02)) OnlinePending-->ProcessingFailure.
000009fc.00005de4::2013/08/05-15:35:01.586 ERR [RCM] rcm::RcmResource::HandleFailure: (SQL Server (SQLCLUSTER02))
000009fc.00005de4::2013/08/05-15:35:01.588 INFO [RCM] resource SQL Server (SQLCLUSTER02): failure count: 3, restartAction: 2.
000009fc.00005de4::2013/08/05-15:35:01.588 INFO [RCM] Greater than restartPeriod time has elapsed since first failure, resetting failureTime and failureCount.
000009fc.00005de4::2013/08/05-15:35:01.588 INFO [RCM] Will restart resource in 500 milliseconds.
000009fc.00005de4::2013/08/05-15:35:01.588 INFO [RCM] TransitionToState(SQL Server (SQLCLUSTER02)) ProcessingFailure-->[WaitingToTerminate to DelayRestartingResource].
000009fc.00005de4::2013/08/05-15:35:01.588 INFO [RCM] TransitionToState(SQL Server (SQLCLUSTER02)) [WaitingToTerminate to DelayRestartingResource]-->[Terminating to DelayRestartingResource].
000009fc.00005de4::2013/08/05-15:35:01.589 INFO [RCM] HandleMonitorReply: TERMINATERESOURCE for 'SQL Server (SQLCLUSTER02)', gen(29) result 0.
000009fc.00005de4::2013/08/05-15:35:01.589 INFO [RCM] TransitionToState(SQL Server (SQLCLUSTER02)) [Terminating to DelayRestartingResource]-->DelayRestartingResource.
000009fc.00005de4::2013/08/05-15:35:02.089 INFO [RCM] Delay-restarting SQL Server (SQLCLUSTER02) and any waiting dependents.
000009fc.00005de4::2013/08/05-15:35:02.094 INFO [RCM] TransitionToState(SQL Server (SQLCLUSTER02)) DelayRestartingResource-->OnlineCallIssued.
000009fc.00005de4::2013/08/05-15:35:02.094 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL Server (SQLCLUSTER02)', gen(29) result 997.
000009fc.00005de4::2013/08/05-15:35:02.094 INFO [RCM] TransitionToState(SQL Server (SQLCLUSTER02)) OnlineCallIssued-->OnlinePending.
00001614.000056d4::2013/08/05-15:35:02.495 ERR [RES] SQL Server <SQL Server (SQLCLUSTER02)>: [sqsrvres] StartResourceService: Failed to start MSSQL$SQLCLUSTER02 service. CurrentState: 1
00001614.000056d4::2013/08/05-15:35:02.495 ERR [RES] SQL Server <SQL Server (SQLCLUSTER02)>: [sqsrvres] OnlineThread: ResUtilsStartResourceService failed (status 435)
00001614.000056d4::2013/08/05-15:35:02.495 ERR [RES] SQL Server <SQL Server (SQLCLUSTER02)>: [sqsrvres] OnlineThread: Error 435 bringing resource online.
00001614.000056d4::2013/08/05-15:35:02.496 ERR [RHS] Online for resource SQL Server (SQLCLUSTER02) failed.
000009fc.000030d8::2013/08/05-15:35:02.496 WARN [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL Server (SQLCLUSTER02)', gen(29) result 5018.
000009fc.000030d8::2013/08/05-15:35:02.496 INFO [RCM] TransitionToState(SQL Server (SQLCLUSTER02)) OnlinePending-->ProcessingFailure.
000009fc.000030d8::2013/08/05-15:35:02.496 ERR [RCM] rcm::RcmResource::HandleFailure: (SQL Server (SQLCLUSTER02))
000009fc.000030d8::2013/08/05-15:35:02.496 INFO [RCM] resource SQL Server (SQLCLUSTER02): failure count: 2, restartAction: 2.
000009fc.000030d8::2013/08/05-15:35:02.496 INFO [RCM] TransitionToState(SQL Server (SQLCLUSTER02)) ProcessingFailure-->[WaitingToTerminate to Failed].
000009fc.000030d8::2013/08/05-15:35:02.496 INFO [RCM] TransitionToState(SQL Server (SQLCLUSTER02)) [WaitingToTerminate to Failed]-->[Terminating to Failed].
000009fc.000030d8::2013/08/05-15:35:02.496 INFO [RCM] Resource SQL Server (SQLCLUSTER02) is causing group SQL Server (SQLCLUSTER02) to failover. Posting worker thread.
000009fc.000030d8::2013/08/05-15:35:02.496 INFO [RCM] TransitionToState(SQL Server Agent (SQLCLUSTER02)) WaitingToComeOnline-->OfflineDueToProvider.
000009fc.000030d8::2013/08/05-15:35:02.497 INFO [RCM] rcm::RcmGroup::Failover: (SQL Server (SQLCLUSTER02))
000009fc.0000535c::2013/08/05-15:35:02.497 INFO [RCM] HandleMonitorReply: TERMINATERESOURCE for 'SQL Server (SQLCLUSTER02)', gen(30) result 0.
000009fc.0000535c::2013/08/05-15:35:02.497 INFO [RCM] TransitionToState(SQL Server (SQLCLUSTER02)) [Terminating to Failed]-->Failed.
000009fc.0000535c::2013/08/05-15:35:02.497 INFO [RCM] rcm::RcmGroup::UpdateStateIfChanged: (SQL Server (SQLCLUSTER02), Pending --> Failed)
000009fc.0000535c::2013/08/05-15:35:02.497 INFO [RCM] Group move for 'SQL Server (SQLCLUSTER02)' has completed.
000009fc.000030d8::2013/08/05-15:35:02.517 INFO [RCM] time since last failure is greater than failover period; resetting failoverCount to 1.

August 5th, 2013 7:17pm

Here's another second part of the log (have to split that due to the limited of 6000 chars of msdn forum):

000009fc.000030d8::2013/08/05-15:35:02.517 WARN [RCM] Failing over group SQL Server (SQLCLUSTER02), failoverCount 1, last time 2013/08/05-23:35:02.517.
000009fc.000030d8::2013/08/05-15:35:02.517 INFO [RCM] rcm::RcmGroup::Move: (SQL Server (SQLCLUSTER02), 1)
000009fc.000030d8::2013/08/05-15:35:02.517 INFO [RCM] rcm::RcmGroup::Move: Bringing group 'SQL Server (SQLCLUSTER02)' offline first...
000009fc.000030d8::2013/08/05-15:35:02.517 INFO [RCM] TransitionToState(SqlClusterDriver2) Online-->WaitingToGoOffline.
000009fc.000030d8::2013/08/05-15:35:02.517 INFO [RCM] rcm::RcmGroup::UpdateStateIfChanged: (SQL Server (SQLCLUSTER02), Failed --> Pending)
000009fc.000030d8::2013/08/05-15:35:02.517 INFO [RCM] Bringing dependent resource 'FileServer-(SqlCluster02)(SqlClusterDriver2)' offline before provider resource 'SqlClusterDriver2'.
000009fc.000030d8::2013/08/05-15:35:02.517 INFO [RCM] TransitionToState(FileServer-(SqlCluster02)(SqlClusterDriver2)) Online-->OfflineCallIssued.
000009fc.000030d8::2013/08/05-15:35:02.517 INFO [RCM] 'SqlClusterDriver2' cannot go offline yet; 'FileServer-(SqlCluster02)(SqlClusterDriver2)' is in state OfflineCallIssued.
000009fc.000030d8::2013/08/05-15:35:02.517 INFO [RCM] 'SQL Network Name (SqlCluster02)' cannot go offline yet; 'FileServer-(SqlCluster02)(SqlClusterDriver2)' is in state OfflineCallIssued.
000009fc.000030d8::2013/08/05-15:35:02.517 INFO [RCM] TransitionToState(SQL Network Name (SqlCluster02)) Online-->WaitingToGoOffline.
000009fc.000030d8::2013/08/05-15:35:02.517 INFO [RCM] TransitionToState(SQL IP Address 1 (SqlCluster02)) Online-->WaitingToGoOffline.
000009fc.000030d8::2013/08/05-15:35:02.517 INFO [RCM] Bringing dependent resource 'SQL Network Name (SqlCluster02)' offline before provider resource 'SQL IP Address 1 (SqlCluster02)'.
000009fc.000030d8::2013/08/05-15:35:02.517 INFO [RCM] 'SQL Network Name (SqlCluster02)' cannot go offline yet; 'FileServer-(SqlCluster02)(SqlClusterDriver2)' is in state OfflineCallIssued.
000009fc.000030d8::2013/08/05-15:35:02.517 INFO [RCM] 'SQL IP Address 1 (SqlCluster02)' cannot go offline yet; 'SQL Network Name (SqlCluster02)' is in state WaitingToGoOffline.
00001468.00004864::2013/08/05-15:35:02.518 INFO [RES] File Server <FileServer-(SqlCluster02)(SqlClusterDriver2)>: FileServerDoTerminate: Terminate called... !!!
000009fc.0000535c::2013/08/05-15:35:02.518 INFO [RCM] HandleMonitorReply: OFFLINERESOURCE for 'FileServer-(SqlCluster02)(SqlClusterDriver2)', gen(2) result 997.
000009fc.0000535c::2013/08/05-15:35:02.518 INFO [RCM] TransitionToState(FileServer-(SqlCluster02)(SqlClusterDriver2)) OfflineCallIssued-->OfflinePending.
00001468.00004864::2013/08/05-15:35:02.518 INFO [RHS] Resource FileServer-(SqlCluster02)(SqlClusterDriver2) has come offline. RHS is about to report resource status to RCM.
00001468.00004864::2013/08/05-15:35:02.518 INFO [RES] File Server <FileServer-(SqlCluster02)(SqlClusterDriver2)>: FileServer is now offline.
000009fc.000018c8::2013/08/05-15:35:02.518 INFO [RCM] HandleMonitorReply: OFFLINERESOURCE for 'FileServer-(SqlCluster02)(SqlClusterDriver2)', gen(2) result 0.
000009fc.000018c8::2013/08/05-15:35:02.518 INFO [RCM] TransitionToState(FileServer-(SqlCluster02)(SqlClusterDriver2)) OfflinePending-->OfflineSavingCheckpoints.
000009fc.000018c8::2013/08/05-15:35:02.518 INFO [RCM] TransitionToState(FileServer-(SqlCluster02)(SqlClusterDriver2)) OfflineSavingCheckpoints-->Offline.
000009fc.000018c8::2013/08/05-15:35:02.518 INFO [RCM] TransitionToState(SqlClusterDriver2) WaitingToGoOffline-->OfflineCallIssued.
000009fc.000018c8::2013/08/05-15:35:02.518 INFO [RCM] TransitionToState(SQL Network Name (SqlCluster02)) WaitingToGoOffline-->OfflineCallIssued.
000009fc.000018c8::2013/08/05-15:35:02.518 INFO [RCM] 'SQL IP Address 1 (SqlCluster02)' cannot go offline yet; 'SQL Network Name (SqlCluster02)' is in state OfflineCallIssued.
00001468.00002a90::2013/08/05-15:35:02.518 INFO [RES] Physical Disk <SqlClusterDriver2>: Offline request.
00001468.00004e18::2013/08/05-15:35:02.518 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Taking resource offline...
00001468.00002498::2013/08/05-15:35:02.519 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: TimerQueueTimer rescheduled to fire after 600 secs
00001468.00002498::2013/08/05-15:35:02.519 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Offline of resource continuing...
000009fc.000030d8::2013/08/05-15:35:02.519 INFO [RCM] HandleMonitorReply: OFFLINERESOURCE for 'SQL Network Name (SqlCluster02)', gen(0) result 997.
000009fc.000030d8::2013/08/05-15:35:02.519 INFO [RCM] TransitionToState(SQL Network Name (SqlCluster02)) OfflineCallIssued-->OfflinePending.
000009fc.000018c8::2013/08/05-15:35:02.519 INFO [RCM] HandleMonitorReply: OFFLINERESOURCE for 'SqlClusterDriver2', gen(0) result 997.
00001468.00002270::2013/08/05-15:35:02.519 INFO [RES] Physical Disk: DriveLetter mask: 0x100
000009fc.000018c8::2013/08/05-15:35:02.519 INFO [RCM] TransitionToState(SqlClusterDriver2) OfflineCallIssued-->OfflinePending.
00001468.00002270::2013/08/05-15:35:02.531 INFO [RES] Physical Disk: HardDiskpScopeShareCallback: Enter resourceName SQL Network Name (SqlCluster02)
00001468.00002270::2013/08/05-15:35:02.531 INFO [RES] Physical Disk: Attempt to REMOVE admin share I$ to/from netname SQLCLUSTER02, returned 0
00001468.00002270::2013/08/05-15:35:02.532 INFO [RES] Physical Disk <SqlClusterDriver2>: HardDiskpCloseSVIHandles: Exit
00001468.00002a90::2013/08/05-15:35:02.532 INFO [RES] Physical Disk: Enter EnumerateDevices: EnumDevice 0
00001468.00002a90::2013/08/05-15:35:02.646 INFO [RES] Physical Disk: Exit EnumerateDevices: status 0
00001468.00003ee4::2013/08/05-15:35:02.647 INFO [RES] Physical Disk: Enter EnumerateDevices: EnumDevice 0
00001468.00003ee4::2013/08/05-15:35:02.683 INFO [RES] Physical Disk: Exit EnumerateDevices: status 0
00001468.00002498::2013/08/05-15:35:02.686 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: DNS name SqlCluster02 successful removed from LSA
000009fc.00000d38::2013/08/05-15:35:02.691 INFO [DCM] filter.Event ->CfsVolumeStateGone FromPNP for ctx=0 status 00000000
000009fc.00000d38::2013/08/05-15:35:02.691 INFO [DCM] Pnp Volume is gone: \\?\Volume{4d50c727-8af3-432d-94f5-ba2669299680}
000009fc.00000d38::2013/08/05-15:35:02.691 INFO [DCM] PnP OnVolumeGone(Disk 10, Part 2)
000009fc.00000d38::2013/08/05-15:35:02.691 INFO [DCM] PnP event for a disk which is no longer managed by CSV
00001468.00002498::2013/08/05-15:35:02.691 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Deleted server name SQLCLUSTER02 from all transports.
00001468.00002498::2013/08/05-15:35:02.691 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Deleted workstation name SQLCLUSTER02 from transport 0.
00001468.00002498::2013/08/05-15:35:02.692 INFO [RHS] Resource SQL Network Name (SqlCluster02) has come offline. RHS is about to report resource status to RCM.
00001468.00002498::2013/08/05-15:35:02.692 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: Resource is now offline
000009fc.000030d8::2013/08/05-15:35:02.692 INFO [RCM] HandleMonitorReply: OFFLINERESOURCE for 'SQL Network Name (SqlCluster02)', gen(0) result 0.
000009fc.000030d8::2013/08/05-15:35:02.692 INFO [RCM] TransitionToState(SQL Network Name (SqlCluster02)) OfflinePending-->OfflineSavingCheckpoints.
00001468.00002270::2013/08/05-15:35:03.217 INFO [RES] Physical Disk: ReleaseDisk: stop reserve succeeded on device 10 (sig d5841c3d)
000009fc.00005de4::2013/08/05-15:35:03.299 INFO [RCM] TransitionToState(SQL Network Name (SqlCluster02)) OfflineSavingCheckpoints-->Offline.
000009fc.0000535c::2013/08/05-15:35:03.300 INFO [RCM] TransitionToState(SQL IP Address 1 (SqlCluster02)) WaitingToGoOffline-->OfflineCallIssued.
00001468.00002a90::2013/08/05-15:35:03.300 INFO [RES] IP Address <SQL IP Address 1 (SqlCluster02)>: Taking resource offline...
00001468.00002a90::2013/08/05-15:35:03.300 INFO [RES] IP Address <SQL IP Address 1 (SqlCluster02)>: Deleting IP interface 36C8C80A.
000009fc.0000535c::2013/08/05-15:35:03.302 DBG [NETFTAPI] 10.200.200.54 NsiDeleteInstance
000009fc.0000535c::2013/08/05-15:35:03.304 WARN [NETFTAPI] 10.200.200.54 ( 80070490)
000009fc.0000535c::2013/08/05-15:35:03.304 DBG [NETFTAPI] 10.200.200.54 NetftLocalRemove
00001468.00002a90::2013/08/05-15:35:03.314 INFO [RES] IP Address <SQL IP Address 1 (SqlCluster02)>: Address 10.200.200.54 on adapter BASP Virtual Adapter offline.
000009fc.0000535c::2013/08/05-15:35:03.314 INFO [RCM] HandleMonitorReply: OFFLINERESOURCE for 'SQL IP Address 1 (SqlCluster02)', gen(0) result 0.
000009fc.0000535c::2013/08/05-15:35:03.314 INFO [RCM] TransitionToState(SQL IP Address 1 (SqlCluster02)) OfflineCallIssued-->OfflineSavingCheckpoints.
000009fc.0000535c::2013/08/05-15:35:03.314 INFO [RCM] TransitionToState(SQL IP Address 1 (SqlCluster02)) OfflineSavingCheckpoints-->Offline.
00001468.00002270::2013/08/05-15:35:03.865 INFO [RHS] Resource SqlClusterDriver2 has come offline. RHS is about to report resource status to RCM.
000009fc.000018c8::2013/08/05-15:35:03.865 INFO [RCM] HandleMonitorReply: OFFLINERESOURCE for 'SqlClusterDriver2', gen(0) result 0.
000009fc.000018c8::2013/08/05-15:35:03.865 INFO [RCM] TransitionToState(SqlClusterDriver2) OfflinePending-->OfflineSavingCheckpoints.
000009fc.0000535c::2013/08/05-15:35:03.865 INFO [RCM] TransitionToState(SqlClusterDriver2) OfflineSavingCheckpoints-->Offline.
000009fc.0000535c::2013/08/05-15:35:03.865 INFO [RCM] rcm::RcmGroup::UpdateStateIfChanged: (SQL Server (SQLCLUSTER02), Pending --> Failed)
000009fc.000055f0::2013/08/05-15:35:03.865 INFO [RCM] rcm::RcmGum::GroupMoveOperation(SQL Server (SQLCLUSTER02),1)
000009fc.00005de4::2013/08/05-15:35:03.927 WARN [RCM] rcm::RcmApi::GetGroupState: retrying: SQL Server (SQLCLUSTER02), 5908.
000009fc.00000834::2013/08/05-15:35:03.981 INFO [GUM] Node 2: Processing RequestLock 1:485267
000009fc.00000834::2013/08/05-15:35:03.981 INFO [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 2250386)
000009fc.000030d8::2013/08/05-15:35:04.929 INFO [NM] Received request from client address fe80::3d48:f03d:c782:380b.
000009fc.000038b8::2013/08/05-15:35:04.929 INFO [NM] Received request from client address fe80::3d48:f03d:c782:380b.
000009fc.00005de4::2013/08/05-15:35:04.930 INFO [NM] Received request from client address fe80::3d48:f03d:c782:380b.
000009fc.000038b8::2013/08/05-15:38:29.631 INFO [NM] Received request from client address DB-CLUSTER1-2.
00001468.000003c4::2013/08/05-15:45:02.485 INFO [RES] Network Name <SQL Network Name (SqlCluster02)>: TimerQueueTimer IS NOT scheduled to fire again. Resource state is3
000009fc.000055f0::2013/08/05-15:49:53.925 INFO [NM] Received request from client address fe80::3d48:f03d:c782:380b.
000009fc.00005708::2013/08/05-15:49:53.927 INFO [NM] Received request from client address fe80::3d48:f03d:c782:380b.
00001438.00003ce0::2013/08/05-15:50:04.001 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
000009fc.00001918::2013/08/05-15:50:04.001 WARN [RCM] Failed to load restype 'MSMQ': error 21.
000009fc.00001918::2013/08/05-15:50:04.001 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00001438.00003ce0::2013/08/05-15:50:04.002 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
000009fc.00005708::2013/08/05-15:50:04.002 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00001438.00003ce0::2013/08/05-15:50:04.048 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
000009fc.00001918::2013/08/05-15:50:04.048 WARN [RCM] Failed to load restype 'MSMQ': error 21.
000009fc.00001918::2013/08/05-15:50:04.048 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00001438.000025e4::2013/08/05-15:50:04.049 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
000009fc.00005708::2013/08/05-15:50:04.049 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
000009fc.00005708::2013/08/05-15:50:04.049 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00001438.000025e4::2013/08/05-15:50:04.051 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
000009fc.00004714::2013/08/05-15:50:04.051 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00001438.00003ce0::2013/08/05-15:50:04.051 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
000009fc.000055f0::2013/08/05-15:50:04.051 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00001438.00003ce0::2013/08/05-15:50:04.082 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
000009fc.00005708::2013/08/05-15:50:04.082 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
000009fc.00005708::2013/08/05-15:50:04.082 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00001438.00003ce0::2013/08/05-15:50:04.084 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
000009fc.00001918::2013/08/05-15:50:04.084 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00001438.00003ce0::2013/08/05-15:50:04.109 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
000009fc.00001918::2013/08/05-15:50:04.109 WARN [RCM] Failed to load restype 'MSMQ': error 21.
000009fc.00001918::2013/08/05-15:50:04.109 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00001438.00003ce0::2013/08/05-15:50:04.110 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
000009fc.000055f0::2013/08/05-15:50:04.110 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00001438.00003ce0::2013/08/05-15:50:04.114 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
000009fc.00001918::2013/08/05-15:50:04.114 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
000009fc.00001918::2013/08/05-15:50:04.114 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00001438.00003ce0::2013/08/05-15:50:04.116 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
000009fc.000055f0::2013/08/05-15:50:04.116 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00001438.00003ce0::2013/08/05-15:50:19.074 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
000009fc.00001918::2013/08/05-15:50:19.074 WARN [RCM] Failed to load restype 'MSMQ': error 21.
000009fc.00001918::2013/08/05-15:50:19.074 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00001438.00003ce0::2013/08/05-15:50:19.076 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
000009fc.000055f0::2013/08/05-15:50:19.076 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00001438.00003ce0::2013/08/05-15:50:19.080 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
000009fc.00001918::2013/08/05-15:50:19.080 WARN [RCM] Failed to load restype 'MSMQ': error 21.
000009fc.00001918::2013/08/05-15:50:19.080 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00001438.000025e4::2013/08/05-15:50:19.081 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
000009fc.000055f0::2013/08/05-15:50:19.081 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
000009fc.000055f0::2013/08/05-15:50:19.081 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00001438.00003ce0::2013/08/05-15:50:19.081 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
000009fc.00005708::2013/08/05-15:50:19.081 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00001438.000025e4::2013/08/05-15:50:19.082 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
000009fc.00004714::2013/08/05-15:50:19.082 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00001438.000025e4::2013/08/05-15:50:19.100 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
000009fc.00001918::2013/08/05-15:50:19.100 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
000009fc.00001918::2013/08/05-15:50:19.100 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00001438.000025e4::2013/08/05-15:50:19.101 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
000009fc.000055f0::2013/08/05-15:50:19.101 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
00001438.000025e4::2013/08/05-15:50:19.115 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
000009fc.00001918::2013/08/05-15:50:19.115 WARN [RCM] Failed to load restype 'MSMQ': error 21.
000009fc.00001918::2013/08/05-15:50:19.115 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node. Attempting to find a good node...
00001438.000025e4::2013/08/05-15:50:19.117 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
000009fc.000055f0::2013/08/05-15:50:19.117 WARN [RCM] Failed to load restype 'MSMQ': error 21.
00001438.000025e4::2013/08/05-15:50:19.120 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
000009fc.00004714::2013/08/05-15:50:19.120 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
000009fc.00004714::2013/08/05-15:50:19.120 WARN [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node. Attempting to find a good node...
00001438.000025e4::2013/08/05-15:50:19.122 ERR [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
000009fc.00001918::2013/08/05-15:50:19.122 WARN [RCM] Failed to load restype 'MSMQTriggers': error 21.
000009fc.00001918::2013/08/05-15:54:47.700 INFO [NM] Received request from client address DB-CLUSTER1-2.

Free Windows Admin Tool Kit Click here and download it now

August 5th, 2013 7:20pm

Hi,

To me it looks like a problem with "SqlClusterDriver2".

Failed to get volume guid for device \\?\GLOBALROOT\Device\Harddisk10\Partition2\. Error 3

Is about the same as saying "Error - path not found".

Can you see this volume on your current node? What's on this drive?

August 5th, 2013 7:51pm

I think that's just a warning, as it continues after that to online resources after that message.

The cluster brings all the required resources online, then tries to start the SQL Server service.

I think the failure is here:

(SQLCLUSTER02)>: [sqsrvres] StartResourceService: Failed to start MSSQL$SQLCLUSTER02 service. CurrentState: 1
00001614.00004ed8::2013/08/05-15:35:01.585 ERR [RES] SQL Server <SQL Server (SQLCLUSTER02)>: [sqsrvres] OnlineThread: ResUtilsStartResourceService failed (status 435)
00001614.00004ed8::2013/08/05-15:35:01.585 ERR [RES] SQL Server <SQL Server (SQLCLUSTER02)>: [sqsrvres] OnlineThread: Error 435 bringing resource online.
00001614.00004ed8::2013/08/05-15:35:01.585 ERR [RHS] Online for resource SQL Server (SQLCLUSTER02) failed.

Which is that the SQL Server failed to start. The SQL Server error log might indicate why.

Free Windows Admin Tool Kit Click here and download it now

August 5th, 2013 8:04pm

Hi David, thanks for your reply.

I check all the sql server logs, and find nothing helpful.

Here's the part of the related logs:

2013-08-05 23:35:18.09 Server (c) Microsoft Corporation.
2013-08-05 23:35:18.09 Server All rights reserved.
2013-08-05 23:35:18.09 Server Server process ID is 13852.
2013-08-05 23:35:18.09 Server System Manufacturer: 'IBM', System Model: 'System x3650 M3 -[7945I45]-'.
2013-08-05 23:35:18.09 Server Authentication mode is MIXED.
2013-08-05 23:35:18.09 Server Logging SQL Server messages in file 'I:\MSSQL10_50.SQLCLUSTER02\MSSQL\Log\ERRORLOG'.
2013-08-05 23:35:18.10 Server This instance of SQL Server last reported using a process ID of 14416 at 2013/8/5 23:24:03 (local) 2013/8/5 15:24:03 (UTC). This is an informational message only; no user action is required.
2013-08-05 23:35:18.10 Server Registry startup parameters:
-d I:\MSSQL10_50.SQLCLUSTER02\MSSQL\DATA\master.mdf
-e I:\MSSQL10_50.SQLCLUSTER02\MSSQL\Log\ERRORLOG
-l I:\MSSQL10_50.SQLCLUSTER02\MSSQL\DATA\mastlog.ldf
2013-08-05 23:35:18.14 Server SQL Server is starting at normal priority base (=7). This is an informational message only. No user action is required.
2013-08-05 23:35:18.14 Server Detected 16 CPUs. This is an informational message; no user action is required.
2013-08-05 23:35:18.15 Server Cannot use Large Page Extensions: lock memory privilege was not granted.
2013-08-05 23:35:18.44 Server Using dynamic lock allocation. Initial allocation of 2500 Lock blocks and 5000 Lock Owner blocks per node. This is an informational message only. No user action is required.
2013-08-05 23:35:18.44 Server Lock partitioning is enabled. This is an informational message only. No user action is required.
2013-08-05 23:35:18.80 Server Node configuration: node 0: CPU mask: 0x000000000000ff00:0 Active CPU mask: 0x000000000000ff00:0. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.
2013-08-05 23:35:18.81 Server Node configuration: node 1: CPU mask: 0x00000000000000ff:0 Active CPU mask: 0x00000000000000ff:0. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.
2013-08-05 23:35:19.05 spid8s Starting up database 'master'.
2013-08-05 23:35:19.59 spid8s 69 transactions rolled forward in database 'master' (1). This is an informational message only. No user action is required.
2013-08-05 23:35:19.59 spid8s 0 transactions rolled back in database 'master' (1). This is an informational message only. No user action is required.
2013-08-05 23:35:19.60 spid8s Recovery is writing a checkpoint in database 'master' (1). This is an informational message only. No user action is required.
2013-08-05 23:35:19.75 spid8s Resource governor reconfiguration succeeded.
2013-08-05 23:35:19.76 spid8s SQL Server Audit is starting the audits. This is an informational message. No user action is required.
2013-08-05 23:35:19.78 spid8s SQL Server Audit has started the audits. This is an informational message. No user action is required.
2013-08-05 23:35:19.79 spid8s FILESTREAM: effective level = 0, configured level = 0, file system access share name = 'SQLCLUSTER02'.
2013-08-05 23:35:19.89 spid8s SQL Trace ID 1 was started by login "sa".
2013-08-05 23:35:19.92 spid8s Starting up database 'mssqlsystemresource'.
2013-08-05 23:35:19.93 spid8s The resource database build version is 10.50.1600. This is an informational message only. No user action is required.
2013-08-05 23:35:20.35 spid12s Starting up database 'model'.
2013-08-05 23:35:20.36 spid8s Server name is 'SQLCLUSTER02\SQLCLUSTER02'. This is an informational message only. No user action is required.
2013-08-05 23:35:20.37 spid8s The NETBIOS name of the local node that is running the server is 'DB-CLUSTER1-1'. This is an informational message only. No user action is required.
2013-08-05 23:35:20.42 spid12s Clearing tempdb database.
2013-08-05 23:35:20.76 spid12s Starting up database 'tempdb'.
2013-08-05 23:35:20.84 spid16s The Service Broker protocol transport is disabled or not configured.
2013-08-05 23:35:20.84 spid16s The Database Mirroring protocol transport is disabled or not configured.
2013-08-05 23:35:20.94 spid16s Service Broker manager has started.
2013-08-05 23:35:20.95 Server A self-generated certificate was successfully loaded for encryption.
2013-08-05 23:35:21.05 Server Server is listening on [ 10.200.200.54 <ipv4> 49659].
2013-08-05 23:35:21.05 Server Server local connection provider is ready to accept connection on [ \\.\pipe\SQLLocal\SQLCLUSTER02 ].
2013-08-05 23:35:21.05 Server Server named pipe provider is ready to accept connection on [ \\.\pipe\$$\SQLCLUSTER02\MSSQL$SQLCLUSTER02\sql\query ].
2013-08-05 23:35:21.14 Server The SQL Server Network Interface library could not register the Service Principal Name (SPN) for the SQL Server service. Error: 0x2098, state: 15. Failure to register an SPN may cause integrated authentication to fall back to NTLM instead of Kerberos. This is an informational message. Further action is only required if Kerberos authentication is required by authentication policies.
2013-08-05 23:35:21.14 Server SQL Server is now ready for client connections. This is an informational message; no user action is required.
2013-08-05 23:35:21.15 spid15s A new instance of the full-text filter daemon host process has been successfully started.
2013-08-05 23:35:21.26 spid36s Starting up database 'XXXXXX'.

August 6th, 2013 5:05am

This of any use?

http://www.symantec.com/business/support/index?page=content&id=TECH155434 - reinstall .Net 3.5 sp1

Maybe you could run a cluster validation test. You'd have to do it out of hours to do the full run

Free Windows Admin Tool Kit Click here and download it now

August 7th, 2013 4:52am

Hello,

I am trying to involve someone more familiar with this topic for a further look at this issue. Sometime delay might be expected from the job transferring. Your patience is greatly appreciated.
Thank you for your understanding and support.

Regards,
Fanny Liu

If you have any feedback on our support, please click here.

August 8th, 2013 4:30am

Hi,

There are few things that you can try:

1). Restart the passive node.

2). As you said that SQL instance(SQLCLUSTER02\SQLCLUSTER02) on passive node(DB-CLUSTER1-1) right now is running. Stopped this SQL server from services console and try bringing SQL Server online from failover cluster manager, if it come online successfully then try also with SQL Agent and try again failback.

3). Try Failover back then check if you found duplicate file share. If yes, then try delete both and restart the service.

Free Windows Admin Tool Kit Click here and download it now

August 9th, 2013 5:18am

Hi all, i repair the .net 3.5 in the broken server, and restart it. and now i can failover to this server back again. i dont know which action takes effect.

August 9th, 2013 11:27pm

This topic is archived. No further replies will be accepted.