Resource Hosting Subsystem process crashes from KERNELBASE.dll (Network Steve Forum)

Resource Hosting Subsystem process crashes from KERNELBASE.dll

Hi,

I have 2 2013 Exchange servers in a DAG configuration. The passive member (I'll refer to it as EX02) is experiencing an issue where the Cluster Service crashes almost immediately after it is started. I've experienced this with this particular DAG before and ultimately ended up rebuilding the entire DAG from scratch to fix the issue, however it seems to have returned and now need a root cause analysis and fix. I've attached 4 Event Logs that have allowed me to drill down to the problem component but am at a loss on how to proceed next. Initially, when checking the cluster using Failover Cluster Manager, the EX02 node goes back and forth between "Joining" and "Down". This is a result of the Cluster Service constantly crashing and restarting since it is set to auto restart after an unexpected termination.

Looking at the 1146 Event ID, I saw this points to the Resource Hosting Subsystem as part of the issue and that this process experienced an issue with a .dll it's using. I then discovered this was the KERNELBASE.dll via the various other log entries I've included below. I haven't been able to find anyone else that's ran into this particular scenario and am not sure what needs to be done to correct this behavior. If anyone has any insight to this issue, it would be greatly appreciated. If any additional information is needed from me, please let me know as well.

 Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          3/3/2015 2:34:59 PM
Event ID:      1146
Task Category: Resource Control Manager
Level:         Critical
Keywords:      
User:          SYSTEM
Computer:      ****-EX02.****.com
Description:
The cluster Resource Hosting Subsystem (RHS) process was terminated and will be restarted. This is typically 
associated with cluster health detection and recovery of a resource. Refer to the System event log to determine which 
resource and resource DLL is causing the issue.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-FailoverClustering" Guid="{BAF908EA-3421-4CA9-9B84-6689B8C6F85F}" />
    <EventID>1146</EventID>
    <Version>0</Version>
    <Level>1</Level>
    <Task>3</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2015-03-03T20:34:59.261068900Z" />
    <EventRecordID>286364</EventRecordID>
    <Correlation />
    <Execution ProcessID="4716" ThreadID="4996" />
    <Channel>System</Channel>
    <Computer>****-EX02.****.com</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="NodeName">****-EX02</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Application Error
Date:          3/3/2015 2:36:13 PM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      ****-EX02.****.com
Description:
Faulting application name: clussvc.exe, version: 6.3.9600.17396, time stamp: 0x5434db7b
Faulting module name: KERNELBASE.dll, version: 6.3.9600.17415, time stamp: 0x54505737
Exception code: 0x80000003
Fault offset: 0x00000000000de002
Faulting process id: 0x31c8
Faulting application start time: 0x01d055f1a860f6f2
Faulting application path: C:\Windows\Cluster\clussvc.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: ee64313c-c1e4-11e4-80da-005056010d14
Faulting package full name: 
Faulting package-relative application ID: 
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-03-03T20:36:13.000000000Z" />
    <EventRecordID>264219</EventRecordID>
    <Channel>Application</Channel>
    <Computer>****-EX02.****.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>clussvc.exe</Data>
    <Data>6.3.9600.17396</Data>
    <Data>5434db7b</Data>
    <Data>KERNELBASE.dll</Data>
    <Data>6.3.9600.17415</Data>
    <Data>54505737</Data>
    <Data>80000003</Data>
    <Data>00000000000de002</Data>
    <Data>31c8</Data>
    <Data>01d055f1a860f6f2</Data>
    <Data>C:\Windows\Cluster\clussvc.exe</Data>
    <Data>C:\Windows\system32\KERNELBASE.dll</Data>
    <Data>ee64313c-c1e4-11e4-80da-005056010d14</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Windows Error Reporting
Date:          3/3/2015 2:36:14 PM
Event ID:      1001
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      ****-EX02.****.com
Description:
Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: clussvc.exe
P2: 6.3.9600.17396
P3: 5434db7b
P4: KERNELBASE.dll
P5: 6.3.9600.17415
P6: 54505737
P7: 80000003
P8: 00000000000de002
P9: 
P10: 

Attached files:

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_clussvc.exe_fcdf2bbcddca922c916880dbb49ce3cd5175efef_f6feed2
e_3a22d925

Analysis symbol: 
Rechecking for solution: 0
Report Id: ee64313c-c1e4-11e4-80da-005056010d14
Report Status: 4100
Hashed bucket: 
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Windows Error Reporting" />
    <EventID Qualifiers="0">1001</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-03-03T20:36:14.000000000Z" />
    <EventRecordID>264220</EventRecordID>
    <Channel>Application</Channel>
    <Computer>****-EX02.****.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>
    </Data>
    <Data>0</Data>
    <Data>APPCRASH</Data>
    <Data>Not available</Data>
    <Data>0</Data>
    <Data>clussvc.exe</Data>
    <Data>6.3.9600.17396</Data>
    <Data>5434db7b</Data>
    <Data>KERNELBASE.dll</Data>
    <Data>6.3.9600.17415</Data>
    <Data>54505737</Data>
    <Data>80000003</Data>
    <Data>00000000000de002</Data>
    <Data>
    </Data>
    <Data>
    </Data>
    <Data>
    </Data>
    <Data>C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_clussvc.exe_fcdf2bbcddca922c916880dbb49ce3cd5175e
fef_f6feed2e_3a22d925</Data>
    <Data>
    </Data>
    <Data>0</Data>
    <Data>ee64313c-c1e4-11e4-80da-005056010d14</Data>
    <Data>4100</Data>
    <Data>
    </Data>
  </EventData>
</Event>

Version=1
EventType=APPCRASH
EventTime=130698881964369580
ReportType=2
Consent=1
ReportIdentifier=0d8e824c-c1e4-11e4-80da-005056010d14
IntegratorReportIdentifier=0d8e824b-c1e4-11e4-80da-005056010d14
NsAppName=clussvc.exe
Response.type=4
Sig[0].Name=Application Name
Sig[0].Value=clussvc.exe
Sig[1].Name=Application Version
Sig[1].Value=6.3.9600.17396
Sig[2].Name=Application Timestamp
Sig[2].Value=5434db7b
Sig[3].Name=Fault Module Name
Sig[3].Value=KERNELBASE.dll
Sig[4].Name=Fault Module Version
Sig[4].Value=6.3.9600.17415
Sig[5].Name=Fault Module Timestamp
Sig[5].Value=54505737
Sig[6].Name=Exception Code
Sig[6].Value=80000003
Sig[7].Name=Exception Offset
Sig[7].Value=00000000000de002
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=6.3.9600.2.0.0.272.7
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=1033
DynamicSig[22].Name=Additional Information 1
DynamicSig[22].Value=a80b
DynamicSig[23].Name=Additional Information 2
DynamicSig[23].Value=a80b5d7481362c8eeb76a77903fa82d6
DynamicSig[24].Name=Additional Information 3
DynamicSig[24].Value=a1f8
DynamicSig[25].Name=Additional Information 4
DynamicSig[25].Value=a1f83374660c205ca4489043b9691b2f
UI[2]=C:\Windows\Cluster\clussvc.exe
UI[5]=Check online for a solution (recommended)
UI[6]=Check for a solution later (recommended)
UI[7]=Close
UI[8]=Microsoft Failover Cluster Service stopped working and was closed
UI[9]=A problem caused the application to stop working correctly. Windows will notify you if a solution is available.
UI[10]=&Close
LoadedModule[0]=C:\Windows\Cluster\clussvc.exe
LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\Windows\system32\KERNEL32.DLL
LoadedModule[3]=C:\Windows\system32\KERNELBASE.dll
LoadedModule[4]=C:\Windows\system32\msvcrt.dll
LoadedModule[5]=C:\Windows\SYSTEM32\sechost.dll
LoadedModule[6]=C:\Windows\system32\ADVAPI32.dll
LoadedModule[7]=C:\Windows\SYSTEM32\CLUSAPI.dll
LoadedModule[8]=C:\Windows\system32\CRYPT32.dll
LoadedModule[9]=C:\Windows\SYSTEM32\dhcpcsvc.DLL
LoadedModule[10]=C:\Windows\SYSTEM32\IPHLPAPI.DLL
LoadedModule[11]=C:\Windows\SYSTEM32\NTDSAPI.dll
LoadedModule[12]=C:\Windows\system32\ole32.dll
LoadedModule[13]=C:\Windows\system32\OLEAUT32.dll
LoadedModule[14]=C:\Windows\SYSTEM32\RESUTILS.dll
LoadedModule[15]=C:\Windows\system32\RPCRT4.dll
LoadedModule[16]=C:\Windows\SYSTEM32\SspiCli.dll
LoadedModule[17]=C:\Windows\SYSTEM32\sscore.dll
LoadedModule[18]=C:\Windows\system32\USER32.dll
LoadedModule[19]=C:\Windows\SYSTEM32\VSSAPI.DLL
LoadedModule[20]=C:\Windows\SYSTEM32\wevtapi.dll
LoadedModule[21]=C:\Windows\system32\WS2_32.dll
LoadedModule[22]=C:\Windows\system32\NSI.dll
LoadedModule[23]=C:\Windows\SYSTEM32\WINNSI.DLL
LoadedModule[24]=C:\Windows\SYSTEM32\FirewallAPI.dll
LoadedModule[25]=C:\Windows\SYSTEM32\bcrypt.dll
LoadedModule[26]=C:\Windows\SYSTEM32\NETAPI32.dll
LoadedModule[27]=C:\Windows\SYSTEM32\FLTLIB.DLL
LoadedModule[28]=C:\Windows\system32\SETUPAPI.dll
LoadedModule[29]=C:\Windows\SYSTEM32\DEVOBJ.dll
LoadedModule[30]=C:\Windows\SYSTEM32\fwpuclnt.dll
LoadedModule[31]=C:\Windows\SYSTEM32\VirtDisk.dll
LoadedModule[32]=C:\Windows\SYSTEM32\clfsw32.dll
LoadedModule[33]=C:\Windows\SYSTEM32\DSPARSE.dll
LoadedModule[34]=C:\Windows\SYSTEM32\cryptdll.dll
LoadedModule[35]=C:\Windows\system32\MSASN1.dll
LoadedModule[36]=C:\Windows\SYSTEM32\combase.dll
LoadedModule[37]=C:\Windows\system32\GDI32.dll
LoadedModule[38]=C:\Windows\SYSTEM32\VssTrace.DLL
LoadedModule[39]=C:\Windows\SYSTEM32\DSROLE.dll
LoadedModule[40]=C:\Windows\SYSTEM32\bcd.dll
LoadedModule[41]=C:\Windows\SYSTEM32\netutils.dll
LoadedModule[42]=C:\Windows\SYSTEM32\srvcli.dll
LoadedModule[43]=C:\Windows\SYSTEM32\wkscli.dll
LoadedModule[44]=C:\Windows\system32\CFGMGR32.dll
LoadedModule[45]=C:\Windows\SYSTEM32\DPAPI.DLL
LoadedModule[46]=C:\Windows\SYSTEM32\CRYPTBASE.dll
LoadedModule[47]=C:\Windows\SYSTEM32\SAMCLI.DLL
LoadedModule[48]=C:\Windows\SYSTEM32\bcryptPrimitives.dll
LoadedModule[49]=C:\Windows\SYSTEM32\sscoreext.dll
LoadedModule[50]=C:\Windows\SYSTEM32\mi.dll
LoadedModule[51]=C:\Windows\SYSTEM32\miutils.dll
LoadedModule[52]=C:\Windows\system32\wmidcom.dll
LoadedModule[53]=C:\Windows\SYSTEM32\kernel.appcore.dll
LoadedModule[54]=C:\Windows\SYSTEM32\ntmarta.dll
LoadedModule[55]=C:\Windows\SYSTEM32\dhcpcsvc6.DLL
LoadedModule[56]=C:\Windows\SYSTEM32\clbcatq.dll
LoadedModule[57]=C:\Windows\system32\wbem\wbemprox.dll
LoadedModule[58]=C:\Windows\SYSTEM32\wbemcomn.dll
LoadedModule[59]=C:\Windows\SYSTEM32\CRYPTSP.dll
LoadedModule[60]=C:\Windows\system32\rsaenh.dll
LoadedModule[61]=C:\Windows\system32\wbem\wbemsvc.dll
LoadedModule[62]=C:\Windows\system32\wbem\fastprox.dll
LoadedModule[63]=C:\Windows\SYSTEM32\pcwum.dll
LoadedModule[64]=C:\Windows\system32\mswsock.dll
LoadedModule[65]=C:\Windows\system32\WINTRUST.dll
LoadedModule[66]=C:\Windows\SYSTEM32\logoncli.dll
LoadedModule[67]=C:\Windows\SYSTEM32\DNSAPI.dll
LoadedModule[68]=C:\Windows\System32\rasadhlp.dll
LoadedModule[69]=C:\Windows\system32\kerberos.DLL
LoadedModule[70]=C:\Windows\SYSTEM32\SAMLIB.dll
LoadedModule[71]=C:\Windows\system32\es.dll
LoadedModule[72]=C:\Windows\system32\PROPSYS.dll
LoadedModule[73]=C:\Windows\SYSTEM32\profapi.dll
FriendlyEventName=Stopped working
ConsentKey=APPCRASH
AppName=Microsoft Failover Cluster Service
AppPath=C:\Windows\Cluster\clussvc.exe
NsPartner=windows
NsGroup=windows8
ApplicationIdentity=78F1110F74732BD871C9A3C553A667E4

March 3rd, 2015 8:58pm

Got a window to reboot the EX02 machine and this seems to have solved the issue. Cluster Service is stable again and DAG is healthy. This issue can be marked as answered.

Marked as answer by GWalus Wednesday, March 04, 2015 3:06 AM

Free Windows Admin Tool Kit Click here and download it now

March 4th, 2015 3:06am

Got a window to reboot the EX02 machine and this seems to have solved the issue. Cluster Service is stable again and DAG is healthy. This issue can be marked as answered.

Marked as answer by GWalus Wednesday, March 04, 2015 3:06 AM

March 4th, 2015 3:06am

Got a window to reboot the EX02 machine and this seems to have solved the issue. Cluster Service is stable again and DAG is healthy. This issue can be marked as answered.

I have this same issue, and it appears to manifest whenever there is a network disruption. After network is restored, the cluster service repeatedly crashes until a reboot. I'm not sure if a reboot is a real fix. I'm curious whether yours returns if you unplug the network cable and plug it back in on your server, like mine.

Free Windows Admin Tool Kit Click here and download it now

May 1st, 2015 6:07pm

This topic is archived. No further replies will be accepted.