2013 CAS server RPC ASP.NET Event 1309, code 3005 unhandled exception

People reporting that outlook does not connect, go into the CAS server event log and it is full off this ASP.NET event 1309 event below...

Log Name:      Application
Source:        ASP.NET 4.0.30319.0
Date:          5/15/2013 5:17:20 PM
Event ID:      1309
Task Category: Web Event
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      wal-vm-cas1.domain.com
Description:
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 5/15/2013 5:17:20 PM
Event time (UTC): 5/15/2013 9:17:20 PM
Event ID: 131b81fd0f8647afacdee56181cebc14
Event sequence: 1
Event occurrence: 1
Event detail code: 0
 
Application information:
    Application domain: /LM/W3SVC/1/ROOT/Rpc-1864-130131262403166620
    Trust level: Full
    Application Virtual Path: /Rpc
    Application Path: C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\rpc\
    Machine name: WAL-VM-CAS1
 
Process information:
    Process ID: 16020
    Process name: w3wp.exe
    Account name: NT AUTHORITY\SYSTEM
 
Exception information:
    Exception type: TargetInvocationException
    Exception message: Exception has been thrown by the target of an invocation.
   at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
   at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
   at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(Type type, Boolean nonPublic)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture)
   at System.Web.HttpRuntime.CreateNonPublicInstance(Type type, Object[] args)
   at System.Web.HttpApplication.BuildIntegratedModuleCollection(List`1 moduleList)
   at System.Web.HttpApplication.GetModuleCollection(IntPtr appContext)
   at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
   at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
   at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)

The type initializer for 'Microsoft.Exchange.Diagnostics.WorkloadManagement.ActivityContext' threw an exception.
   at Microsoft.Exchange.Diagnostics.WorkloadManagement.ActivityContext.RegisterMetadata(Type customMetadataType)
   at Microsoft.Exchange.RpcHttpModules.RpcHttpLoggingModule.GetDefaultLogger()
   at Microsoft.Exchange.RpcHttpModules.RpcHttpLoggingModule..ctor()

The type initializer for 'Microsoft.Exchange.Diagnostics.ExTraceConfiguration' threw an exception.
   at Microsoft.Exchange.Diagnostics.BaseTrace..ctor(Guid guid, Int32 traceTag)
   at Microsoft.Exchange.Diagnostics.WorkloadManagement.ActivityContext.RegisterMetadata(Type customMetadataType)
   at Microsoft.Exchange.Diagnostics.WorkloadManagement.ActivityContext..cctor()

The type initializer for 'Microsoft.Exchange.Diagnostics.ETWTrace' threw an exception.
   at Microsoft.Exchange.Diagnostics.ConfigFileHandler.GetFilePath()
   at Microsoft.Exchange.Diagnostics.ConfigFileHandler..ctor(String key, String defaultFileName)
   at Microsoft.Exchange.Diagnostics.ConfigFiles.get_Trace()
   at Microsoft.Exchange.Diagnostics.ExTraceConfiguration..ctor()
   at Microsoft.Exchange.Diagnostics.ExTraceConfiguration..cctor()

Not enough storage is available to complete this operation
   at Microsoft.Exchange.Diagnostics.ETWTrace.RegisterGuid(EtwTraceGuids& traceGuids, CriticalTraceRegistrationHandle& regHandle, ControlCallback callback, IntPtr& eventHandle)
   at Microsoft.Exchange.Diagnostics.ETWTrace.StartTraceSession()
   at Microsoft.Exchange.Diagnostics.ETWTrace..ctor()
   at Microsoft.Exchange.Diagnostics.ETWTrace..cctor()


 
Request information:
    Request URL: https://wal-vm-cas1.domain.com:443/rpc/rpcproxy.dll?ec0a64fa-b14c-4394-a97c-acf8e67f357f@domain.com:6001
    Request path: /rpc/rpcproxy.dll
    User host address: 172.16.58.156
    User: 
    Is authenticated: False
    Authentication Type: 
    Thread account name: NT AUTHORITY\SYSTEM
 
Thread information:
    Thread ID: 629
    Thread account name: NT AUTHORITY\SYSTEM
    Is impersonating: False
    Stack trace:    at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
   at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
   at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(Type type, Boolean nonPublic)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture)
   at System.Web.HttpRuntime.CreateNonPublicInstance(Type type, Object[] args)
   at System.Web.HttpApplication.BuildIntegratedModuleCollection(List`1 moduleList)
   at System.Web.HttpApplication.GetModuleCollection(IntPtr appContext)
   at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
   at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
   at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)
 
 

So I issue an IIS Reset which then generates some 70+ of the following;

Log Name:      Application
Source:        MSExchange Front End HTTP Proxy
Date:          5/15/2013 5:22:12 PM
Event ID:      1002
Task Category: (1)
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      wal-vm-cas1.domain.com
Description:
[RpcHttp] Application shutting down.

And then outlook clients are able to connect again...


May 16th, 2013 1:52am

Hi there,

It seems to be more about .net app issue since it can be temp resolved by restarting IIS.

Based on my research, event ID 1309 can be resolved by deploying the assemblies again. I'd recommend you post the qeustion in .net forum for better suggestion:http://social.msdn.microsoft.com/Forums/en-US/category/netdevelopment

Your understanding would be appreciated.

From Exchange server side, I'd suggest you verify the IIS settings according to: http://technet.microsoft.com/en-us/library/gg247612(v=exchg.150).aspx

Hope it is helpful.

Free Windows Admin Tool Kit Click here and download it now
May 16th, 2013 5:57am

Hello,

Is there any udpate?

If no more questions on this thread, we may mark it as answered.

If you have any feedback on our support, please click here

May 17th, 2013 6:15am

Fiona, these errors re-appear at least once or twice daily, resulting in some outlook users beeing unable to connect while others seem unaffected.

I did not post in the ASP.NET forum because to me this is an issue with Exchange 2013 CU1, in the area of RPCHTTP proxy on client access server.

Am hoping for either confirmation that this is a bug that will be addressed in next CU, or any other suggestions as to what I can do to work around it.

Thanks.

Free Windows Admin Tool Kit Click here and download it now
May 22nd, 2013 4:07pm

I am afraid that I can't agree with you.

Event Id 1309 may occur in many products that use asp.net, and it could even be a security attack:

http://blogs.technet.com/b/kevinholman/archive/2010/04/07/common-issues-with-the-opsmgr-web-console.aspx

http://blogs.technet.com/b/exchange/archive/2010/09/23/3411149.aspx

http://blogs.technet.com/b/jribeiro/archive/2010/09/27/microsoft-security-advisory-2416728-the-asp-net-vulnerability-and-exchange-server.aspx

However I am not familiar with ASP.net and currently there is less document/experience for Exchange 2013. I'd suggest you create a support ticket on Microsoft customer support service. It it is confirmed to be a bug, the case will be free for charge.

http://support.microsoft.com/default.aspx?scid=fh;EN-US;PHONENUMBERS 

May 24th, 2013 6:24am

Hi Fiona, hi adenhaan

I've setup a fresh Windows 2012 domain and i've installed an exchange server 2013. Everything is new; domain accounts, domain itself, exchange server ... everything and i get the same error messages as adenhaan.

----------------------

Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 27.05.2013 12:57:53
Event time (UTC): 27.05.2013 10:57:53
Event ID: 2c2c585684914066a2730ac502c4f3d5
Event sequence: 11
Event occurrence: 10
Event detail code: 0
 
Application information:
    Application domain: /LM/W3SVC/1/ROOT/Rpc-1-130141255402170437
    Trust level: Full
    Application Virtual Path: /Rpc
    Application Path: D:\Program Files\Exchange Server\v15\FrontEnd\HttpProxy\rpc\
    Machine name: %ExchangeServerHostName%
 
Process information:
    Process ID: 13476
    Process name: w3wp.exe
    Account name: NT AUTHORITY\SYSTEM
 
Exception information:
    Exception type: HttpException
    Exception message: The client disconnected.
   at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.EndProcessRequest(IAsyncResult result)
   at System.Web.HttpApplication.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)


 
Request information:
    Request URL: https://%exchangeserverhostname%.itsp.local:443/rpc/rpcproxy.dll?875b0f59-84fd-4faa-aed6-8f176fabbb93@%domainname.ch%:6001
    Request path: /rpc/rpcproxy.dll
    User host address: 192.168.42.131
    User: ITSP\andreas.hofstetter
    Is authenticated: True
    Authentication Type: NTLM
    Thread account name: NT AUTHORITY\SYSTEM
 
Thread information:
    Thread ID: 26
    Thread account name: NT AUTHORITY\SYSTEM
    Is impersonating: False
    Stack trace:    at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.EndProcessRequest(IAsyncResult result)
   at System.Web.HttpApplication.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)
 

I don't agree with Fiona, this error has an direct link to ms exchange 2013 and Microsoft shoud deliver as soon as possible a bug fix for this issue. Does anyone know a workaround or the source for this error?

Greetings,
Gino


Free Windows Admin Tool Kit Click here and download it now
May 27th, 2013 2:22pm

Hi Gino & adenhaan,

Thanks for your update. The technet forums might be not very suitable for troubleshooting issue that is suspected to be a bug. This is because that we need to more detailed information and/or diagnosing logs and performance files which is beyond what we can do in the forum as the nature of forum support. Again, I'd like to recommend you to contact Microsoft Customer Support Service (CSS) for assistance so that this problem can be resolved efficiently.
Your understanding would be appreciated.

June 3rd, 2013 1:10pm

Now that I've been running a while with this... the ASP.NET 1309 event is still occuring at least twice times daily, I have found it is always preceded by the following;

Log Name:      Application
Source:        MSExchange Front End HTTP Proxy
Date:          6/5/2013 3:16:15 PM
Event ID:      1003
Task Category: Core
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      wal-vm-cas1.domain.com
Description:
[Ews] An internal server error occurred. The unhandled exception was: System.NotSupportedException: The stream does not support concurrent IO read or write operations.
   at System.Net.ConnectStream.InternalWrite(Boolean async, Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   at System.Net.ConnectStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   at Microsoft.Exchange.HttpProxy.StreamProxy.BeginWrite(Byte[] buffer, Int32 count)
   at Microsoft.Exchange.HttpProxy.StreamProxy.BeginSend(Int32 bytesToSend)
   at Microsoft.Exchange.HttpProxy.StreamProxy.OnReadComplete(Object asyncState)

Another indication that this may indeed be a bug.

Unfortunately and I hope you can understand; there is not enough time in a day for me to spend a few hours on the phone trying to get to right level support person and then following up with another few hours "gathering" support data. If you can get me in touch with the right resources directly I will. In the meantime I will continue to provide feedback in this forum.

Free Windows Admin Tool Kit Click here and download it now
June 5th, 2013 11:02pm

Hello, I have same issue, new domain new Exchange 2013 CU1.

8-12 times a day:

At 11:44:29 System LOG Event ID 7031, The Microsoft Exchange RPC Client Access service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service

Att 11:44:31 Application LOG Event ID 1002, [RpcHttp] Application shutting Down

Just before the events I think I have the event that starts it all from the app log at 11:42:20:

Process Microsoft.Exchange.RpcClientAccess.Service.exe (PID=12256). Current policies: PolicyType:WorkloadPolicy;MailboxReplicationService:Discretionary;ELCAssistant:InternalMaintenance;JunkEmailOptionsCommitterAssistant:CustomerExpectation;CalendarRepairAssistant:InternalMaintenance;SharingPolicyAssistant:InternalMaintenance;TopNAssistant:InternalMaintenance;CalendarSyncAssistant:InternalMaintenance;UMReportingAssistant:InternalMaintenance;InferenceTrainingAssistant:InternalMaintenance;StoreMaintenanceAssistant:InternalMaintenance;PublicFolderAssistant:Discretionary;DirectoryProcessorAssistant:InternalMaintenance;PowerShell:InternalMaintenance;PowerShellForwardSync:CustomerExpectation;PowerShellBackSync:CustomerExpectation;PowerShellGalSync:InternalMaintenance;PowerShellLowPriorityWorkFlow:InternalMaintenance;PowerShellDiscretionaryWorkFlow:Discretionary;OABGeneratorAssistant:InternalMaintenance;Ews:InternalMaintenance;Eas:CustomerExpectation;Domt:InternalMaintenance;Owa:InternalMaintenance;TeamMailboxSync:CustomerExpectation;Pop:InternalMaintenance;Imap:InternalMaintenance;Momt:InternalMaintenance;Transport:CustomerExpectation;TransportSync:InternalMaintenance;OwaVoice:InternalMaintenance;OrgContactsSyncAssistant:InternalMaintenance;PublicFolderMailboxSync:CustomerExpectation;InferenceDataCollectionAssistant:Discretionary;PeopleRelevanceAssistant:InternalMaintenance;SiteMailboxAssistant:InternalMaintenance;PushNotificationService:InternalMaintenance;MailboxReplicationServiceHighPriority:Urgent

June 10th, 2013 1:33pm

Hi all, I have the same error.

Event id 1309 asp.net,

the biggest problem is that my clients get disconnected, and then password prompt appears.

This happens a few times a day for each of the users (approx 50 users).

this is the event logged,

Log Name:      Application
Source:        ASP.NET 4.0.30319.0
Date:          7/25/2013 5:06:44 PM
Event ID:      1309
Task Category: Web Event
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      Exchange.Terrogence.com
Description:
Event code: 3005 
Event message: An unhandled exception has occurred. 
Event time: 7/25/2013 5:06:44 PM 
Event time (UTC): 7/25/2013 2:06:44 PM 
Event ID: ff5d894ab8934d39a4e43ec10d0902f2 
Event sequence: 2 
Event occurrence: 1 
Event detail code: 0 
 
Application information: 
    Application domain: /LM/W3SVC/1/ROOT/Rpc-197-130192315327199311 
    Trust level: Full 
    Application Virtual Path: /Rpc 
    Application Path: C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\rpc\ 
    Machine name: EXCHANGE 
 
Process information: 
    Process ID: 2736 
    Process name: w3wp.exe 
    Account name: NT AUTHORITY\SYSTEM 
 
Exception information: 
    Exception type: HttpException 
    Exception message: The client disconnected.
   at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.EndProcessRequest(IAsyncResult result)
   at System.Web.HttpApplication.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)

 
 
Request information: 
    Request URL: https://owa.terrogence.com:443/rpc/rpcproxy.dll?060507d4-5dc4-47f0-9b03-ec5d8c97bac8@terrogence.com:6001 
    Request path: /rpc/rpcproxy.dll 
    User host address: 85.64.172.7 
    User: TERROGENCE\danielg 
    Is authenticated: True 
    Authentication Type: NTLM 
    Thread account name: NT AUTHORITY\SYSTEM 
 
Thread information: 
    Thread ID: 9 
    Thread account name: NT AUTHORITY\SYSTEM 
    Is impersonating: False 
    Stack trace:    at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.EndProcessRequest(IAsyncResult result)
   at System.Web.HttpApplication.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)
 
 
Custom event details: 

Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="ASP.NET 4.0.30319.0" />
    <EventID Qualifiers="32768">1309</EventID>
    <Level>3</Level>
    <Task>3</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2013-07-25T14:06:44.000000000Z" />
    <EventRecordID>795228</EventRecordID>
    <Channel>Application</Channel>
    <Computer>Exchange.Terrogence.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>3005</Data>
    <Data>An unhandled exception has occurred.</Data>
    <Data>7/25/2013 5:06:44 PM</Data>
    <Data>7/25/2013 2:06:44 PM</Data>
    <Data>ff5d894ab8934d39a4e43ec10d0902f2</Data>
    <Data>2</Data>
    <Data>1</Data>
    <Data>0</Data>
    <Data>/LM/W3SVC/1/ROOT/Rpc-197-130192315327199311</Data>
    <Data>Full</Data>
    <Data>/Rpc</Data>
    <Data>C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\rpc\</Data>
    <Data>EXCHANGE</Data>
    <Data>
    </Data>
    <Data>2736</Data>
    <Data>w3wp.exe</Data>
    <Data>NT AUTHORITY\SYSTEM</Data>
    <Data>HttpException</Data>
    <Data>The client disconnected.
   at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.EndProcessRequest(IAsyncResult result)
   at System.Web.HttpApplication.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)

</Data>
    <Data>https://owa.terrogence.com:443/rpc/rpcproxy.dll?060507d4-5dc4-47f0-9b03-ec5d8c97bac8@terrogence.com:6001</Data>
    <Data>/rpc/rpcproxy.dll</Data>
    <Data>85.64.172.7</Data>
    <Data>TERROGENCE\danielg</Data>
    <Data>True</Data>
    <Data>NTLM</Data>
    <Data>NT AUTHORITY\SYSTEM</Data>
    <Data>9</Data>
    <Data>NT AUTHORITY\SYSTEM</Data>
    <Data>False</Data>
    <Data>   at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.EndProcessRequest(IAsyncResult result)
   at System.Web.HttpApplication.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)
</Data>
  </EventData>
</Event>

My configuration is 2012 AD and all 2012 servers.

Two CAS servers and one mailbox Servers,

I get this error on both CAS servers,

And I have tried everything including reinstalling the CAS server.

Nothing works!!!

please help...

thank you

sahar.

Free Windows Admin Tool Kit Click here and download it now
July 25th, 2013 10:53am

Hi All,

Please Microsoft help!!!!!!

no answer yet, searched every forum to no avail....

August 11th, 2013 10:26am

Have no definitive proof, but check that internalhostname and externalhostname are set the same on set-outlookanywhere.

After I made that change, it feels like users would no longer get disconnected periodiaclly.

Also note that this ASP.NET 1309 event is sort of an umbrella for many different things that go wrong in the underbelly of Exchange. I still encounter plenty of them (including the exact one that I started this thread with) however they no longer appear to negatively impact outlook users.

Free Windows Admin Tool Kit Click here and download it now
August 11th, 2013 6:27pm

Here is another possible clue; I found that within seconds before a pack of these events shows up (the exact ones I started this post with) in the application eventlog the following informational message was logged;

The Outlook Anywhere settings for SSL offloading have been configured.

New value: True

And sure enough the CAS server that was plagued by this behavior had SSLOffloading set to $true.

Am now clear of these events for almost 24h after set-outlookanywhere cas-server\rpc* -sssloffloading $false

crossing my fingers.

August 22nd, 2013 9:51am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics