Active Directory = Windows 2003 Server SP2 - W2K3 DFL Mail server OS = Windows 2008 Server SP2 Exchange 2007 SP2 (since August 12, 2010) - note: no RU installed yet. Outlook 2007 SP2 Yes... Outlook Anywhere had been working for 16 months (installed the system in April 2009). This weekend, two users noticed they could not access their email from home using Outlook (Anywhere) on their work laptops. They could access their mail via OWA. They do not remember if they could or could not access email via Outlook Anywhere the first weekend after SP2 for Exchange 2007 was installed. Outlook seems to be working onsite (LAN) with possibly some OAB issues for two users but no complaints elsewhere. EMC - Exchange Tools - Mail tracking shows that mail reaches their mailbox and once again, they can access via OWA. When I attempted the reproduce the problem from an external connection (offsite) using Outlook Anywhere, this error message displayed about a minute after I entered my username and password (domainName\Username, then password): "Microsoft Exchange is unavailable". If I select Retry, I can enter Outlook but Status is "Offline". This did work "before". Once again, first time anyone noticed a problem for the last 16 months was this weekend. I was able to access OWA immediately after the above test and using the same connection. Send and Receive from OWA was successfull. The following error (seen in Even Viewer) seems to coincide with my attempts to connect via Outlook Anywhere. 4 failed attempts and 4 entries as follows, produced within a minute of the attempt to connect: EventID 11 CAPI2 Failed extract of third-party root list from auto update cab at: <http://www.download.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootstl.cab> with error: A required certificate is not within its validity period when verifying against the current system clock or the timestamp in the signed file. Server side Test-SystemHealth Displays no errors about certificates however (some drivers are older than two years, the Filter Pack is not installed, Active Server Pages is not detected or allowed, but that's it). Test-ServiceHealth All is fine here. Test-OWAConnectivity Sometimes passes, sometimes fails (this never happened before SP2 and yes, I right click and run EMS as aministrator). When it fails, this is the error message: [PS] C:\>Test-OwaConnectivity -TrustAnySSLCertificate | fl WARNING: The test was unable to log on to Outlook Web Access because the SSL certificate did not validate. You can force the cmdlet to proceed by re-running it and specifying the ?TrustAnySSLCertificate parameter. AuthenticationMethod : ClientAccessServer : MS1.mydomain.local Scenario : Logon ScenarioDescription : Log on to Outlook Web Access and verify the response page. PerformanceCounterName : Logon Latency Result : Skipped MailboxServer : MS1.mydomain.local StartTime : 8/23/2010 12:01:46 PM Latency : 00:00:00.0156001 SecureAccess : True Error : The test was unable to log on to Outlook Web Access because the SSL certificate did not validate. You can force the cmdlet to proceed by re-running it and specifying the -TrustAnySSLCertificate parameter. UserName : CAS_00xxxxxxxxxxxb VirtualDirectoryName : owa (Default Web Site) Url : https://mail.mydomain.org/owa/ UrlType : Internal EventType : Warning Port : 0 ConnectionType : Plaintext Yet at other times, with no apparent pattern, the test is successful. This was the result a minute or so before the failure above. Same credentials, same logon with right click and "Run as Administrator", same EMS session: [PS] C:\>Test-OwaConnectivity ClientAccessServer MailboxServer URL Scenario Result Latency Error (ms) ------------------ ------------- --- -------- ------ - ------ ----- MS1 MS1 https://mail.mydomain.org/owa Logon Success 62.4 Same phenomenon with.. Test-WebServicesConnectivity I just tried it - All Success (Pass). But other times, it fails with a reference to the SSL cert not validating. ****************** EMC BPA Connectivity Test is a Pass (no errors). EMC BPA Health Test shows same Warnings about old NIC and Storage drivers that the Test-SystemHealth check does. No errors. Nothing about certificates. What do I do next? I'm guessing it has something to do with the certificates?

Here are the ECRA results - only domain name and user name changed, IP address replaced with xx.xx.xx.xx https://www.testexchangeconnectivity.com/ __________________________________________________ ExRCA is testing RPC/HTTP connectivity. The RPC/HTTP test failed . Test Steps Attempting to test Autodiscover for testuser1@MyDomain.org Autodiscover was tested successfully. Test Steps ExRCA is attempting each method of contacting the Autodiscover service. The Autodiscover service was tested successfully. Test Steps Attempting to test potential AutoDiscover URL https://MyDomain.org/AutoDiscover/AutoDiscover.xml Testing of this potential Autodiscover URL failed. NOTE: WE DO NOT USE THIS ADDRESS - OTHER AUTODISCOVER URL IS OK - SEE BELOW Test Steps Attempting to resolve the host name MyDomain.org in DNS. Host successfully resolved Additional Details IP(s) returned: xx.xx.xx.xx Testing TCP Port 443 on host MyDomain.org to ensure it is listening and open. The port was opened successfully. ExRCA is testing the SSL certificate to make sure it's valid. The SSL certificate failed one or more certificate validation checks. Test Steps The certificate name is being validated. Certificate name validation failed. Tell me more about this issue and how to resolve it Additional Details Host name MyDomain.org does not match any name found on the server certificate E=info@plesk.com, CN=plesk, OU=Plesk, O="SWsoft, Inc.", L=Herndon, S=Virginia, C=US NOTE: I HAVE NO IDEA WHAT THIS INFO IS ABOUT - THIS IS NOT MY DOMAIN. IT USED TO COME UP IN BPA HEALTH CHECKS - BUT NOT SINCE SP2 Attempting to test potential AutoDiscover URL https://autodiscover.MyDomain.org/AutoDiscover/AutoDiscover.xml Testing of the Autodiscover URL was successful. Test Steps Attempting to resolve the host name autodiscover.MyDomain.org in DNS. Host successfully resolved Additional Details IP(s) returned: xx.xx.xx.xx Testing TCP Port 443 on host autodiscover.MyDomain.org to ensure it is listening and open. The port was opened successfully. ExRCA is testing the SSL certificate to make sure it's valid. The certificate passed all validation requirements. Test Steps The certificate name is being validated. Successfully validated the certificate name Additional Details Found hostname autodiscover.MyDomain.org in Certificate Subject Alternative Name entry Certificate trust is being validated. The certificate is trusted and all certificates are present in the chain. Additional Details The Certificate chain has be validated up to a trusted root. Root = E=info@valicert.com, CN=http://www.valicert.com/, OU=ValiCert Class 2 Policy Validation Authority, O="ValiCert, Inc.", L=ValiCert Validation Network The certificate date is being confirmed to ensure the certificate is valid. Date validation passed. The certificate hasn't expired. Additional Details Certificate is valid: NotBefore = 2/25/2010 6:12:11 PM, NotAfter = 3/23/2011 8:07:10 PM" The IIS configuration is being checked for client certificate authentication. Client certificate authentication wasn't detected. Additional Details Accept/Require Client Certificates not configured. ExRCA is attempting to send an Autodiscover POST request to potential Autodiscover URLs. Successfully Retrieved AutoDiscover Settings by sending AutoDiscover POST. Test Steps Attempting to Retrieve XML AutoDiscover Response from url https://autodiscover.MyDomain.org/AutoDiscover/AutoDiscover.xml for user testuser1@MyDomain.org The Autodiscover XML response was successfully retrieved. Additional Details AutoDiscover Account Settings XML Response: [...] Autodiscover settings for Outlook Anywhere are being validated. Outlook Anywhere Autodiscover Settings validated Attempting to resolve the host name mail.MyDomain.org in DNS. Host successfully resolved Additional Details IP(s) returned: xx.xx.xx.xx Testing TCP Port 443 on host mail.MyDomain.org to ensure it is listening and open. The port was opened successfully. ExRCA is testing the SSL certificate to make sure it's valid. The certificate passed all validation requirements. Test Steps The certificate name is being validated. Successfully validated the certificate name Additional Details Found hostname mail.MyDomain.org in Certificate Subject Common name Certificate trust is being validated. The certificate is trusted and all certificates are present in the chain. Additional Details The Certificate chain has be validated up to a trusted root. Root = E=info@valicert.com, CN=http://www.valicert.com/, OU=ValiCert Class 2 Policy Validation Authority, O="ValiCert, Inc.", L=ValiCert Validation Network The certificate date is being confirmed to ensure the certificate is valid. Date validation passed. The certificate hasn't expired. Additional Details Certificate is valid: NotBefore = 2/25/2010 6:12:11 PM, NotAfter = 3/23/2011 8:07:10 PM" The IIS configuration is being checked for client certificate authentication. The test passed with some warnings encountered. Please expand the additional details. Additional Details Client Certificate Authentication could not be determined due to an unexpected failure. WinHttpSendRequest failed with error 12002. Testing Http Authentication Methods for URL https://mail.MyDomain.org/rpc/rpcproxy.dll The HTTP authentication test failed. Additional Details An HTTP 500 response was returned from Unknown

I regret that the above is unreadable - font was x-small and when I changed it to small that killed the formatting. That's a lot to reformat. This seems to be the most pertinent: --------------------------------------------------- The IIS configuration is being checked for client certificate authentication. The test passed with some warnings encountered. Please expand the additional details. Additional Details Client Certificate Authentication could not be determined due to an unexpected failure. WinHttpSendRequest failed with error 12002. Testing Http Authentication Methods for URL https://mail.MyDomain.org/rpc/rpcproxy.dll The HTTP authentication test failed. Additional Details An HTTP 500 response was returned from Unknown

FYI: ERCA Autodiscover test passed with and without SSL Trust.

Outlook /RPC Doesn't tell us much. Status is "Connecting" for a Type "Referral" and then "Directory", references the mail server (mailserver.mydomain.tld), then a domain controller (dc1.mydomain.tld). No results are displayed. No Pass, no Fail, nothing. Outlook still displays "Microsoft Exchange is unavailable".

RPCPing? (Is there I troubleshooting tool I have not thought of?). Question: Is RPCPing encrypted over the Internet (I think not)? Is it safe to send real user credentials? Or should I create a test user as I did for https://www.testexchangeconnectivity.com/ Pending your responses, I'm going to try with a test user. Not sure how valid that will be?

I tried this on the mailserver itself - just to see what would happen. Looks like I have to resolve some issues before trying it one the client machine. What is the RPCProxy? I thought it was the ExternalHostName from Get-OutlookAnywhere? I also tried some other combinations: C:\>RPCPing -t ncacn_http -o RPCProxy=mail.myDomain.org -u 10 -a connect -v 3 -E -P "testUser1,SecretPa$$Word,myDomain," -H 1 -F 3 Invalid BindingOption (RPCProxy=mail.myDomain.org). You must specify the RpcProxy C:\>RPCPing -t ncacn_http -o RPCProxy=MailServer1.myDomain.local -u 10 -a connect -v 3 -E -P "testUser1,SecretPa$$Word,myDomain," -H 1 -F 3 Invalid BindingOption (RPCProxy=MailServer1.myDomain.local). You must specify the RpcProxy C:\>RPCPing -t ncacn_http -o RPCProxy=MailServer1 -u 10 -a connect -v 3 -E -P "testUser1,SecretPa$$Word,myDomain," -H 1 -F 3 Invalid BindingOption (RPCProxy=MailServer1). You must specify the RpcProxy

Authentication settings on virtual IIS folders are as described below, except for EWS which also has Basic enabled: http://blogs.technet.com/b/ferris/archive/2010/03/30/default-authentication-settings-exchange-2007-2010-iis-application-virtual-directories.aspx RPCwithCert = All = Disabled

All that for no response? MSFT - any suggestions?

Please browse the URL below, you should get a blank page after authentication https://mail.MyDomain.org/rpc/rpcproxy.dll Does the redirection set on the /RPC virtual directory? Please run the cmdlet below Get-Outlookprovider EXPR |Fl CertPrincipalName,Server Get-OutlookAnywhere | FlJames Luo TechNet Subscriber Support (http://technet.microsoft.com/en-us/subscriptions/ms788697.aspx) If you have any feedback on our support, please contact tngfb@microsoft.com

Thank you James. Good to hear from you! ---------------------------------------------------- https://mail.MyDomain.org/rpc/rpcproxy.dll HTTP 500 Internal Server Error The website cannot display the page. -------------------------------------------------------------- [PS] C:\>Get-OutlookProvider EXPR | fl CertPrincipalName, server CertPrincipalName : Server : That does not look right ^ [PS] C:\>Get-OutlookAnywhere | fl ServerName : MS1 SSLOffloading : False ExternalHostname : mail.myDomain.org ClientAuthenticationMethod : Basic IISAuthenticationMethods : {Basic, Ntlm} MetabasePath : IIS://MS1.myDomain.loc/W3SVC/1/ROOT/Rpc Path : C:\Windows\System32\RpcProxy Server : MS1 AdminDisplayName : ExchangeVersion : 0.1 (8.0.535.0) Name : Rpc (Default Web Site) DistinguishedName : CN=Rpc (Default Web Site),CN=HTTP,CN=Protocols,CN=MS1,CN=Servers,CN=Exchange Administr ative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=myDomain,CN=Microsoft Exchange,CN=Se rvices,CN=Configuration,DC=myDomain,DC=loc Identity : MS1\Rpc (Default Web Site) Guid : 86ea098d-f473-4f71-8f8a-c5e2e95cc74d ObjectCategory : myDomain.loc/Configuration/Schema/ms-Exch-Rpc-Http-Virtual-Directory ObjectClass : {top, msExchVirtualDirectory, msExchRpcHttpVirtualDirectory} WhenChanged : 3/28/2009 6:58:53 PM WhenCreated : 3/28/2009 6:58:38 PM OriginatingServer : dc1.myDomain.loc IsValid : True ----------------------------------------------------------------- Once again, OWA, for example, is working fine. And, for a comparison, I can get this - after authentication - (the EWS/exchange.asmx file is accessible as well). https://mail.myDomain.org/autodiscover/autodiscover.xml <?xml version="1.0" encoding="utf-8" ?> - <Autodiscover xmlns="http://schemas.microsoft.com/exchange/autodiscover/responseschema/2006"> - <Response> - <Error Time="08:24:33.1753685" Id="1668824430"> <ErrorCode>600</ErrorCode> <Message>Invalid Request</Message> <DebugData /> </Error> </Response> </Autodiscover>

As I saw above, “mail.myDomain.org” is the common name in the certificate, right? If so, the output is correct in the EXPR provider I assume redirection isn’t configured on the virtual directory Please check the SSL settings of /RPC virtual directory · Require SSL (Checked) · Require 128-bit SSL (Unchecked) · Client certificates: Ignore Does the error information same no matter whether you browsed the virtual directory internally and externally? Please open the IIS log, check the sub status code after 500 HTTP 500.x -- Internal Server Error Codes Per my research, it seems that RPC proxy component has corrupted on the CAS server, please re-install it and test the URL again: 1. Disable outlook anywhere via EMC 2. Remove RPC proxy component via PowerShell 3. Command: servermanagercmd -r rpc-over-http-proxy 4. Reboot the server 5. Install RPC proxy component via PowerShell 6. Command: servermanagercmd -i rpc-over-http-proxy 7. Enable outlook anywhere 8. Restart Microsoft active directory Topology service 9. Check the issue againJames Luo TechNet Subscriber Support (http://technet.microsoft.com/en-us/subscriptions/ms788697.aspx) If you have any feedback on our support, please contact tngfb@microsoft.com

As I saw above, “mail.myDomain.org” is the common name in the certificate, right? I think so: [PS] C:\>Get-ExchangeCertificate Thumbprint Services Subject ---------- -------- ------- 8606EFxxxxxxxxx62xxxxxxxxxx IP.WS CN=mail.myDomain.org, OU=Domain Control Validated, O=mail.myDomain.org I assume redirection isn’t configured on the virtual directory Correct. We did not configure redirction on any virtual directory, including OWA. I realize this is an option but we opted against it: keep things simple - just deploy a desktop shortcut via GPO so users can access OWA, for example, without entering a long URL (or any URL for that matter). At home, they enter it once, then create a shortcut or add it to favorites. Please check the SSL settings of /RPC virtual directory · Require SSL (Checked) - SAME · Require 128-bit SSL (Unchecked) - mine is checked, as it is for autodiscover, OWA and others. · Client certificates: Ignore - SAME James - my settings are underlined above. They are as you recommend, except for 128 bit SSL which, on my server, is checked. Does the error information same no matter whether you browsed the virtual directory internally and externally? Yes - I just verified: HTTP 500 This is an example of what I found in the IIS logs - not easy to find things in there - I searched for "500" and found entries like this: 2010-08-23 01:43:46 10.0.x.x RPC_IN_DATA /rpc/rpcproxy.dll MS1.abc.loc:6001 443 - x.x.185.29 MSRPC 500 0 21 45037 2010-08-23 01:43:46 10.0.x.x RPC_OUT_DATA /rpc/rpcproxy.dll MS1.abc.loc:6001 443 - x.x.185.29 MSRPC 500 0 21 45053 2010-08-23 01:44:32 10.0.x.x RPC_IN_DATA /rpc/rpcproxy.dll MS1.abc.loc:6002 443 - x.x.185.29 MSRPC 500 0 21 45037 2010-08-23 01:44:32 10.0.x.x RPC_OUT_DATA /rpc/rpcproxy.dll MS1.abc.loc:6002 443 - x.x.185.29 MSRPC 500 0 21 45037 2010-08-23 01:45:17 10.0.x.x RPC_IN_DATA /rpc/rpcproxy.dll MS1.abc.loc:6001 443 - x.x.185.29 MSRPC 500 0 21 45037 2010-08-23 01:45:17 10.0.x.x RPC_OUT_DATA /rpc/rpcproxy.dll MS1.abc.loc:6001 443 - x.x.185.29 MSRPC 500 0 21 45037 Please open the IIS log, check the sub status code after 500 Am I looking for something after 500? If so, I cannot find any 0 or 21 sub status codes in the link you provided. Note: I am running Windows 2008 Server SP2, so IIS 7. Per my research, it seems that RPC proxy component has corrupted on the CAS server, please re-install it and test the URL again: I will attempt to schedule the necessary downtime as soon as possible, hopefully Saturday morning. Thank you so much for your assistance! By the way, does the information above confirm your idea about a corrupt rpcproxy component?

I had some similar issues with Outlook Anywhere recently. For the 500 error, a simple reboot of our Exchange server fixed that. But for the other issues, we also had a problem with .NET Framework v4.0 and ended up having to uninstall .NET 4.0 and then we applied Server 2008 SP2 and also Exchange 2007 SP3.

Quote: “I cannot find any 0 or 21 sub status codes in the link you provided” 21 is Win32 error code, ERROR_NOT_READY (The device is not ready) Win32 Error Codes Quote: “does the information above confirm your idea about a corrupt rpcproxy component?” I have seen several similar outlook anywhere cases that appears such error (500 0 21) when browse the RPC proxy component. It seems the possible cause is the corrupted rpcproxy.dll fileJames Luo TechNet Subscriber Support (http://technet.microsoft.com/en-us/subscriptions/ms788697.aspx) If you have any feedback on our support, please contact tngfb@microsoft.com

Thank you James - and Frosty. We are planning to implement your recommendations as soon as possible. I will keep you posted.

How's the issue currently?James Luo TechNet Subscriber Support (http://technet.microsoft.com/en-us/subscriptions/ms788697.aspx) If you have any feedback on our support, please contact tngfb@microsoft.com

James, I should be able to attempt the solution you recommended above, this Saturday, 7:00 AM local time. Of course, I will keep you posted. Oh yes! I'll do a full server backup (we are on Windows 2008) of the OS drive and a backup of the database (on separate drive) before the "operation". But this should normally have no effect on the other components, right? I don't want to make things worse.

James, The solution seems to have worked - 3 out of the 4 users in question are now able to use Outlook Anywhere again. I have not been able to contact the 4th user to confirm that they too can once again use Outlook Anywhere (on vacation). As a bonus, it seems to have resolved the OAB problem in my other thread: "Outlook Connectivity- problems receiving email": http://social.technet.microsoft.com/Forums/en-US/exchangesvradmin/thread/51903d4d-c2ae-4c20-b61b-27efaa34a5d4 Thank you for your help! You were right on the mark!

Awesome : )James Luo TechNet Subscriber Support (http://technet.microsoft.com/en-us/subscriptions/ms788697.aspx) If you have any feedback on our support, please contact tngfb@microsoft.com

so would this fix apply to Outlook 2007 (mapi)?

Convert database Exchange OST to Outlook 2011 via OST Recovery software. Our OST Recovery software easily recover OST emails, contacts, calendar, notes, tasks, journal items in a new effective .Fix OST Files and Recover MS Exchange OST to Outlook software automatically import recovered PST file into Microsoft Outlook. This Software quickly repairs 2GB PST file and get back to you with original contents of every email.http://www.convertostpst.com/