Problems with TMG MRS Service

Is anyone experiencing any issues with the TMG MRS Service today? We're getting a lot of time out/errors causing a lot of sites to show up as unknown. These are the errors we've been seeing since about 9am this morning (UTC):

The failure is due to error: The remote endpoint was not reachable.

The failure is due to error: The remote endpoint is unable to process the
request due to being overloaded.

March 2nd, 2015 1:35pm

Hi,

You could try to follow the steps in the article to troubleshoot this issue.

URL filtering troubleshooting flow

https://technet.microsoft.com/en-us/library/ff358603.aspx?f=255&MSPPError=-2147217396

Best Regards,

Joyce

Free Windows Admin Tool Kit Click here and download it now
March 3rd, 2015 5:39am

Hi Joyce, I can see successful connections through the web proxy. I get as far as Troubleshoot WinHTTP on the flow chart. One issue that I can see in the logs is that it always uses: 10.ds.mrs.microsoft.com. As far as I can see this only supports SSL which we don't support. Is there any way to ask Microsoft to enable TLS for this - we've had to disable support for SSL as part of the POODLE issue? I can connect to 10.ts.mrs.microsoft.com which as far as I can see uses TLS/a different version of IIS. I've tried removing 10.ds.mrs.microsoft.com from the "Microsoft Reputation Service Sites" Domain Name Set but it doesn't seem to make a difference - it still only uses 10.ds.mrs.microsoft.com - I'm not sure if there's any way to force it to use: 10.ts.mrs.microsoft.com? Can I just update my host file to repoint 10.ds.mrs.microsoft.com to the IP for 10.ts.mrs.microsoft.com - does this change often/do they provide the same functionality?
March 3rd, 2015 10:14am

Hi,

What's the error when you access 10.ts.mrs.microsoft.com?

As the blog below indicates, URL Filtering is a cloud based service and must be able to successfully establish an HTTPS connection to the either of the MRS (Microsoft Reputation Service) sites(https://10.ds.mrs.microsoft.com and https://10.ts.mrs.microsoft.com).

TMG URL Filtering fails

Best Regards,

Joyce

 

Free Windows Admin Tool Kit Click here and download it now
March 4th, 2015 2:57am

Hi Joyce, I don't get any errors for connecting to: https://10.ts.mrs.microsoft.com - however doing a Log search in TMG for anything where the URL contains mrs.microsoft.com I can only see 10.ds.mrs.microsoft.com.
March 4th, 2015 10:16am

I noticed today that I'm able to connect to 10.ds.mrs.microsoft.com with TLS. I also tried a few random queries and they seem to return OK. I'll try expanding it out to see if we get any more errors. I'm still not sure why we don't seem to be using 10.ts.mrs.microsoft.com.

Free Windows Admin Tool Kit Click here and download it now
March 4th, 2015 10:28am

Both MRS sites are working now for us - when I accessed ds.mrs the other day it was a tiled/metro IIS landing page I hadn't seen before; now its IIS 7?

March 4th, 2015 10:35am

The problem started February 22nd. Since this have been reported by a few users in the forums, then I wonder if Microsoft did any changes to any servers in the MRS cloud service? It doesn't seem to be a permanent problem but only occurs from time to time on daily basis.

Description:

An error occurred while trying to communicate with the Microsoft Reputation Service server. If this Forefront TMG server is chained to an upstream server, verify that the WinHTTP proxy is set to localhost. If this issue persists, check that Internet connectivity is available.

The failure is due to error: The remote endpoint was not reachable.

Description:

An error occurred while trying to communicate with the Microsoft Reputation Service server. If this Forefront TMG server is chained to an upstream server, verify that the WinHTTP proxy is set to localhost. If this issue persists, check that Internet connectivity is available.

The failure is due to error: The remote endpoint is unable to process the request due to being overloaded.

Free Windows Admin Tool Kit Click here and download it now
March 4th, 2015 10:36am

Yes, we've been seeing the same since the 23rd Feb, but its increased in frequency since 01 March...4 so far today, 11 yesterday, 15 the day before.  Our errors are either:

The remote endpoint is unable to process the request due to being overloaded

The operation did not complete within the time allotted

Security verification was not successful for the received data

March 4th, 2015 12:09pm

Anyone able to access https://10.ds.mrs.microsoft.com/ in a browser?
Free Windows Admin Tool Kit Click here and download it now
March 4th, 2015 2:27pm

Nope, we can't (but we could a coupe of days ago). We get the IE message...

This page cant be displayed

Turn on TLS 1.0, TLS 1.1, and TLS 1.2 in Advanced settings

We have SSL disabled for POODLE, and have done for ages, and TLS is turned on in the browser (IE11)

https://10.ts.mrs.microsoft.com/ is fine though - goes to IIS holding page.

We're up to 11 timeouts today now.

March 4th, 2015 2:56pm

https://10.ds.mrs.microsoft.com is back to the tiled/metro landing page again...its was IIS7 landing page earlier.

Free Windows Admin Tool Kit Click here and download it now
March 4th, 2015 5:09pm

I'm afraid it's also still breaking for us. I'm not sure how Microsoft do the DNS but looking through the logs it resolves to quite a few different IPs:

65.55.74.113
65.55.74.114
65.55.222.50
65.55.222.51
94.254.112.71
94.254.112.72

This is for the UK so I guess if you are somewhere you may be getting different IPs.

Some of this have what I guess is the IIS 8.5 landing page and some have IIS 7. Some also seem to time out completely. However looking through the proxy logs 65.55.74.113 is the one that logs the time outs. I suspect TMG wouldn't log it as an error in the proxy logs however if the node responded that it was busy.

March 5th, 2015 10:20am

We're still suffering TMG errors and timeouts.  Having dived a little deeper, we are seeing DNS timeouts for lookups of the MRS servers, then successes immediately afterwards?  Our web browsing is, correspondingly, very slow - obviously during the MRS DNS failures and then a website loads once a MRS DNS response is resolved.

I can see that similar MRS DNS issues have occurred in the past and its been traced to MRS European IP's and Microsoft - anyone out there from Microsoft??

Free Windows Admin Tool Kit Click here and download it now
March 9th, 2015 9:01am

We are also seeing DNS timeouts. I thought maybe the DNS issue was a separate problem but as the TTL on the records is only 10 minutes it can cause the intermittent issues we are seeing.
  • Edited by PanosE 16 hours 53 minutes ago
March 9th, 2015 10:40am

Any update on this, we started seeing problems last week.
Free Windows Admin Tool Kit Click here and download it now
March 9th, 2015 11:43am

We are also seeing DNS timeouts. I thought maybe the DNS issue was a separate problem but as the TTL on the records is only 10 minutes it can cause the intermittent issues we are seeing.
  • Edited by PanosE Monday, March 09, 2015 2:39 PM
March 9th, 2015 2:39pm

Our ISP has given us some new DNS servers to resolve against, apparently the ones we were using were "legacy".  Its improved, but we are still seeing TMG communication errors to MRS.  Since we changed our DNS forwarders, the errors have exclusively been "Security verification was not successful for the
received data
".

Free Windows Admin Tool Kit Click here and download it now
March 11th, 2015 5:34am

Still seeing errors against 10.ds.mrs.microsoft.com - the MRS site with the IIS8.5 holding page.  Its slow to load in a browser, we get occasional, but regular, nslookup failures against it.  We have been unable to trap any further details in TMG Diagnostic Logging, but a Connectivity Test with detailed pathping gave us the following:

Time reported by the Microsoft Forefront TMG Firewall Service: 4.052 seconds
Testing https://10.ds.mrs.microsoft.com:443
Category: Connectivity error
Error details: 64 - The specified network name is no longer available.

whereas the same test for 10.ts.mrs.microsoft.com gives us...

Time reported by the Microsoft Forefront TMG Firewall Service: 0.273 seconds
HTTP response: 200 OK.
The test successfully completed for this URL.

It really looks like a DNS resolution issue against the 10.ds.mrs.microsoft.com MRS server (94.245.112.72 in the UK).  Changing our ISP DNS forwarders hasn't helped, and we don't see DNS failures for other queries.

We're going to try switching the order of the MRS servers in the TMG "Microsoft Reputation Service Sites" object, within the Domain Name Sets group....

March 11th, 2015 8:45am

Nope, the dialog just orders the entries alphabetically!  We've taken the 10.ds.mrs.microsoft.com server out of this object and are just using the "ts" entry - lets see if it will run OK with just one MRS server?
Free Windows Admin Tool Kit Click here and download it now
March 11th, 2015 8:47am

No change - we are still getting TMG errors for "Security verification was not successful for the
received data
".  Web browsing for users is generally faster though (using the one MRS server entry).

Have reverted back to standard (two) server MRS server configuration.

March 11th, 2015 11:35am

https://10.ds.mrs.microsoft.com has gone back to a IIS7 holding page?  Are we alone with our TMG DNS resolution errors, or are others still having the same issue??
Free Windows Admin Tool Kit Click here and download it now
March 11th, 2015 1:37pm

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics