TMG 2010 network adapter losing connectivity after application of MS updates for October 2013

Shortly after we applied the Microsoft October 2013 updates to our TMG 2010 SP2 server we started experiencing loss of connectivity on our Internet facing adapter (could not longer ping the gateway etc).  A reboot would resolve the issue.  The problem kept recurring so we removed a couple of the networking related updates for October ( http://support.microsoft.com/kb/2888049 ) and ( http://support.microsoft.com/kb/2882822 ) as a test.  After these were removed the problem stopped.

We inadvertently reapplied these two updates during the November 2013 update cycle and the problem happened again. We removed the updates and everything is back to normal.

Just wondering if anyone else has applied these two updates to their TMG 2010 SP2 server and experienced any unusual issues?

Thanks

November 20th, 2013 4:08am

Hi,

Thanks for your post here.

Do you see any error information on TMG live logging when the problem occurs?

we really need that to help us to analyze issue.

One way to deal with the issue is that you can try to upgrade your TMG to the latest version to see if the problem persists.

Best Regards

Quan Gu


Free Windows Admin Tool Kit Click here and download it now
November 20th, 2013 6:41am

Hi,

Thanks for your post here.

Do you see any error information on TMG live logging when the problem occurs?

we really need that to help us to analyze issue.

One way to deal with the issue is that you can try to upgrade your TMG to the latest version to see if the problem persists.

Best Regards

Quan Gu


November 20th, 2013 6:41am

Hi,

Thanks for your post here.

Do you see any error information on TMG live logging when the problem occurs?

we really need that to help us to analyze issue.

One way to deal with the issue is that you can try to upgrade your TMG to the latest version to see if the problem persists.

Best Regards

Quan Gu


Free Windows Admin Tool Kit Click here and download it now
November 20th, 2013 6:41am

Hi,

Thanks for your post here.

Do you see any error information on TMG live logging when the problem occurs?

we really need that to help us to analyze issue.

One way to deal with the issue is that you can try to upgrade your TMG to the latest version to see if the problem persists.

Best Regards

Quan Gu


November 20th, 2013 9:41am

Hi,

Thanks for your post here.

Do you see any error information on TMG live logging when the problem occurs?

we really need that to help us to analyze issue.

One way to deal with the issue is that you can try to upgrade your TMG to the latest version to see if the problem persists.

Best Regards

Quan Gu


Free Windows Admin Tool Kit Click here and download it now
November 20th, 2013 9:41am

Thanks for your reply Quan Gu. 

Re "upgrade your TMG to the latest version" . As I mentioned the server is running TMG SP2 with rollup 3.   Rollup 4 was only released recently so its a little early to apply that when we have no logical reason to do so.  Nothing out of the ordinary in the logs.

You may have missed the point of my post.  This TMG server has been running without issue for about three years. Shortly after those two Microsoft updates were applied we started experiencing loss of connectivity on our Internet facing adapter.  Removing the updates clearly resolves the issue.  One of those two updates is obviously causing the problem. We're interested in knowing if anyone who has applied those two updates has or hasn't seen the same issue with their network adapters.  So having said that ...

Quan Gu, Do you have personal experience with applying these two updates to a production TMG 2010 SP2 server ?  If so what are your observations with regards to the problem that we describe here?

November 20th, 2013 4:36pm

Hi,

Thanks for your update.

I know what you said. It looks like that something conflict after these updates are installed.

I dont experience this problem.

From my view, TMG loss the connection toward internet. There must be some information on TMG.

We need to know how do these updates impact on TMG and we need to know if the issue is only related to your environment.

If the issue is not special case, we intend to submit it to MS support.

Best Regards

Quan Gu  

Free Windows Admin Tool Kit Click here and download it now
November 21st, 2013 6:01am

Hi there,

Yes we are experiencing the same problems since October, loosing connectivity on the external adapter.

We thought that might be a physical problem with the adapter, the server it's a DELL PowerEdge 710, but now I don't think it's hardware related.

I've just uninstalled the updates you mentioned and waiting for results in the next days. We did not passed more than 4 days lately without a network loss with the TMG server.

Thanks

Adrian

November 25th, 2013 1:33am

Hello,

We are experiencing the same issue on TMG2010-SP2-CU3 or CU4. Sporadically losing WAN connectivity and TMG starts flooding DNS servers.
The conflicting update is KB2888049. After removal things gets back to normal.

Best Regards,

Free Windows Admin Tool Kit Click here and download it now
November 25th, 2013 1:36pm

Hi,

Thank you for your patience and support.

I am trying to involve someone familiar with this topic to further look at this issue. There might be some time delay. Appreciate your patience.

Thank you for your understanding and support.

Best Regards

Quan Gu

November 26th, 2013 6:15am

Hello all,

I am experience the same issue.

Especialy when TMG 2010 with latest CU and updates have L2tp clients. And in parrallel there are permanent one IPSEC tunnel and one PPTP client. 

I hope you should anounce this issue somehow to public, becouse i spend a lot of time to find out whats the problem, especialy in virtual enviroment.

Best Regards

Ievgen

Free Windows Admin Tool Kit Click here and download it now
November 26th, 2013 12:17pm

We have had at least one other report opened with similar symptoms. There is currently a support case opened with us. We do know this has to do with MTU size. There is a hotfix that updates TCPIP.SYS to a newer version. It is here.

http://support.microsoft.com/kb/2896146/EN-US

Can you install the problem update, then install this latest one and let us know if that resolves the issue? If it does not, we will need to open a bug with our Windows Networking team.

Regards,

Keith

December 2nd, 2013 2:44pm

We have had at least one other report opened with similar symptoms. There is currently a support case opened with us. We do know this has to do with MTU size. There is a hotfix that updates TCPIP.SYS to a newer version. It is here.

http://support.microsoft.com/kb/2896146/EN-US

Can you install the problem update, then install this latest one and let us know if that resolves the issue? If it does not, we will need to open a bug with our Windows Networking team.

Regards,

Keith

Free Windows Admin Tool Kit Click here and download it now
December 2nd, 2013 2:44pm

We have had at least one other report opened with similar symptoms. There is currently a support case opened with us. We do know this has to do with MTU size. There is a hotfix that updates TCPIP.SYS to a newer version. It is here.

http://support.microsoft.com/kb/2896146/EN-US

Can you install the problem update, then install this latest one and let us know if that resolves the issue? If it does not, we will need to open a bug with our Windows Networking team.

Regards,

Keith

December 2nd, 2013 2:44pm

We have had at least one other report opened with similar symptoms. There is currently a support case opened with us. We do know this has to do with MTU size. There is a hotfix that updates TCPIP.SYS to a newer version. It is here.

http://support.microsoft.com/kb/2896146/EN-US

Can you install the problem update, then install this latest one and let us know if that resolves the issue? If it does not, we will need to open a bug with our Windows Networking team.

Regards,

Keith

Free Windows Admin Tool Kit Click here and download it now
December 2nd, 2013 5:44pm

We have had at least one other report opened with similar symptoms. There is currently a support case opened with us. We do know this has to do with MTU size. There is a hotfix that updates TCPIP.SYS to a newer version. It is here.

http://support.microsoft.com/kb/2896146/EN-US

Can you install the problem update, then install this latest one and let us know if that resolves the issue? If it does not, we will need to open a bug with our Windows Networking team.

Regards,

Keith

December 2nd, 2013 5:44pm

@ Phillip Windell

You may want to review the rest of the thread.  Others are reporting the same issue. I narrowed the issue down to two MS updates, someone was kind enough to narrow it down to one particular update.  It appears that this has been reported to Microsoft and we're talking specifically about an MS update affecting TMG2010 - no one is concerned about an old issue affecting ISA2006.

Free Windows Admin Tool Kit Click here and download it now
December 4th, 2013 11:10pm

Keith.  Do you know for a fact that the TCPIP.SYS hotfix resolves the issue I and others have reported here with loss of connectivity on a TMG network adapter after the application of an October hotfix ??  or is this speculation?
December 20th, 2013 5:41pm

We have the exact same problem after applying the october updates. We are still running rollup 3. I have installed the hotfix http://support.microsoft.com/kb/2896146/EN-US. Will let you know if this resolves the problem.
Free Windows Admin Tool Kit Click here and download it now
January 9th, 2014 9:56am

I'm sorry to say that our forefront tmg 2010 has lost connectivity again. So the hotfix does not fix the problem. Tonight i'll uninstall the hotfix and october updates.
January 9th, 2014 12:01pm

After applying suggested update KB2896146 on TMG2010-SP2-CU4 the same issue is back as per KB2888049. Update removal set things back to normal.

Best Regards,

Free Windows Admin Tool Kit Click here and download it now
January 10th, 2014 12:29pm

We've been having the same problem here and been searching for a solution since october. In our case not only TMG can no longer communicate with WAN but also LAN (event IDs saying can no longer connect to local DC for authentication and also time services sync fails). We'll uninstall the two KBs tonight and will let you guys know.

January 22nd, 2014 9:42am

El Wayno. Let us know how it works out.

Please be aware that other hotfixes may cause the same issue since they update the same modules that caused the original problem. e.g. the more recent KB2913431 . That HF caused the same problem for us and we had to remove it.

 

Free Windows Admin Tool Kit Click here and download it now
January 22nd, 2014 1:43pm

We had exactly the same issue too. TMG stable for years, then we ran a full windows update.

Big mistake. On removing the hotfixes listed in this thread, it stabilized TMG.

Any solution yet?

February 20th, 2014 2:19pm

Exactly the same condition as described here. TMG is at SP2 RU4.
Free Windows Admin Tool Kit Click here and download it now
February 28th, 2014 10:27am

We have exactly the same Problem:

SP2 RU4

Uninstalling kb2882822, kb2888049 and kb2913431 resolves the Problem.

Is there any other solution (update fr TMG2010 ?)

April 9th, 2014 7:49am

It's good to see that i'm not the only one:

http://social.technet.microsoft.com/Forums/en-US/Forefrontedgegeneral/thread/fbb0a2cb-e79b-4741-befa-7265b0f18665/#3432e291-b991-4259-bb36-66754933bebf

I am running TMG in a hyper-v environment. I have a hunch that maybe there is a problem with the NIC drivers on the host.

Out of curiosity, how many of you are using Broadcom nics with driver version 7.8.6.0/7.8.21.0 ?

Regards,

Henning


  • Edited by dtscaps Tuesday, April 15, 2014 9:20 AM
Free Windows Admin Tool Kit Click here and download it now
April 15th, 2014 9:11am

It's good to see that i'm not the only one:

http://social.technet.microsoft.com/Forums/en-US/Forefrontedgegeneral/thread/fbb0a2cb-e79b-4741-befa-7265b0f18665/#3432e291-b991-4259-bb36-66754933bebf

I am running TMG in a hyper-v environment. I have a hunch that maybe there is a problem with the NIC drivers on the host.

Out of curiosity, how many of you are using Broadcom nics with driver version 7.8.6.0/7.8.21.0 ?

Regards,

Henning


  • Edited by dtscaps Tuesday, April 15, 2014 9:20 AM
April 15th, 2014 9:11am

It's good to see that i'm not the only one:

http://social.technet.microsoft.com/Forums/en-US/Forefrontedgegeneral/thread/fbb0a2cb-e79b-4741-befa-7265b0f18665/#3432e291-b991-4259-bb36-66754933bebf

I am running TMG in a hyper-v environment. I have a hunch that maybe there is a problem with the NIC drivers on the host.

Out of curiosity, how many of you are using Broadcom nics with driver version 7.8.6.0/7.8.21.0 ?

Regards,

Henning


  • Edited by dtscaps Tuesday, April 15, 2014 9:20 AM
Free Windows Admin Tool Kit Click here and download it now
April 15th, 2014 9:11am

I'm the person that started this thread.

Still no resolution but just to summarize the issue and findings since my original post...

We're running TMG 2010 SP2 with rollup 4 on Windows 2008 R2.

The problematic updates to date are specifically ( http://support.microsoft.com/kb/2888049 ) and ( http://support.microsoft.com/kb/2882822 ). We also noted that after we installed IE11 on the machine it reinstalled at least one of these two updates which of course caused the problem to reoccur.  So for now we have these three updates hidden in Windows Updates in Control Panel so that we do not reinstall them by accident. Of course a future MS update that contains the same modules may very well cause the same issue with loss of network connectivity on one of the adapters. 

We also note on two Windows 2008 R2 Hyper-V hosts (no TMG involved) that we occasionally lose connectivity on one of the network adapters on those machines.  Not sure if there is any relation but we're still investigating.

We also note that while several others in this thread report the exact same issue, Microsoft reps disappeared from this thread long ago.  

April 18th, 2014 8:27pm

Hyper-V hosts (no TMG involved) that we occasionally lose connectivity on one of the network adapters on those machines.  Not sure if there is any relation but we're still investigating.

I had the exact same issue on a standalone host, hyper-V but no TMG involved. In fact this is what makes me believe this is a NIC driver related issue.
My exact same problem was resolved by upgrading NIC drivers. NICs in question is a HP Broadcom 1Gb Multifunction NIC. The same NIC is widely used by both HP and DELL.

NIC drivers with the flaw:7.8.6.0/7.8.21.0.

The new driver that corrects the issue:
"This driver corrects an issue that could result in halted traffic when
configuring jumbo frame size to 9000" (a bit misleading)

VBD driver version 7.8.50.0
NDIS 6.0 driver version 7.8.52.0.

The same kind of NICs are involved on the Hyper-V host running the TMG vm in question. I have upgraded the nic drivers but have not reinstalled the windows updates due to the fact that I'm not physically present until after easter.
I come back with more info then.

Could you please check what NICs are involved on your impacted hosts ?

Regards,
Henning






  • Edited by dtscaps Saturday, April 19, 2014 7:01 AM
Free Windows Admin Tool Kit Click here and download it now
April 19th, 2014 6:20am

Hyper-V hosts (no TMG involved) that we occasionally lose connectivity on one of the network adapters on those machines.  Not sure if there is any relation but we're still investigating.

I had the exact same issue on a standalone host, hyper-V but no TMG involved. In fact this is what makes me believe this is a NIC driver related issue.
My exact same problem was resolved by upgrading NIC drivers. NICs in question is a HP Broadcom 1Gb Multifunction NIC. The same NIC is widely used by both HP and DELL.

NIC drivers with the flaw:7.8.6.0/7.8.21.0.

The new driver that corrects the issue:
"This driver corrects an issue that could result in halted traffic when
configuring jumbo frame size to 9000" (a bit misleading)

VBD driver version 7.8.50.0
NDIS 6.0 driver version 7.8.52.0.

The same kind of NICs are involved on the Hyper-V host running the TMG vm in question. I have upgraded the nic drivers but have not reinstalled the windows updates due to the fact that I'm not physically present until after easter.
I come back with more info then.

Could you please check what NICs are involved on your impacted hosts ?

Regards,
Henning






  • Edited by dtscaps Saturday, April 19, 2014 7:01 AM
April 19th, 2014 6:20am

Hyper-V hosts (no TMG involved) that we occasionally lose connectivity on one of the network adapters on those machines.  Not sure if there is any relation but we're still investigating.

I had the exact same issue on a standalone host, hyper-V but no TMG involved. In fact this is what makes me believe this is a NIC driver related issue.
My exact same problem was resolved by upgrading NIC drivers. NICs in question is a HP Broadcom 1Gb Multifunction NIC. The same NIC is widely used by both HP and DELL.

NIC drivers with the flaw:7.8.6.0/7.8.21.0.

The new driver that corrects the issue:
"This driver corrects an issue that could result in halted traffic when
configuring jumbo frame size to 9000" (a bit misleading)

VBD driver version 7.8.50.0
NDIS 6.0 driver version 7.8.52.0.

The same kind of NICs are involved on the Hyper-V host running the TMG vm in question. I have upgraded the nic drivers but have not reinstalled the windows updates due to the fact that I'm not physically present until after easter.
I come back with more info then.

Could you please check what NICs are involved on your impacted hosts ?

Regards,
Henning






  • Edited by dtscaps Saturday, April 19, 2014 7:01 AM
Free Windows Admin Tool Kit Click here and download it now
April 19th, 2014 6:20am



Could you please check what NICs are involved on your impacted hosts ?

Regards,
Henning






Not Broadcom. The adapters are an Intel 82678DM and a Realtek RTL8139. 

I believe that the specific module in the MS updates that causes the issue is netio.sys.

April 19th, 2014 1:01pm

Our HyperV Host is an HP Server DL 380 G7

In my Case the external Interface is an HP NC360T I think it is an Intel-Chip (e1e6232e.sys   Driver Version:9.15.17.0) Jumbo packets are deactivated.

We're running TMG 2010 SP2 with rollup 4 on Windows 2008 R2. (The Updates kb2882822, kb2888049 and kb2913431 are deactivated)

Free Windows Admin Tool Kit Click here and download it now
April 22nd, 2014 6:03am

Thanks for info !

sime3000, I saw you also posted your findings in the TMG team blog back in January. Still nothing from MS anywhere?

April 22nd, 2014 1:00pm

dtscaps,

No, nothing at all from Microsoft.  I also posted the issue in the isaserver.org forums recently.

Back in November Quan Gu   (MSFT CSG) posted here that "If the issue is not special case, we intend to submit it to MS support"

Quan Gu  ?  Microsoft ?   Can you offer any assistance with this issue that several people are reporting?

             


  • Edited by sime3000 Tuesday, April 22, 2014 4:56 PM
Free Windows Admin Tool Kit Click here and download it now
April 22nd, 2014 4:50pm

dtscaps,

No, nothing at all from Microsoft.  I also posted the issue in the isaserver.org forums recently.

Back in November Quan Gu   (MSFT CSG) posted here that "If the issue is not special case, we intend to submit it to MS support"

Quan Gu  ?  Microsoft ?   Can you offer any assistance with this issue that several people are reporting?

             


  • Edited by sime3000 Tuesday, April 22, 2014 4:56 PM
April 22nd, 2014 4:50pm

dtscaps,

No, nothing at all from Microsoft.  I also posted the issue in the isaserver.org forums recently.

Back in November Quan Gu   (MSFT CSG) posted here that "If the issue is not special case, we intend to submit it to MS support"

Quan Gu  ?  Microsoft ?   Can you offer any assistance with this issue that several people are reporting?

             


  • Edited by sime3000 Tuesday, April 22, 2014 4:56 PM
Free Windows Admin Tool Kit Click here and download it now
April 22nd, 2014 4:50pm

hi, we have the some problems.

I opened this thread in the German forum some time ago:

http://social.technet.microsoft.com/Forums/de-DE/66f1c5d6-b0d5-4e77-92a6-1971e64b8ec9/tmg2010probleme-nach-windows-updates-oktober-2013?prof=required

I also stopped updates on TMG since October and everything was working fine.

but since last week the problem occurs 2 times without installed updates on the TMG-server (2008r2)

have you already installed the 3 affected updates on your host systems?

  • KB2882822
  • KB2888049
  • KB2913431

May 12th, 2014 2:58pm

Can someone confirm if RU5 fixes this ?
Free Windows Admin Tool Kit Click here and download it now
August 30th, 2014 7:37am

I can reply to my own question.

IT DOES NOT FIX IT !!

September 1st, 2014 12:06pm

Just ran into this problem today.

None of the three patches mentioned above where installed, most likely they have been superseded by a newer fix having this problem.

It seems to be load related as I have one array that runs happily on this configuration but others aren't but it is not as heavily utilized as the others. As it is affecting business I did not have time to do trial and error but had to revert to snapshots taken before the updating began.

Systems had WU up until yesterday (not october 2014) applied and RU5 afterwards but still experienced the issue. Systems installed in VMWare running NLB. Previously the systems have been able to cope with the load just fine on one node but after WU they where not.

Anyone found any other update contributing to this?

Free Windows Admin Tool Kit Click here and download it now
October 15th, 2014 3:09pm

Can't remember which KB from the top of my head. Just search for netio.sys. This file has to be from August 2013 for it to work. Any KB after that replacing netio.sys makes it crash.

This problem manifests itself specifically so that you cannot reach default gateway, hence no internet connection.

Henning





  • Edited by dtscaps Thursday, October 16, 2014 3:56 PM
October 15th, 2014 3:15pm

Can't remember which KB from the top of my head. Just search for netio.sys. This file has to be from August 2013 for it to work. Any KB after that replacing netio.sys makes it crash.

This problem manifests itself specifically so that you cannot reach default gateway, hence no internet connection.

Henning





  • Edited by dtscaps Thursday, October 16, 2014 3:56 PM
Free Windows Admin Tool Kit Click here and download it now
October 15th, 2014 3:15pm

Can't remember which KB from the top of my head. Just search for netio.sys. This file has to be from August 2013 for it to work. Any KB after that replacing netio.sys makes it crash.

This problem manifests itself specifically so that you cannot reach default gateway, hence no internet connection.

Henning





  • Edited by dtscaps Thursday, October 16, 2014 3:56 PM
October 15th, 2014 3:15pm

having gone through the list of updates for Server 2008 R2 I believe the august 2013 update in question, which provides the latest working copy of netio.sys for use with TMG 2010 will be from this windows update:

http://support2.microsoft.com/default.aspx?kbid=2868623

that's KB 2868623

I am yet to verify and test this as I have a suspected related issue which may be being masked by a large array, figured it would help someone else down the line from having to trawl through the list of updates

Free Windows Admin Tool Kit Click here and download it now
November 28th, 2014 4:54pm

I'm the person that started this thread and I have an update.

We added a new NIC to our TMG server and used it to replace the NIC that was having the issues that are described in this thread. 

The MS updates that we re causing problems for us with the old NIC are not an issue with the new NIC.

November 28th, 2014 5:09pm

I'm the person that started this thread and I have an update.

We added a new NIC to our TMG server and used it to replace the NIC that was having the issues that are described in this thread. 

The MS updates that we re causing problems for us with the old NIC are not an issue with the new NIC.

Free Windows Admin Tool Kit Click here and download it now
November 28th, 2014 5:09pm

I'm the person that started this thread and I have an update.

We added a new NIC to our TMG server and used it to replace the NIC that was having the issues that are described in this thread. 

The MS updates that we re causing problems for us with the old NIC are not an issue with the new NIC.

November 28th, 2014 5:09pm

Well,

I have around 5 deployments of TMGs. All but one is OK, a mix of physical and VMs under Hyper-V.

The one having the specific problem is a VM on a 2008 R2 Hyper-V host with Broadcom adapters, latest drivers.

TMG VM running synthtetic adapters.

Henning






  • Edited by dtscaps Tuesday, December 30, 2014 7:14 PM
Free Windows Admin Tool Kit Click here and download it now
December 30th, 2014 7:09pm

Well,

I have around 5 deployments of TMGs. All but one is OK, a mix of physical and VMs under Hyper-V.

The one having the specific problem is a VM on a 2008 R2 Hyper-V host with Broadcom adapters, latest drivers.

TMG VM running synthtetic adapters.

Henning






  • Edited by dtscaps Tuesday, December 30, 2014 7:14 PM
December 30th, 2014 7:09pm

Well,

I have around 5 deployments of TMGs. All but one is OK, a mix of physical and VMs under Hyper-V.

The one having the specific problem is a VM on a 2008 R2 Hyper-V host with Broadcom adapters, latest drivers.

TMG VM running synthtetic adapters.

Henning






  • Edited by dtscaps Tuesday, December 30, 2014 7:14 PM
Free Windows Admin Tool Kit Click here and download it now
December 30th, 2014 7:09pm

Netio.sys  version 6.1.7601.22177 or above.

It was interesting to see so many issues with KB2888049 and beyond.  Unfortunately it was a black eye for TMG even though its an operating system issue.

To add to this, our symptoms were similar but different.  We have tracked a memory leak that also surfaced from this patch.  We discovered if we patched a server that leveraged ipsec l2tp client vpn's it would slowly consume all memory non paged pool until it ran too low on resources to operate resulting in weekly reboots.

I read in this chain that replacing nic's might influence behavior.  But that didn't help us.  The symptoms can also be repro'ed in physical and virtual nics.

We have been working with Microsoft and their support is excellent.

 

July 13th, 2015 12:38pm

OK, and how you solved the problems?
Free Windows Admin Tool Kit Click here and download it now
July 16th, 2015 3:43am

Guys, does the restart resolve your issues? Cos in my case, restart is not a solution. I will have to uninstall Rollups and reinstall them until TMG starts working again.
August 10th, 2015 5:46am

guys, any updates or conclusions on this?

my TMG is behaving like this from since last week. In my case, it was OK until I activated VPN connectivity.

IF I disable VPN, thinks are OK again...

Tried to patch to latest RU 5 but problem continues

Free Windows Admin Tool Kit Click here and download it now
August 10th, 2015 6:27am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics