System Center 2012 R2 Endpoint Protection Hanging on full scans

Since Monday we multiple machines trying to run a full scan. The scan will get so far and hang up on random files. I have watched the scans and when it hangs it shows the file and the file count stops. When this happens we get the following every 30 seconds to a minute in the system log "A timeout (30000 milliseconds) was reached while waiting for a transaction response from the MsMpSvc service." If you try to do anything with Endpoint Protection after this point it freezes up" You end up having to reboot the computer. Since it never completes the full scan it starts over and does the same thing after logging in.

Our computer turn on early in the morning so some of the computers start running this scan. By the time the users come to work and tries to log in the login just sets and spins. You have to physically turn the computer off and back on in order to log in.

Is any one else having these problems?

August 25th, 2015 11:13am

We are also having the exact same issue describe above plus some other issues. Some machines will sit at the starting windows screen forever, others will sit at the welcome screen.
Free Windows Admin Tool Kit Click here and download it now
August 25th, 2015 3:39pm

As an update in event viewer there are a ton of errors of the following:

"A timeout (30000 millieseconds) was reached while waiting for a transaction response from the MsMpSvc Service." Event 7011.

This starts on August 23rd at 2:10am. It grabbed the following:

"Current Signature Version: 1.205.188.0"

"Current Engine Version "1.1.1200.2"

This appears to be 100% bad definition update.

August 25th, 2015 4:17pm

Here is the message about AV updating:

Free Windows Admin Tool Kit Click here and download it now
August 25th, 2015 5:05pm

And it breaking:

August 25th, 2015 5:06pm

We are also having the exact same issue by one of our customers since Monday.

AV-Def: 1.205.320.0
Spy-Def: 1.205.320.0

Also, latest MS-Updates (last week) are installed.

Free Windows Admin Tool Kit Click here and download it now
August 26th, 2015 5:00am

I have opened a ticket with Microsoft. I have also turned off all scheduled scans until everything gets back to normal. I will post what Microsoft says as soon as I know something but I have yet to here from them.
August 26th, 2015 7:10am

many thanks! I'm curious about your findings.
Free Windows Admin Tool Kit Click here and download it now
August 26th, 2015 7:12am

I have reported the same issue to Premier support. Awaiting their response.
August 26th, 2015 9:24am

Thanks for sharing. We are experiencing the same symptoms as well. Please share your findings after working with Microsoft Support.
Free Windows Admin Tool Kit Click here and download it now
August 26th, 2015 12:21pm

We are having the same issue with one of our file servers, we have had to restart the server once a day to keep it going. I have changed the real-time settings to only Monitor incoming files rather than all files and the CPU usage has stayed at a constant low now, I will also be logging a call with MS Premier support

August 26th, 2015 7:15pm

Same problem for us, started appearing last Saturday (22/08/2015) morning, we run a full scan every Saturday at 02:00.

Agent details of 1 of our affected machines:-

Antimalware Client Version: 4.5.216.0
Engine Version: 1.1.12002.0
Antivirus definition: 1.205.659.0
Antispyware definition: 1.205.659.0
Network Inspection System Engine Version: 2.1.11804.0
Network Inspection System Definition Version: 115.3.0.0

Free Windows Admin Tool Kit Click here and download it now
August 27th, 2015 1:39am

We also are having the error: "A timeout (30000 millieseconds) was reached while waiting for a transaction response from the MsMpSvc Service." It happened since the beginning of this week.

I thought reinstalling the sccm client was the solution but it helps for only 1 day. The next day the problem is back. We don't do scheduled full scans so I have no clue what triggers the error. I hope there will be a fix very soon.

August 27th, 2015 4:37am

My company is also experiencing issues, I've had to fix three PCs already today. Users get stuck at the 'Welcome' logging on screen for at least 15-20 minutes. Any access to network resources appears extremely delayed, non responsive even.

I'm having to use a third party tool called RunAsTI which runs as command prompt as TrustedInstaller, from there I run Process Explorer, from the services tab of the MsMpEng process I can stop the process from running (eventually!). After this I can uninstall the Endpoint Protection and the System Center client. Reboot then reinstall System Center client, which in turn installs the AV.

I've also noticed when I manage to forcibly close the MsMpEng process I get a failed definition update event log:

Installation Failure: Windows failed to install the following update with error 0x80246007: Definition Update for Microsoft Endpoint Protection - KB2461484 (Definition 1.205.681.0).

After reinstalling System Center this update installs with no problems and the PC itself seems happy again. I don't plan on doing this for 500 PCs so it would be good if Microsoft could fix this issue.

Free Windows Admin Tool Kit Click here and download it now
August 27th, 2015 7:46am

I'm so glad I'm not the only one having this problem.

We are experiencing this problem as well and have no solutions up until now.

Waiting for the response of microsoft support as well.

August 27th, 2015 8:35am

It appears that KB3087985 is the culprit.  I removed it from 3 machines and the MsMpSvc Service timeout errors no longer appear.  Scans are running normally.  My short term plan is to change scan policies and turn off scans entirely and hope for a quick resolution from Microsoft, rather than remove the security update.

Free Windows Admin Tool Kit Click here and download it now
August 27th, 2015 8:43am

It appears that KB3087985 is the culprit.  I removed it from 3 machines and the MsMpSvc Service timeout errors no longer appear.  Scans are running normally.  My short term plan is to change scan policies and turn off scans entirely and hope for a quick resolution from Microsoft, rather than remove the security update.

Did you make sure your AV signatures did not update on reboot? We had that update pushed out on the 20th and no issues until the 23rd right after a AV update.  

August 27th, 2015 9:22am

It appears that KB3087985 is the culprit.  I removed it from 3 machines and the MsMpSvc Service timeout errors no longer appear.  Scans are running normally.  My short term plan is to change scan policies and turn off scans entirely and hope for a quick resolution from Microsoft, rather than remove the security update.


well, in my case I don't have this update installed on the affected server, I also rather think it has something to do with the SCEP version, at first my version was the buggy one (4.8.204.0) but in this thread there seems to be another versions as well...
Free Windows Admin Tool Kit Click here and download it now
August 27th, 2015 9:28am

We are using anti-malware version 4.7.209.0.

So far the following 3 versions have been listed here as having issues: 

-4.7.209.0

-4.8.204.0

-4.5.216.0

August 27th, 2015 9:53am

interessting, that makes me think that it couldn't be the client version rather than other components?

One server who had this problem has these SCEP Definition

maybe someone else can recognize some similarities of any versions?


  • Edited by DTICT 17 hours 22 minutes ago
Free Windows Admin Tool Kit Click here and download it now
August 27th, 2015 10:01am

nevermind, I just now realized there are information already posted in this topic. I think it has to be the engine version 1.1.12002.0

I also restored from last week and it was on 1.1.11903.0

now the question of how to downgrade the engine version....

August 27th, 2015 10:34am

I spoke too soon on this one.  Knowing that KB3087985 was the last patch that was deployed to our environment I looked at it as a possibility, but this morning the SCEP problem is back. 

A scan kicked off on my workstation and about an hour later I was getting timeout errors again.  This was preceded by a lot of dynamic signature update retrievals, which were preceded by dynamic signature update deletions, all of which have the same odd timestamp of 1/1/1601 12:02:24 AM. 

Free Windows Admin Tool Kit Click here and download it now
August 27th, 2015 10:40am

We have switched to quick scan as a workaround. This seems to work well while we investigate.

I am not sure if this is a coincidence, but the scan stops in \windows\winsxs\backup on several systems i have checked.

Edit: it stops on files in other folders as well, so it must be a coinci

August 27th, 2015 12:11pm

interessting, that makes me think that it couldn't be the client version rather than other components?

One server who had this problem has these SCEP Definition

maybe someone else can recognize some similarities of any versions?


  • Edited by DTICT Thursday, August 27, 2015 2:02 PM
Free Windows Admin Tool Kit Click here and download it now
August 27th, 2015 2:01pm

interessting, that makes me think that it couldn't be the client version rather than other components?

One server who had this problem has these SCEP Definition

maybe someone else can recognize some similarities of any versions?


  • Edited by DTICT Thursday, August 27, 2015 2:02 PM
August 27th, 2015 2:01pm

interessting, that makes me think that it couldn't be the client version rather than other components?

One server who had this problem has these SCEP Definition

maybe someone else can recognize some similarities of any versions?


  • Edited by DTICT Thursday, August 27, 2015 2:02 PM
Free Windows Admin Tool Kit Click here and download it now
August 27th, 2015 2:01pm

interessting, that makes me think that it couldn't be the client version rather than other components?

One server who had this problem has these SCEP Definition

maybe someone else can recognize some similarities of any versions?


  • Edited by DTICT Thursday, August 27, 2015 2:02 PM
August 27th, 2015 2:01pm

Has anyone heard back from MS Support?

Also, what OSes are involved?  Anecdotally, I can't replicate the issue on my Windows 8.1 workstation but have had the issue reported to me on a couple of Windows 7 workstations.  Is it just Windows 7 and Server 2012?

Free Windows Admin Tool Kit Click here and download it now
August 27th, 2015 4:37pm

Had the same issue today looking forward to hearing what Microsoft says
August 27th, 2015 5:20pm

And to add to that...

I have the scan freezing w/ various definitions...

1.201.681

1.201.538

1.201.415

This problem is causing major productivity issues in my office.  When the SCEP hangs...  the client can no longer save work properly either locally or over the network.  It's as if their entire workstation comes to a stand still.  Items that take seconds, take minutes.  Software that ties to the network, software that teams rely on to work together, lock up, bringing them all to a halt, until the offender reboots his workstation. 

Something changed over the weekend and we need a resolution ASAP.

Free Windows Admin Tool Kit Click here and download it now
August 27th, 2015 5:58pm

Hi all, same issue in our company.

Several people having trouble saving/opening files on the network and outlook is not working good.

Scep is locking up, when you try to do a reboot the computer is hanging, only the trick with the button is working to get it restarted.

Now disabled all full and fast scan on computers.

Thx for the help

August 28th, 2015 5:10am

Has anyone heard back from MS Support?

Also, what OSes are involved?  Anecdotally, I can't replicate the issue on my Windows 8.1 workstation but have had the issue reported to me on a couple of Windows 7 workstations.  Is it just Windows 7 and Server 2012?

still waiting for the answer of microsoft, they said their "escalation manager" is working on it...

windows 2008 r2 is involved on my side

Free Windows Admin Tool Kit Click here and download it now
August 28th, 2015 5:10am

Microsoft support is suspecting this update KB3076895 is causing a deadlock when scanning.

I have initiated tests after removing this patch.

August 28th, 2015 6:39am

Microsoft support is suspecting this update KB3076895 is causing a deadlock when scanning.

I have initiated tests after removing this patch.

Free Windows Admin Tool Kit Click here and download it now
August 28th, 2015 7:38am

Excellent.  I'm going to look into whether affected workstations have that patch, espensh.  Then maybe try some experimentation.

Thanks!

August 28th, 2015 8:44am

The first test had a positive result. The full scan complete without the service freezing. It failed last evening before removing the patch.
Free Windows Admin Tool Kit Click here and download it now
August 28th, 2015 9:07am

Frist off, thanks for everyone's valuable input.  I have a case with Microsoft, but haven't received much assistance yet.  

I removed KB3076895 from my test machine and the re-ran scan that froze yesterday, it passed today since removal of that patch. 

I want to run a few more tests, but then the next tasks would be to figure a way to automate the removal of this KB3076895 from 275+ workstations. 

And also remove it from SCCM from re-distributing it. 

We've always had our SCCM in set-it and forget-it mode for the last year when it came to Endpoint and Window Updates. 

At least we are now making some progress it seems. 

August 28th, 2015 10:20am

This is the case on several computers is my company as well. Not sure why it doesnt apply to all of them, but at least 4-500 computers are affected. It seems like uninstall of update KB3076895 (and reboot) resolves the case. Thanks for sharing!
Free Windows Admin Tool Kit Click here and download it now
August 28th, 2015 11:46am

Does this look like the same issue with Symantec AV?  http://answers.microsoft.com/en-us/windows/forum/windows_7-update/issues-with-update-kb3076895-released-on-13082015/f887adce-9183-4d3d-86fb-2d01a9305981?page=1
August 28th, 2015 4:23pm

https://isc.sans.edu/forums/diary/Microsoft+patch+tuesday+problem+with+Symantec+Cloud+Endpoint+protection/20037/ ?
Free Windows Admin Tool Kit Click here and download it now
August 28th, 2015 4:24pm

Looks like we have a HotFix from Microsoft for KB3076895 on Windows 7 and Windows Server 2008R2.

https://support.microsoft.com/en-us/kb/3090303

I haven't had a chance to test it yet though.  Comes with the usual caveats from Microsoft, "proceed with caution and only deploy to machines actively running into the problem and whatnot."

August 29th, 2015 9:42am

Looks like we have a HotFix from Microsoft for KB3076895 on Windows 7 and Windows Server 2008R2.

https://support.microsoft.com/en-us/kb/3090303

I haven't had a chance to test it yet though.  Comes with the usual caveats from Microsoft, "proceed with caution and only deploy to machines actively running into the problem and whatnot."

  • Proposed as answer by superbug73 15 hours 21 minutes ago
Free Windows Admin Tool Kit Click here and download it now
August 29th, 2015 1:41pm

Looks like we have a HotFix from Microsoft for KB3076895 on Windows 7 and Windows Server 2008R2.

https://support.microsoft.com/en-us/kb/3090303

I haven't had a chance to test it yet though.  Comes with the usual caveats from Microsoft, "proceed with caution and only deploy to machines actively running into the problem and whatnot."

  • Proposed as answer by superbug73 Monday, August 31, 2015 4:04 PM
August 29th, 2015 1:41pm

Looks like we have a HotFix from Microsoft for KB3076895 on Windows 7 and Windows Server 2008R2.

https://support.microsoft.com/en-us/kb/3090303

I haven't had a chance to test it yet though.  Comes with the usual caveats from Microsoft, "proceed with caution and only deploy to machines actively running into the problem and whatnot."

  • Proposed as answer by superbug73 Monday, August 31, 2015 4:04 PM
Free Windows Admin Tool Kit Click here and download it now
August 29th, 2015 1:41pm

Looks like we have a HotFix from Microsoft for KB3076895 on Windows 7 and Windows Server 2008R2.

https://support.microsoft.com/en-us/kb/3090303

I haven't had a chance to test it yet though.  Comes with the usual caveats from Microsoft, "proceed with caution and only deploy to machines actively running into the problem and whatnot."

  • Proposed as answer by superbug73 Monday, August 31, 2015 4:04 PM
August 29th, 2015 1:41pm

Looks like we have a HotFix from Microsoft for KB3076895 on Windows 7 and Windows Server 2008R2.

https://support.microsoft.com/en-us/kb/3090303

I haven't had a chance to test it yet though.  Comes with the usual caveats from Microsoft, "proceed with caution and only deploy to machines actively running into the problem and whatnot."

Yes, i just got the information from Microsoft :)
Free Windows Admin Tool Kit Click here and download it now
August 31st, 2015 4:18am

Looks like we have a HotFix from Microsoft for KB3076895 on Windows 7 and Windows Server 2008R2.

https://support.microsoft.com/en-us/kb/3090303

I haven't had a chance to test it yet though.  Comes with the usual caveats from Microsoft, "proceed with caution and only deploy to machines actively running into the problem and whatnot."

did anybody have success with this hotfix and didnt remove KB3076895?
August 31st, 2015 5:25am

We do not install the HotFix KB3090303 or uninstall KB3076895.

We just downgrade SCEP to version 4.3.215.0. It seems that this works for us, until Microsoft has a better solution for it.

Update:

Answer from Microsoft today ... to install the Update above... we will do it now.

Based on memory dump analysis (dump provided by another customer), the hang issue is likely caused by the following security update :
fix KB3076895/ MS15-084  (https://www.microsoft.com/en-us/download/details.aspx?id=48320).
The above XML security fix contains code that causes deadlock related to loader lock and produce applications hang.

There is a hotfix available for Windows 7 :
https://support.microsoft.com/en-us/kb/3090303

[abstract]
Windows freezes or applications freeze after you install security update 3076895

<...>
Microsoft Common Antimalware Platform (CAMP) Server stops responding.
Microsoft Forefront Endpoint Protection (FEP) Application stops responding.
<...>

A GDR (Genral Distribution Release) of the fix will be available later this week.

  • Edited by bkliever 18 hours 0 minutes ago
Free Windows Admin Tool Kit Click here and download it now
August 31st, 2015 7:43am

We applied the hotfix mentioned above and it appears to have fixed our problems.
August 31st, 2015 8:36am

We do not install the HotFix KB3090303 or uninstall KB3076895.

We just downgrade SCEP to version 4.3.215.0. It seems that this works for us, until Microsoft has a better solution for it.

Update:

Answer from Microsoft today ... to install the Update above... we will do it now.

Based on memory dump analysis (dump provided by another customer), the hang issue is likely caused by the following security update :
fix KB3076895/ MS15-084  (https://www.microsoft.com/en-us/download/details.aspx?id=48320).
The above XML security fix contains code that causes deadlock related to loader lock and produce applications hang.

There is a hotfix available for Windows 7 :
https://support.microsoft.com/en-us/kb/3090303

[abstract]
Windows freezes or applications freeze after you install security update 3076895

<...>
Microsoft Common Antimalware Platform (CAMP) Server stops responding.
Microsoft Forefront Endpoint Protection (FEP) Application stops responding.
<...>

A GDR (Genral Distribution Release) of the fix will be available later this week.

  • Edited by bkliever Monday, August 31, 2015 1:26 PM
Free Windows Admin Tool Kit Click here and download it now
August 31st, 2015 11:41am

We do not install the HotFix KB3090303 or uninstall KB3076895.

We just downgrade SCEP to version 4.3.215.0. It seems that this works for us, until Microsoft has a better solution for it.

Update:

Answer from Microsoft today ... to install the Update above... we will do it now.

Based on memory dump analysis (dump provided by another customer), the hang issue is likely caused by the following security update :
fix KB3076895/ MS15-084  (https://www.microsoft.com/en-us/download/details.aspx?id=48320).
The above XML security fix contains code that causes deadlock related to loader lock and produce applications hang.

There is a hotfix available for Windows 7 :
https://support.microsoft.com/en-us/kb/3090303

[abstract]
Windows freezes or applications freeze after you install security update 3076895

<...>
Microsoft Common Antimalware Platform (CAMP) Server stops responding.
Microsoft Forefront Endpoint Protection (FEP) Application stops responding.
<...>

A GDR (Genral Distribution Release) of the fix will be available later this week.

  • Edited by bkliever Monday, August 31, 2015 1:26 PM
August 31st, 2015 11:41am

We do not install the HotFix KB3090303 or uninstall KB3076895.

We just downgrade SCEP to version 4.3.215.0. It seems that this works for us, until Microsoft has a better solution for it.

Update:

Answer from Microsoft today ... to install the Update above... we will do it now.

Based on memory dump analysis (dump provided by another customer), the hang issue is likely caused by the following security update :
fix KB3076895/ MS15-084  (https://www.microsoft.com/en-us/download/details.aspx?id=48320).
The above XML security fix contains code that causes deadlock related to loader lock and produce applications hang.

There is a hotfix available for Windows 7 :
https://support.microsoft.com/en-us/kb/3090303

[abstract]
Windows freezes or applications freeze after you install security update 3076895

<...>
Microsoft Common Antimalware Platform (CAMP) Server stops responding.
Microsoft Forefront Endpoint Protection (FEP) Application stops responding.
<...>

A GDR (Genral Distribution Release) of the fix will be available later this week.

  • Edited by bkliever Monday, August 31, 2015 1:26 PM
Free Windows Admin Tool Kit Click here and download it now
August 31st, 2015 11:41am

We just tested the hotfix on 2 PCs and it has resolved the issues for us as well.

Now to roll out to all the other affected PC's...

Thanks everyone!

August 31st, 2015 12:03pm

We just tested the hotfix on 2 PCs and it has resolved the issues for us as well.

Now to roll out to all the other affected PC's...

Thanks everyone!

  • Edited by superbug73 Monday, August 31, 2015 4:04 PM
Free Windows Admin Tool Kit Click here and download it now
August 31st, 2015 4:02pm

We just tested the hotfix on 2 PCs and it has resolved the issues for us as well.

Now to roll out to all the other affected PC's...

Thanks everyone!

  • Edited by superbug73 Monday, August 31, 2015 4:04 PM
August 31st, 2015 4:02pm

We just tested the hotfix on 2 PCs and it has resolved the issues for us as well.

Now to roll out to all the other affected PC's...

Thanks everyone!

  • Edited by superbug73 Monday, August 31, 2015 4:04 PM
Free Windows Admin Tool Kit Click here and download it now
August 31st, 2015 4:02pm

This should be fixed in KB3092627 (which is also distributed via WSUS)

https://support.microsoft.com/de-de/kb/3092627

  • Proposed as answer by Rotronic 24 minutes ago
September 2nd, 2015 3:01am

This should be fixed in KB3092627 (which is also distributed via WSUS)

https://support.microsoft.com/de-de/kb/3092627

  • Proposed as answer by Rotronic Wednesday, September 02, 2015 7:00 AM
Free Windows Admin Tool Kit Click here and download it now
September 2nd, 2015 7:00am

This should be fixed in KB3092627 (which is also distributed via WSUS)

https://support.microsoft.com/de-de/kb/3092627

  • Proposed as answer by Rotronic Wednesday, September 02, 2015 7:00 AM
September 2nd, 2015 7:00am

This should be fixed in KB3092627 (which is also distributed via WSUS)

https://support.microsoft.com/de-de/kb/3092627


This has worked for us so far.
Free Windows Admin Tool Kit Click here and download it now
September 2nd, 2015 9:00am

The above hotfix (KB3090303) took care of our earlier problem but now we are seeing something new 

I am now noticing on some machines when they run a full scan and hit a recovery partition on the machine called (Q:\FactoryRecovery\cdrivebackup.wim) it is crashing the Antimalware service and not starting back up. This recovery partition is common to Lenovo products and every machine with this partition is crashing the Antimalware service. It never did this in the past. Anyone else see this problem with recovery partitions, drives or directories.

September 3rd, 2015 7:50am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics