Now and then some grey agents SCOM.

If I look at the server it looks like the event ID 4503 in the Operations manager eventlog is connected to the agent being greyed out.

"A module reported an error 0x80070057 from a callback which was running as part of rule "Microsoft.SystemCenter.CollectPublishedEntityState.Resend" running for instance".

A restart of the service and clearing the "health service state"-folder will fix the problem.

Anyone no of a reason for why this is happening?

Please see if this helps -

http://www.eventid.net/display-eventid-4503-source-HealthService-eventno-9697-phase-1.htm

This will provided you and indept insight to the greyed agents 

http://support.microsoft.com/kb/2288515

Thanks for the tips. But none of them applies to my case.

I found something interesting though. I have a scheduled a job that sets computers in maintenance mode when updates are installed and computers are restarted. And it looks like the error appears when the computer exits maintenance mode.

And then it gets grey in the console. So it seems to be connected to the data it has to send when it exits maintenance mode. Hence the "Microsoft.SystemCenter.CollectPublishedEntityState.Resend" in the error.

But why should that kill the service?

I have also recently implemented a script to set maintenance mode for server patching windows.  Approximately half of the servers put into maintenance mode threw the "Microsoft.SystemCenter.CollectPublishedEntityState.Resend" error after exiting maintenance mode.

At this early stage in my troubleshooting, I'm going to assume it's a load issue with having a fairly large number of servers exit maintenance mode at the same time.  I'm going to update my script to spread out the maintenance mode end times, and I will reply back with results.

Despite limiting the number of servers exiting maintenance mode to 10 every 2 minutes I am still experiencing the grey agent issues after maintenance mode.

Has anyone else had any luck resolving this issue?