Hello all, I have a SCOM 2007 SP1 environment Version 6.0.6278.0 Total Servers in this environment: 278 Over 7 days period the Health Service Unloaded System Rule(s) alerts are (the success events after the error event.) : The health service {044D97F4-33A7-28B0-F52C-F581BA7126B6} running on host WPS03 and serving management group SCOM01 with id {69C9579E-8021-BEF0-E8C7-6EC8EC36235B} is available through the server inf07. The health service {046AC1C8-A8C2-53C2-4CB3-CA7370133492} running on host lns14 and serving management group SCOM01 with id {69C9579E-8021-BEF0-E8C7-6EC8EC36235B} is available through the server inf06. The health service {08D92AE4-CE43-E666-62A7-9B2BA8354468} running on host MAN5 and serving management group SCOM01 with id {69C9579E-8021-BEF0-E8C7-6EC8EC36235B} is available through the server inf06. The health service {0A1CA4ED-460B-F917-215B-E2AB723E26B5} running on host sym02 and serving management group SCOM01 with id {69C9579E-8021-BEF0-E8C7-6EC8EC36235B} is available through the server inf06. The health service {261FB6C0-6777-5DBB-9D59-EAEC0CCD6EDB} running on host leo01 and serving management group SCOM01 with id {69C9579E-8021-BEF0-E8C7-6EC8EC36235B} is available through the server inf06. The health service {2F3E20E0-A089-EF78-219F-5D407BD21B8C} running on host wps01 and serving management group SCOM01 with id {69C9579E-8021-BEF0-E8C7-6EC8EC36235B} is available through the server inf07. The health service {501DAF69-38A6-A012-43C6-49EC263EB286} running on host lep01 and serving management group SCOM01 with id {69C9579E-8021-BEF0-E8C7-6EC8EC36235B} is available through the server inf06. The health service {627A6BF3-FDBC-A37B-5C46-24B5CD69B15F} running on host chzrhad9 and serving management group SCOM01 with id {69C9579E-8021-BEF0-E8C7-6EC8EC36235B} is available through the server inf06. The health service {75EAB872-2FCC-98A4-7447-793FA8849FCC} running on host cmire and serving management group SCOM01 with id {69C9579E-8021-BEF0-E8C7-6EC8EC36235B} is available through the server inf07. The health service {8C2D5379-E626-0941-780F-ED94C59F45A1} running on host WPS16 and serving management group SCOM01 with id {69C9579E-8021-BEF0-E8C7-6EC8EC36235B} is available through the server inf07. The health service {A8E44100-750C-56D1-FA35-353BB569DBF3} running on host WPS05 and serving management group SCOM01 with id {69C9579E-8021-BEF0-E8C7-6EC8EC36235B} is available through the server inf07. The health service {D933D9D5-9761-1853-0EA1-F157850C9386} running on host WPS08 and serving management group SCOM01 with id {69C9579E-8021-BEF0-E8C7-6EC8EC36235B} is available through the server inf07. The health service {F2AA54EC-33C2-E523-2868-6D52497A09B3} running on host lep02 and serving management group SCOM01 with id {69C9579E-8021-BEF0-E8C7-6EC8EC36235B} is available through the server inf06. The health service {F2AA54EC-33C2-E523-2868-6D52497A09B3} running on host lep02 and serving management group SCOM01 with id {69C9579E-8021-BEF0-E8C7-6EC8EC36235B} is available through the server inf06. In over 97% of the cases the problem is fixed by itself without any kind of interventions.Like you see the servers are different and the health service is different. Any ideas what is the problem and how to fix it?

Hi SP1 seemed more prone to this problem than R2 but it could also be management pack related. Do you have the actual errors that are generated? And is there something in common between the agents e.g. 1) Are they all Windows 2000? 2) Are they all desktops? Low on resource? 3) Are you using the Dell MP? Or SNMP monitoring from any of these servers? This might help: http://ianblythmanagement.wordpress.com/2009/02/16/health-service-unloaded-system-rules-event-1102/ Good Luck GrahamView OpsMgr tips and tricks at http://systemcentersolutions.wordpress.com/

1) No. They are 2003 Server SP2 2) In general they are Virtual Machine on ESX Server, and very few HP Blade Servers. I will have to check if something is happening regarding performances exactly in the time when the allert arives. 3) No Dell MP, There is a HP management pack that should not have affect on the Virtual Servers. I never seen a such a message described in the post on the servers

Can you upload copies of the actual alert? You seem to have copied something that looks like success (is available)Microsoft Corporation

Like I already told.. the alert is resolved by itself ( in approximately 97% of the cases )after some time. ( between 10min and 2hours ). The rest of the 3 procent i need to clear the cache from the server. More data ( for one server): The alert is automatically closed. The alert Properties after it was closed looks like this: General Tab; Alert Description: Alert raised by monitor when system rules have been unloaded by the Health Service. Product Knowledge: ..... use Repair Action.... History: 05.06.2010 04:22 -> Alert Activated by the System; 05.06.2010 04:42 -> Alert Resolved by the System. ( so automatically closed when the system reload the configuration ) Alert content ( the data that i gave you before ) ( IF the alert is automatically resolved here you will have the event that closed the alert )=> the agent reloaded all it`s configuration Date and Time: 05.06.2010 04:41:54 Log Name: Operations Manager Source: OpsMgr Connector Event Number: 20021 Level: 0 Logging Computer: SERVER User: N/A Description: The health service {6372A011-44EF-EB71-531E-9357D515695F} running on host SERVER and serving management group SCOM01 with id {69C9579E-8021-BEF0-E8C7-6EC8EC36235B} is available through the server MANAGEMENT SERVER The alerts that appear in that time ( I checked the operation manager event logs ) that are related to this error ( from my checks) are: Event Type: Error Event Source: HealthService Event Category: None Event ID: 4503 Date: 05.06.2010 Time: 04:20:42 User: N/A Computer: SERVER Description: A module reported an error 0x80FF0004 from a callback which was running as part of rule "Microsoft.SystemCenter.DataWarehouse.CollectPerformanceData" running for instance "SERVER " with id:"{6372A011-44EF-EB71-531E-9357D515695F}" in management group "SCOM01". Event Type: Warning Event Source: HealthService Event Category: Health Service Event ID: 1103 Date: 05.06.2010 Time: 04:20:52 User: N/A Computer: SERVER Description: Summary: 1 rule(s)/monitor(s) failed and got unloaded, 1 of them reached the failure limit that prevents automatic reload. Management group "SCOM01". This is summary only event, please see other events with descriptions of unloaded rule(s)/monitor(s). Event Type: Error Event Source: HealthService Event Category: None Event ID: 4503 Date: 05.06.2010 Time: 04:20:58 User: N/A Computer: SERVER Description: A module reported an error 0x80FF0004 from a callback which was running as part of rule "Microsoft.SystemCenter.CollectPerformanceData" running for instance "SERVER " with id:"{6372A011-44EF-EB71-531E-9357D515695F}" in management group "SCOM01". Event Type: Warning Event Source: HealthService Event Category: Health Service Event ID: 1103 Date: 05.06.2010 Time: 04:21:11 User: N/A Computer: SERVER Description: Summary: 1 rule(s)/monitor(s) failed and got unloaded, 1 of them reached the failure limit that prevents automatic reload. Management group "SCOM01". This is summary only event, please see other events with descriptions of unloaded rule(s)/monitor(s). Like you see from the first data from me the ID is different. -------------------------------------------- Data from another server: 02:44 ->Alert activated by system 03:34 -> allert resolved by the system ( resolved when the server reloads the configuration) Event Type: Error Event Source: HealthService Event Category: None Event ID: 4503 Date: 31.05.2010 Time: 02:43:12 User: N/A Computer: SERVER Description: A module reported an error 0x80FF0004 from a callback which was running as part of rule "Microsoft.SystemCenter.CollectPerformanceData" running for instance "SERVER.mydomain.com" with id:"{501DAF69-38A6-A012-43C6-49EC263EB286}" in management group "SCOM01". Event Type: Warning Event Source: HealthService Event Category: Health Service Event ID: 1103 Date: 31.05.2010 Time: 02:43:15 User: N/A Computer: SERVER Description: Summary: 1 rule(s)/monitor(s) failed and got unloaded, 1 of them reached the failure limit that prevents automatic reload. Management group "SCOM01". This is summary only event, please see other events with descriptions of unloaded rule(s)/monitor(s). Event Type: Error Event Source: HealthService Event Category: None Event ID: 4503 Date: 31.05.2010 Time: 02:43:17 User: N/A Computer: SERVER Description: A module reported an error 0x80FF0004 from a callback which was running as part of rule "Microsoft.Windows.Server.2003.OperatingSystem.TotalDPCTime" running for instance "Microsoft(R) Windows(R) Server 2003, Enterprise Edition" with id:"{F82C1876-41AC-2714-CA09-C65EE9C57713}" in management group "SCOM01". Event Type: Warning Event Source: HealthService Event Category: Health Service Event ID: 1103 Date: 31.05.2010 Time: 02:43:18 User: N/A Computer: SERVER Description: Summary: 1 rule(s)/monitor(s) failed and got unloaded, 1 of them reached the failure limit that prevents automatic reload. Management group "SCOM01". This is summary only event, please see other events with descriptions of unloaded rule(s)/monitor(s).

Hi, Regarding the Event ID: 4503, I would like to share the following with you for your reference: Operations Manager 2007 Service Pack 1 may stop monitoring SNMP devices http://support.microsoft.com/kb/961363 Hope this helps. Thanks. Nicholas Li - MSFT

I checked the file versions for MOMNetworkModules.dll 6.0.6278.0. But.... all my servers are Virtual! Do you think that the update will fix this problem?

yes. A server is a server is a server (if you are a scom agent)Microsoft Corporation

I tough that only if the SNMP monitr reach the server then the agent fails... Ok... I will try but first i need a approval... it will take some time... I will get back with a result!

Hi, Sadly I still wait for the OK of my boss. I have seen a Updates collection http://support.microsoft.com/kb/971541/ ( that includes also the KB Nicholas recommended ) Do you think if I apply the Update Collection only to the Agents and not to the SCOM Infrastrcutre Environment it will have any effect?

Hi, I wanted to update the status.. I did not got the accept (big company.... :( ) but strangely I did not do anything and I only get around 1-2 alerts/week.

Hi, I will mark this as answer due to no activities for a long time. Feel free to re-open it. ThanksAnders Bengtsson | Microsoft PFE | blog at http://www.contoso.se