ScInterop In Maintenance Mode (Network Steve Forum)

ScInterop In Maintenance Mode

Hi,

We recently did maintenance on our SCOM servers and decided to put the connector for HP Openview Operations into maintenance mode to prevent forwarding of alerts.

1. We put the connector into maintenance mode

2. Disabled/stopped the RMS Connector Service

After completing our work we turned the service back on and removed maintenance. We had a flood of messages coming across into Openview of WSMAN unauthorized access and No providers coul be contacted alerts. We stopped the connector service and took a look at the Interop Database cache.dbo table and saw 500+ entries remaining most of which had the same id. Did we do something wrong that created all these in the cache?

Thanks,

Bob

March 29th, 2010 8:29pm

Bob,

My first guess is that the connector service was not actually stopped but you stopped the provider on the OVO side and the connector service continued to try an poll the provider and failed with the 'No providers could be contacted" alerts thus caching the failures.

- When you put the connector into maintenance mode can you explain the steps used to do this?

- When you disabled/stopped the RMS Connector Service was this actually the 'OpsMgr Connector for HP Operations' service as I'm not sure what the RMS connector service is?

Regards,

-Steve

Free Windows Admin Tool Kit Click here and download it now

March 31st, 2010 12:56am

Steve,

To put the connector into maintance mode we open up diagram view for the rms server and find the "Connector into HP OVO" object.

We launch Operations Manager Shell... for the object to get the directory path of the object.

We then schedule a powershell script to change to that directory and run New-MaintenanceWindow.

We verfied that the "Connector into HP OVO" object was in maintenance mode.

We then did a Stop/disable RMS Connector service: OpsMgr Connector for HP OpenView Operations

we did not stop the provider service "System Center OpsMgr Event Consumer for HP OpenView"

-Bob

March 31st, 2010 4:10pm

Bob and I work together and I wanted to provide additional information since we are still not comfortable with what/why this happened.

In the application event log on the RMS, hundreds of times on the day this occurred, (after we started the OpsMgr Connector for HP OpenView Operations service and then manually removed it from Maintenance in the OpsMgr console) the following event was written:

Event ID 1411:

2010-03-28 07:19:43.590 [Microsoft.SystemCenter.Interop.Framework.Modules.OpsMgrModule] ERROR Could not update alerts due to internal error. The GetMonitoringAlerts did not return the expected number of alerts

At a much lesser amount, but still written multiple times, was another error:

Event ID 3000:

2010-03-28 09:25:31.848 [Microsoft.SystemCenter.Interop.Framework.Utilities.SmartTimer] ERROR Timer callback threw an InteropException. The GetMonitoringAlerts did not return the expected number of alerts.

In addition, the night before, starting at 9:00pm until we stopped the connector at 3:55 am, I see lots of errors occurring throughout the night.

Event ID 2803:

2010-03-27 21:00:27.799 [Microsoft.SystemCenter.Interop.Framework.HighAvailability.HighAvailability] ERROR SqlException Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. occurred while sending heartbeat

Event ID 2800:

2010-03-27 21:00:27.815 [Microsoft.SystemCenter.Interop.Framework.HighAvailability.Agent] ERROR Exception caught while heartbeating: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

Event ID 1202:

2010-03-27 21:09:36.219 [Microsoft.SystemCenter.Interop.Caching.DBCacheManager] ERROR SqlException Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. occurred while retrieving cache data

Event ID 1102:

2010-03-27 21:09:36.219 [Microsoft.SystemCenter.Interop.Framework.Connector] ERROR SqlException caught while starting modules. DB cache will not be loaded

Event ID 1201:

2010-03-27 21:20:08.030 [Microsoft.SystemCenter.Interop.Caching.DBCacheManager] ERROR SqlException Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. occurred while removing cache data from storage

Event ID 1207:

2010-03-27 21:43:24.424 [Microsoft.SystemCenter.Interop.Framework.Modules.Cache] ERROR An item with the same key has already been added.

Event ID 3000:

2010-03-27 21:43:24.424 [Microsoft.SystemCenter.Interop.Framework.Utilities.SmartTimer] ERROR Timer callback threw an InteropException. An item with the same key has already been added.

Event ID 1207 and 3000 repeated every 30 seconds throughout the night.

What are these telling me? Also, what is happening that the cache table had 500+ entries?

Ideas?

Thank you,

Kate

Free Windows Admin Tool Kit Click here and download it now

April 14th, 2010 11:24pm

The situation above has happened again. This time in our QA environment. I updated our properties on our OpsMgr Connector for HP OpenView Operations. The pop-up reminded me I need to restart the Connector to make the changes take effect.

I put the connector into maintenance mode for 5 minutes. I stopped and restarted the connector service on our RMS. When the connector service came out of maintenance, hundreds of alerts started arriving in our OMW system. On the OMW system, the provider service (System Center OpsMgr Event Consumer for HP OpenView) was up during this entire time. I had not shut it down.

I stopped the connector, I stopped the provider. I checked the dbo.cache table. 2000+ rows. It appeared to be a repeat of about four different alerts in OpsMgr. The main two were:

The Connector service threw an exception because no providers could be contacted to send alerts.

The Connector service threw a WSMan COM exception when attempting to send alerts to provider.

On our OMW system, I disabled the policies that forward on these messages to our ticketing and alerting system.

I restarted the connector and let it go. The 2000+ entries arrived in OMW (thankfully not going anywhere beyond because of shutting off the forwarding policies) and finally stopped. During this, I watched the dbo.cache table diminish to 0 rows.

Here is the really interesting part. Our OMW system had issues over the weekend. It was rebooted and back online earlier this week. These alerts that came in today during this message storm appeared to be those from the weekend. How would these have been re-issued? The alerts are no longer active in the OMW or the OpsMgr side.

Is there something with coming out of maintenance mode causing this?

Thank you,

Kate

May 27th, 2010 11:44pm

Hello!

I'm currently having a similar issue.... were you able to fix this problem? Thanks!!

Free Windows Admin Tool Kit Click here and download it now

August 26th, 2010 12:25pm

Unfortunately, we were never able to pinpoint why this happens. If you are able to find an explanation/solution, please share. Thanks!

Kate

August 26th, 2010 4:16pm

Here's what we did: The problem was that the config of our Connector was set to forward historical alerts up to 72 hours old... so we set it to 0!

On the System Center Operations Manager Console go to Administration -> Product Connectors -> HP OpenView Operations Connectors, then on the Connectors page right-click on OpsMgr Connector for HP OpenView.