Who watches the watchers?
This Saturday, the SQL server hosting our OM database went down, requiring 2 failovers. The RMS lost connection to the SQL server and did not recover (I didn't notice it until Monday morning), meaning SCOM was broken the WHOLE WEEKEND.
So, how is anyone out there "watching" SCOM to make sure it's running? I've added a task (like a scheduled task) to a couple of the events in the OM log (like event ID 33333 from DataAccessLayer) to send the SCOM admins a page, but there has to
be a better way. Our NOC doesn't watch the SCOM console, and even if they did, there were some alerts generated during the "outage" so that wouldn't have helped."Fear disturbs your concentration"
December 13th, 2010 12:28pm
Hey Andy.
Check out Dan Kregors tool
http://opsm.gr/2010/05/28/1e-system-center-watcher-tool-v1-now-available/ I have tried it in test/lab enviroments and like it alot..
mats.w | www.opsmode.com
Free Windows Admin Tool Kit Click here and download it now
December 13th, 2010 2:45pm
This wouldn't have caught the fugue state that our RMS was in; the RMS was up, pingable, all services were running, just the SDK (err...Data Access Service) was off in la-la-land after it lost connection to the OM DB."Fear disturbs your concentration"
December 13th, 2010 5:42pm
I guess it's just how far you want to go. i think it's already questionable to monitor the monitor, because don't you need to monitor the monitor's monitor too? (i have people looking at the console 24/7 and they will notice when the console
doesn't respond).
To capture all errors related to the sdk not responding you could run a powershell test that does a simple query (e.g. find the name of the rms). If that fails raise an alert. In this case you might be able to use scom itself as according to you only the
sdk service was not working (assuming the scom workflows still worked), but i'm not sure as i don't use notifications myself; but you can also use an external script to mail/sms/whatever notification you want
Rob Korving
http://jama00.wordpress.com/
Free Windows Admin Tool Kit Click here and download it now
December 14th, 2010 4:57am