The Beanspy Alert monitor reports alerts with missing data. See example below, specifically the {} returned parameters appear empty/incorrect.

Application server connection lost

Alert Description

Source:
Full Path Name:
Alert Monitor:		BeanSpy
Created:

The Health Service lost the connection to BeanSpy on port {2} to the machine {1} for the application server ID '{0}'.

Why o why is the first answer always assuming not having read and/or used the manual, by posting a RTFM reply. I find this kind of replies discriminating. (I almost clicked report for abuse).

It is instantly assumed I made a configuration error. Without asking me any question(s) first to verify anything that might cause this behaviour. Maybe I am right and maybe there is a problem with the monitor in the MP?

I have a counter proposal: Assume I did read the documentation already and everything is configured properly. What'd then be the next step?

• Edited by Manni Calavera Friday, February 13, 2015 10:38 AM

Bumping this one with some additional information:

This particular alert is related to a Jboss 6EAP instance.

With Jboss 5AS instances the port and machine name are properly shown in the alert. I don't know why the monitor is not capable of showing the values properly.

Secondly,

The reason why this alert occurs seems to be related to enabling Deep Monitoring, which I enabled using the Task (Enable deep monitoring) provided by the Management Pack.

The Jboss instances 5AS en 6EAP are all automatically discovered and showing up in the Configuration view.

However 1 detail is sometimes incorrect. The automatic discovery lists incorrect HTTP and HTTPS ports. I am absolutely clueless where the scripts get this information from. I can't find these ports being used in the boot.log nor are ports in use on the server, verified with netstat.

At the moment I believe the incorrect ports are also provided to the task Enable Deep Monitoring, which causes the Deep Monitoring to fail because it tries to contact beanspy on an incorrect port.

Can anyone explain where exactly the MP gets the HTTP and HTTPS port information from? I'm not experienced enough interpreting the extensive/impressive/complex discovery and monitor scripts.

• Edited by Manni Calavera 12 hours 18 minutes ago

Bumping this one with some additional information:

This particular alert is related to a Jboss 6EAP instance.

With Jboss 5AS instances the port and machine name are properly shown in the alert. I don't know why the monitor is not capable of showing the values properly.

Secondly,

The reason why this alert occurs seems to be related to enabling Deep Monitoring, which I enabled using the Task (Enable deep monitoring) provided by the Management Pack.

The Jboss instances 5AS en 6EAP are all automatically discovered and showing up in the Configuration view.

However 1 detail is sometimes incorrect. The automatic discovery lists incorrect HTTP and HTTPS ports. I am absolutely clueless where the scripts get this information from. I can't find these ports being used in the boot.log nor are ports in use on the server, verified with netstat.

At the moment I believe the incorrect ports are also provided to the task Enable Deep Monitoring, which causes the Deep Monitoring to fail because it tries to contact beanspy on an incorrect port.

Can anyone explain where exactly the MP gets the HTTP and HTTPS port information from? I'm not experienced enough interpreting the extensive/impressive/complex discovery and monitor scripts.

• Edited by Manni Calavera Thursday, March 05, 2015 7:47 PM

Bumping this one with some additional information:

This particular alert is related to a Jboss 6EAP instance.

With Jboss 5AS instances the port and machine name are properly shown in the alert. I don't know why the monitor is not capable of showing the values properly.

Secondly,

The reason why this alert occurs seems to be related to enabling Deep Monitoring, which I enabled using the Task (Enable deep monitoring) provided by the Management Pack.

The Jboss instances 5AS en 6EAP are all automatically discovered and showing up in the Configuration view.

However 1 detail is sometimes incorrect. The automatic discovery lists incorrect HTTP and HTTPS ports. I am absolutely clueless where the scripts get this information from. I can't find these ports being used in the boot.log nor are ports in use on the server, verified with netstat.

At the moment I believe the incorrect ports are also provided to the task Enable Deep Monitoring, which causes the Deep Monitoring to fail because it tries to contact beanspy on an incorrect port.

Can anyone explain where exactly the MP gets the HTTP and HTTPS port information from? I'm not experienced enough interpreting the extensive/impressive/complex discovery and monitor scripts.

• Edited by Manni Calavera Thursday, March 05, 2015 7:47 PM

Bumping this one with some additional information:

This particular alert is related to a Jboss 6EAP instance.

With Jboss 5AS instances the port and machine name are properly shown in the alert. I don't know why the monitor is not capable of showing the values properly.

Secondly,

The reason why this alert occurs seems to be related to enabling Deep Monitoring, which I enabled using the Task (Enable deep monitoring) provided by the Management Pack.

The Jboss instances 5AS en 6EAP are all automatically discovered and showing up in the Configuration view.

However 1 detail is sometimes incorrect. The automatic discovery lists incorrect HTTP and HTTPS ports. I am absolutely clueless where the scripts get this information from. I can't find these ports being used in the boot.log nor are ports in use on the server, verified with netstat.

At the moment I believe the incorrect ports are also provided to the task Enable Deep Monitoring, which causes the Deep Monitoring to fail because it tries to contact beanspy on an incorrect port.

Can anyone explain where exactly the MP gets the HTTP and HTTPS port information from? I'm not experienced enough interpreting the extensive/impressive/complex discovery and monitor scripts.

• Edited by Manni Calavera Thursday, March 05, 2015 7:47 PM

So the next step I took was taking a deep dive into the JBoss.Monitored.Configuration.Discovery scripts and I found that the script tries to find and/or create registry keys about its discovered Jboss instances.

The key being created is: HKLM\Software\Microsoft\Microsoft Operations Manager\3.0\Modules\{GUID}\.\Script\PersistedDiscovery

In that key, values are stored called HTTP and HTTPS.

The persistedDiscovery key is deleted and recreated when you stop the Microsoft Monitoring Agent service and/or flush the health state cache. Thus manually adding the values is no use.

The HTTP and HTTPS values in a Jboss6 instance running domain mode cannot be retrieved because the port configuration is done at the Jboss domain controller and 'streamed' to the Jboss domain hosts.

In JBoss 5 instance however I see HTTP and HTTPS values being inserted into the registry. In a few cases incorrect information about ports is found and added to the registry.

So for now I assume something in the discovery is not finding the actual running configuration of HTTP and HTTPS ports, but it finds and uses something else. Now if I only can pinpoint where exactly the script is getting the information from....

• Edited by Manni Calavera 22 minutes ago

So the next step I took was taking a deep dive into the JBoss.Monitored.Configuration.Discovery scripts and I found that the script tries to find and/or create registry keys about its discovered Jboss instances.

The key being created is: HKLM\Software\Microsoft\Microsoft Operations Manager\3.0\Modules\{GUID}\.\Script\PersistedDiscovery

In that key, values are stored called HTTP and HTTPS.

The persistedDiscovery key is deleted and recreated when you stop the Microsoft Monitoring Agent service and/or flush the health state cache. Thus manually adding the values is no use.

The HTTP and HTTPS values in a Jboss7 instance running domain mode cannot be retrieved because the port configuration is done at the Jboss domain controller and 'streamed' to the Jboss domain hosts.

In JBoss 5 instance however I see HTTP and HTTPS values being inserted into the registry. In a few cases incorrect information about ports is found and added to the registry.

In the Jboss discovery vbs I found the pieces for Jboss5:

' Get the HTTP & HTTPS ports of the JBoss 5 installation that runs a specific configuration 
' Do this by adding the offset for the port binding to the base ports found in the XML file 
' bindings-jboss-beans.xml located in <configuration>\conf\bindingservice.beans\META-INF\

Apparantly what is written in this file does not actually reflect the running config.

For Jboss7 it appears quite a bit more complex (especially running domain mode):

Step1:

' Get the HTTP and HTTPS ports for a JBoss 7 or Wildfly installation that is running in domain mode 
' In order to get the HTTP Ports we must first retrieve the group that the server belongs too, as well as the port offset for each server 
' These values both resiode within the host.xml file

Step2:

' We then look within domain.xml to find the profile reference based on the group name that we retrieved from host.xml 
' After the profile reference we then can find the socket-bindings for http and https ports

While the information is there, these values are not added to the registry.

Thus, so far the script is looking in the right place to find the used ports for that particular Jboss7 Service, yet the monitor for the BeanSpy is not using that port to contact the BeanSpy.

My investigation continues...

• Edited by Manni Calavera 18 hours 59 minutes ago

So the next step I took was taking a deep dive into the JBoss.Monitored.Configuration.Discovery scripts and I found that the script tries to find and/or create registry keys about its discovered Jboss instances.

The key being created is: HKLM\Software\Microsoft\Microsoft Operations Manager\3.0\Modules\{GUID}\.\Script\PersistedDiscovery

In that key, values are stored called HTTP and HTTPS.

The persistedDiscovery key is deleted and recreated when you stop the Microsoft Monitoring Agent service and/or flush the health state cache. Thus manually adding the values is no use.

The HTTP and HTTPS values in a Jboss7 instance running domain mode cannot be retrieved because the port configuration is done at the Jboss domain controller and 'streamed' to the Jboss domain hosts.

In JBoss 5 instance however I see HTTP and HTTPS values being inserted into the registry. In a few cases incorrect information about ports is found and added to the registry.

In the Jboss discovery vbs I found the pieces for Jboss5:

' Get the HTTP & HTTPS ports of the JBoss 5 installation that runs a specific configuration 
' Do this by adding the offset for the port binding to the base ports found in the XML file 
' bindings-jboss-beans.xml located in <configuration>\conf\bindingservice.beans\META-INF\

Apparantly what is written in this file does not actually reflect the running config.

For Jboss7 it appears quite a bit more complex (especially running domain mode):

Step1:

' Get the HTTP and HTTPS ports for a JBoss 7 or Wildfly installation that is running in domain mode 
' In order to get the HTTP Ports we must first retrieve the group that the server belongs too, as well as the port offset for each server 
' These values both resiode within the host.xml file

Step2:

' We then look within domain.xml to find the profile reference based on the group name that we retrieved from host.xml 
' After the profile reference we then can find the socket-bindings for http and https ports

While the information is there, these values are not added to the registry.

Thus, so far the script is looking in the right place to find the used ports for that particular Jboss7 Service, yet the monitor for the BeanSpy is not using that port to contact the BeanSpy.

My investigation continues...

• Edited by Manni Calavera Saturday, March 07, 2015 12:06 PM

So the next step I took was taking a deep dive into the JBoss.Monitored.Configuration.Discovery scripts and I found that the script tries to find and/or create registry keys about its discovered Jboss instances.

The key being created is: HKLM\Software\Microsoft\Microsoft Operations Manager\3.0\Modules\{GUID}\.\Script\PersistedDiscovery

In that key, values are stored called HTTP and HTTPS.

The persistedDiscovery key is deleted and recreated when you stop the Microsoft Monitoring Agent service and/or flush the health state cache. Thus manually adding the values is no use.

The HTTP and HTTPS values in a Jboss7 instance running domain mode cannot be retrieved because the port configuration is done at the Jboss domain controller and 'streamed' to the Jboss domain hosts.

In JBoss 5 instance however I see HTTP and HTTPS values being inserted into the registry. In a few cases incorrect information about ports is found and added to the registry.

In the Jboss discovery vbs I found the pieces for Jboss5:

' Get the HTTP & HTTPS ports of the JBoss 5 installation that runs a specific configuration 
' Do this by adding the offset for the port binding to the base ports found in the XML file 
' bindings-jboss-beans.xml located in <configuration>\conf\bindingservice.beans\META-INF\

Apparantly what is written in this file does not actually reflect the running config.

For Jboss7 it appears quite a bit more complex (especially running domain mode):

Step1:

' Get the HTTP and HTTPS ports for a JBoss 7 or Wildfly installation that is running in domain mode 
' In order to get the HTTP Ports we must first retrieve the group that the server belongs too, as well as the port offset for each server 
' These values both resiode within the host.xml file

Step2:

' We then look within domain.xml to find the profile reference based on the group name that we retrieved from host.xml 
' After the profile reference we then can find the socket-bindings for http and https ports

While the information is there, these values are not added to the registry.

Thus, so far the script is looking in the right place to find the used ports for that particular Jboss7 Service, yet the monitor for the BeanSpy is not using that port to contact the BeanSpy.

My investigation continues...

• Edited by Manni Calavera Saturday, March 07, 2015 12:06 PM

So the next step I took was taking a deep dive into the JBoss.Monitored.Configuration.Discovery scripts and I found that the script tries to find and/or create registry keys about its discovered Jboss instances.

The key being created is: HKLM\Software\Microsoft\Microsoft Operations Manager\3.0\Modules\{GUID}\.\Script\PersistedDiscovery

In that key, values are stored called HTTP and HTTPS.

The persistedDiscovery key is deleted and recreated when you stop the Microsoft Monitoring Agent service and/or flush the health state cache. Thus manually adding the values is no use.

The HTTP and HTTPS values in a Jboss7 instance running domain mode cannot be retrieved because the port configuration is done at the Jboss domain controller and 'streamed' to the Jboss domain hosts.

In JBoss 5 instance however I see HTTP and HTTPS values being inserted into the registry. In a few cases incorrect information about ports is found and added to the registry.

In the Jboss discovery vbs I found the pieces for Jboss5:

' Get the HTTP & HTTPS ports of the JBoss 5 installation that runs a specific configuration 
' Do this by adding the offset for the port binding to the base ports found in the XML file 
' bindings-jboss-beans.xml located in <configuration>\conf\bindingservice.beans\META-INF\

Apparantly what is written in this file does not actually reflect the running config.

For Jboss7 it appears quite a bit more complex (especially running domain mode):

Step1:

' Get the HTTP and HTTPS ports for a JBoss 7 or Wildfly installation that is running in domain mode 
' In order to get the HTTP Ports we must first retrieve the group that the server belongs too, as well as the port offset for each server 
' These values both resiode within the host.xml file

Step2:

' We then look within domain.xml to find the profile reference based on the group name that we retrieved from host.xml 
' After the profile reference we then can find the socket-bindings for http and https ports

While the information is there, these values are not added to the registry.

Thus, so far the script is looking in the right place to find the used ports for that particular Jboss7 Service, yet the monitor for the BeanSpy is not using that port to contact the BeanSpy.

My investigation continues...

• Edited by Manni Calavera Saturday, March 07, 2015 12:06 PM

The BeanSpy monitor (for deep monitoring) might not be correct either.

In the Health Explorer looking at the Details of the context the Request executed to verify BeanSpy response is using the physical host name. If case are multiple Jboss Instances with unique IP addresses running on this host only the BeanSpy loaded in the Jboss Instance that is also using the IP adress of the host would answer.

If there are no instances on that IP address the monitor would fail, in case of multiple instances the result might not reflect the status of the appropriate BeanSpy. Instead the BeanSpy monitor should use the DNS name of JBoss service

Jboss 3 is not supported by the Jboss MP. But we currently do have a such Jboss3 instance sitting on a server together with a couple Jboss5 instances and it's making the Jboss scripts crash, thus the Jboss5 instances cannot be properly discovered until we stop the Jboss3 and let the discovery scripts do their job at least once.

I strongly believe I found the problem in the Jboss ManagementPack.

For JBoss7 running in domain mode it is perfectly possible to have a JBoss domaincontroller running on a different server than the hostcontroller(s).

Function getJBossDomainPorts (oAPI, objProcessInfo, userSuppliedFSO, userSuppliedHostXmlDom, userSuppliedDomainXmlDom). Is used to retrieve the xml.

This Function is being called by:

Set thePorts = getJBossDomainPorts (oAPI, objProcessInfo, userSuppliedFSO, userSuppliedXmlDom, userSuppliedXmlDom)

However if this is true, than the first function starts with wrong information. The value userSuppliedDomainXmlDom is not necessarily equal to userSuppliedXmlDom.

This makes the vbs script only look on the localhost for the domain.xml file.

I confirmed this by copying the domain.xml file to the host in same folder where the host.xml file resides.

Can anyone from Microsoft verify these findings ?

 

• Marked as answer by Manni Calavera 18 hours 55 minutes ago

I strongly believe I found the problem in the Jboss ManagementPack.

For JBoss7 running in domain mode it is perfectly possible to have a JBoss domaincontroller running on a different server than the hostcontroller(s).

Function getJBossDomainPorts (oAPI, objProcessInfo, userSuppliedFSO, userSuppliedHostXmlDom, userSuppliedDomainXmlDom). Is used to retrieve the xml.

This Function is being called by:

Set thePorts = getJBossDomainPorts (oAPI, objProcessInfo, userSuppliedFSO, userSuppliedXmlDom, userSuppliedXmlDom)

However if this is true, than the first function starts with wrong information. The value userSuppliedDomainXmlDom is not necessarily equal to userSuppliedXmlDom.

This makes the vbs script only look on the localhost for the domain.xml file.

I confirmed this by copying the domain.xml file to the host in same folder where the host.xml file resides.

Can anyone from Microsoft verify these findings ?

 

• Marked as answer by Manni Calavera Tuesday, March 10, 2015 12:14 PM

I strongly believe I found the problem in the Jboss ManagementPack.

For JBoss7 running in domain mode it is perfectly possible to have a JBoss domaincontroller running on a different server than the hostcontroller(s).

Function getJBossDomainPorts (oAPI, objProcessInfo, userSuppliedFSO, userSuppliedHostXmlDom, userSuppliedDomainXmlDom). Is used to retrieve the xml.

This Function is being called by:

Set thePorts = getJBossDomainPorts (oAPI, objProcessInfo, userSuppliedFSO, userSuppliedXmlDom, userSuppliedXmlDom)

However if this is true, than the first function starts with wrong information. The value userSuppliedDomainXmlDom is not necessarily equal to userSuppliedXmlDom.

This makes the vbs script only look on the localhost for the domain.xml file.

I confirmed this by copying the domain.xml file to the host in same folder where the host.xml file resides.

Can anyone from Microsoft verify these findings ?

 

• Marked as answer by Manni Calavera Tuesday, March 10, 2015 12:14 PM

Hi Manni,

Currently monitoring of JBoss Application Servers in domain mode requires every Windows Server/ Linux Server running application servers in the domain to be monitored with the OpsMgr Windows or Linux agent. Each application server in the domain is monitored individually with unique BeanSpy statistics.

In order to have a correct automatic discovery the profiles specified in the domain xml from the domain-controller should also be present on the monitored host-controllers. The easiest way is to copy the domain.xml file from the domain-controller into the domain/configuration directory of the host-controllers. An example of domain mode monitoring is found here:

http://blogs.technet.com/b/momteam/archive/2015/02/12/managing-jboss-application-server-7-with-system-center-operations-manager-2012-r2-part-2-domain-mode.aspx

If there are specific questions email me at anugup at the usual Microsoft

Hi,

I'm very aware that JBoss has quite a unique structure to service applications. In our environment we have several JBoss domain controllers, each domain contains multiple JBoss hosts, some of these hosts run multiple Jboss servers, some of them with a port offset. Within at least one domain we also servers running with portoffset on different hosts. All of the (required) Jboss servers that need deep monitoring have their own beanspy installed and working fine.

Considering this the configuration of JBoss and beanspy is ok. What I don't understand or perhaps what I'm curious about is why do I need to 'manually' copy domain.xml files to hosts rather than the discovery script obtaining the domaincontroller name from the host.xml file and retrieve the domain.xml from it and/or cache and/or compare it for changes everytime the discovery runs. I can imagine this could be added in a next version of the MP.

The fact that the domain.xml retrieval only happens local on the Jboss hosts also results in the basic discovery missing HTTP/HTTPS port information (regardless of turning on manual/deep monitoring). But since the basic monitoring only watches for java processes running that meet the filters, the basic process monitoring is working.

• Edited by Manni Calavera 21 hours 28 minutes ago

Hi,

I'm very aware that JBoss has quite a unique structure to service applications. In our environment we have several JBoss domain controllers, each domain contains multiple JBoss hosts, some of these hosts run multiple Jboss servers, some of them with a port offset. Within at least one domain we also servers running with portoffset on different hosts. All of the (required) Jboss servers that need deep monitoring have their own beanspy installed and working fine.

Considering this the configuration of JBoss and beanspy is ok. What I don't understand or perhaps what I'm curious about is why do I need to 'manually' copy domain.xml files to hosts rather than the discovery script obtaining the domaincontroller name from the host.xml file and retrieve the domain.xml from it and/or cache and/or compare it for changes everytime the discovery runs. I can imagine this could be added in a next version of the MP.

The fact that the domain.xml retrieval only happens local on the Jboss hosts also results in the basic discovery missing HTTP/HTTPS port information (regardless of turning on manual/deep monitoring). But since the basic monitoring only watches for java processes running that meet the filters, the basic process monitoring is working.

• Edited by Manni Calavera Thursday, March 12, 2015 9:40 AM

Hi,

I'm very aware that JBoss has quite a unique structure to service applications. In our environment we have several JBoss domain controllers, each domain contains multiple JBoss hosts, some of these hosts run multiple Jboss servers, some of them with a port offset. Within at least one domain we also servers running with portoffset on different hosts. All of the (required) Jboss servers that need deep monitoring have their own beanspy installed and working fine.

Considering this the configuration of JBoss and beanspy is ok. What I don't understand or perhaps what I'm curious about is why do I need to 'manually' copy domain.xml files to hosts rather than the discovery script obtaining the domaincontroller name from the host.xml file and retrieve the domain.xml from it and/or cache and/or compare it for changes everytime the discovery runs. I can imagine this could be added in a next version of the MP.

The fact that the domain.xml retrieval only happens local on the Jboss hosts also results in the basic discovery missing HTTP/HTTPS port information (regardless of turning on manual/deep monitoring). But since the basic monitoring only watches for java processes running that meet the filters, the basic process monitoring is working.

• Edited by Manni Calavera Thursday, March 12, 2015 9:40 AM