Hi,

SCOM 2012 R2

We have trouble to run scom agent on centos6 coexistation with OpenVPN server. When openvpn server service is running omiserver log errors and scom server lost heart beating to scom agent. When stop OpenVpn service, connection is back

Does anybody know solution?

Omiserver log file
-----------------------------
2014/04/15 15:14:00: WARNING: lost connection to agent running as [0]
2014/04/15 15:15:01: WARNING: lost connection to agent running as [0]
2014/04/15 15:15:45: WARNING: _WriteV - Error on writev for socket 11: Broken pipe
2014/04/15 15:17:55: WARNING: lost connection to agent running as [0]
2014/04/15 15:17:55: WARNING: cannot send message: invalid handler (msg->clientID) (nil)
2014/04/15 15:17:55: WARNING: cannot send message: invalid handler (msg->clientID) (nil)
2014/04/15 15:17:55: WARNING: cannot send message: invalid handler (msg->clientID) (nil)
2014/04/15 15:17:55: WARNING: cannot send message: invalid handler (msg->clientID) (nil)
2014/04/15 15:17:55: WARNING: cannot send message: invalid handler (msg->clientID) (nil)
2014/04/15 15:17:55: WARNING: cannot send message: invalid handler (msg->clientID) (nil)
2014/04/15 15:17:55: WARNING: cannot send message: invalid handler (msg->clientID) (nil)
2014/04/15 15:17:55: WARNING: cannot send message: invalid handler (msg->clientID) (nil)
2014/04/15 15:17:55: WARNING: _Sock_Read - SSL_read returned OS error: 9
2014/04/15 15:17:56: ERROR: Error on read for socket 12: Connection reset by peer
2014/04/15 15:17:56: WARNING: lost connection to agent running as [0]
2014/04/15 15:20:45: ERROR: Error on read for socket 12: Connection reset by peer
2014/04/15 15:20:45: WARNING: lost connection to agent running as [0]
2014/04/15 15:20:48: ERROR: Error on read for socket 15: Connection reset by peer
2014/04/15 15:20:48: WARNING: lost connection to agent running as [0]
2014/04/15 15:30:45: WARNING: _WriteV - Error on writev for socket 15: Broken pipe
2014/04/15 15:30:48: WARNING: lost connection to agent running as [0]
2014/04/15 15:30:48: WARNING: _Sock_Read - SSL_read returned OS error: 9
2014/04/15 15:30:49: WARNING: lost connection to agent running as [0]
2014/04/15 15:30:53: WARNING: lost connection to agent running as [0]
2014/04/15 15:30:57: WARNING: lost connection to agent running as [0]

My guess here is that once OpenVpn is enabled it blocks all traffic to the system unless it connects via VPN. Once OpenVpn is enabled can you ping the system from the SCOM server? What about SSH, can you SSH into the system when OpenVpn is enabled. The agent listens on port 1270, if you can configure OpenVpn to not block this I would think it would work whether OpenVpn is enabled or not.

Regards,

-Steve

• Edited by Steve Weber [MSFT]Microsoft employee, Moderator Wednesday, April 16, 2014 6:00 PM

My guess here is that once OpenVpn is enabled it blocks all traffic to the system unless it connects via VPN. Once OpenVpn is enabled can you ping the system from the SCOM server? What about SSH, can you SSH into the system when OpenVpn is enabled. The agent listens on port 1270, if you can configure OpenVpn to not block this I would think it would work whether OpenVpn is enabled or not.

Regards,

-Steve

• Edited by Steve Weber [MSFT]Microsoft employee, Moderator Wednesday, April 16, 2014 6:00 PM

Hi,

SSH port and all other ports are not blocked by OpenVpn. OPenVPN used just UDP port 1194. LinuxSE is disabled, Linux firewall is disabled.

Configuration of my OpenVPN server

port 1194
proto udp
dev tun0
ca keys/oltistestca/ca.crt
cert keys/oltistestca/server1.crt
key keys/oltistestca/server1.key
dh keys/oltistestca/dh2048.pem
server 10.8.0.0 255.255.255.0
crl-verify keys/oltistestca/crl.pem
cipher DES-CBC
user nobody
group adm
status servers/changeme/logs/openvpn-status.log
log-append servers/changeme/logs/openvpn.log
verb 2
mute 20
max-clients 100
keepalive 10 120
client-config-dir /etc/openvpn/servers/changeme/ccd
comp-lzo
persist-key
persist-tun
ccd-exclusive

With OpenVpn running try to run the following command from a command prompt on the SCOM server.

winrm enumerate  http://schemas.microsoft.com/wbem/wscim/1/cim-schema/2/SCX_Agent?__cimnamespace=root/scx -username:<Linux User> -password:<Linux Password> -r:https://<Linux Server>:1270/wsman -auth:basic -encoding:utf-8  

Does this work? My guess will be it does not. With OpenVpn still running log onto the Linux server and try running following commands:

.  /opt/microsoft/scx/bin/tools/setup.sh  (note the space between the dot and slash as you are sourcing the setup.sh file)

omicli ei root/scx SCX_agent

Does this work? If so that tells us the agent is running fine but we fail to communicate to it via port 1270 using the wsman protocol. If the omicli command fails then something is interfering with the agent itself.

You can also try telneting to port 1270 on the Linux server when OpenVpn is running and see if you can connect. It will not return anything but it should at least connect.

Regards,

-Steve

• Edited by Steve Weber [MSFT]Microsoft employee, Moderator Thursday, April 17, 2014 1:44 PM

With OpenVpn running try to run the following command from a command prompt on the SCOM server.

winrm enumerate  http://schemas.microsoft.com/wbem/wscim/1/cim-schema/2/SCX_Agent?__cimnamespace=root/scx -username:<Linux User> -password:<Linux Password> -r:https://<Linux Server>:1270/wsman -auth:basic -encoding:utf-8  

Does this work? My guess will be it does not. With OpenVpn still running log onto the Linux server and try running following commands:

.  /opt/microsoft/scx/bin/tools/setup.sh  (note the space between the dot and slash as you are sourcing the setup.sh file)

omicli ei root/scx SCX_agent

Does this work? If so that tells us the agent is running fine but we fail to communicate to it via port 1270 using the wsman protocol. If the omicli command fails then something is interfering with the agent itself.

You can also try telneting to port 1270 on the Linux server when OpenVpn is running and see if you can connect. It will not return anything but it should at least connect.

Regards,

-Steve

• Edited by Steve Weber [MSFT]Microsoft employee, Moderator Thursday, April 17, 2014 1:44 PM

Hi,

Thank you for reaction.

What does the first command winrm enumerate ... do? There is a calling web interface with sending password ?...

Second command on linux > result is here

----------------------------

[root@centostest ~]# . /opt/microsoft/scx/bin/tools/setup.sh
[root@centostest ~]# omicli ei root/scx SCX_agent
omicli: failed to connect to /var/opt/microsoft/scx/run/omiserver.sock
[root@centostest ~]#

---------------------------

same result as in log I send first. There is a failed with sock comunications. Port 1270 on Linux is running and I can connect by telnet to it.

Regards, Ales

• Edited by Alftech Tuesday, April 22, 2014 11:34 AM

Hi,

Thank you for reaction.

What does the first command winrm enumerate ... do? There is a calling web interface with sending password ?...

Second command on linux > result is here

----------------------------

[root@centostest ~]# . /opt/microsoft/scx/bin/tools/setup.sh
[root@centostest ~]# omicli ei root/scx SCX_agent
omicli: failed to connect to /var/opt/microsoft/scx/run/omiserver.sock
[root@centostest ~]#

---------------------------

same result as in log I send first. There is a failed with sock comunications. Port 1270 on Linux is running and I can connect by telnet to it.

Regards, Ales

• Edited by Alftech Tuesday, April 22, 2014 11:34 AM

The first command is calling a schema on the agent which tells the agent what provider is being called. It is not going out to the web. This is how WSMan works. You can check out the following links if you want more details.

http://msdn.microsoft.com/en-us/library/aa384470%28v=vs.85%29.aspx

http://en.wikipedia.org/wiki/WS-Management

For the second command, can you run it again and if it fails run the following to verify the OMI daemon is running.

ps -ef | grep omiserver

Which should return something similar:

 root 24264     1  0 11:27:40 ?        17:57 /opt/microsoft/scx/bin/omiserver -d

If the OMI daemon is running but the omicli command still fails, OpenVpn is somehow blocking this as my guess is once OpenVpn is shutdown the omicli command will work.

If the OMI daemon is not running try starting it by running:

scxadmin -start

then try running the omicli command again.

Regards,

-Steve

• Edited by Steve Weber [MSFT]Microsoft employee, Moderator Tuesday, April 22, 2014 2:37 PM

The first command is calling a schema on the agent which tells the agent what provider is being called. It is not going out to the web. This is how WSMan works. You can check out the following links if you want more details.

http://msdn.microsoft.com/en-us/library/aa384470%28v=vs.85%29.aspx

http://en.wikipedia.org/wiki/WS-Management

For the second command, can you run it again and if it fails run the following to verify the OMI daemon is running.

ps -ef | grep omiserver

Which should return something similar:

 root 24264     1  0 11:27:40 ?        17:57 /opt/microsoft/scx/bin/omiserver -d

If the OMI daemon is running but the omicli command still fails, OpenVpn is somehow blocking this as my guess is once OpenVpn is shutdown the omicli command will work.

If the OMI daemon is not running try starting it by running:

scxadmin -start

then try running the omicli command again.

Regards,

-Steve

• Edited by Steve Weber [MSFT]Microsoft employee, Moderator Tuesday, April 22, 2014 2:37 PM

Hi Steve,

Omiserver is running, it still log to omniserver log as I wrote at top of this thread. After openvpn is started then not connection to omiserver port 1270 is not broken, it still work.

How is all that working? Omiserver is just gateway to omicli that getting Linux state? What is Linux agent infrastructure of getting Linux states?

Question - why just omiserver is broken? Tested on two clear instalation of centos 6.4 and 6.5 with same result.

Regards,

Ales

• Edited by Alftech Wednesday, April 23, 2014 5:54 AM

Hi Steve,

Omiserver is running, it still log to omniserver log as I wrote at top of this thread. After openvpn is started then not connection to omiserver port 1270 is not broken, it still work.

How is all that working? Omiserver is just gateway to omicli that getting Linux state? What is Linux agent infrastructure of getting Linux states?

Question - why just omiserver is broken? Tested on two clear instalation of centos 6.4 and 6.5 with same result.

Regards,

Ales

• Edited by Alftech Wednesday, April 23, 2014 5:54 AM

I have no idea why OpenVpn breaks our agent at this time as we have never tested it and you are the first to ever report it that I'm aware of.

If this is something you would like addressed then you will need to open a support ticket with Microsoft and someone can work directly with you to investigate the problem and try to come up with a solution as trying to troubleshoot this over the forums is not feasible.

Regards,

-Steve

Hi Steve,

thank's for your patience. Strange thing is that omiserver still communicating some time after openvpn service is started and than communication is stopped. Omiserver and agent still running, port of omiserver is active and can connect to it.

When communication stops I can Kill processes omiserver and omiagent, start it a communication some time running again.

Scom error message perhaps for memory information is WSManFault ......

I thing OpenVPN make some changes to network communication, packet length or some other change that kill communication or it can be omiserver trouble.

Regards,

Ales

• Edited by Alftech Thursday, April 24, 2014 9:35 AM

Hi Steve,

thank's for your patience. Strange thing is that omiserver still communicating some time after openvpn service is started and than communication is stopped. Omiserver and agent still running, port of omiserver is active and can connect to it.

When communication stops I can Kill processes omiserver and omiagent, start it a communication some time running again.

Scom error message perhaps for memory information is WSManFault ......

I thing OpenVPN make some changes to network communication, packet length or some other change that kill communication or it can be omiserver trouble.

Regards,

Ales

• Edited by Alftech Thursday, April 24, 2014 9:35 AM

did you ended up finding the cause of this?  

I'm seeing the same error on 5 different servers, all have oracle but no OpenVPN installed. some of these server have solidcore file integrity monitoring installed.