SCOM 2012 event 29120 on Management Server

Hi,

In our SCOM 2012 environment we have 4 management servers spread over 4 datacenters.

Everything was fine until I started deploying Linux agents. Therefor I needed to import the necessary management packs.

After the import I'm seeing event 29120 on 2 of my management servers:

OpsMgr Management Configuration Service failed to process configuration request (Xml configuration file or management pack request) due to the following exception

Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.DataAccessException: Data access operation failed
   at Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.DataAccessOperation.ExecuteSynchronously(Int32 timeoutSeconds, WaitHandle stopWaitHandle)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.CmdbDataProvider.GetManagementPack(Guid managementPackVersionDependentId)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.TracingConfigurationDataProvider.GetManagementPack(Guid managementPackVersionDependentId)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.AgentRequestProcessor.ProcessManagementPackRequest(ManagementPackRequestDescriptor requestDescriptor)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.AgentRequestProcessor.Execute()
   at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.ThreadManager.ResponseThreadStart(Object state)
-----------------------------------
System.Data.SqlClient.SqlException (0x80131904): Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
   at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.ReadBuffer()
   at System.Data.SqlClient.TdsParserStateObject.ReadByteArray(Byte[] buff, Int32 offset, Int32 len)
   at System.Data.SqlClient.TdsParserStateObject.ReadUInt32()
   at System.Data.SqlClient.TdsParserStateObject.ReadPlpLength(Boolean returnPlpNullIfNull)
   at System.Data.SqlClient.TdsParserStateObject.ReadPlpBytes(Byte[]& buff, Int32 offst, Int32 len)
   at System.Data.SqlClient.TdsParser.ReadSqlValue(SqlBuffer value, SqlMetaDataPriv md, Int32 length, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlDataReader.ReadColumnData()
   at System.Data.SqlClient.SqlDataReader.ReadColumn(Int32 i, Boolean setTimeout)
   at System.Data.SqlClient.SqlDataReader.GetSqlBinary(Int32 i)
   at System.Data.SqlClient.SqlDataReader.GetBytesInternal(Int32 i, Int64 dataIndex, Byte[] buffer, Int32 bufferIndex, Int32 length)
   at System.Data.SqlClient.SqlDataReader.GetBytes(Int32 i, Int64 dataIndex, Byte[] buffer, Int32 bufferIndex, Int32 length)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.ManagementPackGetOperation.ReadResourceList(SqlDataReader reader)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.ManagementPackGetOperation.ReadData(SqlDataReader reader)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.ReaderSqlCommandOperation.SqlCommandCompleted(IAsyncResult asyncResult)

When I check the "AgentManagement" folder (under server) and the "Management Packs" (under Health Service State) folder on those servers, the files are not there.

Anybody has an idea why this timeout is happening? I have to say the first 2 management servers and database server are located in Europe and the 2 other management server where I'm experiencing the problem are located in North America. But a copy of the files goes fast and everything else seems to work fine.

Another strange thing is that the Library management packs are transferred.

Example: Microsoft.Linux.Library and Microsoft.Linux.RedHat.Library are there, Microsoft.Linux.RHEL.6 isn't.

This is causing the All Management Group resource pool and others to be unavailable and greyed out.

Kind regards,

Peter

February 26th, 2014 11:08am

Hi

I think you have 2 problems:

1) The error above Looks like a timeout between Management Server and SCOM configuration service. This could be due to high latency or delays in responding. Do you see any other events in the OperationsManager Event log?

System.Data.SqlClient.SqlException (0x80131904): Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.

2) Concerning the missing Linux Agent kits, you missed to Import all MP's check this post here http://social.technet.microsoft.com/Forums/en-US/0e2007b2-34b1-40aa-a3c4-50f4b9663608/scom-2012-sp1-redhat-enterprise-63-no-installable-unixlinux-agent-is-available?forum=operationsmanagerunixandlinux .

Cheers,

Stefan

Free Windows Admin Tool Kit Click here and download it now
February 26th, 2014 4:34pm

Latency is a possibility, but it only started when importing Linux management packs.

Those mp's indeed contain the install kits, which is much more, but you would expect that an XML of about 300kb should not cause any trouble. The libraries of 13kb are there.

The other error I see is event 1209: Management Pack with ID... has been requested X times.

What do you mean with "all MP's"?

I've imported the unix, linux and redhat linux libraries and redhat MP's 4,5 and 6.

February 27th, 2014 7:36am

Latency is a possibility, but it only started when importing Linux management packs.

Those mp's indeed contain the install kits, which is much more, but you would expect that an XML of about 300kb should not cause any trouble. The libraries of 13kb are there.

The other error I see is event 1209: Management Pack with ID... has been requested X times.

What do you mean with "all MP's"?

I've imported the unix, linux and redhat linux libraries and redhat MP's 4,5 and 6.

Free Windows Admin Tool Kit Click here and download it now
February 27th, 2014 3:34pm

Latency is a possibility, but it only started when importing Linux management packs.

Those mp's indeed contain the install kits, which is much more, but you would expect that an XML of about 300kb should not cause any trouble. The libraries of 13kb are there.

The other error I see is event 1209: Management Pack with ID... has been requested X times.

What do you mean with "all MP's"?

I've imported the unix, linux and redhat linux libraries and redhat MP's 4,5 and 6.

February 27th, 2014 3:34pm

Latency is a possibility, but it only started when importing Linux management packs.

Those mp's indeed contain the install kits, which is much more, but you would expect that an XML of about 300kb should not cause any trouble. The libraries of 13kb are there.

The other error I see is event 1209: Management Pack with ID... has been requested X times.

What do you mean with "all MP's"?

I've imported the unix, linux and redhat linux libraries and redhat MP's 4,5 and 6.

Free Windows Admin Tool Kit Click here and download it now
February 27th, 2014 3:34pm

Hi,

Please try to update SP1 with latest Rollup for a test.

February 28th, 2014 12:38am

Hi,

We are already at UR4.

I did another test:

- removed all OS management packs for linux + library's -> all management servers became healthy again instead of greyed out.

- imported Windows 2003 and 2008 management packs: all MP's where distributed to all management servers. -> everything was still fine.

- imported the redhat library and RHEL 6 MP: events 29120 started to come in.

Although I also received events 29121 which states "OpsMgr Management Configuration Service successfully processed configuration request (Xml configuration file or management pack request)" the xml files are not located under the management packs folder.

For the moment all management servers are healthy, but I doubt that the 2 in question will be able to monitor Linux agents.

Some other error events I see now:

- OpsMgr SDK Service ID 26319: 

An exception was thrown while processing GetObjectsFromReader for session ID uuid:56799f6e-b37f-4584-88e6-ea7538bd48cc;id=2.
 Exception message: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)
 Full Exception: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)

- Health Service Modules ID 22409: 

The PowerShell script failed with below exception

System.Management.Automation.ActionPreferenceStopException: Command execution stopped because the preference variable "ErrorActionPreference" or common parameter is set to Stop: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)
At line:10 char:32
+ $alertsByObject = Get-SCOMAlert <<<<  -ResolutionState 0 | group MonitoringObjectId
   at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input, Hashtable errorResults, Boolean enumerate)
   at System.Management.Automation.PipelineNode.Execute(Array input, Pipe outputPipe, ArrayList& resultList, ExecutionContext context)
   at System.Management.Automation.ParseTreeNode.Execute(Array input, Pipe outputPipe, ExecutionContext context)
   at System.Management.Automation.AssignmentStatementNode.Execute(Array input, Pipe outputPipe, ExecutionContext context)
   at System.Management.Automation.StatementListNode.ExecuteStatement(ParseTreeNode statement, Array input, Pipe outputPipe, ArrayList& resultList, ExecutionContext context)

 

Script Name: GetMGAlertsCount.ps1 

One or more workflows were affected by this.  

Workflow name: ManagementGroupCollectionAlertsCountRule 
Instance name: All Management Servers Resource Pool 
Instance ID: {4932D8F0-C8E2-2F4B-288E-3ED98A340B9F} 
Management group: DSGOpsMgr2012PRD

Free Windows Admin Tool Kit Click here and download it now
March 4th, 2014 4:30am

Geographically dispersed management servers are a bit of a grey area I tend to avoid them as there are question marks around their supportability.

Im not sure how much help this will be but theres a good discussions around this subject from a previous forum.

http://social.technet.microsoft.com/Forums/systemcenter/en-US/f089e1d3-378b-4602-b1ee-d45852bacd3b/scom-2012-event-29120-on-management-server?forum=operationsmanagermgmtpacks&prof=required

March 4th, 2014 5:46am

Do you have another link? The link you posted is this thread.

I found this one and it seems that I might need a design change.

http://social.technet.microsoft.com/Forums/en-US/85ca6bab-9c80-43b8-a2e2-17a3cbdbe1b9/system-center-2012-operations-manager-in-multiple-sites?forum=operationsmanagerdeployment

Free Windows Admin Tool Kit Click here and download it now
March 4th, 2014 7:37am

Sorry about that - I've updated the post with the following url:

http://social.technet.microsoft.com/Forums/systemcenter/en-US/c5c14c95-be00-44a4-8ae8-fa13875b6f2d/scom-2012-datacentre-site-resiliency?forum=operationsmanagerdeployment

March 4th, 2014 9:04am

Geographically dispersed management servers are a bit of a grey area I tend to avoid them as there are question marks around their supportability.

Im not sure how much help this will be but theres a good discussions around this subject from a previous forum.

http://social.technet.microsoft.com/Forums/systemcenter/en-US/c5c14c95-be00-44a4-8ae8-fa13875b6f2d/scom-2012-datacentre-site-resiliency?forum=operationsmanagerdeployment

http://social.technet.microsoft.com/Forums/en-US/85ca6bab-9c80-43b8-a2e2-17a3cbdbe1b9/system-center-2012-operations-manager-in-multiple-sites?forum=operationsmanagerdeployment

  • Edited by Chunky.1 21 hours 30 minutes ago
Free Windows Admin Tool Kit Click here and download it now
March 4th, 2014 1:44pm

Geographically dispersed management servers are a bit of a grey area I tend to avoid them as there are question marks around their supportability.

Im not sure how much help this will be but theres a good discussions around this subject from a previous forum.

http://social.technet.microsoft.com/Forums/systemcenter/en-US/c5c14c95-be00-44a4-8ae8-fa13875b6f2d/scom-2012-datacentre-site-resiliency?forum=operationsmanagerdeployment

http://social.technet.microsoft.com/Forums/en-US/85ca6bab-9c80-43b8-a2e2-17a3cbdbe1b9/system-center-2012-operations-manager-in-multiple-sites?forum=operationsmanagerdeployment

  • Edited by Chunky.1 Tuesday, March 04, 2014 2:04 PM
March 4th, 2014 1:44pm

Do you have another link? The link you posted is this thread.

I found this one and it seems that I might need a design change.

http://social.technet.microsoft.com/Forums/en-US/85ca6bab-9c80-43b8-a2e2-17a3cbdbe1b9/system-center-2012-operations-manager-in-multiple-sites?forum=operationsmanagerdeployment

Free Windows Admin Tool Kit Click here and download it now
March 4th, 2014 3:36pm

Sorry about that - I've updated the post with the following url:

http://social.technet.microsoft.com/Forums/systemcenter/en-US/c5c14c95-be00-44a4-8ae8-fa13875b6f2d/scom-2012-datacentre-site-resiliency?forum=operationsmanagerdeployment

March 4th, 2014 5:02pm

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics