So today I noticed that the drive containing the inboxes on my primary site was filled.  Upon further investigation it appears that the BGB.box folder was filled with .__M files and the inbox\bgb.box\bad folder was filled with .BOS files (BGB online status files).  The bgbmgr.log file has several entries indicating the following:

Inbox changes detected. Sleep for 1 second and process inbox files    SMS_NOTIFICATION_MANAGER    3/4/2014 2:30:55 PM    3240 (0x0CA8)
Begin to get file list to process    SMS_NOTIFICATION_MANAGER    3/4/2014 2:30:56 PM    3240 (0x0CA8)
Begin to handle file 1dyj1xcg.BOS    SMS_NOTIFICATION_MANAGER    3/4/2014 2:30:56 PM    3240 (0x0CA8)
Begin to process file D:\Program Files\Microsoft Configuration Manager\inboxes\bgb.box\1dyj1xcg.BOS    SMS_NOTIFICATION_MANAGER    3/4/2014 2:30:56 PM    3240 (0x0CA8)
ERROR: Failed to parse online status file    SMS_NOTIFICATION_MANAGER    3/4/2014 2:30:56 PM    3240 (0x0CA8)
ERROR: Failed to execute task class OnlineStatusParser    SMS_NOTIFICATION_MANAGER    3/4/2014 2:30:56 PM    3240 (0x0CA8)
ERROR: Failed to parse file    SMS_NOTIFICATION_MANAGER    3/4/2014 2:30:56 PM    3240 (0x0CA8)
ERROR: Failed to execute task class OnlineStatusProcessTask    SMS_NOTIFICATION_MANAGER    3/4/2014 2:30:56 PM    3240 (0x0CA8)
WARNING: Failed to process file 1dyj1xcg.BOS, move it to bad inbox    SMS_NOTIFICATION_MANAGER    3/4/2014 2:30:56 PM    3240 (0x0CA8)
Begin to move file from D:\Program Files\Microsoft Configuration Manager\inboxes\bgb.box\1dyj1xcg.BOS to D:\Program Files\Microsoft Configuration Manager\inboxes\bgb.box\bad\1dyj1xcg.BOS    SMS_NOTIFICATION_MANAGER    3/4/2014 2:30:56 PM    3240 (0x0CA8)
BgbManager is waiting for file and registry change notification or timeout after 60 seconds    SMS_NOTIFICATION_MANAGER    3/4/2014 2:30:56 PM    3240 (0x0CA8)
SQL>>>set quoted_identifier on;set ansi_warnings on;set ansi_padding on;set ansi_nulls on;set concat_null_yields_null on;set arithabort on;set numeric_roundabort off;set DATEFORMAT mdy;    SMS_NOTIFICATION_MANAGER    3/4/2014 2:31:44 PM    3248 (0x0CB0)
SQL>>>>> Done.    SMS_NOTIFICATION_MANAGER    3/4/2014 2:31:44 PM    3248 (0x0CB0)

I cleared out (moved the BOS files) the outbox on the MP server and inbox on the PS which slowed the assault but as soon as a new BOS file is created, the errors appear again.  Looking into the BOS files, it looks like the entries are consistent so I am not sure how the parser is failing.

Just to cover my bases on supplying info I ran the following query against the primary site server DB:

select B.ServerName, A.OnlineClients as 'Online Clients' from Bgb_Server B left
join dbo.v_BgbServerCurrent A on A.ServerID=B.ServerID order by 2

The response was the FQDN of the MP server and the online clients count was 0.

It seems this all started when I had Premier Support track down a corrupted Status message SQL file that was preventing status messages from processing on the affected primary site.  The other primaries are as happy as can be.

If this first occurred when Premier worked on your environment then you might try reaching back out to them. Maybe this is something they can/should help you directly with.

Outside of that I can honestly say I've never seen this issue, nor is Google any help either.  I hope someone on these forums or at Microsoft is able to help.  Good luck!

• Proposed as answer by Garth JonesMVP, Moderator 19 hours 34 minutes ago
• Unproposed as answer by Keith P. Sumlar 12 hours 13 minutes ago

If this first occurred when Premier worked on your environment then you might try reaching back out to them. Maybe this is something they can/should help you directly with.

Outside of that I can honestly say I've never seen this issue, nor is Google any help either.  I hope someone on these forums or at Microsoft is able to help.  Good luck!

• Proposed as answer by Garth JonesMVP, Moderator Thursday, March 05, 2015 1:03 PM
• Unproposed as answer by Keith P. Sumlar Thursday, March 05, 2015 8:24 PM

I just replied to the last email I got from the engineer on the previous issue. 

Not sure if they will do anything without opening another Premier Support ticket.  We will see.  I will be sure to post a fix when one is figured out since this doesn't appear in any web searches I've made.

For the benefit of those out on the web below is the answer in my case.  I forgot about this thread until some email reminded me it existed and I thought posting the solution would be helpful.

I have resolved this issue myself. 

 

When setting up a new HTTPS MP I noticed the same behavior with the BOS files failing to process correctly.  I am not sure if the logs read something different on the HTTP MP to which this ticket is open for but the entry that stuck out at me indicated that the sending server (the site system with the MP role) is what moved the BOS file from filename.__M to filename.BOS in the inbox\bgb.box folder on the primary site server.  Looking at the special perms of the inbox\bgb.box folder and comparing it to another primary site server, it turns out that the permissions to the bgb.box folder were incorrect for the SMS_SiteSystemToSiteServerConnection_MP_sitecode group.  It displayed the basic perms to read and write, but lacked perms to append data or modify attributes.

 

My guess is that work on a previous ticket resulted in the perm getting changed, causing the issue.  Because the perms in question were only visible in the advanced security settings screen, it was easily overlooked.

• Marked as answer by Keith P. Sumlar 12 hours 27 minutes ago

For the benefit of those out on the web below is the answer in my case.  I forgot about this thread until some email reminded me it existed and I thought posting the solution would be helpful.

I have resolved this issue myself. 

 

When setting up a new HTTPS MP I noticed the same behavior with the BOS files failing to process correctly.  I am not sure if the logs read something different on the HTTP MP to which this ticket is open for but the entry that stuck out at me indicated that the sending server (the site system with the MP role) is what moved the BOS file from filename.__M to filename.BOS in the inbox\bgb.box folder on the primary site server.  Looking at the special perms of the inbox\bgb.box folder and comparing it to another primary site server, it turns out that the permissions to the bgb.box folder were incorrect for the SMS_SiteSystemToSiteServerConnection_MP_sitecode group.  It displayed the basic perms to read and write, but lacked perms to append data or modify attributes.

 

My guess is that work on a previous ticket resulted in the perm getting changed, causing the issue.  Because the perms in question were only visible in the advanced security settings screen, it was easily overlooked.

• Marked as answer by Keith P. Sumlar Thursday, March 05, 2015 8:10 PM