PKI Client starts to intialize, then 7 minutes later client agents go back to disabled for upwards of an hour

Starting a new thread on this as I have done much digging and no longer believe its a conflicting record issue.

I am SCCM 2012 SP1 on Server 2008 R2/SQL 2012 CU2.  My "Lan" based management point is HTTP and I also have an Internet Management point that is HTTPS.  All scenerios discussed in this thread are installing the 2012 SP1 client while connected to the Lan.  We have a fully functional PKI infrastructure with auto enrol enabled.

The issue at hand:

Wether I install the client during OSD, or manually install the client I get the same result.  This is that the client agent upon successful install begins communicating with the HTTP management point and starts retrieving policy.  If I open the ConfigMgr applet in Control Panel I see the client shows "Client Certificate: PKI", "Connection Type: Currently Intranet".  If I view actions I see all actions with the exception of Discovery Data Collection Cycle and Hardware Inventory Cycle.

I have watched the client logs the only thing that seems to stick out is in the CcmNotificationAgent.log which shows the bgb client agent actions.  it repeats the following aprox every 5 minutes:

bgb client agent is starting...
bgb client agent is disabled
TCP Listener is disabled
bgbController main thread us started with settings: [bgb enable = 0], {tcp enable = 0} and {http enable = 0}.
Wait 3600 seconds for even notification

The ClientIDManagerStartup.log files shows:

PopulateRegistrationHint: Using the Certificate selected by the current version of SCCM to set the hint. ClientIDManagerStartup 1/23/2013 5:00:51 PM 2100 (0x0834)
CCMCreateAuthHeadersEx failed (0x80004005). ClientIDManagerStartup 1/23/2013 5:00:52 PM 2100 (0x0834)
PopulateRegistrationHint failed (0x80004005), expected upon first start of non-upgrade client. ClientIDManagerStartup 1/23/2013 5:00:52 PM 2100 (0x0834)
Begin searching client certificates based on Certificate Issuers ClientIDManagerStartup 1/23/2013 5:00:52 PM 2100 (0x0834)
Certificate Issuer 1 [CN=myrootca; DC=mydomain; DC=com] ClientIDManagerStartup 1/23/2013 5:00:52 PM 2100 (0x0834)
Finding certificate by issuer chain returned error 80092004 ClientIDManagerStartup 1/23/2013 5:00:52 PM 2100 (0x0834)
Completed searching client certificates based on Certificate Issuers ClientIDManagerStartup 1/23/2013 5:00:52 PM 2100 (0x0834)
Unable to find any Certificate based on Certificate Issuers ClientIDManagerStartup 1/23/2013 5:00:52 PM 2100 (0x0834)
Raising event:

instance of CCM_ServiceHost_CertRetrieval_Status
{
 ClientID = "GUID:b4aacc70-6de3-4829-88e0-498777c49379";
 DateTime = "20130123230052.204000+000";
 HRESULT = "0x87d00215";
 ProcessID = 2352;
 ThreadID = 2100;
};
 ClientIDManagerStartup 1/23/2013 5:00:52 PM 2100 (0x0834)
Failed to submit event to the Status Agent. Attempting to create pending event. ClientIDManagerStartup 1/23/2013 5:00:52 PM 2100 (0x0834)
Raising pending event:

instance of CCM_ServiceHost_CertRetrieval_Status
{
 ClientID = "GUID:b4aacc70-6de3-4829-88e0-498777c49379";
 DateTime = "20130123230052.204000+000";
 HRESULT = "0x87d00215";
 ProcessID = 2352;
 ThreadID = 2100;
};
 ClientIDManagerStartup 1/23/2013 5:00:52 PM 2100 (0x0834)
PKI Client Certificate matching SCCM certificate selection criteria is not available. ClientIDManagerStartup 1/23/2013 5:00:52 PM 2100 (0x0834)
Generated a new Signing certificate ClientIDManagerStartup 1/23/2013 5:00:54 PM 2100 (0x0834)
Generated a new Encryption certificate ClientIDManagerStartup 1/23/2013 5:00:54 PM 2100 (0x0834)
Initializing registration renewal for potential PKI issued certificate changes. ClientIDManagerStartup 1/23/2013 5:00:56 PM 2348 (0x092C)
Succesfully intialized registration renewal. ClientIDManagerStartup 1/23/2013 5:00:56 PM 2348 (0x092C)
[RegTask] - Executing registration task synchronously. ClientIDManagerStartup 1/23/2013 5:00:56 PM 2348 (0x092C)
Read SMBIOS (encoded): 32003100350039002D0039003900320037002D0032003500360036002D0038003300320035002D0037003500340032002D0035003200370031002D0033003900 ClientIDManagerStartup 1/23/2013 5:00:56 PM 2348 (0x092C)
Evaluated SMBIOS (encoded): 32003100350039002D0039003900320037002D0032003500360036002D0038003300320035002D0037003500340032002D0035003200370031002D0033003900 ClientIDManagerStartup 1/23/2013 5:00:56 PM 2348 (0x092C)
No SMBIOS Changed ClientIDManagerStartup 1/23/2013 5:00:56 PM 2348 (0x092C)
SMBIOS unchanged ClientIDManagerStartup 1/23/2013 5:00:56 PM 2348 (0x092C)
SID unchanged ClientIDManagerStartup 1/23/2013 5:00:56 PM 2348 (0x092C)
HWID unchanged ClientIDManagerStartup 1/23/2013 5:00:57 PM 2348 (0x092C)
Windows To Go requires a minimum operating system of Windows 8 ClientIDManagerStartup 1/23/2013 5:00:59 PM 2348 (0x092C)
GetSystemEnclosureChassisInfo: IsFixed=TRUE, IsLaptop=FALSE ClientIDManagerStartup 1/23/2013 5:00:59 PM 2348 (0x092C)
Windows To Go requires a minimum operating system of Windows 8 ClientIDManagerStartup 1/23/2013 5:00:59 PM 2348 (0x092C)
Computed HardwareID=2:9C8C08C4B3E16249A2F1457998D16528B656DE30
 Win32_SystemEnclosure.SerialNumber=2159-9927-2566-8325-7542-5271-39
 Win32_SystemEnclosure.SMBIOSAssetTag=9344-3677-7824-5579-3797-0729-37
 Win32_BaseBoard.SerialNumber=2159-9927-2566-8325-7542-5271-39
 Win32_BIOS.SerialNumber=2159-9927-2566-8325-7542-5271-39
 Win32_NetworkAdapterConfiguration.MACAddress=00:15:5D:0B:78:20 ClientIDManagerStartup 1/23/2013 5:00:59 PM 2348 (0x092C)
[RegTask] - Client is not registered. Sending registration request for GUID:b4aacc70-6de3-4829-88e0-498777c49379 ... ClientIDManagerStartup 1/23/2013 5:00:59 PM 2348 (0x092C)
[RegTask] - Client registration is pending. Server assigned ClientID is GUID:b4aacc70-6de3-4829-88e0-498777c49379 ClientIDManagerStartup 1/23/2013 5:01:00 PM 2348 (0x092C)
[RegTask] - Sleeping for 60 seconds ... ClientIDManagerStartup 1/23/2013 5:01:00 PM 2348 (0x092C)
RenewalTask: Executing renewal task. ClientIDManagerStartup 1/23/2013 5:01:43 PM 2620 (0x0A3C)
Begin searching client certificates based on Certificate Issuers ClientIDManagerStartup 1/23/2013 5:01:43 PM 2620 (0x0A3C)
Certificate Issuer 1 [CN=myrootca; DC=mydomain; DC=com] ClientIDManagerStartup 1/23/2013 5:01:43 PM 2620 (0x0A3C)
Based on Certificate Issuer 'MyrootCA' found Certificate [Thumbprint 52BFCF407B7071512CE65F8868D66578244ABDD9] issued to 'myclient.mydomain.com' ClientIDManagerStartup 1/23/2013 5:01:43 PM 2620 (0x0A3C)
Begin validation of Certificate [Thumbprint 52BFCF407B7071512CE65F8868D66578244ABDD9] issued to 'myclient.mydomain.com' ClientIDManagerStartup 1/23/2013 5:01:43 PM 2620 (0x0A3C)
Completed validation of Certificate [Thumbprint 52BFCF407B7071512CE65F8868D66578244ABDD9] issued to 'myclient.mydomain.com' ClientIDManagerStartup 1/23/2013 5:01:43 PM 2620 (0x0A3C)
Completed searching client certificates based on Certificate Issuers ClientIDManagerStartup 1/23/2013 5:01:43 PM 2620 (0x0A3C)
Begin to select client certificate ClientIDManagerStartup 1/23/2013 5:01:43 PM 2620 (0x0A3C)
Begin validation of Certificate [Thumbprint 52BFCF407B7071512CE65F8868D66578244ABDD9] issued to 'myclient.mydomain.com' ClientIDManagerStartup 1/23/2013 5:01:43 PM 2620 (0x0A3C)
Completed validation of Certificate [Thumbprint 52BFCF407B7071512CE65F8868D66578244ABDD9] issued to 'myclient.mydomain.com' ClientIDManagerStartup 1/23/2013 5:01:43 PM 2620 (0x0A3C)
>>> Client selected the PKI Certificate [Thumbprint 52BFCF407B7071512CE65F8868D66578244ABDD9] issued to 'myclient.mydomain.com' ClientIDManagerStartup 1/23/2013 5:01:43 PM 2620 (0x0A3C)
Raising event:

instance of CCM_ServiceHost_CertRetrieval_Status
{
 ClientID = "GUID:b4aacc70-6de3-4829-88e0-498777c49379";
 DateTime = "20130123230143.106000+000";
 HRESULT = "0x00000000";
 ProcessID = 2352;
 ThreadID = 2620;
};
 ClientIDManagerStartup 1/23/2013 5:01:43 PM 2620 (0x0A3C)
Client PKI cert is available. ClientIDManagerStartup 1/23/2013 5:01:43 PM 2620 (0x0A3C)
RenewalTask: Certificate has changed, initiating a renewal. ClientIDManagerStartup 1/23/2013 5:01:43 PM 2620 (0x0A3C)
Aborting any pending registration. ClientIDManagerStartup 1/23/2013 5:01:43 PM 2620 (0x0A3C)
Re-registration/renewal initiated. Restarting the service. ClientIDManagerStartup 1/23/2013 5:01:43 PM 2620 (0x0A3C)
[----- SHUTDOWN -----] ClientIDManagerStartup 1/23/2013 5:01:44 PM 2100 (0x0834)
[----- STARTUP -----] ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Machine: W21599927256 ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
OS Version: 6.1 Service Pack 1 ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
SCCM Client Version: 5.00.7804.1000 ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Client is set to use HTTPS when available. The current state is 448. ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Begin searching client certificates based on Certificate Issuers ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Certificate Issuer 1 [CN=myrootca; DC=mydomain; DC=com] ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Based on Certificate Issuer 'MyrootCA' found Certificate [Thumbprint 52BFCF407B7071512CE65F8868D66578244ABDD9] issued to 'myclient.mydomain.com' ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Begin validation of Certificate [Thumbprint 52BFCF407B7071512CE65F8868D66578244ABDD9] issued to 'myclient.mydomain.com' ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Completed validation of Certificate [Thumbprint 52BFCF407B7071512CE65F8868D66578244ABDD9] issued to 'myclient.mydomain.com' ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Completed searching client certificates based on Certificate Issuers ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Begin to select client certificate ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Begin validation of Certificate [Thumbprint 52BFCF407B7071512CE65F8868D66578244ABDD9] issued to 'myclient.mydomain.com' ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Completed validation of Certificate [Thumbprint 52BFCF407B7071512CE65F8868D66578244ABDD9] issued to 'myclient.mydomain.com' ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
>>> Client selected the PKI Certificate [Thumbprint 52BFCF407B7071512CE65F8868D66578244ABDD9] issued to 'myclient.mydomain.com' ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Raising event:

instance of CCM_ServiceHost_CertRetrieval_Status
{
 ClientID = "GUID:b4aacc70-6de3-4829-88e0-498777c49379";
 DateTime = "20130123230145.722000+000";
 HRESULT = "0x00000000";
 ProcessID = 3612;
 ThreadID = 3888;
};
 ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Failed to submit event to the Status Agent. Attempting to create pending event. ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Raising pending event:

instance of CCM_ServiceHost_CertRetrieval_Status
{
 ClientID = "GUID:b4aacc70-6de3-4829-88e0-498777c49379";
 DateTime = "20130123230145.722000+000";
 HRESULT = "0x00000000";
 ProcessID = 3612;
 ThreadID = 3888;
};
 ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
'RDV' Identity store does not support backup. ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
CCM Identity is in sync with Identity stores ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Begin searching client certificates based on Certificate Issuers ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Certificate Issuer 1 [CN=myrootca; DC=mydomain; DC=com] ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Based on Certificate Issuer 'MyrootCA' found Certificate [Thumbprint 52BFCF407B7071512CE65F8868D66578244ABDD9] issued to 'myclient.mydomain.com' ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Begin validation of Certificate [Thumbprint 52BFCF407B7071512CE65F8868D66578244ABDD9] issued to 'myclient.mydomain.com' ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Completed validation of Certificate [Thumbprint 52BFCF407B7071512CE65F8868D66578244ABDD9] issued to 'myclient.mydomain.com' ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Completed searching client certificates based on Certificate Issuers ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Begin to select client certificate ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Begin validation of Certificate [Thumbprint 52BFCF407B7071512CE65F8868D66578244ABDD9] issued to 'myclient.mydomain.com' ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Completed validation of Certificate [Thumbprint 52BFCF407B7071512CE65F8868D66578244ABDD9] issued to 'myclient.mydomain.com' ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
>>> Client selected the PKI Certificate [Thumbprint 52BFCF407B7071512CE65F8868D66578244ABDD9] issued to 'myclient.mydomain.com' ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Raising event:

instance of CCM_ServiceHost_CertRetrieval_Status
{
 ClientID = "GUID:b4aacc70-6de3-4829-88e0-498777c49379";
 DateTime = "20130123230145.752000+000";
 HRESULT = "0x00000000";
 ProcessID = 3612;
 ThreadID = 3888;
};
 ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Failed to submit event to the Status Agent. Attempting to create pending event. ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Raising pending event:

instance of CCM_ServiceHost_CertRetrieval_Status
{
 ClientID = "GUID:b4aacc70-6de3-4829-88e0-498777c49379";
 DateTime = "20130123230145.752000+000";
 HRESULT = "0x00000000";
 ProcessID = 3612;
 ThreadID = 3888;
};
 ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Client PKI cert is available. ClientIDManagerStartup 1/23/2013 5:01:45 PM 3888 (0x0F30)
Initializing registration renewal for potential PKI issued certificate changes. ClientIDManagerStartup 1/23/2013 5:01:48 PM 3928 (0x0F58)
Succesfully intialized registration renewal. ClientIDManagerStartup 1/23/2013 5:01:48 PM 3928 (0x0F58)
[RegTask] - Executing registration task synchronously. ClientIDManagerStartup 1/23/2013 5:01:48 PM 3928 (0x0F58)
Read SMBIOS (encoded): 32003100350039002D0039003900320037002D0032003500360036002D0038003300320035002D0037003500340032002D0035003200370031002D0033003900 ClientIDManagerStartup 1/23/2013 5:01:48 PM 3928 (0x0F58)
Evaluated SMBIOS (encoded): 32003100350039002D0039003900320037002D0032003500360036002D0038003300320035002D0037003500340032002D0035003200370031002D0033003900 ClientIDManagerStartup 1/23/2013 5:01:48 PM 3928 (0x0F58)
No SMBIOS Changed ClientIDManagerStartup 1/23/2013 5:01:48 PM 3928 (0x0F58)
SMBIOS unchanged ClientIDManagerStartup 1/23/2013 5:01:48 PM 3928 (0x0F58)
SID unchanged ClientIDManagerStartup 1/23/2013 5:01:48 PM 3928 (0x0F58)
HWID unchanged ClientIDManagerStartup 1/23/2013 5:01:49 PM 3928 (0x0F58)
Windows To Go requires a minimum operating system of Windows 8 ClientIDManagerStartup 1/23/2013 5:01:53 PM 3928 (0x0F58)
GetSystemEnclosureChassisInfo: IsFixed=TRUE, IsLaptop=FALSE ClientIDManagerStartup 1/23/2013 5:01:53 PM 3928 (0x0F58)
Windows To Go requires a minimum operating system of Windows 8 ClientIDManagerStartup 1/23/2013 5:01:53 PM 3928 (0x0F58)
Computed HardwareID=2:9C8C08C4B3E16249A2F1457998D16528B656DE30
 Win32_SystemEnclosure.SerialNumber=2159-9927-2566-8325-7542-5271-39
 Win32_SystemEnclosure.SMBIOSAssetTag=9344-3677-7824-5579-3797-0729-37
 Win32_BaseBoard.SerialNumber=2159-9927-2566-8325-7542-5271-39
 Win32_BIOS.SerialNumber=2159-9927-2566-8325-7542-5271-39
 Win32_NetworkAdapterConfiguration.MACAddress=00:15:5D:0B:78:20 ClientIDManagerStartup 1/23/2013 5:01:53 PM 3928 (0x0F58)
[RegTask] - Client is not registered. Sending registration request for GUID:b4aacc70-6de3-4829-88e0-498777c49379 ... ClientIDManagerStartup 1/23/2013 5:01:53 PM 3928 (0x0F58)
[RegTask] - Client registration is pending. Server assigned ClientID is GUID:b4aacc70-6de3-4829-88e0-498777c49379 ClientIDManagerStartup 1/23/2013 5:01:54 PM 3928 (0x0F58)
[RegTask] - Sleeping for 60 seconds ... ClientIDManagerStartup 1/23/2013 5:01:54 PM 3928 (0x0F58)
[RegTask] - Client registration is pending. Sending confirmation request for GUID:b4aacc70-6de3-4829-88e0-498777c49379 ... ClientIDManagerStartup 1/23/2013 5:02:54 PM 3928 (0x0F58)
[RegTask] - Client is registered. Server assigned ClientID is GUID:b4aacc70-6de3-4829-88e0-498777c49379. Approval status 2 ClientIDManagerStartup 1/23/2013 5:02:54 PM 3928 (0x0F58)

After almost 7 minutes exactly, the all the client agent actions besides Machine Policy Retrieval and UserPolicy Retrieval disappear (as if client policy was Reset)

If I let the client sit for 45-minutes to an hour, everything starts working again and works fine from there on out. (cycling the SMS Agent service nor rebooting the machine makes it recover until the 45-min to an hour pass)

My command line I am using to install the client is:

ccmsetup.exe /mp:MYLANBASEDMP /UsePKICert /NOCRLCheck CCMHOSTNAME="myinternetmp.mydomain.com" SMSSITECODE=P10 SMSCACHESIZE=7000 FSP=MYLANBASEDMP CCMLOGMAXSIZE=1000000

If I take out the /usePKICert, /NOCRLCheck and CCMHOSTNAME Entries, the client install and continues to function without issue.

Anyone have any others ideas on where to troubleshoot this issue?  It would make more sense if the client NEVER worked after install.  Tearing my hair out trying to figure out why it starts to intialize, then reverts, then comes back online and works fine.  This happens at both my primary site MP as well as my secondary site/mp.  It happens on my standard Win7 image as well as Windows 8 test machines so I dont believe its a client OS issue.


January 24th, 2013 2:48am

I should add that this apears to only affect client installs where an object already exists in the SCCM database.  On the machine in reference above, I can uninstall and reinstall the client over and over and witness the same behavior,  if i delete the object from the SCCM Database, then install the client the problem does not happen.  the client comes online and stays online.  :-/
Free Windows Admin Tool Kit Click here and download it now
January 24th, 2013 4:38am

More information:

I noticed that Windows Management Framwork 3.0 (beta) was installed on the primary site server.  I uninstalled this version and installed the latest released version.  I was then able to reinstall the client on the machine above and it did not go dormant after 7 minutes and continued to work without issue (using the PKI stuff).  I then rebuilt this machine completely and the client once again did not exhibit the behavior and continued work successfully. 

I then pushed the client install to a Windows Server 2012 box and once again the client came online and stayed online.

Both of these clients are located at my secondary site server subnet using a proxy management point.

I also have a VMware client thats in the boundaries of the Primary Site server.  I built the machine a few days ago to see if the same thing would happen in a different location and it did indeed go dormant after 7 minutes and after about an hour came back online.

After my success against the secondary I rebuilt the VM in the Primary boundary and it still goes dormant after 7 minutes.  :-/

I noticed on the 2 clients here that worked that the CcmNotificationAgent.log shows "bgbController main thread is started with settings: {bgb enabled = 1}, {tcp enabled = 1}, {tcp port = 10123} and {http enabled = 1}.

On the client that exhibits the behavior (the vm in the primary site boundaries) the CcmNotificationAgent.log shows "bgbController main thread is started with settings: {bgb enabled = 0}, {tcp enabled = 0}, {tcp port = 0} and {http enabled = 0}.

Can't say for sure that this is whats causing the behavior but it does appear to be consistent for client that "fail" and then recover.

Anyone have any suggestions on where to look next?



January 24th, 2013 8:14pm

Just a bit more info...just reinstalled the client again on one of the secondary machines where the behavior didnt reappear earlier and its back to going dormant after 7 minutes.

I have opened a case with Microsoft since I am getting nowhere fast.  if I find a resolution I will post back to the forums.

I would however still LOVE to hear from anyone out there who has SCCM 2012 SP1 on wether they see this behavior or not..

Free Windows Admin Tool Kit Click here and download it now
January 24th, 2013 10:49pm

I've done some quick tests in my lab (http, not https) and do see the same behavior. I already contacted the product group but did not get any reply yet. Please update this thread if you got feedback from MS.
January 25th, 2013 10:57am

Thank you Torsten!  Very glad to hear its not just me (and that i have not gone mad).  I should be hearing from the CSS guy today so I'll certainly post back if/when we find the root cause.  Not sure about you but it "feels" like it has something to do with the new "Client Notification" system in SP1 (BGB Agent). 

Thanks again for the reply and testing.

Free Windows Admin Tool Kit Click here and download it now
January 25th, 2013 5:35pm

Just got off the phone with MS Support and as expected, the gentleman had never seen this behavior.  I captured 2 sets of logs from different machines and uploaded to them.  He is going to escelate the issue with the logs for analysis.  Likely wont hear back until Monday.
January 25th, 2013 9:50pm

Just out of curiosity, if you check ccmeval.log is there any sort of client remediation taking place during this time?

Free Windows Admin Tool Kit Click here and download it now
January 26th, 2013 3:33am

Hi Adam, CcmEval.log is not yet created at the time the issue appears. 
January 27th, 2013 11:29pm

William, thanks for posting this!  I am having the exact same issue!  It has been driving me nuts trying to figure it out.  I was just able to validate, as a workaround, I can delete the record from database and reinstall client and it does appear to resolve issue as best I can tell.  I've had tons of clients go "Inactive" (Disabled) just like yours.  I haven't been able to find a pattern to why some clients work and some don't, besides deleting records from database.  You found any other info yet?
Free Windows Admin Tool Kit Click here and download it now
January 29th, 2013 5:52pm

Hey bcehr, I am still waiting to hear back from microsoft support.  I sent them client logs as well as ran a diagnostic util on the primary site server and uploaded the results.  I did this on Friday.  I sent an email yesterday asking for a status and was told they are still working on it.  Hoping I hear back today.  :-/
January 29th, 2013 6:31pm

Hey, ok sounds good.  Let us know what you find out.  I'm also working on issue from my side, if I can find anything else, I'll post back an update.
Free Windows Admin Tool Kit Click here and download it now
January 29th, 2013 6:35pm

Just an update for anyone watching this thread..

I sent an email yesterday and asked for an update on the issue from the support person and never got a response.  :-/  Just sent another email requesting an update.  So at its stands, day 5 with CSS and still nothing to go on yet.  I will traveling the rest of the week so hoping to have something by the time I return on Monday.

January 30th, 2013 5:56pm

Just another update, finally got a response from the support tech who unfortunately is requesting more time.  So no closer to a resolution at this time.  I'll update again when i get more info.
Free Windows Admin Tool Kit Click here and download it now
February 5th, 2013 1:08am

William, thanks for keeping us in the loop.  I also wanted to provide a little more info.  I found that my secondary management servers, also have the same issue, but the workaround for them does not work, best I can tell, no way to fix those machines.  Are you seeing this too?
February 8th, 2013 4:21pm

I do see it at both my primary site and at my secondary site.  I "thought" at one point that deleting the record from the database was the definitive workaround however have since then experienced the issue against a machine that was not already in the DB there really doesnt seem to be any consistency with why/when it happens.  I pushed 3 clients yesterday (upgrade to 2012 SP1 pushed via SCCM 2007) and 2 of 3 of then installed and stayed online, while the 3rd went dormat for about an hour and a half before coming back online.

I have still yet to hear back from MS on my support case.  If I inquire I am unfortunately told "we here at PRO and PRE are affected by an increased influx of case volume so I will ask for some more time on this" so have no clue if/when we will actually work towards resolving this issue.  :-/

Free Windows Admin Tool Kit Click here and download it now
February 8th, 2013 5:51pm

hello William,

i found your thread because i have a similar problem (maybe even the exact same), with the error message "Finding certificate by issuer chain returned error 80092004". i have a similar setup (lan MP with http and another internet MP with https).

when i do a client push install i see this in ClientIDManagerStartup.log:

i tried already a lot to fix this thing, and i'm short before reinstalling the entire PKI/cert things from scratch. so far i have found one workaround:

http://austrianalex.com/2012/10/system-center-configuration-manager-sccm-2012-client-pki-and-subordinate-ca-woes/

this actually works for me - we have our root CA certificate distributed through active directory to our clients, so when i remove the cert from SCCM primary site properties / client computer communication / trusted root certification authorities, installation of the clients works again, like described in the link above.

however, i am not sure if this is a correct "solution" because i now got errors on other parts of the infrastructure (e.g. pxe driver installation), which may be because i do not have a trusted root cert set in SCCM....

i thought i share this here so maybe you want to try this on your end too.

EDIT:

after a lot of testing, this "solution" does not help me, as PXE no longer works without the trusted root CA cert set in site properties.

February 12th, 2013 3:16pm

hello again, i have done a lot of analysis in the last couple of days and i am now confident that i have the exact same issue as the OP. i opened a microsoft ticket too.

Free Windows Admin Tool Kit Click here and download it now
February 14th, 2013 2:40pm

Thanks for the post back guys.  My case has been opened with MS Support for approaching a month now and unfortunately all I get back from the assigned engineer is that they are super busy and need more time.  I have escelated to our AM who I hope can spark a flame with the support team.  The only other hope I have is that they are delaying response because the product team is working on a hotfix.  Might be wishful thinking..

This is a show stopper issue for us so we cannot move our migration forward until this is resolved.

I'll post back if/when we ever start troubleshooting this issue with MS Support.

February 18th, 2013 6:36pm

A couple of quick questions:

Do you have a trusted root certification authority set up in the admin console?

If you do have this set up, are your client certificates issued by the specified trusted root CA?

Free Windows Admin Tool Kit Click here and download it now
February 18th, 2013 11:42pm

Hey Adam, yes we do have our Root CA configured and its the same CA that issues client certs.  Once the client comes back "online" it works perfectly fine using PKI.  We have succesfully tested the devices rolling over to the Internet MP when connected a network other than our domain network.  The problem only happens on intial client install (or reinstall).  Once its dies (for a lack of a better term) it lays dormant for awhile but then eventually comes online and everything works as expected and never goes dormant again.

Thanks!

February 19th, 2013 12:14am

Any news on this bug?
Free Windows Admin Tool Kit Click here and download it now
February 26th, 2013 2:01am

Not yet.  I sent them a bunch of information on Client Settings last week.  Still waiting on MS Support who have escelated the issue internally. 
February 26th, 2013 2:20am

same here. no news from MS except that they are in delay.
Free Windows Admin Tool Kit Click here and download it now
February 26th, 2013 10:37am

update, a microsoft tech has been in touch with me yesterday, and we looked at the issue on our server for roughly 2 hours. no solution was found but they will try to replicate the issue internally and get back to me asap.
February 27th, 2013 12:46pm

Thanks for the update Robert..
Free Windows Admin Tool Kit Click here and download it now
February 27th, 2013 5:34pm

For those of you that are having this issue, what's the issuer name of the trusted root CA?
March 20th, 2013 3:16am

Adam, for me uses the default CA naming convention (domain-servername-CA); ie. CONTOSO-PKISRV-CA
Free Windows Admin Tool Kit Click here and download it now
March 20th, 2013 4:28am

CN = ABCD Root CA

DC = domain

DC = eu

(domain.eu is our internal DNS domain, ABCD is the company name, no special characters)

fyi, yesterday, microsoft tech has been on our server again, still no solution. i have requested this to be escalated to the dev team, to me it looks like a bug in sccm, especially after all that the MS tech has tried on our server.

March 20th, 2013 1:21pm

Robert, if you have configured a trusted root CA in the admin console, the client will only use a certificate for client authentication that was issued by that CA. This may explain what you were observing.

Free Windows Admin Tool Kit Click here and download it now
March 20th, 2013 9:16pm

On my side, I can confirm all my clients have a certificate from the CA that SCCM trusts.  Also, I can confirm its not a certificate issue, because if I uninstall the SCCM agent, delete the record from SCCM database, then reinstall client, it works.  However, this is very inconvenient because this scenario can happen to any client.
March 20th, 2013 9:28pm

I can also confirm its not a trust issue with the cert.  The Root CA I have configured in SCCM is the same CA that hands out certs to clients.  As stated above, once the client "comes back" after letting it sit for a while it works fine from then on out.  This means it properly talks over HTTP to the Intranet MP/DP when on LAN and successfully flips over to HTTPS and talks to the Internet MP/DP when off LAN.

Free Windows Admin Tool Kit Click here and download it now
March 20th, 2013 9:50pm

Thanks to both of you for your detailed reports on the issues you are experiencing. Unfortunately, I'm out of ideas at this time as to what may be happening and I hope that you'll get the assistance that need from CSS to get to the bottom of what's happening here.

March 21st, 2013 9:53pm

Thanks Adam.  Appreciate the effort.  :-)
Free Windows Admin Tool Kit Click here and download it now
March 21st, 2013 10:10pm

Any update on this guys?
April 8th, 2013 11:48pm

Had a PFE remoted in last week digging through more server logs and have sent additional verbose client logs.  Still waiting :-/
Free Windows Admin Tool Kit Click here and download it now
April 9th, 2013 4:48pm

nothing new.

MS tech insisted that the problem must be our root certificate, so i should recreate it.

i contacted our hosting partner, who created the certificate and is using it for MS lync, and we decided not to go ahead with this as lync is working fine and the cert is showing no sign of problems. ticket back to MS...

April 9th, 2013 5:00pm

Our Root Cert comes from our internal CA which works fine for every other application we use it for.  Given the fact that this issue also happen if I do not provide the IBCM settings during install and install only as Intranet sure doesn't sound like a certificate issue to me, unless there is still cert checking stuff happening even when not using PKI for client communication?
Free Windows Admin Tool Kit Click here and download it now
April 9th, 2013 5:56pm

Not to mention that when the client does finally come back online, it works fine from there on out, including via PKI across the internet..Seems that if the cert itself was the issue then the client would never work properly..
April 9th, 2013 5:58pm

last update, microsoft asked me for a fresh set of logs, which i provided (http working state and https failed state). after checking through them, they again told me  that it must be a problem with the certs, so we should recreate them. i still cannot do this (root certs are in use by our lync infrastructure, all other certs i already recreated three times without effect), so finally management cancelled this project and i closed the ms ticket. bottom line, we can't use sccm 2012 for https/IBCM.

not to hijack this thread for the wrong reasons, i posted here because i get the same error messages as the thread OP in the client logs:

Certificate Issuer 1 [CN=myrootca; DC=mydomain; DC=com] ClientIDManagerStartup 1/23/2013 5:00:52 PM 2100 (0x0834)
Finding certificate by issuer chain returned error 80092004 ClientIDManagerStartup 1/23/2013 5:00:52 PM 2100 (0x0834

so our issue may have been for the same reasons, maybe not.
i hope you find a solution for this, william, good luck!

Free Windows Admin Tool Kit Click here and download it now
April 23rd, 2013 4:33pm

Robert, in this specific case do you have a trusted root issuer configured in the admin console? If you do, the logging indicates that the trusted root issuer configured in the admin console hasn't issued a usable certificate on the client machine. See the earlier post I made on that piece of the issue and see if that helps you any.

April 24th, 2013 12:12am

Hello Adam, yes we do (trusted root CA cert in console, as in your screenshot, and the client certs are issued by the same root CA through active directory enrollment)

this root CA cert is the cause of our issue. once we remove the root CA cert from sccm console, clients immediately detect their machine certificate as valid and start communicating with the MP without the slightest delay. sadly we can't leave it with this, as pxe/osd now no longer works correctly. not going into too much detail here, we would need to fix the failure of sccm client to detect the machine cert as beeing valid one way or another, and MS tech's approach was to recreate the certs.

we recreated all the client and IIS certs many times without success, strictly following the MS docs. i cannot recreate the root CA cert because too much production systems are depending on it (lync 2010) and apart from that, the certificate chain validates correctly from client cert to root CA cert so i don't see the issue here.

bottom line, we were running in circles with MS tech support for over a month, and have now cancelled this project to move on with other things.

Free Windows Admin Tool Kit Click here and download it now
April 24th, 2013 9:33am

Hi

Just want to inform you, and Microsoft if they are listening, this issue is not isolated to PKI environment. I havent got a PKI setup and i am experiencing the exact same issues. I have been battling with it for approx a month now and can find very little info out there about, unitl that was unitil i found this post.

It is serously hindering my project, i am expecting to go 'LIVE' with SCCM based OS Deployment and then this starts happening, at first i thought it was becuase i was using ZTIWindowsUpdate.wsf as part of my task sequence so told everyone we cant use windows update, but i can see now that windowd update it not the cause.

Could someone tell me how to log a call about this so i can also try to expediate things to get this resolved.

Cheers..

April 25th, 2013 7:25pm

Hi Torsten,

Im just wondering if you heard anything else regarding this ?

Cheers

April 26th, 2013 2:03pm

Hey Overfiend, I can tell you that my support engineer is still looking into it and have no additional updates at this time.   Going on 3 months now with having this case open with Microsoft with no movement.  :-/
Free Windows Admin Tool Kit Click here and download it now
April 26th, 2013 5:54pm

Hi William,

im sitting at a clinet right now and seeing this over and over again. The annoying thing is that its so random aswell, sometimes it 'appears' to be ok, then bang, it fails.

This is not good, it renders the computer unusable for my purpose.

I can't beleive its taken this long to fix..

Have you applied CU1 yet ?

April 26th, 2013 5:57pm

Yeah, we upgraded to CU1 this week.  No change for this issue..
Free Windows Admin Tool Kit Click here and download it now
April 26th, 2013 6:45pm

Also seeing this same issue. Happens very randomly and as far as I can tell with no environment changes, or warnings for changes.

2012 Configuration Manager only environment, client has a the new SP1 agent. Was working just fine and even though there was no changes all of the sudden says "Client is set to use HTTPS when available. The current state is 448" and PKI none.


April 29th, 2013 8:57pm

Just another update to state we still have not received any movement from Microsoft on this issue.  Its been over 3 months now and I am rapidly approaching the point where I just punt SCCM 2012 and any potential Windows 8 tablet projects we have in the pipeline sadly and maybe try to revisit next year (assuming the issue has been identified and corrected by then of course).  This sucks.  :-)  Anyone else hear anything by chance from MS on this?

Free Windows Admin Tool Kit Click here and download it now
May 1st, 2013 5:02pm

Hi William,

I had some difficulty logging a  case with Microsoft so am looking here for answers and responses.

For the past two days i havent had any failures, all seems to be well. I had made one slight change to my setup, i was getting a lot of errors in my SMS_INVENTORY_DATA_LOADER log, stating that mesages or inventory data couldnt be processed due to the MAX MIF Size being to small. The recommendation was to increase it larger than its default of 5MB (5000000), so i increased it to 20MB (20000000) just to get things working. Since then i havent had any errors. This could be just coincidence or good indeed be bad data getting stored in the database for the client machines expereinceing the problems..

The reg key in question was ..HKLM\Software\Microsoft\SMS\Components\SMS_INVENTORY_DATA_LOADER, the key your looking for is Max MIF Size

Let me know if this works for you, or anyone else.

I only found this by looking at the errors being flagged up in my Sites Components Status, worked through trying to eliminate some of the errors that were appearing.

Cheers.

May 1st, 2013 6:31pm

Overfiend, are you saying you were having the issue of client installs coming online, then go offline for awhile and then come back on and start working (As is the basis for this thread) and when you increased your MIF size the client install issue disappeared?  Just want to make sure we are still talking about the same (original) issue.

If its any consolation, I am not getting inventory processing errors in SMS_INVENTORY_DATA_LOADER so assuming this is not the same issue (for me at least).

Free Windows Admin Tool Kit Click here and download it now
May 2nd, 2013 8:40pm

On my side, I still have the issue as well.  I noticed today, the issue typically comes back when I either update for SCCM Agent, or reinstall Agent.  If I ever do that, the agent gets Disabled like you have posted, and I have to uninstall, then delete the record from the Database, then reinstall from scratch.  Until someone finds a solution on this, I'm terrified to update to any CUs.  You see this same behavior?
May 3rd, 2013 9:17pm

I see this behavior on almost every install, whether its a new install or a reinstall.  In ALL cases however if I just let the machine sit, the client will come back online (typically about 45 minutes) and then work just fine from there on out.  Do your clients eventually recover bcehr or once they are disabled they never "self-correct"?
Free Windows Admin Tool Kit Click here and download it now
May 6th, 2013 4:32pm

Best I can tell, they don't recover, they stay stuck disabled forever. I have to delete them from database and reinstall to get them to wake up.  The one way that agents do work correctly are ones that are installed via OSD, best I can tell, those agents work immediately and continue to work if untouched.  I don't do many "new" installs outside of OSD, so not sure if those might break too.
May 6th, 2013 4:35pm

Hmm, well ok, that different then from what I am seeing.  :-/  On your side have you checked the conflicting records node to ensure there isn't an issue there?  They are supposed to auto merge for any domain joined clients who's hardware hasn't changed but might be worth a look just in case...
Free Windows Admin Tool Kit Click here and download it now
May 7th, 2013 12:36am

Hi.

Just got word from Microsoft support that they have raised a hotfix for this problem. It's beeing tested out at some other customer now. They should get back to me in a couple of weeks.

I'll keep up updated.

Bjrn Erik

May 22nd, 2013 10:50pm

Seeing exactly the same issue here in a PKI only environment.  Root CA is configured in site properties and is also present on clients under Trust Root Certification Authorities.  Certificates issued to the SCCM environment, including server and client certificates are issues by subordinates.  Be keen to see a fix for this...as the OP said the client does come back to life within around an hour but this means that any machine rebuilt for a customer is missing key software for quite some time after build.

 
Free Windows Admin Tool Kit Click here and download it now
May 23rd, 2013 12:09pm

I can also confirm that this has now been submitted to the Product Team as a bug and will be corrected with a hotfix or CU.  Now we just wait.  :-/
May 29th, 2013 12:47am

That's good news.  We are stuck until we have a resolution.  Any idea if this will be just a client hotfix?
Free Windows Admin Tool Kit Click here and download it now
May 29th, 2013 11:06am

Nothing official on what systems the hotfix will apply to.  If I were a betting man, I would bet its a server hotfix but only time will tell.
May 29th, 2013 4:13pm

Just curious, but is this the hotfix you are referring to http://support.microsoft.com/kb/2841764 ?  it was just released yesterday.
Free Windows Admin Tool Kit Click here and download it now
May 30th, 2013 4:45pm

Unfortunately no.  This appears to fix an issue with having a 3rd party cert with the & symbol in it causing clients to not ever become assigned. 

May 30th, 2013 9:33pm

This fix makes some changes to code responsible for site assignment and HTTPS certificate publishing into AD. I would recommend trying it out to see if it manages to resolve any of your issues.

It contains both client and server-side fixes.

Free Windows Admin Tool Kit Click here and download it now
May 31st, 2013 10:52pm

If anyone else tries this please post back your results.  I am going to hold out for the "official" hot fix for my particular issue from MS.  ;-)
June 3rd, 2013 9:30pm

Did MS give you any sort of ETA for hotfix or CU that will contain fix?
Free Windows Admin Tool Kit Click here and download it now
June 3rd, 2013 9:34pm

Unfortunately no.  I was simply told that the issue has been identified as a bug and that the product team was working on a fix that would be released sometime in the future as a hotfix or as part of a CU.  :-/
June 4th, 2013 4:44pm

Any hotfix ? I've spent days and days on this and finally found this thread.

MS support person working on this hasn't got anywhere yet.

Either client registration works (clear the trust root cert in site client communication tab) and pxe fails, or the other way around.

:(

Free Windows Admin Tool Kit Click here and download it now
June 27th, 2013 5:44pm

The last update I got was that this was identified as a bug and will be released with CU3.  No ETA on when CU3 will be released.  CU2 was just released a few days ago so I would think several weeks before CU3.    :-/
June 27th, 2013 5:55pm

William, I want to follow up internally to make sure this issue is being tracked properly. Could you let me know what the ticket number is for your incident?
Free Windows Admin Tool Kit Click here and download it now
July 2nd, 2013 1:46am

Hey Adam, the Incident number is "113012410165068".  The incident was "closed" several weeks ago however and I was told the resolution is that it's been identified and to wait for a fix.

The information I gave above was from another MS employee who I met during a sessions at the local MS office here in STL who has been kind enough to dig around for information for me. 

July 2nd, 2013 12:14pm

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics