Root Certificate Update stopped working via Proxy last week (CAPI2 event ID 4110)
Hi, according to Certificate Support and Resulting Internet Communication in Windows 7 and Windows Server 2008 R2 the root certificate update in Windows 7 works differently and we cannot download the certificate packages via WSUS anymore (as we did with XP). Apart from security perspective (I do not really like the idea that each PC connect to the internet for security updates), I do not understand how the proxy configuration works. We use just each users's proxyconfiguration and that seemed to work, but now since last week I get lots of requests for unvalid certificates due to the fact that the CAPI2 update functionality does not get through our proxy anymore. We made a lab-test and set the proxy with "Netsh winhttp" and even used WPAD - both solve the problem, but we cannot enable that for all our clients, but it proofs that proxy configuration is the root cause for our problem. So, main question is: did Microsoft change something in the process of using proxy settings from the user? Shouldn't the setting of a proxy in Internet Explorer 8 be sufficient as described in this article . Thanks, Torsten.
February 21st, 2011 7:59am

We have the same problem here! For now we download it manually. Hope this gets resolved very soon! Cheers,- Moutalib
Free Windows Admin Tool Kit Click here and download it now
February 22nd, 2011 6:22am

Hi Torsten, We have also seen issues just in the last week or so. Our experience with Windows 7 was that until last week the dynamic root certificate update process worked fine. From what I can tell, it used to perform this process as the "user" which meant that it went through the same proxy that the user had set in IE and it worked fine. Something has changed which now causes this process to happen within the SYSTEM (or maybe NETWORK SERVICE) context which means it now fails (as these process do not know about the proxy server and are trying to access the internet directly). As you said, we can fix it with "netsh winhttp" or using WPAD as well. I would really like to understand what caused this change in the first place. It must have been a Windows patch, but I haven't taken the time to work out which one it is yet. I will have a go at that today. Cheers, David
February 28th, 2011 6:37pm

Hi all, Just want to mention that we have exactly the same problem since some time, difficult to say since when exactly. I hope that the issue will be fixed soon too! Otherwhise it seems that we would have to configure our proxies as transparent ones, which would surely fix the problem. Thanks, Benjamin
Free Windows Admin Tool Kit Click here and download it now
March 1st, 2011 11:41am

We're also experiencing this problem. The workaround I've been using is to install the most recent Root Certificate Update package for XP (http://www.microsoft.com/downloads/en/details.aspx?FamilyID=25249786-2b8e-4c51-8f4b-727ce25cc2c5), which just adds all Trusted Root CAs as of October to the Trusted Roots store. Obviously it would be desirable to fix the Windows 7 dynamic update process instead of using a band-aid. Sorry, I mean "adhesive bandage". :-)
March 2nd, 2011 1:56pm

Hi Jerry, Have you only been having the problem relatively recently (3 or 4 weeks or so)? There are a number of ways to fix it - either by allowing the computer account access through the proxy (wpad, transparent proxy, netsh winhttp) or by populating the root certificate store with the required roots somehow (like using the update package for Windows XP). I am more curious as to what has caused it in the first place. We have been using Windows 7 happily for 6 months without any issue and suddenly in the last month or so this started happening. Something fundamental about the way Windows 7 performs the root certificate dynamic update process must have changed but I've yet to get to the bottom of what caused it. I assume there must have been a patch in the last month or so which changed the behaviour of the update system. I am also perplexed as to why there is hardly any "internet chatter" about it. This is a huge problem which I would have thought almost every major corporation in the world would be seeing - yet apart from a couple of threads on forums hardly anyone seems to have noticed it. Maybe most companies have yet to deploy whatever patch has caused this and the storm is yet to come (or maybe nobody uses Windows 7 yet)! :) Cheers, David
Free Windows Admin Tool Kit Click here and download it now
March 2nd, 2011 5:54pm

From my testing, I believe the issue began starting January 25, 2011 between 8:50 PM UTC and 8:55 PM UTC and is not related to a patch. I took a new build of Windows 7, without any patches, and gradually changed the date until the dynamic root certificate updates began working. Any time before 8:50 PM UTC on January 25th, 2011 is when it started working again. One interesting thing I did find was that once you get the root certificate download to work for a particular site, it is very hard to break it again. This may explain why there is not very much internet chatter about the issue. So far, I have only seen the issue manifest itself with any meaningful impact on new PC builds. Possible workaround: If you install the Microsoft Certificate Trust List PCA certificate from http://www.microsoft.com/pki/certs/MicCerTruLisPCA_2009-04-02.crt to the Computer (not user!) Intermediate Certificate store, the dynamic root certificate download succeeds without having to make any special changes. I haven't tested this to make sure it works in all cases, but looked promising from the limited testing I did today.
March 2nd, 2011 6:26pm

Thanks for the tip Steven, I'll try that out. We first noticed this issue about 2 weeks ago. We had deployed Windows 7 only to laptops about 6 months ago. Most of those users already had the trusted roots they needed, and could also pick up new trusted roots when working away from the corporate network. Now we're deploying to desktops, so hearing a lot of screaming. One thing I found was that the SYSTEM account is not affected by the problem. If you launch IE as SYSTEM using a tool like psexec, trusted roots are downloaded and installed successfully, even though the SYSTEM account does not have an IE proxy. Very strange.
Free Windows Admin Tool Kit Click here and download it now
March 2nd, 2011 6:45pm

A follow up to Steven's proposed workaround: The solution of adding the Microsoft Certificate Trust List PCA certificate from http://www.microsoft.com/pki/certs/MicCerTruLisPCA_2009-04-02.crt to the local computer Intermediate Certificate store was effective in resolving this problem in our environment. I have already deployed the certificate company-wide via GPO and am crossing my fingers that we'll hear no more about this problem. Steven, I bow down in awe of your awesomeness. Thank you, sir!
March 2nd, 2011 7:28pm

I believe I am getting closer to the root cause of the problem: At some point Microsoft published a new Certificate Trust List (CTL) with an effective date of January 25, 2011 at 8:54:12 PM UTC. The file is available here: http://download.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootstl.cab or http://www.download.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootstl.cab In order for PCs to validate the authenticity of the CTL, it is digitally signed. The certificate used to digitally sign the CTL happens to be issued by the Microsoft Certificate Trust List PCA and is the certificate I mentioned in the above workaround. The Microsoft Certificate Trust List PCA certificate is issued by the Microsoft Root Certificate Authority. A new install of Windows 7 contains the certificate for the Microsoft Root Certificate Authority but it does not contain the intermediate certificate Microsoft Certificate Trust List PCA. When the user goes to an SSL site that the machine does not have a root certificate for, it downloads the CTL from the windowsupdate.com links I mentioned above. It then appears to verify that the CTL is trusted by validating the certificate chain. In the case of the CTL, the first certificate in the chain is Microsoft Certificate Trust List PCA. Looking at packet captures, Windows 7 will download the Microsoft Trust List PCA certificate, but when no system proxy is enabled, the Microsoft Trust List PCA certificate doesn't seem to get installed in the system. The validation of the certificate chain for the CTL downloaded from windowsupdate.com then fails, and the root certificates from the CTL cannot be installed. An interesting thing happens though if you block both download.windowsupdate.com and www.download.windowsupdate.com. Windows 7 seems to have an internal CTL that it can use if there is none available from either of the windowsudpate.com sites or the windowsupdate.com sites are unavailable. This internal CTL does not have any validation issues, and if both windowsupdate.com sites are blocked, Windows 7 will successfully add the root certificates if it can find the necessary root in the internal CTL. These roots can be for the same sites that fail when the updated CTL is downloaded from windowsupdate.com. At this point, validation of CTLs downloaded from windowsupdate.com fail when the certificate used to sign the CTL is not already present in the machine certificate store and there is no direct internet access or machine account proxy configured.
Free Windows Admin Tool Kit Click here and download it now
March 2nd, 2011 10:20pm

Hi Steven, it works for us as well. Thanks for your investigations. I was wondering why it works for the normal Windows Update, but for Root-CA update we'd need to set a system-wide proxy... BR, Torsten.
March 3rd, 2011 3:19am

Hi Steven, Excellent sleuthing - very impressive! Everything you said makes sense. Out of interest - have you been able to figure out where Windows actually keeps the CTL? In my testing, Windows only seemed to download this once (authrootstl.cab). Even if it was unable to verify the CTL signature and ultimately failed (due to the missing Intermediate cert), Windows still didn't seem to download the CAB file again. It was as if the CAB file is already on the system somewhere and it continually refers to it locally (which only becomes successful once the Intermediate cert is imported into the store, or if the computer account is able to download the Intermediate cert from the Internet). Thanks again for the great work! Cheers, David
Free Windows Admin Tool Kit Click here and download it now
March 3rd, 2011 3:40am

-edit- it's getting more strange: after testing the solution above, I cannot reproduce the problem anymore on a freshly installed system. Maybe the certificate is cached somewhere?... But now it also works without any Proxy configured. - 2 hours later - And another 2 hours later the problem is back. I installed the intermediate certificate on computer A, which solved the problem again. But immediately it also works on computer B. That's pure Voodoo! BR, Torsten.
March 3rd, 2011 5:34am

Hi, it's getting more strange: over night the problem has disappeared (on running PCs just a reboot needed). I cannot reproduce it anymore. Maybe Microsoft fixed something without further announcement?... BR, Torsten.
Free Windows Admin Tool Kit Click here and download it now
March 3rd, 2011 5:34am

Hi all, Thanks for the great troubleshooting. I'm facing the same problem since mid february, but the intermediate certificate is in place and chain validation is successful for the signed CTL, if downloaded manually. But I've noticed something else. If I bring up the file properties for the CTL file (authroot.stl) and check the certificate "Microsoft Certificate Trust List Publisher", there's an exclamation mark indicating the signature should not be trusted. Certificate status: "This certificate does not appear to be valid for the selected purpose." When checking the key usage attribute for the certificate, I find it only contains "Digital Signature (80)" This certificate was issued on January 25th, so my thought is that Microsoft started using this certificate by mid february when publishing an updated list. And that's when the problems started. Thoughts on this, anyone?Cheers
March 3rd, 2011 11:26am

David, As you implied in your note, Windows caches the CTL it downloads from windowsupdate.com. You can easily view the user version of this cache by running the command "certutil -urlcache -v". The user version of this cache is located at %USERPROFILE%\AppData\LocalLow\Microsoft\CryptnetUrlCache with the files located in the Content folder and details about the files located in the MetaData folder. There is also a system version of the cache, which appears to be where the intermeidate cert needs to appear for the CTL to work properly. It is located at C:\Windows\System32\config\systemprofile\AppData\LocalLow\Microsoft\CryptnetUrlCache The intermediate certificate we are concerned with is named 52FE9FFE4780FF24EC690DB2F1D013CE
Free Windows Admin Tool Kit Click here and download it now
March 3rd, 2011 1:09pm

StoffeB, I noticed the same thing. The older versions of the CTL have the same issue though. According to http://msmvps.com/blogs/bradley/archive/2010/09/02/capi2-errors-driving-you-crazy.aspx it is only a GUI error.
March 3rd, 2011 1:11pm

just tried to reproduce the issue thats been bugging me the last few weeks. but to my suprise the problem wasnt there when i atempted to test on a newly imaged client. im no longer seeing the capi2 error and the client can validate the cert from the https site with no issue. you guys still seeing the issue?
Free Windows Admin Tool Kit Click here and download it now
March 3rd, 2011 6:17pm

Hi Everyone, I want to let you know that the CTL issue has been resolved - we re-signed and posted a new CTL to Windows Update servers yesterday, and that version should have made its way to all users already, or will shortly. There is an explanation of the issue available at http://social.technet.microsoft.com/wiki/contents/articles/troubleshooting-root-certificate-update-failure-march-3-2011.aspx. Best Regards, Tom Albertson, Program Manager Windows Root Certificate ProgramProgram Manager Windows Root Certificate Program
March 3rd, 2011 8:24pm

I can confirm the issue is resolved for us. It would also explain why I couldn't reproduce the issue when I forced the proxy to send down an old CTL I pulled from the cache of a VM snapshot... If anyone needs to confirm a system is using the new CTL, you can do so by running the following command: certutil -urlcache -v authrootstl.cab If the output displays a file size of 30436, as below, it is the new CTL. WinHttp Cache entry: 52 Bytes Source Url Name: "http://www.download.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootstl.cab" Local File Name: "C:\Users\LocalAdmin\AppData\LocalLow\Microsoft\CryptnetUrlCache\Content\94308059B57B3142E455B38A6EB92015" Meta File Name: "C:\Users\LocalAdmin\AppData\LocalLow\Microsoft\CryptnetUrlCache\MetaData\94308059B57B3142E455B38A6EB92015" File Size: 30436 Last Sync Time: 3/3/2011 9:00 PM If the output displays a file size of 29367, as below, it is the old file. You may want to try clearing the cache on your proxy server. WinHttp Cache entry: 52 Bytes Source Url Name: "http://www.download.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootstl.cab" Local File Name: "C:\Users\LocalAdmin\AppData\LocalLow\Microsoft\CryptnetUrlCache\Content\94308059B57B3142E455B38A6EB92015" Meta File Name: "C:\Users\LocalAdmin\AppData\LocalLow\Microsoft\CryptnetUrlCache\MetaData\94308059B57B3142E455B38A6EB92015" File Size: 29367 Last Sync Time: 3/3/2011 8:59 PM
Free Windows Admin Tool Kit Click here and download it now
March 3rd, 2011 9:23pm

I want to let you know that the CTL issue has been resolved - we re-signed and posted a new CTL to Windows Update servers yesterday Hi Tom, thanks for making this solution possible. We had lots of trouble due to this issue as we could not explain that is was working fine before February. And the needed proxy settings are hardly documented - even from Microsoft I did not get a clear answer, if setting the proxy in user's Internet Settings is sufficent for this feature to work. BR, Torsten.
March 4th, 2011 5:07am

All, I would like to add to this thread an important question in my view. The issue was touch on at the start of this thread. That being that even though the proxy option is NOT checked, the proxy server is being used by my Win7 PC to connect to this site. And I believe another side affect of this issue is that our admin account keeps getting locked out because of the attempts to go to the download path through the proxy. My question is, Why does this underling applet discussed in this thread use the proxy server when I have it unchecked in IE? I was happy to ignore the errors, but i believe its causes the admin account to be disabled because of all the failed proxy attempts. (I have looked at the calls on wireshark and attempts to connect to the download link are being rejected by the proxy) Geoff.
Free Windows Admin Tool Kit Click here and download it now
March 25th, 2011 9:35am

Hi Geoff, Do a search for "WinHTTP proxy" and it should answer your question. In a nutshell, the certificate update mechanism discussed in this thread uses the WinHTTP interface to make HTTP calls. This interface (WinHTTP) ignores the proxy settings in IE and instead has separate settings for the proxy. By default, WinHTTP uses proxy autodetection (WPAD) - if no WPAD mechanism is in place then WinHTTP will go directly to the Internet. There are a set of netsh commands to manually set the WinHTTP proxy. In older versions of Windows a utility called proxycfg.exe controlled WinHTTP proxy settings. If you have never configured WinHTTP and you can see your machine using the proxy this would indicate that you have proxy autodetection (WPAD) in place. I cannot think of any way that this could cause the administrator account to lockout (as WinHTTP does not run in the context of the administrator) but who knows - there might be a link. My suspicion would be to look elsewhere though. Cheers, David
March 27th, 2011 11:51pm

Thanks David, That answers my question. I will investigate the wpad as there may be one in place. Regards, Geoff.
Free Windows Admin Tool Kit Click here and download it now
April 5th, 2011 8:32pm

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics