An IPsec main mode negotiation failed
Hello, I have setup an enterprise root CA in a test AD domain (2008r2), and am attempting to get file sharing traffic secure using IPSec. I'm able to secure the traffic between a Windows 7 client and a 2008r2 file server that are both part of the domain using either the default Kerberos method, or a computer certificate. However, I have been unable to get a Windows server 2008r2 box that is NOT part of the domain to communicate with my file server using a computer certificate for IPSec. On the 2008r2 client I created a RequestConf.inf file with the following info: [Version] Signature = "$Windows NT$" [NewRequest] RequestType = PKCS10 ProviderName = "Microsoft Software Key Storage Provider" Subject = "CN=2008R2-1" KeyLength = 2048 MachineKeySet = TRUE KeySpec = 2 KeyUsage = 0x80 [EnhancedKeyUsageExtension] OID = 1.3.6.1.5.5.8.2.2 ;IP Security IKE Intermediate From the command prompt I ran 'certreq -N -f RequestConf.inf CertRequest.req' to generate my CSR. I then connected to my CA via https and selected 'Request a certificate' > 'Advanced certificate request' > 'Submit a certificate request by using a base-64.....' and pasted the CSR I generated above into the Saved Request field. I then selected IPSec (Offline request) from the Template drop down box and hit submit. Once the certificate was created, I selected the 'Base 64 encoded' option and downloaded the certificate. I then import the certificate via the MMC (Computer Account), and the certificate successfully imports into the Local Computer > Personal > Certificates store (I've also imported the certificate using the 'certreq -accept' method). I should also mention that I imported the trusted root from my CA into the clients certificate store as well. When I attempt to access the file share, I receive the following in the event log on the client: -------------------------------------------------------------------------- An IPsec main mode negotiation failed. Local Endpoint: Local Principal Name: - Network Address: x.x.x.x Keying Module Port: 500 Remote Endpoint: Principal Name: - Network Address: x.x.x.x Keying Module Port: 500 Additional Information: Keying Module Name: AuthIP Authentication Method: Unknown authentication Role: Initiator Impersonation State: Not enabled Main Mode Filter ID: 65930 Failure Information: Failure Point: Local computer Failure Reason: IKE authentication credentials are unacceptable State: Sent second (KE) payload Initiator Cookie: b148c67e631af726 Responder Cookie: 97f6a8eb093568ea ----------------------------------------------------------- The server has a similar error as well, pointing towards the client as the failure point. When I got the Windows 7 client (that is part of the domain) working, I requested the certificates via the mmc on the client (I had to install both a computer AND IPSec (Offline request) certificate before things worked). In other words I did not use the web interface to request the certificate. So... does anyone see what I'm doing wrong when trying to setup IPSec with certificates on a system that is not part of the domain? And is it really necessary to install two certificates (computer and IPSec (offline request) in order to use certificates for IPSec authentication? Thanks, Dasani
April 27th, 2011 1:58pm

I would also sugest checking whether the computer account has a private key for the certificate. You can also create the request directly from the local computer store by using the GUI wizard and not using the text file. However, the certificate can be ok, while there also can be a problem with the IPSEc policy that is not configured correctly to find the certificate. recheck whether the policy really specifies the correct trusted CA as the certificate issuer. if everything still seems correct, try obtaining the certificate not for the "key storage provider", but use a CSP instead. Although the "key storage providers" are supported by ipsec, as many applications have problems with the new CNG, I would try the older solid CSP as a means of test. ondrej.
Free Windows Admin Tool Kit Click here and download it now
April 28th, 2011 3:53am

When I view the certificate it says "You have a private key that corresponds to this certificate", so I suspect the certificate/key are healthy. I've double checked the IPSec policy to ensure I was using the correct CA. I also tried using both "Microsoft Enhanced Cryptographic Provider v1.0" and "Microsoft Base Cryptographic Provider v1.0" for ProviderName with no luck. Thanks, Dasani
April 28th, 2011 11:48am

Hi Dasani, Thank you for the post. From the post I can understand that you are not part of the domain to communicate with my file server using a computer certificate for IPSec. - So you are using CISCO VPN / Juniper ? These companies have changed the rules of ipsec for customer's convinience. If it is Juniper, then they use network connect - with split tunneling and allow access to local subnet etc. Further, the file server you have mentioned use : Certificate for authentication ? (please confirm) Please confirm my understanding. Regards, Dhruv
Free Windows Admin Tool Kit Click here and download it now
April 28th, 2011 3:34pm

Sorry for my late response.... In my test scenario, the client that I am attempting to connect to a file server is not part of the active directory domain the file server is in - the client is simply in a workgroup. I'm testing on a campus LAN, no VPN or provider is in the equation. I'm trying to setup a scenario in which all file sharing traffic will be secured by IPSec. Clients that are domain members will use the default (Kerberos) authentication for IPSec. Clients that are not domain members will use certificates for IPSec authentication. Dasani
May 3rd, 2011 2:30pm

Hi Customer, Please add server/client authentication in your RequestConf.inf file to generate CSR. [EnhancedKeyUsageExtension] OID = 1.3.6.1.5.5.8.2.2 ;IP Security IKE Intermediate OID = 1.3.6.1.5.5.7.3.1 ;Server Authentication OID = 1.3.6.1.5.5.7.3.2 ;Client Authentication If still not work, please install certificate refer to KB 323342. How to install a certificate for use with IP Security in Windows Server 2003 http://support.microsoft.com/kb/323342 Regards, Rick Tan
Free Windows Admin Tool Kit Click here and download it now
May 4th, 2011 12:07am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics