An IPsec main mode negotiation failed
Hello,
I have setup an enterprise root CA in a test AD domain (2008r2), and am attempting to get file sharing traffic secure using IPSec. I'm able to secure the traffic between a Windows 7 client and a 2008r2 file server that are both part
of the domain using either the default Kerberos method, or a computer certificate.
However, I have been unable to get a Windows server 2008r2 box that is NOT part of the domain to communicate with my file server using a computer certificate for IPSec. On the 2008r2 client I created a RequestConf.inf file with the following info:
[Version]
Signature = "$Windows NT$"
[NewRequest]
RequestType = PKCS10
ProviderName = "Microsoft Software Key Storage Provider"
Subject = "CN=2008R2-1"
KeyLength = 2048
MachineKeySet = TRUE
KeySpec = 2
KeyUsage = 0x80
[EnhancedKeyUsageExtension]
OID = 1.3.6.1.5.5.8.2.2 ;IP Security IKE Intermediate
From the command prompt I ran 'certreq -N -f RequestConf.inf CertRequest.req' to generate my CSR.
I then connected to my CA via https and selected 'Request a certificate' > 'Advanced certificate request' > 'Submit a certificate request by using a base-64.....' and pasted the CSR I generated above into the Saved Request field. I then selected
IPSec (Offline request) from the Template drop down box and hit submit.
Once the certificate was created, I selected the 'Base 64 encoded' option and downloaded the certificate.
I then import the certificate via the MMC (Computer Account), and the certificate successfully imports into the Local Computer > Personal > Certificates store (I've also imported the certificate using the 'certreq -accept' method). I should also
mention that I imported the trusted root from my CA into the clients certificate store as well.
When I attempt to access the file share, I receive the following in the event log on the client:
--------------------------------------------------------------------------
An IPsec main mode negotiation failed.
Local Endpoint:
Local Principal Name: -
Network Address: x.x.x.x
Keying Module Port: 500
Remote Endpoint:
Principal Name: -
Network Address: x.x.x.x
Keying Module Port: 500
Additional Information:
Keying Module Name: AuthIP
Authentication Method: Unknown authentication
Role: Initiator
Impersonation State: Not enabled
Main Mode Filter ID: 65930
Failure Information:
Failure Point: Local computer
Failure Reason: IKE authentication credentials are unacceptable
State: Sent second (KE) payload
Initiator Cookie: b148c67e631af726
Responder Cookie: 97f6a8eb093568ea
-----------------------------------------------------------
The server has a similar error as well, pointing towards the client as the failure point.
When I got the Windows 7 client (that is part of the domain) working, I requested the certificates via the mmc on the client (I had to install both a computer AND IPSec (Offline request) certificate before things worked). In other words I did not use
the web interface to request the certificate.
So... does anyone see what I'm doing wrong when trying to setup IPSec with certificates on a system that is not part of the domain? And is it really necessary to install two certificates (computer and IPSec (offline request) in order to use certificates
for IPSec authentication?
Thanks,
Dasani
April 27th, 2011 1:58pm
I would also sugest checking whether the computer account has a private key for the certificate. You can also create the request directly from the local computer store by using the GUI wizard and not using the text file.
However, the certificate can be ok, while there also can be a problem with the IPSEc policy that is not configured correctly to find the certificate. recheck whether the policy really specifies the correct trusted CA as the certificate issuer.
if everything still seems correct, try obtaining the certificate not for the "key storage provider", but use a CSP instead. Although the "key storage providers" are supported by ipsec, as many applications have problems with the new CNG, I would try the
older solid CSP as a means of test.
ondrej.
Free Windows Admin Tool Kit Click here and download it now
April 28th, 2011 3:53am
When I view the certificate it says "You have a private key that corresponds to this certificate", so I suspect the certificate/key are healthy.
I've double checked the IPSec policy to ensure I was using the correct CA.
I also tried using both "Microsoft Enhanced Cryptographic Provider v1.0" and "Microsoft Base Cryptographic Provider v1.0" for ProviderName with no luck.
Thanks,
Dasani
April 28th, 2011 11:48am
Hi Dasani,
Thank you for the post. From the post I can understand that you are not part of the domain to communicate with my file server using a computer certificate for IPSec.
- So you are using CISCO VPN / Juniper ? These companies have changed the rules of ipsec for customer's convinience. If it is Juniper, then they use network connect - with split tunneling and allow access to local subnet etc.
Further, the file server you have mentioned use : Certificate for authentication ? (please confirm)
Please confirm my understanding.
Regards,
Dhruv
Free Windows Admin Tool Kit Click here and download it now
April 28th, 2011 3:34pm
Sorry for my late response....
In my test scenario, the client that I am attempting to connect to a file server is not part of the active directory domain the file server is in - the client is simply in a workgroup.
I'm testing on a campus LAN, no VPN or provider is in the equation.
I'm trying to setup a scenario in which all file sharing traffic will be secured by IPSec. Clients that are domain members will use the default (Kerberos) authentication for IPSec. Clients that are not domain members will
use certificates for IPSec authentication.
Dasani
May 3rd, 2011 2:30pm
Hi Customer,
Please add server/client authentication in your RequestConf.inf file to generate CSR.
[EnhancedKeyUsageExtension]
OID = 1.3.6.1.5.5.8.2.2 ;IP Security IKE Intermediate
OID = 1.3.6.1.5.5.7.3.1 ;Server Authentication
OID = 1.3.6.1.5.5.7.3.2 ;Client Authentication
If still not work, please install certificate refer to KB 323342.
How to install a certificate for use with IP Security in Windows Server 2003
http://support.microsoft.com/kb/323342
Regards, Rick Tan
Free Windows Admin Tool Kit Click here and download it now
May 4th, 2011 12:07am