We have been getting quite a few calls lately where Kerberos authentication fails intermittently and users are unable to log on.  By itself, that’s a type of call that we’re used to and we help our customers with all the time.  Most experienced AD admins know that this can happen because of broken AD replication, unreachable DCs on your network, or a variety of other environmental issues that all of you likely work hard to avoid as much as possible – because let’s face it, the last thing any admin wants is to have users unable to log in – especially intermittently.

Anyway, we’ve been getting more calls than normal about this lately, and that led us to take a closer look at what was going on.  What we found is that there’s a problem that can manifest when you have Windows Server 2003 and Windows Server 2012 R2 domain controllers serving the same domain.  Since many of you are trying very hard to get rid of your last Windows Server 2003 domain controllers, you might be running into this.  In the case of the customers that called us, the login issues were actually preventing them from being able to complete their migration to Windows Server 2012 R2.

We want all of our customers to be running their Active Directory on the latest supported OS version, which is frankly a lot more scalable, robust, and powerful than Windows Server 2003.  We realize that upgrading an enterprise environment is not easy, and much less so when your users start to have problem during your upgrade.  So we’re just going to come out and say it right up front:

We are working on a hotfix for this issue, but it’s going to take us some time to get it out to you. In the meantime, here are some details about the problem and what you can do right now.

Symptoms include:

When a user tries to log on to their computer, the logon may fail with “unknown username or bad password”

If you look in the system event log, you may notice Kerberos event IDs 4 that look like this:

Event ID: 4
Source: Kerberos
Type: Error
"The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server host/myserver.domain.com.  This indicates that the password used to encrypt the Kerberos service ticket is different than that on the target server. Commonly, this is due to identically named machine accounts in the target realm (domain.com), and the client realm.   Please contact your system administrator."

 

Why this happens:

The Kerberos client depends on a “salt” from the KDC in order to create the AES keys on the client side. These AES keys are used to hash the password that the user enters on the client, and protect it in transit over the wire so that it can’t be intercepted and decrypted. The “salt” refers to information that is fed into the algorithm used to generate the keys, so that the KDC is able to verify the password hash and issue tickets to the user.

When a Windows 2012 R2 DC is promoted in an environment where Windows 2003 DCs are present, there is a mismatch in the encryption types that are supported on the KDCs and used for salting. Windows Server 2003 DCs do not support AES and Windows Server 2012 R2 DCs don’t support DES for salting.

You might be wondering why these encryption types matter.  As computer hardware gets more powerful, older encryption methods become easier and easier to break.  Thus, we are constantly incorporating newer, more powerful encryption into Windows and Kerberos in order to help protect your user passwords (and your data and your network).

 

Workaround:

If users are having the problem:

Restart the computer that is experiencing the issue. This recreates the AES key as the client machine or member server reaches out to the KDC for Salt. Usually, this will fix the issue temporarily.

How to prevent this from happening:

Option 1: Query against Active Directory the list of computers which are about to change their machine account password and proactively reset their password against a Windows Server 2012 R2 DC and follow that by a reboot.

There’s an advantage to doing it this way: since you are not disabling any encryption type and keeping things set at the default, you shouldn’t run into any other other authentication related issue as long as the machine account password is reset successfully.

Unfortunately, doing this will mean a reboot of machines that are about to change their passwords, so plan on doing this during non-business hours when you can safely reboot workstations.

We’ve created a quick PowerShell script that you can run to do this.

Sample PS script:

> Import-module ActiveDirectory

> Get-adcomputer -filter * -properties PasswordLastSet | export-csv machines.csv

Reset Password:

nltest /SC_CHANGE_PWD:<DomainName>

Option 2: Disable machine password change or increase duration to 120 days.

You should not run into this issue at all if password change is disabled. Normally we don’t recommend doing this since machine account passwords are a core part of your network security and should be changed regularly. However because it’s an easy workaround, the best mitigation right now is to set it to 120 days. That way you buy time while you wait for the hotfix.

If you go with this approach, make sure you set your machine account password duration back to normal after you’ve applied the hotfix that we’re working on.

Here’s the relevant Group Policy settings to use for this option:

Computer Configuration\Windows Settings\Security Settings\Local Polices\Security Options

Domain Member:  Maximum machine account password age:

Domain Member: Disable machine account password changes:

Option 3: Disable AES in the environment by modifying Supported Encryption Types for Kerberos using Group Policy. This tells your domain controllers to use RC4-HMAC as the encryption algorithm, which is supported in both Windows Server 2003 and Windows Server 2012 and Windows Server 2012 R2.

You may have heard that we had a security advisory recently to disable RC4 in TLS. Such attacks don’t apply to Kerberos authentication, but there is ongoing research in RC4 which is why new features such as Protected Users do not support RC4. Deploying this option on a domain computer will make it impossible for Protected Users to sign on, so be sure to remove the Group Policy once the Windows Server 2003 DCs are retired.

The advantage to doing this is that once the policy is applied consistently, you don’t need to chase individual workstations. However, you’ll still have to reset machine account passwords and reboot computers to make sure they have new RC4-HMAC keys stored in Active Directory.

You should also make sure that the hotfix https://support.microsoft.com/kb/2768494  is in place on all of your Windows 7 clients and Windows Server 2008 R2 member servers, otherwise they may have other issues.

Remember if you take this option, then after the hotfix for this particular issue is released and applied on Windows Server 2012 R2 KDCs, you will need to modify it again in order to re-enable AES in the domain. The policy needs to be changed again and all the machines will require reboot.

Here are the relevant group policy settings for this option:

Computer Configuration\Windows Settings\Security Settings\Local Polices\Security Options

Network Security:  Configure encryption types allowed for Kerberos:

Be sure to check:  RC4_HMAC_MD5

If you have unix/linux clients that use keytab files that were configured with DES enable:  DES_CBC_CRC, DES_CBC_MD5

Make sure that AES128_HMAC_SHA1, and AES256_HMAC_SH1 are NOT Checked

 

–The Directory Services Team


Ask the Directory Services Team

{ Comments on this entry are closed }

Twitter’s Diversity Report Is More Of The Same

July 23, 2014

 Twitter released its diversity numbers on Wednesday afternoon after mounting pressure from civil rights groups — and they’re pretty bad if you’re not a white or Asian guy. Overall, Twitter’s employees are 70 percent male and 30 percent female. However when you look specifically at employees in tech, the gender gap grows, and 90 percent […]

Read the full article →

Microsoft named a Leader in Agile Business Intelligence by Forrester

July 23, 2014

We are pleased to see Microsoft acknowledged by Forrester Research as a Leader in The Forrester Wave™: Agile Business Intelligence Platforms, Q3 2014.   We are happy to see what we believe to be an affirmation in our approach and in the strength of our technologies. Our placement in this report reflects both high scores from […]

Read the full article →

No Man’s Sky: a vast game crafted by algorithms

July 23, 2014

No Man’s Sky is a video game quite unlike any other. Developed for Sony’s PlayStation 4 by an improbably small team (the original four-person crew has grown only to 10 in recent months) at Hello Games, an independent studio in the south of England, it’s a game that presents a traversable universe in which every […]

Read the full article →

Announced: Microsoft’s unified technology event for enterprises

July 22, 2014

Quick note to let you know we've announced a new unified technology event for enterprises. If you've attended Microsoft Exchange Conference (MEC), SharePoint Conference, Lync Conference or Project Conference, this event is for you! Head over to Office Blogs for more. Bharat Suneja Exchange Team Blog

Read the full article →

How to link Azure RemoteApp to an existing VNET

July 22, 2014

My name is Daniel Campos, and I am a program manager intern on the RemoteApp team. On May 12th, we announced the availability of the preview for the Microsoft Azure RemoteApp Service. Some of our customers asked for more information on how to connect Azure RemoteApp to their existing Azure VNET. This tutorial walks you […]

Read the full article →

As Popular As It Is, We Can No Longer Unbundle Microsoft Office

July 20, 2014

 This spring we witnessed the big three OS manufacturers — Microsoft, Apple and Google — continue to unbundle their versions of Microsoft Office apps for mobile, including versions of PowerPoint, Excel and Word for iPad. You’ve also seen Craig Federighi’s endurance test at Apple’s WWDC where he juggled all three Apple devices in order to […]

Read the full article →

Backdoors and surveillance mechanisms in iOS devices

July 19, 2014

Jonathan Zdziarski’s paper about backdoors, attack points and surveillance mechanisms built into iOS is quite, quite interesting. recent revelations exposed the use (or abuse) of operating system features in the surveillance of targeted individuals by the National Security Agency (NSA), of whom some subjects appear to be American citizens. This paper identifies the most probable […]

Read the full article →

Sentiment Analysis with Microsoft APS and StreamInsight

July 19, 2014

In this overview and demo, we will show you what sentiment analysis is and how to build a quick mashup that combines real-time access to multiple data sources using tools from Microsoft. Sentiment analysis is one of the hottest topics in the Big Data space. Sentiment analysis is the process of analyzing customer comments and […]

Read the full article →

Moving users to different Sync Shares

July 18, 2014

User folders are associated with a sync share, and there are times you will need to move a user to different sync shares. For example, maybe the user changed jobs from HR to Finance, which has its own sync share. The process is pretty straightforward, by changing the user’s security memberships. This blog post goes […]

Read the full article →