Notes from the field: The Kerberos chronicles, the one with VMware TrueSSO

After a lengthy and cumbersome troubleshoot on a VMware TrueSSO setup finally had the time to blog this one.

In summary the situation with a customer was a working VMware TrueSSO setup which stopped working, after lengthy troubleshooting we opened a support case with VMware and later also with Microsoft.

The issue was manifesting in an always denied by policy module message form the CA which also cohosts the enrollment server role of VMware TrueSSO. Logging etc. nothing was making any sense and had multiple sessions with VMware and Microsoft around this. We already had some suspicions that CIS hardening around this was perhaps breaking items in the chain and eventually it was the culprit.

After focusing on some spn errors, Kerberos messages regarding the parent/child domain we found out that the CIS hardening which in turn strengthens the Kerberos allowed ciphers and disabled the older ones like DES and RC4 for the member servers was causing this behavior. But why? Well, the CIS hardening was in place for multiple other customers which didn’t have a parent forest/domain and this setup has. And now comes the kicker in which Microsoft support really helped. By default, every parent/child or trust setup is using the RC4 encryption type. The only way to enable AES support there is to click on the item box in domains and trusts on all your trusts which say: The other domain supports Kerberos AES Encryption. After enabling this or even enabling the RC4 cipher again everything started to work as expected.

Disabling RC4 Encryption type for Kerberos can be a tedious process, if you want to avoid an outage. Generally, you want to first introduce AES into the environment for a while (at least 60 days, if possible), then remove RC4.

Key points to remember:

–User accounts by default do not have MsDS-SupportedEncryptionTypes populated. When this account authenticates, it may still use AES if the computer it is running on supports it. However, when requesting a ticket for a service hosted by that account, it will default to RC4 because when the DC looks up the supported encryption, nothing is specified. So, it is possible to have a SQL service account that logs on as a service on a SQL server obtain an AES ticket for itself while any user’s requesting tickets for that account will only be able to obtain RC4 tickets. When RC4 is disabled in this scenario, services will start, but no one can authenticate using Kerberos. In order to enable AES on these service accounts, you’ll need to check the boxes regarding AES on the ‘Account’ tab under ‘Account options’.

–By default, all trusts, including parent and child, are RC4 only. In the domain and trusts snap-in, there is an option called “The other domain supports Kerberos AES Encryption” Do no check that box without knowing the result. Checking this box both turns off RC4 and enables AES. In order to support a graceful transition, use ksetup instead to add AES to the trusts. You’ll need to do this for all trusts. For example, consider contoso.com and child.contoso.local:

ksetup /setenctypeattr child.contoso.local RC4-HMAC-MD5 AES128-CTS-HMAC-SHA1-96 AES256-CTS-HMAC-SHA1-96 (On a Contoso DC)

ksetup /setenctypeattr contoso.com RC4-HMAC-MD5 AES128-CTS-HMAC-SHA1-96 AES256-CTS-HMAC-SHA1-96 (On a child DC)

–Passwords set prior to the introduction of Server 2008 may not have AES keys generated. This can include the krbtgt account for the domain as well. These accounts must have their password reset if you find that after enabling AES, they are still using RC4.

–You’ll want to leverage the security logs on the DC throughout this transition effort looking for RC4 tickets being issued. You need to enable auditing for “Kerberos Authentication Service” and “Kerberos Service Ticket Operations” on all Domain Controllers. Events 4768 and 4769 will be logged that show the encryption type used. The field you’ll need to focus on is called “Ticket Encryption Type” and you’re looking for 0x17. 0x17 indicates RC4 was issued. This XML query below can be used to filter for these:

*[EventData[Data[@Name=’TicketEncryptionType’] and (Data=’0x17′)]]

</Select>

</Query>

</QueryList>

You may want to consider leveraging a SIEM or event forwarding to collect these events in a central location for review.

–Its possible that you will have devices/applications on the network that do not support AES. You will need to work with the vendor to upgrade or replace them.

MORE INFORMATION:

The RC4 Removal Files Part 2: In AES We Trust – Microsoft Tech Community

Technologies for Federating Multiple Forests | Microsoft Docs

Tough Questions Answered: Can I disable RC4 Etype for Kerberos on Windows 10 ? (microsoft.com)

Preventing Kerberos change password that use RC4 secret keys | Microsoft Docs

Kerberos Unsupported etype error – Windows Server | Microsoft Docs

When the Kerberos EType needs adoption:

977321 KDC Event ID 16 or 27 is logged if DES for Kerberos is disabled

https://support.microsoft.com/en-US/help/977321

Blog on Kerberos Supported Encryption Types:

https://blogs.msdn.microsoft.com/openspecification/2011/05/30/windows-configurations-for-kerberos-supported-encryption-type/

Netlogon Debug Logging: