The same as my previous Kerberos blog but this time we have Citrix NetScaler in the mix with drumrolls… Kerberos Constrained Delegation henceforth to be known as KCD.
This in an setup derived from the following article: Tutorial: Azure Active Directory single sign-on integration with Citrix ADC SAML Connector for Azure AD (Kerberos-based authentication) – Microsoft Entra | Microsoft Learn
The scenario is a working setup from one of my customers and regarding all the latest exploits we needed to upgrade the NetScaler versions to a higher and higher and higher version. Along the way we encountered a faulty KCD which never would work again. Only a downgrade to the working version firmware would resolve the issue for us.
The working version was 13.0 Build 85.19 and everything higher would break Kerberos logons, eventually we also deployed an acceptance environment to reproduce the issue and that was instantly visible because the new deployment was on the latest 13.0 Build, when we downgraded to the 85.19 release everything started working again.
After troubleshooting this into oblivion and just couldn’t find anything wrong in the setup this most certainly needed to be a bug in the higher releases of version 13.0, even on 13.1 it would have the same behavior. So, concluding there is a change! Obviously at this point you would think didn’t you read the release notes? And yes we did like we always do and the only change was in Build 86.17 see Release Notes for Citrix ADC 13.0-86.17 Release | Citrix ADC 13.0 with NSHELP-28681, this doesn’t really state that something “changed” but keep with me on this one as later on this is the change that would have an undocumented change not being mentioned regarding security purposes of the firmware.
KB5020805: How to manage Kerberos protocol changes related to CVE-2022-37967 – Microsoft Support is the common ground in this, we now know that Microsoft is pushing the envelope with their phases etc. ok but in turn my issues are also around this! After troubleshooting and troubleshooting this with support, even an angry me to them as well because we weren’t getting anywhere.
It turns out that the ciphers regarding Kerberos were altered in this build and higher, the negotiating isn’t much negotiating but just dropping the ball and favoring AES and higher so basically, we wouldn’t see a working KCD logon anymore. After this we got the information that the username and password challenge wouldn’t work anymore if you only set that one up in the NetScaler setup, and yes this was the case for the customer because why would we need to change that in a working production environment.
To summarize the working setup after this, we needed to put in place a keytab instead of password challenge. The username/password value wasn’t working in all or some scenario’s and the keytab setup basically also changed the User logon name in ADDS to HOST\serviceaccount.fqdn which was the resolution. The switch to AES preferred also mandates this sort of configuration so the HOST\serviceaccount.fqdn and your SPN of the account need to be the same with the backend server as a delegation and then it works as expected.
One caveat though is that when domain hardening is in place the ciphers need to align exactly and that means the -all ciphers won’t work then. You would need to specify the ciphers like AES and higher, see ktpass | Microsoft Learn for the details in making a keytab account.
My solution for the customer then was a new service account or more a duplicate of the current one validatin all delegations and spn and then create a keytab with all ciphers, and AES only and use that in the NetScaler setup. After this everything started working for the acceptance environment and after an upgrade to the latest and greatest it kept on going strong.
Did the same in production and there had a site or two which needed to have the username/password but then as HOST\serviceaccount.fqdn as the logonname in the NetScaler KCD configuration to get it to work.
Citrix support admits the changes in the code of Kerberos in that particular build and higher, I’ve worked with them on aligning an updated document which take these changes into account, hope that one is getting out sooner than later.
Some reference articles for troubleshooting:
NetScaler Kerberos WIKI – Troubleshooting (citrix.com)
SUPPORT WIKI: Kerberos with NetScaler (citrix.com)
And a very big thanks to Anton van Pelt and Mark Brilman for keeping me sane in the months of troubleshooting and sparring actions with them.
Hope it helps!