HOWTO: Fix IIS FTP server error 431 Failed to setup secure session
Published on 2021/06/10 by Igor Levicki
Today I was setting up an FTP server on a Windows Server using IIS, and I ran into a problem with establishing the TLS connection.
When trying to connect using FTP client I was getting the following error:
431 Failed to setup secure session.
After checking the IIS FTP logs I noticed a Win32 error code 2148074253 (0x8009030d) which translates into SEC_E_UNKNOWN_CREDENTIALS.
Given that I haven't even reached the point where I would be asked for user credentials, I suspected that this has something to do with the SSL certificate private key access and it turned out I was right.
The mistake I made is that I imported the SSL certificate using IIS Manager -- it turns out you have to import it using MMC Certificates snap-in (select Computer Account, Personal store), but you also have to mark the private key as exportable in order for TLS to work.
The reason is that the IIS is using SChannel for TLS and according to Microsoft Security Developer John Banes, SChannel has to be able to export the SSL certificate's private key:
On NT4 private keys used for server-side SSL must be marked exportable. The CSPs on NT4 don't support the operation necessary for SSL and so our SSL implementation (schannel) exports out the private key and uses it directly.
One would hope that 17 years later new CSPs would be available with a proper support for SSL/TLS which would eliminate the need for this requirement, but here we are.
TL;DR -- re-import your SSL certificate using MMC and mark the key as exportable and your IIS FTP server will work fine.