Windows, IKEv2, and error 13801

2018-03-28

I have been working with a fork of the Algo VPN, and I am trying to add functionality that will assign the same IP address to a cliente each time it connects.

During this process, I found myself debugging an error from Windows clients - “IKE authentication credentials are unacceptable”. In the Event Log, this was reported as error number 13801. This error is pretty bad, but until I ran into this problem, I didn’t realize how bad it was.

This error is about the client authenticating the server, not the server authenticating the client. That is, this error is generated when the client cannot verify that it is talking to the correct server. It is not an indication that the client’s credentials (its certificate and private key) are incorrect. However, even that explanation does not cover the breadth of scenarios when this error can crop up, as we will se below.

I had actually hit this error before, but this time seemed to be different. Algo was generating and signing the certs the same way before and after my modification; the only thing that changed was strongSwan’s ipsec.conf file itself. It didn’t seem to be the Extended Key Usage (EKU) properties that had tripped me up before.

Microsoft has published the document Troubleshooting IKEv2 VPN Connections, which says:

Error 13801 occurs on the client when:

  • The certificate is expired.
  • The trusted root for the certificate is not present on the client.
  • The subject name of the certificate does not match the remote computer.
  • The certificate does not have the required Enhanced Key Usage (EKU) values assigned.

However, this list is not complete.

My issue was wasn’t related to certificates at all. It was instead that my ipsec.conf file did not accept the client’s ID. I was defining individual connection stanzas for each client, in order to assign them the same IP address each time they connected. The stanzas looked like this:

conn AlgoUser-testuser
    rightid="testuser"
    rightsourceip=10.19.48.2
    auto=add

And the problem was in the rightid assignment. The certificate was generated with a CommonName of /CN=testuser, and this is submitted as a client identifier. However different IKEv2 implementations submit this identifier in different ways. For instance, macOS and iOS would submit their client identifier as simply testuser, while strongSwan’s default configuration and apparently Windows will submit their client identifier as /CN=testuser. Since this did not match any of my connection stanzas, the server was not able to accept the connection.

So, in summary, error 13801 is very generic, and issues unrelated to certificates can cause it.