IIS 7 and Above
Application Request Routing (ARR)
TLS 1.2 issues on ARR
Last post Oct 06, 2020 07:48 PM by Rovastar
Sep 06, 2018 06:37 AM|carlwerner|LINK
I have ARR setup on Server 2016 with 2x Server 2008 web servers in the webfarm. This setup has been working fine for the last year. We are expecting an influx of traffic on our websites for the next 2 months so have decided to add 2 more web servers in the
webfarm. Herewith the problem... I added 2x Server 2016 web servers to the webfarm and when the connection is routed to one of these 2 servers I get the following error:
HTTP Error 502.3 - Bad Gateway
The connection with the server was terminated abnormally
Most likely causes:
The CGI application did not return a valid set of HTTP errors.
A server acting as a proxy or gateway was unable to process the request due to an error in a parent gateway.
On the ARR server I see the following error in the System log:
Schannel: A fatal error occurred while creating a TLS client credential. The internal error is 10013
I have enabled Schannel debug logging and can see that when ARR connects to one of the server 2008 web servers it negotiates to communicate using TLS 1.0 but when it connects to one of the server 2016 web servers it fails the negotiation.
If I completely disable TLS 1.0 (which we plan to do after this busy period), the connections to both server 2008 and server 2016 web servers fail.
TLS 1.0 is currently enabled on all web servers and the websites work if I connect to them directly from the ARR server bypassing ARR.
I have come to the conclusion that the problem lies with ARR itself not supporting using TLS 1.2.
I have tried the following solutions on a test environment and still dont have a working solution.
- Confirm all servers are fully up to date and we are using the latest version of ARR 3.0
Any other ideas or info?
Sep 06, 2018 12:19 PM|lextm|LINK
What does a tool like OpenSSL say about TLS 1.2 when talking to your ARR server？
Start from a more specific error, and that should lead you to the solution. You'd better have a testing environment to mirror the production environment, where you can make all necessary changes.
Sep 06, 2018 07:55 PM|Rovastar|LINK
Although I have not tested with the same setup as you (I tried on Windows 2016 ARR to windows 2012r2 backend webservers that both only had TLS 1.2 enabled) I disagree with your conclusion.
The way you talk it implies that you are communicating with your backend server via https. Often you can offload you SSL so it terminates on teh ARR and then communication is on http to the backend servers. But if your URLrewrite rule in the ARR says they
must be Https communication you are adding an adding a new https connection from ARR to your web servers. That is fine just trying to understand your env.
You can do a couple of simple test here.
Setup a new farm that has in its farm just has a new "hello world" html site on the backend that is HTTPS site (with a valid cert that ARR can understand).
IF your server setup is TLS 1.2 only then see if this works. If this works no problem then it shows ARR works via TLS 1.2 only.
I have experienced issues disabling TLS 1.2 on my ARR but these haven't been because ARR doesn't talk via TLS 1.2 but because the backend webservers are talking back to the ARR via TLS 1.0
To test this get started with netmon/wireshark is see the actual traffic and its protocols to see if there is any traffic going from your webserver back to your ARR.
If you error "Schannel: A fatal error occurred while creating a TLS client credential. The internal error is 10013" is on the ARR as you imply, then that is communication from some client (a server, an end user web browser, etc) using a commuinication method
that is not applicable on your ARR server. Schannel error messages are not very user friendly. This is likely communication on SSL 3 or TLS 1.0 that is (now) disabled.
You need to establish what is communicating on this protocol.
USe wireshark/netmon on your ARR.
It could also be an invalid cert in some way but either way the SSL/TLS handshake will tell you more info.
Sep 07, 2018 07:36 AM|carlwerner|LINK
Yes I specify HTTPS in the URL rewrite rule on ARR for communication to the backend servers.
I enabled SCHANNEL debug logging and can see that something is happening during the SSL/TLS handshake.
I followed your recommendation with the test site and Wireshark.
I saw the following:
- When the 2016 ARR server communicates to the 2016 backend server it uses tls 1.2 for communication. The ARR server sends the "client hello" but the backend server doesnt send the "server hello". The backend server just replies with an rst ack - connection
- When I disable TLS1 on the 2008 backend server, the same issue happens as when I connect to the 2016 backend server
So with this investigation my issue looks similar to your issue where the backend 2016 webservers are not talking back to the ARR via TLS 1.2.
I dont think its a cert issue because it work when using TLS1.
Sep 07, 2018 07:39 AM|carlwerner|LINK
I will try this later today...
Yes I have a mirrored testing environment.
We are running our web serves on VMWare, so was easy to mirror for a test environment
Sep 10, 2018 07:54 AM|deepakpanchal10|LINK
did you try to make a test on your side?
If yes, Whats the testing the result?
If you did not test it, then once you done with it. Let us know about the testing results.
We will try to provide further suggestions, If needed.
Nov 22, 2018 03:30 PM|marbor|LINK
Hi, did you find a solution to your problem? ARR uses WinHTTP and it defaults to a lower version of TLS. Se the following article on how to fix it:
Feb 18, 2020 07:00 PM|zolibatan|LINK
Thank you for your answer. I had this same problem and the solution was to set DefaultSecureProtocols for WinHTTP. Also needed a reboot for ARR to work.
Mar 26, 2020 10:03 AM|_Tubbs_|LINK
I have the same problem only with 2019 ARR and backend servers. I tried changing the DefaultSecureProtocols for winhttp, but it doesnt seem to have any effect as far as i can see in wireshark. The connection seems to get made properly, but the backend server
responds with a reset.
If i connect directly to the backend servers with https it works just fine.
If anyone has a sollution i can try?
Oct 01, 2020 09:10 PM|MetUys|LINK
Did you come right here?
From what I read about the winHTTP, is that on Servers from Server 2012 R2 and newer it already supports TLSv1.1 and TLSv1.2, its only for Servers 2012 and older than need the registry updated.
(see applies to in this article: https://support.microsoft.com/en-us/help/3140245/update-to-enable-tls-1-1-and-tls-1-2-as-default-secure-protocols-in-wi)
This is further validated on a newish setup of an ARR on Server 2012 R2 with a back-end server that has had its TLS hardened to only support the TLSv1.2 protocol.
In this setup, if I sniff the traffic with wireshark on the ARR destined for the back-end server, with the ARR set to use the HTTPS scheme in the URL_rewrite rule to the web-farm, I can see the ARR making a TLSv1.2 connection.
However! What I did notice in the packet captures is that the ARR may be doing a TLSv1.2 connection, but it does not have the SNI extension set in the Client Hello... what this means is if the backend server is only configured with an SNI certificate and no
"base certificate" then the connection will fail.
Although the above might shed some light, I have yet to find a way to get the ARR to set the SNI TLS extension. if anyone has ideas on that please let me know. alternatively we will have to see if we cant get a base certificate setup on the back-end servers.
Oct 02, 2020 04:10 AM|Rovastar|LINK
Just a few weeks ago I had had to troubleshoot this issue in the wild and it is indeed SNI not being sent by ARR and the backend servers need 1 site that is not forcing SNI.
To be honest the workaround is so simple (just have a site, any site, that has any valid cert (doesn't have to be for the domain you are on) that has non SNI in the bindings and all connections will work to that server from the ARR. That I haven't looked
any further into it. It may be that ARR cannot send sni.
I saw someone has posted on this thread and thought I know what this is now and I can update it but I see you have already informed them.
Oct 02, 2020 10:45 PM|lextm|LINK
That I haven't looked any further into it. It may be that ARR cannot send sni.
ARR by default does not send the original Host header, so "preserveHostHeader" must be set to true
https://github.com/lextm/iis_schema/blob/master/arr_schema.xml#L29 Maybe that's why SNI bindings on upstream server do not work (SNI requires Host header to match the certificate
mapping in HTTP API).
Oct 06, 2020 07:48 PM|Rovastar|LINK
"ARR by default does not send the original Host header, so "preserveHostHeader"
This isn't entirely true. All the setups I have ever done for ARR the host header is preserved. Maybe this is because it use the web farms part of ARR rather than just the ARR for a reverse proxy.
Anyway by default the farms have "preserveHostHeader" set to true.
So maybe that setting is for those that use ARR without any webfarms do that reverse proxy setup. But they are edge cases for most users of ARR.
I thought may be that that it the initial check it is defaulting to the proxy setting of ARR like for the initial SSL handshake and somehow this preseving the value and choosing to send the SNI but in the health check but that doesn't seem to be the case.
NIce idea though.
I think that the winhttp that ARR is using to send out these connections isn't sending out the SNI by default. I guess maybe there is some registry hack to send this but it seems like more and more work to test it.
Anyway there is a good workaround in place. I'm not going ot spend another 4 hours testing different scenerios (well for now at least) for something that isn't really an issue,.