IIS 7 and Above
Application Request Routing (ARR)
TLS 1.2 issues on ARR
Last post Mar 26, 2020 10:03 AM by _Tubbs_
Sep 06, 2018 06:37 AM|carlwerner|LINK
I have ARR setup on Server 2016 with 2x Server 2008 web servers in the webfarm. This setup has been working fine for the last year. We are expecting an influx of traffic on our websites for the next 2 months so have decided to add 2 more web servers in the
webfarm. Herewith the problem... I added 2x Server 2016 web servers to the webfarm and when the connection is routed to one of these 2 servers I get the following error:
HTTP Error 502.3 - Bad Gateway
The connection with the server was terminated abnormally
Most likely causes:
The CGI application did not return a valid set of HTTP errors.
A server acting as a proxy or gateway was unable to process the request due to an error in a parent gateway.
On the ARR server I see the following error in the System log:
Schannel: A fatal error occurred while creating a TLS client credential. The internal error is 10013
I have enabled Schannel debug logging and can see that when ARR connects to one of the server 2008 web servers it negotiates to communicate using TLS 1.0 but when it connects to one of the server 2016 web servers it fails the negotiation.
If I completely disable TLS 1.0 (which we plan to do after this busy period), the connections to both server 2008 and server 2016 web servers fail.
TLS 1.0 is currently enabled on all web servers and the websites work if I connect to them directly from the ARR server bypassing ARR.
I have come to the conclusion that the problem lies with ARR itself not supporting using TLS 1.2.
I have tried the following solutions on a test environment and still dont have a working solution.
- Confirm all servers are fully up to date and we are using the latest version of ARR 3.0
Any other ideas or info?
Sep 06, 2018 12:19 PM|lextm|LINK
What does a tool like OpenSSL say about TLS 1.2 when talking to your ARR server？
Start from a more specific error, and that should lead you to the solution. You'd better have a testing environment to mirror the production environment, where you can make all necessary changes.
Sep 06, 2018 07:55 PM|Rovastar|LINK
Although I have not tested with the same setup as you (I tried on Windows 2016 ARR to windows 2012r2 backend webservers that both only had TLS 1.2 enabled) I disagree with your conclusion.
The way you talk it implies that you are communicating with your backend server via https. Often you can offload you SSL so it terminates on teh ARR and then communication is on http to the backend servers. But if your URLrewrite rule in the ARR says they
must be Https communication you are adding an adding a new https connection from ARR to your web servers. That is fine just trying to understand your env.
You can do a couple of simple test here.
Setup a new farm that has in its farm just has a new "hello world" html site on the backend that is HTTPS site (with a valid cert that ARR can understand).
IF your server setup is TLS 1.2 only then see if this works. If this works no problem then it shows ARR works via TLS 1.2 only.
I have experienced issues disabling TLS 1.2 on my ARR but these haven't been because ARR doesn't talk via TLS 1.2 but because the backend webservers are talking back to the ARR via TLS 1.0
To test this get started with netmon/wireshark is see the actual traffic and its protocols to see if there is any traffic going from your webserver back to your ARR.
If you error "Schannel: A fatal error occurred while creating a TLS client credential. The internal error is 10013" is on the ARR as you imply, then that is communication from some client (a server, an end user web browser, etc) using a commuinication method
that is not applicable on your ARR server. Schannel error messages are not very user friendly. This is likely communication on SSL 3 or TLS 1.0 that is (now) disabled.
You need to establish what is communicating on this protocol.
USe wireshark/netmon on your ARR.
It could also be an invalid cert in some way but either way the SSL/TLS handshake will tell you more info.
Sep 07, 2018 07:36 AM|carlwerner|LINK
Yes I specify HTTPS in the URL rewrite rule on ARR for communication to the backend servers.
I enabled SCHANNEL debug logging and can see that something is happening during the SSL/TLS handshake.
I followed your recommendation with the test site and Wireshark.
I saw the following:
- When the 2016 ARR server communicates to the 2016 backend server it uses tls 1.2 for communication. The ARR server sends the "client hello" but the backend server doesnt send the "server hello". The backend server just replies with an rst ack - connection
- When I disable TLS1 on the 2008 backend server, the same issue happens as when I connect to the 2016 backend server
So with this investigation my issue looks similar to your issue where the backend 2016 webservers are not talking back to the ARR via TLS 1.2.
I dont think its a cert issue because it work when using TLS1.
Sep 07, 2018 07:39 AM|carlwerner|LINK
I will try this later today...
Yes I have a mirrored testing environment.
We are running our web serves on VMWare, so was easy to mirror for a test environment
Sep 10, 2018 07:54 AM|deepakpanchal10|LINK
did you try to make a test on your side?
If yes, Whats the testing the result?
If you did not test it, then once you done with it. Let us know about the testing results.
We will try to provide further suggestions, If needed.
Nov 22, 2018 03:30 PM|marbor|LINK
Hi, did you find a solution to your problem? ARR uses WinHTTP and it defaults to a lower version of TLS. Se the following article on how to fix it:
Feb 18, 2020 07:00 PM|zolibatan|LINK
Thank you for your answer. I had this same problem and the solution was to set DefaultSecureProtocols for WinHTTP. Also needed a reboot for ARR to work.
Mar 26, 2020 10:03 AM|_Tubbs_|LINK
I have the same problem only with 2019 ARR and backend servers. I tried changing the DefaultSecureProtocols for winhttp, but it doesnt seem to have any effect as far as i can see in wireshark. The connection seems to get made properly, but the backend server
responds with a reset.
If i connect directly to the backend servers with https it works just fine.
If anyone has a sollution i can try?