TLS 1.2 issues on ARRRSS

8 replies

Last post Mar 26, 2020 10:03 AM by _Tubbs_

  • TLS 1.2 issues on ARR

    Sep 06, 2018 06:37 AM|carlwerner|LINK

    Hi,

    I have ARR setup on Server 2016 with 2x Server 2008 web servers in the webfarm. This setup has been working fine for the last year. We are expecting an influx of traffic on our websites for the next 2 months so have decided to add 2 more web servers in the webfarm. Herewith the problem... I added 2x Server 2016 web servers to the webfarm and when the connection is routed to one of these 2 servers I get the following error:


    HTTP Error 502.3 - Bad Gateway

    The connection with the server was terminated abnormally

    Most likely causes:
    •The CGI application did not return a valid set of HTTP errors.
    •A server acting as a proxy or gateway was unable to process the request due to an error in a parent gateway.

    On the ARR server I see the following error in the System log:

    Schannel: A fatal error occurred while creating a TLS client credential. The internal error is 10013

    I have enabled Schannel debug logging and can see that when ARR connects to one of the server 2008 web servers it negotiates to communicate using TLS 1.0 but when it connects to one of the server 2016 web servers it fails the negotiation.

    If I completely disable TLS 1.0 (which we plan to do after this busy period), the connections to both server 2008 and server 2016 web servers fail.

    TLS 1.0 is currently enabled on all web servers and the websites work if I connect to them directly from the ARR server bypassing ARR.

    I have come to the conclusion that the problem lies with ARR itself not supporting using TLS 1.2.

    I have tried the  following solutions on a test environment and still dont have a working solution.

    - Confirm all servers are fully up to date and we are using the latest version of ARR 3.0

    • - Set different combinations using IISCrypto tool, including completely enabling all protocols and ciphers.
    • - Set read permission for the "NETWORK SERVICE" on the C:\Program Data\Microsoft\Crypto\RSA\MachineKeys folder
    • - Set the default protocol for WINHTTP using the Internet Settings\WinHttp\DefaultSecureProtocols registry key to use TLS 1.2 for both 32 and 64 bit.
    • - Set the default protocol for DotNet using the .NETFramework\v4.0.30319\SchUseStrongCrypto to use TLS 1.2 for both 32 and 64 bit.
    • - Enabled the FIPS trusted algorithm local security policy.

    Any other ideas or info?

    Regards

    Carl

  • Re: TLS 1.2 issues on ARR

    Sep 06, 2018 12:19 PM|lextm|LINK

    What does a tool like OpenSSL say about TLS 1.2 when talking to your ARR server?

    https://wiki.openssl.org/index.php/SSL/TLS_Client

    Start from a more specific error, and that should lead you to the solution. You'd better have a testing environment to mirror the production environment, where you can make all necessary changes.

    Lex Li
    Affordable IIS Consulting Services at https://support.lextudio.com/services/consulting.html
    ---------------------------
    This posting is provided "AS IS" with no warranties, and confers no rights.
  • Rovastar Rovastar

    5473 Posts

    MVP

    Moderator

    Re: TLS 1.2 issues on ARR

    Sep 06, 2018 07:55 PM|Rovastar|LINK

    Although I have not tested with the same setup as you (I tried on Windows 2016 ARR to windows 2012r2 backend webservers that both only had TLS 1.2 enabled) I disagree with your conclusion.

    The way you talk it implies that you are communicating with your backend server via https. Often you can offload you SSL so it terminates on teh ARR and then communication is on http to the backend servers. But if your URLrewrite rule in the ARR says they must be Https communication you are adding an adding a new https connection from ARR to your web servers. That is fine just trying to understand your env.

     

    You can do a couple of simple test here.

    Setup a new farm that has in its farm just has a new "hello world" html site on the backend  that is HTTPS site (with a valid cert that ARR can understand).

    IF your server setup is TLS 1.2 only then see if this works. If this works no problem then it shows ARR works via TLS 1.2 only.

    I have experienced issues disabling TLS 1.2 on my ARR but these haven't been because ARR doesn't talk via TLS 1.2 but because the backend webservers are talking back to the ARR via TLS 1.0

    To test this get started with netmon/wireshark is see the actual traffic and its protocols to see if there is any traffic going from your webserver back to your ARR.

    If you error "Schannel: A fatal error occurred while creating a TLS client credential. The internal error is 10013" is on the ARR as you imply, then that is communication from some client (a server, an end user web browser, etc) using a commuinication method that is not applicable on your ARR server. Schannel error messages are not very user friendly. This is likely communication on SSL 3 or TLS 1.0 that is (now) disabled.

    You need to establish what is communicating on this protocol.

    USe wireshark/netmon on your ARR.

    It could also be an invalid cert in some way but either way the SSL/TLS handshake will tell you more info.

    Troubleshoot IIS in style
    https://www.leansentry.com/
  • Re: TLS 1.2 issues on ARR

    Sep 07, 2018 07:36 AM|carlwerner|LINK

    @Rovastar...

    Yes I specify HTTPS in the URL rewrite rule on ARR for communication to the backend servers.

    I enabled SCHANNEL debug logging and can see that something is happening during the SSL/TLS handshake.

    I followed your recommendation with the test site and Wireshark.

    I saw the following:

     - When the 2016 ARR server communicates to the 2016 backend server it uses tls 1.2 for communication. The ARR server sends the "client hello" but the backend server doesnt send the "server hello". The backend server just replies with an rst ack - connection reset.

    • - When I bypass ARR and connect to the backend 2016 server direct from the ARR 2016 server using IE the connection works. Communication is done using TLS1.2 and the TLS connection setup shows the client hello, then the server hello and then the client key exchange.
    • - When the 2016 ARR server communicates to the 2008 backend server it uses TLS1 for communication. The TLS connection setup works correctly  and shows the client hello, then the server hello and then the client key exchange.

    - When I disable TLS1 on the 2008 backend server, the same issue happens as when I connect to the 2016 backend server

    So with this investigation my issue looks similar to your issue where the backend 2016 webservers are not talking back to the ARR via TLS 1.2.

    I dont think its a cert issue because it work when using TLS1.

    Regards

    Carl

  • Re: TLS 1.2 issues on ARR

    Sep 07, 2018 07:39 AM|carlwerner|LINK

    @lextm

    I will try this later today...

    Yes I have a mirrored testing environment.

    We are running our web serves on VMWare, so was easy to mirror for a test environment

  • Re: TLS 1.2 issues on ARR

    Sep 10, 2018 07:54 AM|deepakpanchal10|LINK

    Hi carlwerner,

    did you try to make a test on your side?

    If yes, Whats the testing the result?

    If you did not test it, then once you done with it. Let us know about the testing results.

    We will try to provide further suggestions, If needed.

    Regards

    Deepak

    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue.
    If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.
  • Re: TLS 1.2 issues on ARR

    Nov 22, 2018 03:30 PM|marbor|LINK

    Hi, did you find a solution to your problem? ARR uses WinHTTP and it defaults to a lower version of TLS. Se the following article on how to fix it:

    https://support.microsoft.com/en-us/help/3140245/update-to-enable-tls-1-1-and-tls-1-2-as-a-default-secure-protocols-in

    /M

  • Re: TLS 1.2 issues on ARR

    Feb 18, 2020 07:00 PM|zolibatan|LINK

    @marbor

    Thank you for your answer. I had this same problem and the solution was to set DefaultSecureProtocols for WinHTTP. Also needed a reboot for ARR to work.

  • Re: TLS 1.2 issues on ARR

    Mar 26, 2020 10:03 AM|_Tubbs_|LINK

    I have the same problem only with 2019 ARR and backend servers. I tried changing the DefaultSecureProtocols for winhttp, but it doesnt seem to have any effect as far as i can see in wireshark. The connection seems to get made properly, but the backend server responds with a reset.

    If i connect directly to the backend servers with https it works just fine.

    If anyone has a sollution i can try?