WebFarm Takes 10 Seconds to Relay to New Site [Answered]RSS

4 replies

Last post Nov 04, 2020 04:28 PM by philiptkd

‹ Previous Thread|Next Thread ›
  • WebFarm Takes 10 Seconds to Relay to New Site

    Oct 30, 2020 05:17 PM|philiptkd|LINK

    Hi, everyone. This is my first time working with IIS, so please forgive any obvious mistakes.

    Installation

    I'm using IIS 10 on Windows 10. I installed it with the "Turn Windows Features on or off" interface.

    I then installed Application Request Routing 3 (ARR 3), which came with the URL Rewrite module.

    Setup

    I have two identical sites/servers in a server farm for the purposes of blue-green deployment. That is, the goal is to have one site up and one site down at all times. I have a deployment script that publishes my application to the stopped site, starts it, and then stops the site originally running. I basically followed this blog post except wrote my script in C#, not powershell. My sites are both barebones ASP.NET Core MVC applications that say either "Blue" or "Green" on the home page.

    Script Procedure

    Let's call running and healthy site "Blue" and the stopped and unhealthy site "Green". Here's the order of events:

    Start Green. -> Wait until Green can return a 200 status fairly quickly. -> Set Green to be healthy. -> Wait for the health check to see that Green is healthy. -> Set Blue to be unhealthy. -> Stop Blue.

    Error

    For about 10 seconds after stopping Blue, the server farm does not switch to Green and instead returns a 502.3 Error (A connection with the server could not be established). After those 10 seconds, the server farm finally switches to Green and returns pages correctly. 

    The same error is also returned for the first request that I make (and 10 seconds thereafter) after a long delay. It seems that the webfarm needs about 10 seconds to do something after the initial request before it starts working.

    Also, oddly enough, the error does not occur when switching from Green to Blue. Only Blue to Green. As far as I can tell, the two sites and their configurations are completely identical aside from their names, ids, application pools, and what ports they bind to. The two application pools are identical, as well. (Default Web Site has id=1. Blue has id=2. Green has id=3.)

    What It Isn't

    I've determined that the problem isn't with the content servers/sites. They always respond as expected when I browse to them directly and not through the web farm.

    Also, I don't get this error when simply changing the sites' health status and leaving them both always running. In that case, the web farm switches to the healthy site promptly. So it's not that the webfarm doesn't do health checks frequently enough (it does every second). 

    The problem isn't the script that starts/stops sites. I can reproduce the problem when performing the same actions by clicking around manually in IIS.

    All of my app pools are set to the "AlwaysRunning" Start Mode.

    Something is making the webfarm stubborn about which site it routes to, and I can not for the life of me find a setting that will tell it to give up on the initial site more quickly.

  • Re: WebFarm Takes 10 Seconds to Relay to New Site

    Nov 02, 2020 07:41 AM|Jalpa Panchal|LINK

    Hi,

    Could you please share your Arr routing rule? try to run the failed request tracing in iis.make sure you did not set disallow new connection in monitoring and management.what is your arr serer proxy setting?

    .NET forums are moving to a new home on Microsoft Q&A, we encourage you to go to Microsoft Q&A for .NET for posting new questions and get involved today.
  • Re: WebFarm Takes 10 Seconds to Relay to New Site

    Nov 02, 2020 02:45 PM|philiptkd|LINK

    Thanks for the reply, Jalpa.

    See below for my routing rule and other web farm config. I'm actually having trouble getting any output from Failed Request Tracing; see my other post about that. I did not set "Disallow New Connections" in Monitoring and Management.

    I left proxy settings on their defaults. I'll list them here since I don't think they're shown in my config file.

    "Pass through" HTTP version. "Keep alive" is checked. "Time-out" is 30 seconds. "Reverse rewrite host" is not checked. The custom-headers settings are "X-Forwarded-For", "X-ARR-ClientCert", and "X-ARR-LOG-ID", in that order. "Include TCP port from client IP" is checked. "Response buffer" is 4096 KB. "Response buffer threshold" is 256 KB.

            <rewrite>
                <globalRules>
                    <rule name="alwaysup" stopProcessing="true">
                        <match url=".*" />
                        <conditions>
                            <add input="{HTTP_HOST}" pattern="^alwaysup.com$" />
                            <add input="{SERVER_PORT}" pattern="^80$" />
                        </conditions>
                        <action type="Rewrite" url="http://alwaysup/{R:0}" />
                    </rule>
                </globalRules>
            </rewrite>
            <aspNetCore stdoutLogEnabled="true" />
    
        </system.webServer>
        <webFarms>
            <webFarm name="alwaysup" enabled="true">
                <server address="alwaysup-blue" enabled="true">
                    <applicationRequestRouting weight="100" httpPort="8001" />
                </server>
                <server address="alwaysup-green" enabled="true">
                    <applicationRequestRouting weight="100" httpPort="8002" />
                </server>
                <applicationRequestRouting>
                    <healthCheck url="http://alwaysup/html/status.html" interval="00:00:01" responseMatch="up" />
                    <protocol timeout="00:00:30">
                        <cache enabled="false" />
                    </protocol>
                    <loadBalancing />
                </applicationRequestRouting>
            </webFarm>
            <applicationRequestRouting>
                <hostAffinityProviderList>
                    <add name="Microsoft.Web.Arr.HostNameRoundRobin" />
                </hostAffinityProviderList>
            </applicationRequestRouting>
        </webFarms>

  • Re: WebFarm Takes 10 Seconds to Relay to New Site

    Nov 04, 2020 02:46 AM|Jalpa Panchal|LINK

    may I know what is the value in your status file for both the site? did you stop the site manually or web farm? try to remove the health test and then set both the server healthy in the monitoring and management. and restart IIS server after doing all the changes. after doing this when you browse the URL http://alwaysup/ it will show the running site without any issue.

    if you just want to use the IIS load balancing it is not necessary to add a health test.

    .NET forums are moving to a new home on Microsoft Q&A, we encourage you to go to Microsoft Q&A for .NET for posting new questions and get involved today.
  • Re: WebFarm Takes 10 Seconds to Relay to New Site

    Nov 04, 2020 04:28 PM|philiptkd|LINK

    Jalpa,

    Thank you so much for your help.

    What you posted led me to finding the solution.

    I had not experimented with setting the sites as Healthy or Unhealthy in Monitoring and Management before.

    For some reason, my server farm does not automatically route to the only running site if one of them is stopped. If the stopped site is still marked "Healthy", then it will still sometimes attempt to route to it and then return an error. So it seems that the health checks are necessary for me.

    This made me realize that I was stopping the Blue site immediately after setting it to be "Unhealthy". I was not waiting for the health check to register the change in the "status.html" file.

    So, I was wrong. The error was in my script after all. I needed to add a delay for that second health check. The steps that I gave in my original post work when changed to the following:

    Start Green. -> Wait until Green can return a 200 status fairly quickly. -> Set Green to be healthy. -> Wait for the health check to see that Green is healthy. -> Set Blue to be unhealthy. -> Wait for the health check to see that Blue is unhealthy. -> Stop Blue.

    Thanks again for your help,

    Philip R.

‹ Previous Thread|Next Thread ›