ARR and 502 errorsRSS

14 replies

Last post Aug 08, 2011 03:39 PM by MaxASPSteve

  • ARR and 502 errors

    Mar 21, 2010 03:28 PM|MaxASPSteve|LINK

    I am running an IIS 7.5 webfarm using ARR to spread requests.  I have enabled ARR caching (both local and secondary), and I have noticed a strange phenomenon occurring frequently. While troubleshooting a problem for a customer, I enabled FRT on the default website on all ARR servers. I noticed that there were a high number (sometimes as high as 10 per hour) of 502 errors like this:

      -MODULE_SET_RESPONSE_ERROR_STATUS 

    <div style="display: block;" id="section_detail_281_details" class="hidden">
    ModuleName ApplicationRequestRouting
    Notification 16
    HttpStatus 502
    HttpReason Bad Gateway
    HttpSubStatus 4
    ErrorCode 0
    ConfigExceptionInfo
    Notification MAP_REQUEST_HANDLER
    ErrorCode The operation completed successfully. (0x0)
    </div>

     -SET_RESPONSE_ERROR_DESCRIPTION 

    <div style="display: block;" id="section_detail_282_details" class="hidden">
    ErrorDescription No appropriate server could be found to route the request.
    </div><div style="display: block;" id="section_detail_282_details" class="hidden"> </div><div style="display: block;" id="section_detail_282_details" class="hidden">I have seen this occur most often on requests for images (.gif's, .png's,  and .jpeg's mainly), but I have also seen it happen on requests for css and js files as well. Also, there are no event entries indicating that any of the application servers were marked as unhealthy, and in the FRT logs there are ARR DISK CACHE GET FAILED errors immediately preceding the 502 for both the local cache and secondary (which makes sense because this site has a never cache rule set up, but I wonder why ARR even looks to the cache when these rules are enabled). If I pull the request URL from the logs and hit the link in a browser the image always displays correctly. I am wondering if anyone else has seen this behavior, this is a production web farm so I need to figure this out pretty quick. Any help is appreciated.
    </div>
  • Re: ARR and 502 errors

    Mar 21, 2010 04:45 PM|steve schofield|LINK

    Just a thought is TCP Chimney Offload disabled? 

    http://support.microsoft.com/kb/951037

    Steve Schofield
    Windows Server MVP - IIS
    http://iislogs.com/steveschofield
    http://www.IISLogs.com
    Log archival solution
    Install, Configure, Forget

  • Re: ARR and 502 errors

    Mar 22, 2010 12:18 PM|MaxASPSteve|LINK

     RSS is set to enabled, Chimney offload state is automatic.

     When I do a netstat-t it shows the offload state of all connections as In Host, so I am not sure if TCP Chimney Offloading is the problem.

  • Re: ARR and 502 errors

    Mar 22, 2010 12:30 PM|steve schofield|LINK

    are your NICs broadcom by chance?  If so I'd suggest disabling the whole SNP settings.  It's a worthless setting IMHO.

    Steve Schofield
    Windows Server MVP - IIS
    http://iislogs.com/steveschofield
    http://www.IISLogs.com
    Log archival solution
    Install, Configure, Forget

  • Re: ARR and 502 errors

    Mar 22, 2010 02:03 PM|MaxASPSteve|LINK

     no they are Intel Pro/1000 MT's.

  • Re: ARR and 502 errors

    Apr 23, 2010 05:27 AM|harrisanderson|LINK

    Hi Gentleman!!

    I wanna state that i have  a problem that is related to the subject  if you wanna help me ,then you may

    Kindly response me directly through e-mail.Problem is states as under

    Proxy Error
    The proxy server received an invalid response from an upstream server.
    The proxy server could not handle the request GET /readTopic.do.
    Reason: Error reading from remote server

  • Re: ARR and 502 errors

    May 31, 2010 02:56 PM|CedricBoivin|LINK

    I got exactly the same problem. If i disable the cache everyting working fine, if i enable it i get this error message. ARR_DISK_CACHE_GET_FAILED FilePath \\?\c:\ArrCache\www.ducksandgifts.com\Client\_Altitude_Formulaire\Save_usagers.asp%3fredirecturl=%2fclient%2fRetailer.asp&retailer=0&Prenom=Cédric&Nom=Boivin&Entreprise=Nmedia&Telephone=&Fax=&Email=cedricboivin@gmail.com&adresse=&Ville=Drummondville&Region=225&Pays=45&codepostal=&textarea=&UserName=testced&Password=test&Cpassword=test&Submit=Complete+registration&hidGroupe=.full ErrorCode 2147942403 ErrorCode The system cannot find the path specified. (0x80070003)
  • Re: ARR and 502 errors

    Oct 28, 2010 04:45 AM|gleysen|LINK

    Anyone found an answer for this?? I kinda having the same type of problem here.. ARR proxy is continuessly checking application servers as unhealthy while no errors are logged in the windows application logs on the application servers.. a minuter after the unhealthy setting, the server is set back to healthy. Any idea on what could be causing this?
  • Re: ARR and 502 errors

    Oct 28, 2010 04:21 PM|MaxASPSteve|LINK

    If you look in the Application Event Viewer on the ARR machine, look for Event ID 1000, Source = Application Request Routing. What do the details say? (should give you an indication why the servers are marked unhealthy)
  • Re: ARR and 502 errors

    Oct 29, 2010 04:17 AM|gleysen|LINK

    It says "unknown health check failure" ;-(
  • Re: ARR and 502 errors

    Jun 23, 2011 07:17 AM|adigraham|LINK

    Hi,

    Did you ever get this fixed please?

    Cheers

    Adi

  • Re: ARR and 502 errors

    Jun 23, 2011 11:27 AM|Rinzler|LINK

     I have the same issue, I receive intermittent 502 errors. I have been unable to fix this issue.

  • Re: ARR and 502 errors

    Jun 23, 2011 12:20 PM|MaxASPSteve|LINK

     In my case a hotfix was released that addressed my issue, something to do with the External Cache (I use multiple ARR servers). What you need to do is distinguish between 502.3 and 502.4...502.4 was the problem I had, I still see 502.3's, however, 99% of the time they have been related to a problem on the application server.

  • Re: ARR and 502 errors

    Aug 08, 2011 12:26 PM|Rinzler|LINK

     What hotfix? And how did you set up your Logs to catch just 502.3 and 502.4 ?

     

    Thanks!

  • Re: ARR and 502 errors

    Aug 08, 2011 03:39 PM|MaxASPSteve|LINK

    Here are links to download the hotfix:

    X86: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=392f4441-814e-4bb9-ad37-79fe525918f5

    X64: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=0451b8dd-711d-4b3d-9d83-bba033385b06

    As for how to parse your W3C logs, I used logparser (http://www.microsoft.com/download/en/details.aspx?id=24659) and the query below:


    SELECT
    Date,
    STRCAT(TO_STRING(sc-status), STRCAT('.', TO_STRING(sc-substatus))) As Status,
    COUNT(*) AS Hits
    FROM M:\Logs\LogFiles\W3SVC1\JAN2011\*.log
    WHERE (sc-status = 502 AND sc-substatus = 4)
    GROUP BY Date, Status
    ORDER BY Date ASC

    It is also fair to point out that my problem was specific to the use of the External Cache feature (when using multiple ARR servers), and the hotfix above addresses that specific issue. I am not sure (although I would imagine it is) if this update is rolled up in the ARR v 2.5 release or not (so if you are using that version this may be a moot point). Happy hunting anyway!