« Previous Next »

Answered Thread: IIS 7 http status 400 errors and traffic drop off

Last post 04-29-2010 5:31 AM by CevetS. 6 replies.

 

RSS

Page 1 of 1 (7 items)

Sort Posts:

  • 11-20-2009, 5:17 PM

    • CevetS
    • Not Ranked
    • Joined on 11-20-2009, 3:31 PM
    • UK
    • Posts 4

    IIS 7 http status 400 errors and traffic drop off


    I am running windows 2008 x64 with IIS7 and .net 2.0 powered website. Default configurations are on most items. The web site is hosted on 4 identical servers operating as web farm behind a hardware loadbalancer. A firewall sites directly behind the internet router restricting access to port 80 http requests.

    The primary user channel website is encountering a problem where the IIS responses to client requests just grinds to a halt.
    User requests for both active and static content seem to be equally effected.

    The problem lasts for a short while and has repeated 5 times in the last 2 months and seems to affect all servers at once. During the event the occurences of http status 400 errors from the httpsys log go up dramatically whilst bandwidth consumption falls. Traffic recorded in the iis logs reflects the drop in "successfull" traffic with a higher than normal level of http status 400 events being logged.

    End users experience incredibly slow response times in their browsers and occasional timeouts, though occasionally some traffic was successfull was noted in the iis logs.

    Extensive network diagnostics have been undertaken and Wireshark has also been logging network traffic during the last couple of events. It showed connections were being handled successfully at the stack level but a high number of RST were being sent out from the web server.  The network window size reduces to a small value seemingly indicating the webservers are in distress.

    The effect noted on network traffic was as if we had just gone down to running the site on a 9600Kbps dial up modem line.
    A netstat on the servers revealed nearly all connections were in state TIME_WAIT.
    Connecting to the TS console of the servers indicates the management network connections are unaffected and when the site is surfed from the console to localhost it responds without a problem, however (I think significantly) if i try and access the site on its [local] front end address I get a very poor response; or if i try and surf the site from one server to another using the front end addresses I get the same poor operation as the external users.

    The whole problem seems to be as if there is a bottleneck appearing in the stack through to http.sys and iis which is causing new connections being operated with small window sizes.

    I would drop a ferb on but the traffic is just not coming through, I have carried out quite a lot of diagnostics so far but frankly I am stumped.

    Any suggestions would be greatly appreciated.

    Thanks

  • 11-21-2009, 10:23 AM In reply to

    Re: IIS 7 http status 400 errors and traffic drop off

    This could be related to TCP Chimney and related 'networking enhancements' in w2k8.

    http://support.microsoft.com/kb/951037

    Run netsh int tcp show global from a command prompt.  Please post the results.  I've seen this issue with DELL and HP servers.

    Steve Schofield
    Windows Server MVP - IIS
    http://weblogs.asp.net/steveschofield

    http://www.IISLogs.com
    Log archival solution
    Install, Configure, Forget
  • 11-21-2009, 3:09 PM In reply to

    • CevetS
    • Not Ranked
    • Joined on 11-20-2009, 3:31 PM
    • UK
    • Posts 4

    Re: IIS 7 http status 400 errors and traffic drop off

    Hi,

    We are running on HP BL460c 2x x5260(3.33ghz) 6gb ram with onboard nics plus 1x nc360m 1GbE 

    The current settings are:

    Querying active state...
    TCP Global Parameters
    ----------------------------------------------
    Receive-Side Scaling State          : enabled
    Chimney Offload State               : disabled
    Receive Window Auto-Tuning Level    : normal
    Add-On Congestion Control Provider  : ctcp
    ECN Capability                      : disabled
    RFC 1323 Timestamps                 : disabled

    Thanks 

     

  • 11-21-2009, 8:11 PM In reply to

    Answered Re: IIS 7 http status 400 errors and traffic drop off

    I would disable RSS too + update all firmware for NIC drivers.  What is your switch port speed set too?  Is it Auto?  I would try setting the duplex speed to match so the NIC doesn't have to auto-negotiate.

    Edit - I would be remise to say have you looked in the c:\windows\system32\logfiles\httperr logs?

    Steve Schofield
    Windows Server MVP - IIS
    http://weblogs.asp.net/steveschofield

    http://www.IISLogs.com
    Log archival solution
    Install, Configure, Forget
  • 11-22-2009, 8:15 AM In reply to

    • CevetS
    • Not Ranked
    • Joined on 11-20-2009, 3:31 PM
    • UK
    • Posts 4

    Re: IIS 7 http status 400 errors and traffic drop off

    I had examined the httperr logs and they supported the weird general major network / nic slow down taking place but didn't at the time give me any new clues. It shows lots of :

    Connection_Dropped for s-queuename <main_app_pool_name>

    http 400 Timer_EntityBody for s-queuename <main_app_pool_name>

    and loads of Timer_MinBytesPerSecond  for the duration of the event. The nics are set to full duplex and 1000gb (non auto negotiate)

    I now have call out with the hardware vendor currently based on your suggestion of having seen this before.  Very quickly they have come back with a pretty complex procedure for removing the old drivers and installing some updated (newer) ones, and disabling TCP offload and RSS. I suspect they may have seen this problem before.

    I think you suggestions above were spot on the money, many many thanks for your help 

     

  • 04-28-2010, 9:13 PM In reply to

    Re: IIS 7 http status 400 errors and traffic drop off

     Hey Steve,

     Did the new vendor drivers work? I believe I'm experiencing this same issue. And I've been stumped for 4 weeks! I'm really excited to have found this post.

  • 04-29-2010, 5:31 AM In reply to

    • CevetS
    • Not Ranked
    • Joined on 11-20-2009, 3:31 PM
    • UK
    • Posts 4

    Re: IIS 7 http status 400 errors and traffic drop off

    Hi,

    Ensuring your drivers are up to date is important, but in our case updating them made no difference.

    FREB was usefull in trapping error events but for us in the end we put some network sniffing tools in place and managed to catch one of the events in the logs. What we noticed was the number of active connections went through the roof...... using nestat -an also showed similar numbers once we knew what to look for.

    Going through the process pipeline from NIC to ASP.Net worker we suddenly hit on a significant difference between our old implementation (IIS5) and the new one (IIS7). In the old models the .Net is offloaded whilst static content is handled in line; in the new intergrated model all is handled in the same pipeline by default. We had effectively reduced our processing pipelines from 2 to 1! 

    Once we realised this a whole bunch of stuff suddenly came under scrutiny. What we did in the end was:

    ( note we don't have keepalives turned)

    1) created seperate application pools for active and static content.

    2) increased the maximum concurrect connections from the default of 12 to 5000, depending on your version/patch level this will either require a reg key for windows 2008sp1 without MS09-36 or an entry in the aspnet.config file. reg=Windows Registry Editor Version 5.00

    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ASP.NET\2.0.50727.0]
    "maxConcurrentRequestsPerCPU"=dword:00001388

    or  config file

    <?xml version="1.0" encoding="utf-8"?><configuration>          <system.web>          <applicationPool maxConcurrentRequestsPerCPU="5000" maxConcurrentThreadsPerCPU="0" requestQueueLimit="5000"/>          </system.web></configuration>

    3) For our application we also implemented our old machine.configs,  IIS7 config viewer editor under system.web/processModel shows the running values.

    <system.web>

    <processModel autoConfig = "false" maxWorkerThreads = "150" maxIoThreads = "150"/>

     

    4) we are also planning on lowering the ConnectionTimeout in order to force closure should transfers take too long a time to complete. In IIS the default is a 2 mins, however if the value is set too low, say 10 secs, devices on slow connections ( like mobiles) will suffer. Note: we have not done this yet.....

     

    I hope this helps.

     

    Regards

     

     

Page 1 of 1 (7 items)