« Previous Next »

Thread: ASP.Net website stops responding at regular intervals in IIS6

Last post 07-02-2009 2:19 PM by Mike426. 12 replies.

Average Rating Rate It (5)

RSS

Page 1 of 1 (13 items)

Sort Posts:

  • 06-24-2009, 11:50 AM

    • azpcc
    • Not Ranked
    • Joined on 09-04-2008, 4:29 PM
    • Posts 8

    ASP.Net website stops responding at regular intervals in IIS6

     Hello, I have an ASP.Net AJAX website that needs to run continuously and it's not.  Originally, the website would randomly throw "ScriptResource.axd" errors, and an associate recommended adjusting the App Pool recycling settings in IIS away from the defaults. My familiarity with IIS is limited, but I managed to spend the past week adjusting settings to recycle the app pool, and no matter what I change it always stops responding (no errors this time) at 22:24 each night.  Here's the troubleshooting and findings I've done so far:

    • The IIS logs show it stopping every night at 22:23-22:24.  No errors, just completely stops.  The app itself reflects the same, because it captures the current minutes on the screen for every refresh, and it always shows a 23 or 24.
    • Event Log shows nothing at or even close to this time, for both Application and System logs.
    • Our network team confirmed that there are no backups or other process-intensive tasks running on that box at that time. 
    I have the following settings set on the app pool properties:
    • Under the Recycling tab, only the “Recycle Worker Processes (in minutes)” item is checked, and set to 60 minutes.  Setting this to 8 hours made no difference.
    • Under the Performance tab, everything is unchecked, and the “Maximum number of worker processes” item is set to 3.
    • Under the Health tab, “Enable Pinging” is checked and set to 30 seconds, “Enable Rapid Fail Protection” is unchecked, and both the Startup and Shutdown time limits are set to 90 seconds.
    • Under the Identity tab, “Predefined” is selected and set to Local System. 
    Likewise, under the actual website settings, I have “Enable HTTP Keep-Alives” checked, bandwidth throttling is not checked, website connections is set to unlimited, “Enable Content Expiration” is not checked, and Isolation mode is not checked (it’s IIS6).  The server it’s running on is my dev server (Server 2003) and I’m the only developer, so I know nobody’s messing with the box at all, ‘cause nobody else can get to it. The goal is for the web app to run continuously.  There's a Timer control built into the app that causes a postback every 10 seconds.  Users will walk past the screen and see current information displayed automatically. 

     What else could be causing this to happen every night at the same time? 

    UPDATE: I apologize, I read the IIS log wrong, it actually dies between 15:23 and 16:18 every day.  There's nothing running at those times that could interfere either, however.

  • 06-26-2009, 5:29 PM In reply to

    • Mike426
    • Top 500 Contributor
    • Joined on 06-26-2009, 4:56 PM
    • Posts 17

    Re: ASP.Net website stops responding at regular intervals in IIS6

    Does the web server on your dev machine become unresponsive at that time (i.e. can it still serve static HTML pages), or does your web page just stop posting back on its timer at that time?  Can you access your web app from another machine on your network and see if it also stops doing the post-backs at the same time?  If it turned out to be a browser issue there wouldn't be anything logged.

    mike aht cognitier doht com
    http://www.cognitier.com
  • 06-29-2009, 10:07 AM In reply to

    • azpcc
    • Not Ranked
    • Joined on 09-04-2008, 4:29 PM
    • Posts 8

    Re: ASP.Net website stops responding at regular intervals in IIS6

    The web site just stops posting back on the timer.  If you refresh the page on that machine, or open the page from another machine, it runs fine.  Nothing gets logged to IIS when this happens, but if it's a browser issue, then why is it that the scenario changes slightly when I tweak the worker pool recycle settings?  Although they don't change much, the changes are visible in the length of time the page stays up.  On the average, it's approximately 8 hours.

  • 06-29-2009, 11:23 AM In reply to

    Re: ASP.Net website stops responding at regular intervals in IIS6

    Hi,

    I would suggest not using a web garden (in other words I would set the maximum number of worker processes to 1) and I would also suggest configuring your application pool to recycle once every 24 hours at an appropriate time.

    You might also want to consider enabling application pool recycle event logging as described here:

    How to Configure Application Pool Recycle Event Logging in IIS 6.0

    Also, is there a reason why you are using the Local System account as your app pool ID ?

    Regards

     

    Paul Lynch | www.iisadmin.co.uk
  • 06-29-2009, 11:30 AM In reply to

    • azpcc
    • Not Ranked
    • Joined on 09-04-2008, 4:29 PM
    • Posts 8

    Re: ASP.Net website stops responding at regular intervals in IIS6

    Ok, I set the max # of worker processes to 1 and set the app pool to recycle every 24 hours during the night.  I'll run a test with those settings and see how it works.

    Why is the local system account the app pool ID?  This is a test server, not production.  I seldom change the IIS settings when deploying a web app to dev, but perhaps I should from now on?  = )

    I'll post the results here in 24 hours.

  • 06-29-2009, 11:51 AM In reply to

    • Mike426
    • Top 500 Contributor
    • Joined on 06-26-2009, 4:56 PM
    • Posts 17

    Re: ASP.Net website stops responding at regular intervals in IIS6

    Another quick test to run while you're waiting:

    1.  Bring up the web site in your browser.  Wait a minute or so and confirm that it's posting back and operating as expected.

    2.  Go into Task Manager and terminate the w3wp.exe process.

    3.  Wait another minute or so and confirm that another w3wp.exe process has started up and that your app continues to post back as expected.

    This would help confirm that your app can keep posting back, even when it's being serviced by a different instance of w3wp.exe.

    mike aht cognitier doht com
    http://www.cognitier.com
  • 06-29-2009, 11:56 AM In reply to

    • azpcc
    • Not Ranked
    • Joined on 09-04-2008, 4:29 PM
    • Posts 8

    Re: ASP.Net website stops responding at regular intervals in IIS6

    I just noticed 4 instances of w3wp.exe running on the server, killed them all, and when the postback was set to fire it created a new instance of w3wp.exe.  When another postback fired, it used the existing w3wp.exe process.  Good call!  I'll continue to keep an eye on it, thanks!

  • 06-30-2009, 10:55 AM In reply to

    • azpcc
    • Not Ranked
    • Joined on 09-04-2008, 4:29 PM
    • Posts 8

    Re: ASP.Net website stops responding at regular intervals in IIS6

    Well, the website remains up and running today after setting the web garden to 1, and scheduling an app recycle at 11:30 PM local time.  I also opened up greater app pool logging as suggested previously (http://www.iisadmin.co.uk/?p=17).  I checked the event log and it shows the scheduled recycle firing on time and no other errors.  I plan to leave it for another 24 hours to ensure stability, but I'll check on it throughout the day today.  So far so good!

  • 07-01-2009, 10:48 AM In reply to

    • azpcc
    • Not Ranked
    • Joined on 09-04-2008, 4:29 PM
    • Posts 8

    Re: ASP.Net website stops responding at regular intervals in IIS6

    Ok, looks like the website failed yesterday around 12:43pm local time.  I checked the Event Log and found the following:

    Event Log entry

    So I looked up the error code (8007006d) and is comes back to "App Pool Crash".  I located many articles with tools to diagnose this, and also came across a great blog by David Wang (MSFT) discussing this at length http://blogs.msdn.com/david.wang/archive/2005/08/29/HOWTO_Understand_and_Diagnose_an_AppPool_Crash.aspx

    So, I started iisstate running on that server against that PID, and I'll just have to wait and see what it tells me.  As soon as it crashes and I have a result I'll post it here.

    Oh yeah, and I changed the App Pool Identity to a service account with local admin permissions, just to see if that changes anything.

  • 07-01-2009, 11:10 AM In reply to

    • Mike426
    • Top 500 Contributor
    • Joined on 06-26-2009, 4:56 PM
    • Posts 17

    Re: ASP.Net website stops responding at regular intervals in IIS6

    If you control the source code for the web app, can you put in some detailed logging so you can determine exactly what it was trying to do right before the crash?  Are you using legacy COM components or other unmanaged components (i.e. anything likely to throw an unhandled exception) in your web app code?

    mike aht cognitier doht com
    http://www.cognitier.com
  • 07-01-2009, 11:14 AM In reply to

    • azpcc
    • Not Ranked
    • Joined on 09-04-2008, 4:29 PM
    • Posts 8

    Re: ASP.Net website stops responding at regular intervals in IIS6

    There's no legacy COM or unmanaged resources, but I can try to add in some detailed tracing and see if that corresponds to the app pool crashing event.  The next time it crashes I'll post those details as well.

  • 07-02-2009, 1:10 PM In reply to

    • azpcc
    • Not Ranked
    • Joined on 09-04-2008, 4:29 PM
    • Posts 8

    Re: ASP.Net website stops responding at regular intervals in IIS6

    Ok, the app pool crashed this morning at 9:07 local time.  IIS State caught the error and reported a "C++ EH Exception code e06d7363".  Event Log had no errors or warnings, or even informations events, even after opening up additional logging using healthMonitoring in the web.config file.  After initial research this points back to very little.  I did find several posts that showed Microsoft recommending to reinstall IIS, and if that doesn't work to rebuild the server.  That's unacceptable, the problem is happening on multiple servers, both actual boxes and VM servers.  Further research pointed to a Microsoft KB article (911816) that recommended modifying the IHTTP Module to report all unhandled exceptions to the Event Log so we can hopefully trap and see the error at the time it occurs.  I made the changes locally and will test this over the long weekend.  I'll report back on Monday on what happened.

  • 07-02-2009, 2:19 PM In reply to

    • Mike426
    • Top 500 Contributor
    • Joined on 06-26-2009, 4:56 PM
    • Posts 17

    Re: ASP.Net website stops responding at regular intervals in IIS6

    Our company's approach is to take the problem code outside of the w3wp.exe process in situations like these (i.e. so process crashes don't affect IIS).  Please send me an email if you'd like to get more information on this.

    mike aht cognitier doht com
    http://www.cognitier.com
Page 1 of 1 (13 items)
Microsoft Communities