Error 500.13 (Server too busy) with many sites on one machine [Answered]RSS

10 replies

Last post Jun 02, 2010 04:29 PM by jn00s

  • Error 500.13 (Server too busy) with many sites on one machine

    Sep 07, 2007 03:33 AM|jn00s|LINK

    <div class="post_message">I am running 3 windows 2003 servers that host a combined total of 290 websites, which are all running the same classic ASP portal application off the same codebase (all websites on each server point to the same physical directory on that server). The problem is that once a server gets past around 70-80 websites running simultaneously, it can't seem to get any new sites running. The first 70-80 sites to get hit by requests will work fine and will continue to run indefinitely. But after a certain point no additional websites will initialize and work correctly. Instead they will perpetually return a 500.13 error. I've seen a couple of mentions on the net of people running into the same problem with lots of sites on a machine, but never any resolution. Any help that could point me in the right direction would be greatly appreciated!</div><div class="post_message">-j</div><div class="post_message"> </div><div class="post_message"> </div>
  • Re: Error 500.13 (Server too busy) with many sites on one machine

    Sep 07, 2007 08:15 AM|jeff@zina.com|LINK

    You may have found your limit for the number of sites per box.  This varies depending on the load on the sites and the hardware, but you will hit this at some point on every system.  More RAM can help, but eventually you need another box.

    Hosters range from a few dozen sites on a box to hundreds, but it depends on loads.  A place like WebHost4Life will run 500+ sites on a box, but they are primarily unused most of the time.

    You might be able to alter some of your app functions to help, but you have less possibilities with ASP Classic than a .NET app.  Changing caching times might help, using a caching proxy or load balancer that caches might help, but you may find it easier and less costly to add another box.  For tuning information, check the IIS 6 Resource Kit, available online in the downloads section.

    Jeff

    Have you Binged a solution before posting?
  • Re: Error 500.13 (Server too busy) with many sites on one machine

    Sep 07, 2007 11:01 AM|jn00s|LINK

     I should mention that this does not appear to have anything to do with actual load on the machines. Most of the sites are not active yet so adding the new ones didn't add any extra load as far as requests or use of resources. Memory and CPU usage are not anywhere near their functional limits, and none of the performance counters I've examined would suggest that the box is overloaded with work to do. In other words, each of these machines could easily handle several times as much traffic, on the first 80 or so sites. And even with zero traffic, no more than 80 sites will function at once.

  • Re: Error 500.13 (Server too busy) with many sites on one machine

    Sep 07, 2007 12:51 PM|KevinKohn|LINK

    Run perfmon on the box and pay attention to ASP counters.  Chances are your ASP Queue max is being reached, thus the 500 server error.  While tweaking ASP request queue max value in the metabase might buy you a bit, I imagine some ASP pages are running very slowly (db access and such) and might be causing your problems.  Generally in this scenario you would not see CPU and Memory issues as the ASP's are stuck in the queue waiting for external dependancies (database for example) to return data.

     In your IIS log's, ensure the time-taken, cs-bytes, and sc-bytes are being logged.  Look at time-taken field to see which ASP's might be causing you problems.
     

  • Re: Error 500.13 (Server too busy) with many sites on one machine

    Mar 04, 2008 12:03 AM|jn00s|LINK

    Hi,
    I am still trying to find the cause and solution to this problem. No matter what I do I cannot host more than 80 sites per web server, regardless of available resources. Performance monitor shows ASP Requests Queued hovering at around 140 with the server fully loaded. Given there are 3000 available queue slots this seems just about right for a busy server. I tried turning on IIS logging for one of the sites that was throwing the 500.13 error and it did not seem to give me any clues but here's a sample anyway:

    #Software: Microsoft Internet Information Services 6.0
    #Version: 1.0
    #Date: 2008-03-04 04:45:33
    #Fields: date time cs-method cs-uri-stem cs-uri-query s-port cs-username sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken 
    2008-03-04 04:45:33 GET /Default.asp |-|ASP_0148|Server_Too_Busy 80 - 500 0 0 1765 459 31
    2008-03-04 04:45:33 GET /Default.asp |-|ASP_0148|Server_Too_Busy 80 - 500 0 0 1765 459 31
    2008-03-04 04:45:34 GET /Default.asp |-|ASP_0148|Server_Too_Busy 80 - 500 0 0 1765 459 15
    2008-03-04 04:45:34 GET /Default.asp |-|ASP_0148|Server_Too_Busy 80 - 500 0 0 1765 459 15
    2008-03-04 04:45:34 GET /Default.asp |-|ASP_0148|Server_Too_Busy 80 - 500 0 0 1765 459 15
    2008-03-04 04:45:35 GET /Default.asp |-|ASP_0148|Server_Too_Busy 80 - 500 0 0 1765 459 31
    2008-03-04 04:45:35 GET /Default.asp |-|ASP_0148|Server_Too_Busy 80 - 500 0 0 1765 459 31
    

    The pages on the working sites don't seem to be hanging up at all, everything is responding very quickly. I am using ASPCache for dataset caching so db access is minimal, my SQL server is nearly idle.

    One other observation I made is that sometimes the non-working sites will start working a few minutes after first being tried. However simply hitting them does not seem to cure the problem. I set up a script to hit all 340 sites in succession every 20 minutes and record how many are up and how many are down. Regardless of how many servers I have or how fast their hardware is, I can get no more than about 80 sites to work on each server. I recently consolidated from 3 weaker servers to one very fast server and the results were quite predictable given the 80 site/server rule. Before I was averaging about 200 sites up (out of 340 total) and now with everything on one server I'm averaging about 60, occasionally peaking in the 80s.


  • Re: Error 500.13 (Server too busy) with many sites on one machine

    Apr 04, 2008 01:24 AM|jn00s|LINK

     Just a bump/update on this problem. I still have found no solution. I tried moving some of my servers to win2000 advanced, which turned out to be far too unstable and constantly hanging the server to the point of needing a power cycle. I tried moving to windows web server 2008 x64 trial, this OS also would eventually hang all the servers until a power cycle. So now I've got them on Win2003 Standard x64 trials and this seems to be the most stable configuration, although it still does not permit me to break 40-80 working sites per server. So far I've bought 7 brand new servers trying to load balance these 340 sites to a point where they are stable. Again I want to emphasize that all of these servers are extremely underutilized. It seems that IIS5/6/7 are all unable to run this simple classic ASP app with stability even with an abundance of resources. I've been unable to find any hardware bottleneck, the limitation seems to be somewhere in the software, possibly some performance tuning feature I'm unaware of that would allow my servers to use their full potential.

  • Re: Error 500.13 (Server too busy) with many sites on one machine

    Apr 04, 2008 02:33 AM|ganeshanekar|LINK

    There are 4 main reasons in IIS 6 why the Server Too Busy error will be displayed
    in response to an ASP request:

    1) The application pool's ASP queue is full (or in other words, the metabase value
    ASPRequestQueueMax has been reached)

    2) A new App Pool cannot be spawned due to OS process limits (or in other words, we
    have reached the maximum number of concurrently running application pools.

    3) An ASP request has been in the ASP queue longer than the value set for the
    metabase key ASPQueueTimeout

    4) We have run out of desktop heap.

    I would take a Hang dump using DebugDiag tool and see what is happening and why the requests are taking long time to execute. Normally this would turn out to be code issue.

    Important : Are you running these sites in Seperate app pools or 1 single app pool? If they are in single pool, then try seperating them in different app pool.

    Another thing is system resources, are you running many products like SQL/Exchange on the same box as IIS and also are they any third party apps, many processes running on this box? If there are many processes running it might cause desktop heap issue. you could use Dheapmonitor and see if there is enough desktop heap available to spwan a new process.

    AFAIK, I have seen more than 50000 sites running on 1 single IIS6.0 box and running Classic ASP and ASP.NET..you are not there yet.

    HTH.

    ~ Ganesh

    Please: Don't forget to click "Mark as Answer" on the post that helped you. That way future readers will know which post solved your issue.
  • Re: Error 500.13 (Server too busy) with many sites on one machine

    Apr 04, 2008 11:44 AM|jn00s|LINK

     Hi Ganesh,

    Thanks for your informative reply. Each site is running in its own application pool. I am not running any other apps or services on my webservers, and they all have an abundance of resources. They are on core2quad q6600s and the cpu never peaks past 5%. They have 4 gigs of ram and never use more than 2.5 gigs. As mentioned above, the Performance Monitor shows that queued ASP requests queued is staying well within its limit. At the moment I have managed to get all 340 sites to run at the same time, but at the expense of spreading them across 6 webservers (plus one more dedicated SQL server box which is also hardly using its CPU or memory.)

    I have tried making dumps using DebugDiag in the past and never found anything that looked unusual to me, always lots of threads waiting to be used. IIS never really hangs, it simply throws a 500.13 immediately.

    However I will give it one more try right now. I will shift 100-200 sites onto one box and run my script which will hit all sites sequentially. At the same time I will run Performance Monitor and DHeapMonitor, as well as try to create a dump using debugdiag, and I'll post the results here.

    Thanks again for your help, I have been fighting with this problem for over a year now.

    -Justin 

     

  • Re: Error 500.13 (Server too busy) with many sites on one machine

    Dec 18, 2008 05:25 PM|jn00s|LINK

    Hey everyone,

    I wanted to post the solution to this problem as it cost me a $300 call to MS tech support to find the answer. The fix is very simple and is applicable to MANY other situations on nearly every recent windows OS. Basically I was running out of non-interactive desktop heap even though I had a vast surplus of memory. The reason for this is that every single version of windows is CRIPPLED by default with very restrictive heap limits which make absolutely no sense even for hardware from 5 or 10 years ago. There are 2 things you can do to fix this. First, you can increase the maximum total heap size using the following steps:

    1.  Start Registry Editor.

    2.  Locate and then click the following registry subkey:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management

    3.  Configure the following entries:

    Name: SessionViewSize
    Data Type: REG_DWORD
    Value data: 120 (decimal)

    4.  Configure the following entries:

    Name: SessionPoolSize
    Data Type: REG_DWORD
    Value data: 60 (decimal)


    That alone will not fix the problem because now you need to increase the limits on the specific sections of the desktop heap, namely the Interactive and Non-Interactive parts:

    1.  Start Registry Editor.

    2.  Locate and then click the following registry subkey:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Subsytems

    3.  Find the value in this key called Windows, it will be a long string of name-value pairs that looks something like this:

     %SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,3072,512 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off MaxRequestThreads=16

    The portion of this section that we're interested in is the "SharedSection=1024,3072,512" part. Those 3 numbers define, in kb, the size of various specific heap limits in this order: global, interactive, non-interactive. You most likely will not ever need to mess with the first one (global).

    The third (non-interactive) value is the one that enabled me to host hundreds/thousands of sites on a single server without getting the 500.13 errors. If you're running more than about 60-80 IIS websites and getting those 500.13's like I was, you need to increase this number. I increased it to 8192 on my server (an 8-core dual xeon with 8 GB RAM) and am now able to run more websites on that one machine than I was able to run on 6 separate windows servers without the adjustment. Needless to say this clever limitation is making MS a lot of $$$ and I am living proof. I bought 6 copies of server 2k3 x64 when I could have done it with 1! In light of this and hardware costs the $300 it cost me to find out was well worth it.

    The second (interactive) value can be increased to allow you to scale your usage of interactive desktop heap, or in other words you can now run way more windows desktop apps simultaneously. I found this to be of particular value on a windows xp machine that I was trying to run many (100+) copies of the same app. After a certain points the instances would become unstable and could not spawn any more processes. I increased the middle value and the limit was lifted.

    After adjusting these numbers reboot your machine and do performance tests to verify that you are now able to run everything without problems. Increasing the individual numbers too much can make you use up too much memory and eventually hit that total heap maximum we adjusted in the first part (which is why we increased it to help prevent this from happening.) I suggest increasing in 2 MB increments until you hit a point where you are no longer hitting those limits, but don't push it much further than you need to. As I mentioned before, I'm having no problems with my numbers set to 1024,10240,8192 and can run over 600 websites (and probably thousands more) without a hitch.

    I find it hard to believe that I've never run across anything on the web that really talks about the significance of these numbers and what a massive artificially-imposed limitation they are on the windows platform. There are a couple of reference pages here and there which talk about adjusting the heap but they carefully avoid talking in detail about the real-world implications. Also I did see in at least one or two places people posting on forums who had the exact same problem/scenario as me but they never got the answer they needed and NOWHERE on the web have I ever been able to find a link between the 500.13 problem (which plagues even 2k8 servers) and the desktop heap adjustments fix to it. Also I haven't found any page at all which really explains,  in a way that most people would find useful, how to go about tweaking these numbers to your particular situation.

    To most home users this is probably no big deal but for we business users who are trying to maximize our performance and fully take advantage of new hardware (especially those doing windows web hosting), this heap issue is massively important. Personally I think MS should have refunded my $300 on grounds that is a (mostly) undocumented bug but I'm sure they have a different "perspective" on the whole issue and I didn't bother trying to argue it. At best I can hope to prevent them from making some more money off other poor suckers who are trying to find the answer to the same problem.

     I should also mention that the MS techs were not especially forthcoming about WHY I was having this problem and what these various tweaks were doing. It was as if they wanted to solve my problem just enough to make me go away without actually letting me understand what was going on. The only reason I know as much as I do is because I badgered them relentlessly for the missing info, the why's and how's. This after I paid them $300 for the support call. And it took them nearly 2 weeks to finally get to a resolution. They did not seem to want to encourage me to explore the upper limits of these numbers and they did not really seem to want to explain the rationale behind how to tweak them. I had to drag it out of them.

    Also I should note that this specific problem will most likely not affect you if you are running IIS in purely x64 mode. I am running classic ASP apps which make use of 32-bit DLLs like ASPUpload so my IIS has to be set to run in a 32 bit compatible mode.

     I hope this helps at least one person out there because I spent literally years trying to find a solution, and $10,000's trying to work around not having a solution. (Buying and hosting lots of servers and windows licenses.)

    As a cross-platform dev who has always been the devils advocate for MS, I am now thoroughly disgusted and alienated by this company and although I'll continue to develop and support my existing applications I am done doing any new development on this platform.

     

  • Re: Error 500.13 (Server too busy) with many sites on one machine

    Jun 01, 2010 04:16 AM|armaita|LINK

    Hello jn00s,

    First of all thank you for this useful help. I applied the everything you said in your message above but i have still problems.
    The thing is my server is an IBM with 8 dual core xeon and 10GB Ram. I did the
    Memory Management settings thats ok. But i could not set the SharedSection keys second and third numbers properly. If i increase them i am starting to get "Service Unavailable" errors on my sites, if I decrease them i get "Web Server is too Busy". And i got insane because i could not solve this since two months now. I have nearly 150 sites in this server.

    Here are the values i tried and the results, i never changed the first value of SharedSection=1024,3072,512 , below are the second and third numver changes.

    SharedSection=1024,3072,1024 : i get server too busy
    3072,1300 : i get service unavailable
    3072,1200 : i get server too busy
    3072,1210 : i get server too busy
    3072,1220 : i get server too busy
    3072,1250 : i get server too busy
    6144,4096 : i get service unavailable
    4096,2048 : i get service unavailable
    3072,2048 : i get service unavailable
    4096,1024 : i get server too busy

    I dont know what should be the ratio between second and third number. But the third number gives different errors between the values 1300 and 1200.

    If anyone can reply me as soon as possible i would appreciate because it is nearly 1,5 month past and i couldnot solve this mess.
    Thanks.

  • Re: Error 500.13 (Server too busy) with many sites on one machine

    Jun 02, 2010 04:29 PM|jn00s|LINK

    Hi armaita,

    First of all, I'm not 100% certain that this matters but just to be on the safe side, I would recommend that you always set your numbers in multiples of 1024.

    I also have been having to mess with these settings again recently, because I again started to get more 500.13 errors as the size of my website applications grows. I tried pushing them as high as 1024,4096,20480 but upon reboot I found that I had an error in the system event log - event ID 243, "A desktop heap allocation failed." So I tried scaling it back to 1024,3072,16384 and was able to reboot my servers and not get the allocation failed error. I also increased the SessionPoolSize and SessionViewSize to 100 and 200 to make more room for the larger heap numbers.

    I noticed that you are increasing your 2nd number, and this may be your problem. The 2nd number is for interactive heap, which is used by desktop applications. On a server you don't need very much of that, so try to get that number as low as possible without causing problems. The third number is the one you want to increase for non-interactive heap, which is used by services like IIS.

    I would suggest that you experiment with the following settings, and please post your results back.

    First, try using the default interactive value of 3072, while scaling the 3rd number upward in increments of 1024:

    1024,3072,1024
    1024,3072,2048
    1024,3072,3072
    1024,3072,4096
    1024,3072,5120
    1024,3072,6144
    1024,3072,7168
    1024,3072,8192
    1024,3072,9216
    1024,3072,10240
    1024,3072,11264
    1024,3072,12288
    1024,3072,13312
    1024,3072,14336
    1024,3072,15360
    1024,3072,16384

    Then, try the same thing but decrease the 2nd number in increments of 1024:

    1024,2048,1024
    1024,2048,2048
    1024,2048,3072
    1024,2048,4096
    1024,2048,5120
    1024,2048,6144
    1024,2048,7168
    1024,2048,8192
    1024,2048,9216
    1024,2048,10240
    1024,2048,11264
    1024,2048,12288
    1024,2048,13312
    1024,2048,14336
    1024,2048,15360
    1024,2048,16384

    ...

    1024,1024,1024
    1024,1024,2048
    1024,1024,3072
    1024,1024,4096
    1024,1024,5120
    1024,1024,6144
    1024,1024,7168
    1024,1024,8192
    1024,1024,9216
    1024,1024,10240
    1024,1024,11264
    1024,1024,12288
    1024,1024,13312
    1024,1024,14336
    1024,1024,15360
    1024,1024,16384

    Of course, you will probably not need to try ALL of those combinations (that would take a lot of reboots) so use your common sense... first scout out the range in increments of 4096, then you can observe the pattern and then you can fine tune down in increments of 2048 and 1024.

    One last thing, what else are you running on your servers besides IIS that might also be using up the heap?