« Previous Next »

• 09-07-2007, 3:33 AM

  jn00s Joined on 09-07-2007, 3:11 AM Posts 8

  Error 500.13 (Server too busy) with many sites on one machine Reply Contact I am running 3 windows 2003 servers that host a combined total of 290 websites, which are all running the same classic ASP portal application off the same codebase (all websites on each server point to the same physical directory on that server). The problem is that once a server gets past around 70-80 websites running simultaneously, it can't seem to get any new sites running. The first 70-80 sites to get hit by requests will work fine and will continue to run indefinitely. But after a certain point no additional websites will initialize and work correctly. Instead they will perpetually return a 500.13 error. I've seen a couple of mentions on the net of people running into the same problem with lots of sites on a machine, but never any resolution. Any help that could point me in the right direction would be greatly appreciated! -j

• 09-07-2007, 8:15 AM

  In reply to

  jeff@zina.com Joined on 09-26-2003, 10:43 AM Naples, FL, USA Posts 2,216

  Re: Error 500.13 (Server too busy) with many sites on one machine Reply Contact You may have found your limit for the number of sites per box.  This varies depending on the load on the sites and the hardware, but you will hit this at some point on every system.  More RAM can help, but eventually you need another box. Hosters range from a few dozen sites on a box to hundreds, but it depends on loads.  A place like WebHost4Life will run 500+ sites on a box, but they are primarily unused most of the time. You might be able to alter some of your app functions to help, but you have less possibilities with ASP Classic than a .NET app.  Changing caching times might help, using a caching proxy or load balancer that caches might help, but you may find it easier and less costly to add another box.  For tuning information, check the IIS 6 Resource Kit, available online in the downloads section. Jeff Look for Wrox's new book Professional IIS 7 in your local bookstore, or order now at Amazon.com

• 09-07-2007, 11:01 AM

  In reply to

  jn00s Joined on 09-07-2007, 3:11 AM Posts 8

  Re: Error 500.13 (Server too busy) with many sites on one machine Reply Contact  I should mention that this does not appear to have anything to do with actual load on the machines. Most of the sites are not active yet so adding the new ones didn't add any extra load as far as requests or use of resources. Memory and CPU usage are not anywhere near their functional limits, and none of the performance counters I've examined would suggest that the box is overloaded with work to do. In other words, each of these machines could easily handle several times as much traffic, on the first 80 or so sites. And even with zero traffic, no more than 80 sites will function at once.

• 09-07-2007, 12:51 PM

  In reply to

  KevinKohn Joined on 07-03-2007, 10:58 AM Columbus, OH Posts 20

  Re: Error 500.13 (Server too busy) with many sites on one machine Reply Contact Run perfmon on the box and pay attention to ASP counters.  Chances are your ASP Queue max is being reached, thus the 500 server error.  While tweaking ASP request queue max value in the metabase might buy you a bit, I imagine some ASP pages are running very slowly (db access and such) and might be causing your problems.  Generally in this scenario you would not see CPU and Memory issues as the ASP's are stuck in the queue waiting for external dependancies (database for example) to return data.  In your IIS log's, ensure the time-taken, cs-bytes, and sc-bytes are being logged.  Look at time-taken field to see which ASP's might be causing you problems.   There are only 10 types of people in the world: Those who understand binary, and those who don't

• 03-04-2008, 12:03 AM

  In reply to

  jn00s Joined on 09-07-2007, 3:11 AM Posts 8

  Re: Error 500.13 (Server too busy) with many sites on one machine Reply Contact Hi, I am still trying to find the cause and solution to this problem. No matter what I do I cannot host more than 80 sites per web server, regardless of available resources. Performance monitor shows ASP Requests Queued hovering at around 140 with the server fully loaded. Given there are 3000 available queue slots this seems just about right for a busy server. I tried turning on IIS logging for one of the sites that was throwing the 500.13 error and it did not seem to give me any clues but here's a sample anyway: #Software: Microsoft Internet Information Services 6.0 #Version: 1.0 #Date: 2008-03-04 04:45:33 #Fields: date time cs-method cs-uri-stem cs-uri-query s-port cs-username sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken 2008-03-04 04:45:33 GET /Default.asp |-|ASP_0148|Server_Too_Busy 80 - 500 0 0 1765 459 31 2008-03-04 04:45:33 GET /Default.asp |-|ASP_0148|Server_Too_Busy 80 - 500 0 0 1765 459 31 2008-03-04 04:45:34 GET /Default.asp |-|ASP_0148|Server_Too_Busy 80 - 500 0 0 1765 459 15 2008-03-04 04:45:34 GET /Default.asp |-|ASP_0148|Server_Too_Busy 80 - 500 0 0 1765 459 15 2008-03-04 04:45:34 GET /Default.asp |-|ASP_0148|Server_Too_Busy 80 - 500 0 0 1765 459 15 2008-03-04 04:45:35 GET /Default.asp |-|ASP_0148|Server_Too_Busy 80 - 500 0 0 1765 459 31 2008-03-04 04:45:35 GET /Default.asp |-|ASP_0148|Server_Too_Busy 80 - 500 0 0 1765 459 31 The pages on the working sites don't seem to be hanging up at all, everything is responding very quickly. I am using ASPCache for dataset caching so db access is minimal, my SQL server is nearly idle. One other observation I made is that sometimes the non-working sites will start working a few minutes after first being tried. However simply hitting them does not seem to cure the problem. I set up a script to hit all 340 sites in succession every 20 minutes and record how many are up and how many are down. Regardless of how many servers I have or how fast their hardware is, I can get no more than about 80 sites to work on each server. I recently consolidated from 3 weaker servers to one very fast server and the results were quite predictable given the 80 site/server rule. Before I was averaging about 200 sites up (out of 340 total) and now with everything on one server I'm averaging about 60, occasionally peaking in the 80s.

• 04-04-2008, 1:24 AM

  In reply to

  jn00s Joined on 09-07-2007, 3:11 AM Posts 8

  Re: Error 500.13 (Server too busy) with many sites on one machine Reply Contact  Just a bump/update on this problem. I still have found no solution. I tried moving some of my servers to win2000 advanced, which turned out to be far too unstable and constantly hanging the server to the point of needing a power cycle. I tried moving to windows web server 2008 x64 trial, this OS also would eventually hang all the servers until a power cycle. So now I've got them on Win2003 Standard x64 trials and this seems to be the most stable configuration, although it still does not permit me to break 40-80 working sites per server. So far I've bought 7 brand new servers trying to load balance these 340 sites to a point where they are stable. Again I want to emphasize that all of these servers are extremely underutilized. It seems that IIS5/6/7 are all unable to run this simple classic ASP app with stability even with an abundance of resources. I've been unable to find any hardware bottleneck, the limitation seems to be somewhere in the software, possibly some performance tuning feature I'm unaware of that would allow my servers to use their full potential.

• 04-04-2008, 2:33 AM

  In reply to

  ganeshanekar Joined on 11-15-2006, 12:29 AM India Posts 532

  Re: Error 500.13 (Server too busy) with many sites on one machine Reply Contact There are 4 main reasons in IIS 6 why the Server Too Busy error will be displayed in response to an ASP request: 1) The application pool's ASP queue is full (or in other words, the metabase value ASPRequestQueueMax has been reached) 2) A new App Pool cannot be spawned due to OS process limits (or in other words, we have reached the maximum number of concurrently running application pools. 3) An ASP request has been in the ASP queue longer than the value set for the metabase key ASPQueueTimeout 4) We have run out of desktop heap. I would take a Hang dump using DebugDiag tool and see what is happening and why the requests are taking long time to execute. Normally this would turn out to be code issue. Important : Are you running these sites in Seperate app pools or 1 single app pool? If they are in single pool, then try seperating them in different app pool. Another thing is system resources, are you running many products like SQL/Exchange on the same box as IIS and also are they any third party apps, many processes running on this box? If there are many processes running it might cause desktop heap issue. you could use Dheapmonitor and see if there is enough desktop heap available to spwan a new process. AFAIK, I have seen more than 50000 sites running on 1 single IIS6.0 box and running Classic ASP and ASP.NET..you are not there yet. HTH. ~ Ganesh

• 04-04-2008, 11:44 AM

  In reply to

  jn00s Joined on 09-07-2007, 3:11 AM Posts 8

  Re: Error 500.13 (Server too busy) with many sites on one machine Reply Contact  Hi Ganesh, Thanks for your informative reply. Each site is running in its own application pool. I am not running any other apps or services on my webservers, and they all have an abundance of resources. They are on core2quad q6600s and the cpu never peaks past 5%. They have 4 gigs of ram and never use more than 2.5 gigs. As mentioned above, the Performance Monitor shows that queued ASP requests queued is staying well within its limit. At the moment I have managed to get all 340 sites to run at the same time, but at the expense of spreading them across 6 webservers (plus one more dedicated SQL server box which is also hardly using its CPU or memory.) I have tried making dumps using DebugDiag in the past and never found anything that looked unusual to me, always lots of threads waiting to be used. IIS never really hangs, it simply throws a 500.13 immediately. However I will give it one more try right now. I will shift 100-200 sites onto one box and run my script which will hit all sites sequentially. At the same time I will run Performance Monitor and DHeapMonitor, as well as try to create a dump using debugdiag, and I'll post the results here. Thanks again for your help, I have been fighting with this problem for over a year now. -Justin

• 12-18-2008, 5:25 PM

  In reply to

  jn00s Joined on 09-07-2007, 3:11 AM Posts 8

  Re: Error 500.13 (Server too busy) with many sites on one machine Reply Contact Hey everyone, I wanted to post the solution to this problem as it cost me a $300 call to MS tech support to find the answer. The fix is very simple and is applicable to MANY other situations on nearly every recent windows OS. Basically I was running out of non-interactive desktop heap even though I had a vast surplus of memory. The reason for this is that every single version of windows is CRIPPLED by default with very restrictive heap limits which make absolutely no sense even for hardware from 5 or 10 years ago. There are 2 things you can do to fix this. First, you can increase the maximum total heap size using the following steps: 1.  Start Registry Editor. 2.  Locate and then click the following registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management 3.  Configure the following entries: Name: SessionViewSize Data Type: REG_DWORD Value data: 120 (decimal) 4.  Configure the following entries: Name: SessionPoolSize Data Type: REG_DWORD Value data: 60 (decimal) That alone will not fix the problem because now you need to increase the limits on the specific sections of the desktop heap, namely the Interactive and Non-Interactive parts: 1.  Start Registry Editor. 2.  Locate and then click the following registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Subsytems 3.  Find the value in this key called Windows, it will be a long string of name-value pairs that looks something like this:  %SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,3072,512 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off MaxRequestThreads=16 The portion of this section that we're interested in is the "SharedSection=1024,3072,512" part. Those 3 numbers define, in kb, the size of various specific heap limits in this order: global, interactive, non-interactive. You most likely will not ever need to mess with the first one (global). The third (non-interactive) value is the one that enabled me to host hundreds/thousands of sites on a single server without getting the 500.13 errors. If you're running more than about 60-80 IIS websites and getting those 500.13's like I was, you need to increase this number. I increased it to 8192 on my server (an 8-core dual xeon with 8 GB RAM) and am now able to run more websites on that one machine than I was able to run on 6 separate windows servers without the adjustment. Needless to say this clever limitation is making MS a lot of $$$ and I am living proof. I bought 6 copies of server 2k3 x64 when I could have done it with 1! In light of this and hardware costs the $300 it cost me to find out was well worth it. The second (interactive) value can be increased to allow you to scale your usage of interactive desktop heap, or in other words you can now run way more windows desktop apps simultaneously. I found this to be of particular value on a windows xp machine that I was trying to run many (100+) copies of the same app. After a certain points the instances would become unstable and could not spawn any more processes. I increased the middle value and the limit was lifted. After adjusting these numbers reboot your machine and do performance tests to verify that you are now able to run everything without problems. Increasing the individual numbers too much can make you use up too much memory and eventually hit that total heap maximum we adjusted in the first part (which is why we increased it to help prevent this from happening.) I suggest increasing in 2 MB increments until you hit a point where you are no longer hitting those limits, but don't push it much further than you need to. As I mentioned before, I'm having no problems with my numbers set to 1024,10240,8192 and can run over 600 websites (and probably thousands more) without a hitch. I find it hard to believe that I've never run across anything on the web that really talks about the significance of these numbers and what a massive artificially-imposed limitation they are on the windows platform. There are a couple of reference pages here and there which talk about adjusting the heap but they carefully avoid talking in detail about the real-world implications. Also I did see in at least one or two places people posting on forums who had the exact same problem/scenario as me but they never got the answer they needed and NOWHERE on the web have I ever been able to find a link between the 500.13 problem (which plagues even 2k8 servers) and the desktop heap adjustments fix to it. Also I haven't found any page at all which really explains,  in a way that most people would find useful, how to go about tweaking these numbers to your particular situation. To most home users this is probably no big deal but for we business users who are trying to maximize our performance and fully take advantage of new hardware (especially those doing windows web hosting), this heap issue is massively important. Personally I think MS should have refunded my $300 on grounds that is a (mostly) undocumented bug but I'm sure they have a different "perspective" on the whole issue and I didn't bother trying to argue it. At best I can hope to prevent them from making some more money off other poor suckers who are trying to find the answer to the same problem.  I should also mention that the MS techs were not especially forthcoming about WHY I was having this problem and what these various tweaks were doing. It was as if they wanted to solve my problem just enough to make me go away without actually letting me understand what was going on. The only reason I know as much as I do is because I badgered them relentlessly for the missing info, the why's and how's. This after I paid them $300 for the support call. And it took them nearly 2 weeks to finally get to a resolution. They did not seem to want to encourage me to explore the upper limits of these numbers and they did not really seem to want to explain the rationale behind how to tweak them. I had to drag it out of them. Also I should note that this specific problem will most likely not affect you if you are running IIS in purely x64 mode. I am running classic ASP apps which make use of 32-bit DLLs like ASPUpload so my IIS has to be set to run in a 32 bit compatible mode.  I hope this helps at least one person out there because I spent literally years trying to find a solution, and $10,000's trying to work around not having a solution. (Buying and hosting lots of servers and windows licenses.) As a cross-platform dev who has always been the devils advocate for MS, I am now thoroughly disgusted and alienated by this company and although I'll continue to develop and support my existing applications I am done doing any new development on this platform.