« Previous Next »

• 09-09-2008, 1:09 AM

  lkevinl Joined on 09-09-2008, 4:27 AM Posts 10

  IIS 6.0 ASP application bogs down despite free CPU & free memory Reply Contact OK, here is the configuration: Quad Core Xeon, 4GB RAM, 500GB sata running Win2K3 R2 / IIS 6.0.  This server is also running a mail server and SQL Server 2000 is running as the backend for the ASP app. During a high traffic time I started to notice the browser requests were being serviced very slowly.  I did some investigating and here is what I found: 3000+ concurrent users, well, 3000+ session were active, the server has a 20 min timeout. IIS had about 250MB of memory, SQL Server had about 1.7GB of memory. The CPU(s) never got over 50%.  Yet with plenty of free memory and CPU cycles, the web requests were not being handled quickly.  I did a recycle on IIS which booted all sessions but shouldn't affect SQL Server and suddenly the web requests started being handled quickly again, so this defiitely looks IIS related.  I had upgraded from a P4 server which was pretty robust but would simply max out on CPU during these high traffic times, so now I have a more powerful server but seem to be getting limited elsewhere. I was unfamiliar with the Web Garden feature in IIS 6.0, read about it and it sounded great, so I made the mistake of adding some worker processes to my classic ASP app.  Bad idea!  User sessions started getting dropped because the processes don't share session data!  I quickly went back to a single worker process. The app consists of a single index.asp file with many #include files which are conditionalized based on a page querystring variable, so I assume IIS sees this as one HUGE file.  I figured that this may not be so bad because IIS could cache this one large file which is really the only page being accessed.  I hope my assumption is true.  Should I be using different ASP File cache option settings?  Currenlty I have Cache limited APS files in memory set to 500 and Cache limited ASP files on disk set to the 2000.  Number of script engines cahced is 250.  These OK? This app uses it's own app pool and I have since adjusted my Request queue limit from 1000 to 8000 thinking that maybe this limit was bottlenecking this more powerful processor.  I don't want to do too much recycling because with 3000 users, bumping them will simply cause a rush as they try to log back in. I figured that with the extra resources this server has, maybe if I find a way to spread these requests across a web garden that I'd make better use of resources.  Since I can't use a web garden, I got the idea of creating subdomains for the same website www, www2, www3, etc. all pointing to the same app using the same SQL Server database.  Now each would get it's own w3wp.exe process like a web garden.  I'd simply round robin distribute my visitors across these subdomains.  Two problems with this though, 1. I have a secure page which has to be on a specific subdomain and if I transfer users from another subdomain to the secure page, then their session will be lost.  2. A minor issue but my web logs will now be spread across 4 subdomains... unless there is a way to combine.  Is this idea a good way to go? Sorry for the long post!!!  Hopefully someone will have some insight and someone else will be helped by this information! Thanks...

• 09-09-2008, 2:21 AM

  In reply to

  mukhtard Joined on 07-07-2008, 9:13 PM Redmond, WA Posts 202	Re: IIS 6.0 ASP application bogs down despite free CPU & free memory Reply Contact What is the bitness of the OS? How about disk usage and network utilization during the high traffic time? Thanks, Mukhtar Desai IIS Performance Team

• 09-09-2008, 4:09 AM

  In reply to

  lkevinl Joined on 09-09-2008, 4:27 AM Posts 10

  Re: IIS 6.0 ASP application bogs down despite free CPU & free memory Reply Contact OS is 32 bit but it's a quad core xeon and motherboard supports visibility of full 4GB of RAM.  I checked disk utilization and didn't look like it would be limiting.  Network bandwidth was very high but I my hosting provider provides good bandwidth and the server isn't throttled in anyway.  Also, as I mentioned, when I restarted IIS, requests started happening quickly again, so it did seem related to the number of session or the number of requests overtime causing lack of resources for the worker process.

• 09-09-2008, 4:52 AM

  In reply to

  qbernard Joined on 03-25-2003, 10:12 PM Malaysia Posts 3,229

  Re: IIS 6.0 ASP application bogs down despite free CPU & free memory Reply Contact Looks like hardware bottlenecks to me, can you do some performance analysis to look at ASP - request executing, in queue, etc. NLB is a good choice for you, say move out the SQL, then have 2 nodes NLB of IIS infront, with sticky session, no need to worry about the client sesion running around the two nodes, it will stick until the session end / expired. Cheers, Bernard Cheah

• 09-09-2008, 8:26 AM

  In reply to

  tomkmvp Joined on 03-20-2003, 10:27 AM Central NJ Posts 6,238

  Re: IIS 6.0 ASP application bogs down despite free CPU & free memory Reply Contact What does the application do?  Are you following good ASP coding practices (i.e. not putting db objects into session vars)?  Any chance you could offload SQl to another box, at least for testing? Tom Kaminski IIS MVP http://mvp.support.microsoft.com/ http://blogs.iis.net/tomkmvp/ http://www.linkedin.com/pub/8/690/408 http://www.bluesrepublic.org/

• 09-09-2008, 9:39 AM

  In reply to

  Rovastar Joined on 03-13-2008, 10:00 AM London, UK Posts 1,797

  Re: IIS 6.0 ASP application bogs down despite free CPU & free memory Reply Contact First off don't use web gardens. 99.99% of the time there is no need for them. Really they are only good for a short term fix for certain coding issues. Is it just slow? How slow is 'slow'? The memory consumption of SQL looks high what is it doing there?  Do a trace on your SQL server box and see if there is a bottleneck there. Run performance stats on the SQL server look out for locks, latches, etc. Compare this to your baseline stats for when the server was last know to be working well. What has changed? If you are getting trafic like 3000+ concurrent users then I would look at least tiering your setup - so have your SQL on another box. When it is slow can you perform an typical SQL query like in your app - compare the speed peak and off peak.   But depending on your setup it could well be your mail server (although I would expect to see an increase in your CPU) I had to move mail of one of our budget mass hosters to another box as this was affecting the websites. Do you get any errors at all in your IIS logs in these busy times? It could be that you rare running out of TCP/IP connections. Maybe there are other factors like you mail, etc  eating up all those TCP connections. I think Windows 2003 has 6000 (or is it 8,000) by default but that is total connections for all apps not just the obvious ones in IIS. If it is this you will see server busy type HTTP staus error messages.  Also related to this do you have anything in the http sys error log? It could be asp as otehr have said look out for queues, etc there.  http://www.iisportal.com

• 09-09-2008, 6:04 PM

  In reply to

  lkevinl Joined on 09-09-2008, 4:27 AM Posts 10

  Re: IIS 6.0 ASP application bogs down despite free CPU & free memory Reply Contact tomkmvp: What does the application do?  Are you following good ASP coding practices (i.e. not putting db objects into session vars)?  Any chance you could offload SQl to another box, at least for testing?  Well, the application is a website... I call it an application because that's the way I think of it.  I started coding it years ago and have hit most of the poor coding practice bumps and made corrections.  No db objects in session vars, minimal use of session vars except to maintain a few user state variables consisting of integer values or short strings.  I have clean-up code that runs at the end of every page to close DB connections and set record sets to nothing, etc. I can't easily move SQL to another box but what I can do and tried to do a bit of during the heavy traffic time was connect my dev machine to the SQL Server on the busy box using the IIS and the copy of the website on the dev box.  I need to do some more testing during the next rush, but I don't believe I saw slow downs on the dev box which means I had a lightly loaded IIS paired with a heavily loaded SQL Server 2000 which seemed to be OK.  I'm going to try this config during the next big rush. There is some pretty heavy disk activity going on.  I have some pages with active content that updates periodically.  To enhance performance, when and update occurs, I redraw the page and store the content to disk using a file system object.  During the time inbetween updates, I just read from the disk file using a file system object and response.write instead of the other processing that would normally do to generate the page.  Basically, I wrote a custom disk caching system for a couple of heavily hit pages on the site.  These pages can be unique among groups of users, so I can have a several thousands of these pages being read from disk or being written during updates.  This never was an issue on the old server.  I have the ability to turn this feature off which would cause these pages to always be generated dynamically.  This, in theory, should make the disk I/O go way down and the CPU usage go up, by how much, I don't know.  With this upgraded hardware maybe the benefit of using the disk on the old system is now outweighed by the power of the processor in the new system meaning that now the Disk I/O is the bottleneck where on the old system, the CPU was.  Hmmmm.  This is very easy for me to test by switching a flag.  I'll definitely try this.

• 09-09-2008, 6:27 PM

  In reply to

  lkevinl Joined on 09-09-2008, 4:27 AM Posts 10

  Re: IIS 6.0 ASP application bogs down despite free CPU & free memory Reply Contact In reply to Rovastar: Hmmm... I read articles about resource contention which made using several worker processes sound appealing, but I can't use web gardens anyway since this is Classic ASP... I found out the hard way! It is just slow.  Slow means several seconds, up to 10 seconds or more for a simple ASP page.  I have one page where I show a flash "Processing" animation and then redirect to the actual page content.  The page with the flash on it is very simple.  I actually had a 10 second pause getting to that page rather than the actual page that had heavier processing.  The SQL Server has tables, stored procedures, views.  During these updates, some stored procedures are running, some using table variables which consume memory.  I actually wasn't so surprised by the memory consumption of SQL Server as I was by IIS.  IIS had well over 200MB.  On the old server, I never saw it get over 150MB.  Great suggestion about checking the SQL querys during these busy times.  I didn't do it while it was happening because at the time, it really didn't seem like SQL was the bottle neck based on my test using the dev box running IIS and the same web site connecting to the busy box's SQL Server.  Regardless, I'll give this a try.  Regarding the Mail Server, I was concerned about it, but it really wasn't taking up much of the CPU cycles.  I need to check the IIS logs, another good suggestion.  I personally didn't see any Server Busy error pages but that doesn't mean other users didn't.  How will I know if I've reached the TCP connection limit?  The HTTP status error messages will appear in the system event log or in the IIS logs?  Where do I adjust the connection limit?  Is this a good idea? Did see anything else in the event logs.  I'm looking through the IIS logs now.  If I find anything I'll post.  Thanks for the tips!

• 09-10-2008, 9:05 AM

  In reply to

  Rovastar Joined on 03-13-2008, 10:00 AM London, UK Posts 1,797

  Re: IIS 6.0 ASP application bogs down despite free CPU & free memory Reply Contact Maybe it is not the database but worth checking. Nothing in the http error log? C:\WINDOWS\system32\LogFiles\HTTPERR     That is processing http requests before getting to IIS log files. Here is a link where it explain about the TCP/IP connection limits in Windows 2003 http://forums.iis.net/p/1149707/1871069.aspx#1871069 See my post at the bottom - it is (5,000-1,024 nearly 4,000) maximum TCP connections on Windows 2003 And see here http://blogs.msdn.com/david.wang/archive/2006/04/12/HOWTO-Maximize-the-Number-of-Concurrent-Connections-to-IIS6.aspx I cannot remember what http status/error message you get but look for anything not a success and investigate that (so any 4xx or 5xx errors in that time) When having a lot of concurrent connections. Also use perfmon to see how many concurrent connections you have and use the max counter value as well this will tell you the max amout you have had since last reboot. I always get a little concern when people have say 3,000+ connections on a single box there are a lot of others things to consider and more loops to jump through to get a smooth running system - hence why I (and others) recommend additional servers. ALthough WIndows/IIS can cope with more you have to tweak it some. I am not saying that this is your problem but you should look at this especially as normally websites increase in traffic/concurrent connections over time and the higher numbers means more potential problems if you are not used to those sort of environments. Ask yourself/your clients is it better to spend money now on new kit/infrastructure or risk have it fall over and then try and fix problems when you are really busy/successful. A downed website is not good for business. http://www.iisportal.com

• 09-14-2008, 11:59 AM

  In reply to

  lkevinl Joined on 09-09-2008, 4:27 AM Posts 10

  Re: IIS 6.0 ASP application bogs down despite free CPU & free memory Reply Contact OK...  I'm monitoring the busy traffic now.  I have one browser with my site at the standard subdomain www, another browser at the subdomain www2 which should now get it's own IIS process, and a third browser which is opened to localhost with a copy of the website on it but connecting to the same, busy remote server.  This should show whether a different IIS process is as slow as the busy process and whether or not the slowdowns are coming from IIS or SQL Server.  I see slowness and so far, perfmon looks OK... I hope I'm tracking the right stats.  I'll see how things going with these browser tests.

• 09-14-2008, 12:08 PM

  In reply to

  lkevinl Joined on 09-09-2008, 4:27 AM Posts 10

  Re: IIS 6.0 ASP application bogs down despite free CPU & free memory Reply Contact OK, I can say definitively that this problem seems to be with IIS.  Both the localhost AND the www2 subdomain perform nicely while the www subdomain is very unresponsive to requests for the exact same page.   I created another website in IIS for the www2 subdomain pointing to the exact same directory that the default www site points to and created it's own Application Pool.  This way, I can test to see if it is IIS related only.  Both of these are still pulling data from the same database and any disk I/O related issues will be shared since they are on the same box.   So one IIS process is slow and the other is not which seems to point to IIS getting bogged down despite having plenty of resources.   Any ideas?  What can I check??

• 09-14-2008, 12:45 PM

  In reply to

  lkevinl Joined on 09-09-2008, 4:27 AM Posts 10

  Re: IIS 6.0 ASP application bogs down despite free CPU & free memory Reply Contact Clearly I'm not benefitting from the availability of the Quad Core Xeon and splitting up the processes helps in this case.  But I'm not sure why this is an issue.  Should the one pocess just use more of the resources?  Here are some numbers: Busy w3wp.exe process - 227MB mem usage, 283MB peak mem usage.  Perfmon numbers: ASP Server Pages - Requests Executing Avg 91, Max 98 duration 1:40, Requests Queued Avg 408, Max 508, Request Rejected 0, Request/sec Avg 28, Max 152, Sessions Current 2319, Max 2339 Memory - Pages/sec Avg 1.1, Max 41.966 Physical Disk Total - Avg Disk Queue Length Avg 104, Max 294 Processor Total - % Processor Time Total - Avg 11%, Max 66% SQL Server:Database - Active Transactions Avg 5, Max 9, Transactions/sec Avg 36, Max 170 SQL Server:General Stats - User Connections Avg 104, Max 107

• 09-14-2008, 9:03 PM

  In reply to

  lkevinl Joined on 09-09-2008, 4:27 AM Posts 10

  Re: IIS 6.0 ASP application bogs down despite free CPU & free memory Reply Contact dear god man...  what a horrendous day....  I read an article about AspProcessorThreadMax in IIS 6.0.  It menation about increasing the number of worker threads from the default of 25 to 50.  http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/d4ed262f-11d0-4a87-b15d-57a8c800539e.mspx?mfr=true  This sounds like it could be related to what I'm seeing.  The CPU usage is not maxing out, there is free RAM available, yet the server is very unresponsive.  I have seen some times where the request queue was high and the requests excuting was low.  Doesn't this mean that IIS simply can't catch up?  Even the numbers above for Active Server Pages show that.   I've bumped the AspProcessorThreadMax to 50 but my traffic has died down right now.  This is horrible.  Processing power to spare and I really don't know where things were getting choked. At least on my old server, it was clear that the CPU was the limiting factor. Any thoughts?

• 09-15-2008, 2:03 AM

  In reply to

  qbernard Joined on 03-25-2003, 10:12 PM Malaysia Posts 3,229

  Re: IIS 6.0 ASP application bogs down despite free CPU & free memory Reply Contact Well, asp queue is very high, while you are maxing on the asp executing, 25 x 4 = 100, and you getting 9X, so it is doing its job. Next, I would look at long running query, the one that stuck in the executing queue for so long and force the request queue to jam up. Take a look at the time taken field in the log file, use log parser to help. ---Ch02Top10WebRequests.sql--- SELECT     TOP 10     STRCAT(EXTRACT_PATH(cs-uri-stem),'/') AS RequestPath,     EXTRACT_FILENAME(cs-uri-stem) AS RequestedFile,     COUNT(*) AS Hits,     MAX(time-taken) AS MaxTime,     AVG(time-taken) AS AvgTime,     AVG(sc-bytes) AS AvgBytesSent FROM %source% TO %destination% GROUP BY cs-uri-stem ORDER BY MaxTime, TotalHits DESC ---Ch02Top10WebRequests.sql--- Damn... the IIS Insider archive screwed up, the Sep 2006 was missing. that was the last IIS insider column published. sigh.  Run the above via C:\LogParser\Samples>Logparser.exe file:Ch02Top10WebRequests.sql?source=”<//localhost/w3svc/1>” +destination=”Top10WebRequests.txt” -o:NAT   check if you have high process time request, look inside the scripts and do time-stamping or each function or data call, etc. next, you want wan to try iis debugdiag to see if it capture more valuable info to help you identify what's wrong.   Cheers, Bernard Cheah

• 09-15-2008, 9:01 AM

  In reply to

  lkevinl Joined on 09-09-2008, 4:27 AM Posts 10

  Re: IIS 6.0 ASP application bogs down despite free CPU & free memory Reply Contact Hmmm... not really sure I followed all of that.  Are you saying I need to figure out which page requests are running slowly causing the queue to back up?  If so, I can try that.  But again, it seems that on my system where I have quite a bit of spare memory and CPU cycles that things are getting choked off by some other limiting factor.  That's why I raised the AspProcessorThreadMax from 25 to 50.  I've also created www2, www3, and www4 each as their own website in IIS to spread users across 3 extra w3wp.exe processes, kind of like a web garden for ASP where they're session will be maintained for that process.  I've implemented some code to round robin users who first hit to be sent to www, www2, www3, or www4 and stay there.  I'll be able to turn this on and off at will. Thoughts??