Classic ASP

Re: IIS 6.0 ASP application bogs down despite free CPU & free memory

qbernard — Sat, 14 Nov 2009 06:30:36 GMT

You might want to start a new thread and post your setup info, problem symptom etc

Re: IIS 6.0 ASP application bogs down despite free CPU & free memory

aspire2x — Thu, 12 Nov 2009 19:25:27 GMT

Hi All, we're also experiencing the same issue. Did either of you guys (lkevinl, or Dublin) get closer to solving the problem? Thanks in advance!

Re: IIS 6.0 ASP application bogs down despite free CPU & free memory

Dublin — Mon, 08 Jun 2009 17:48:40 GMT

Hey lkevinl, we are having somewhat the same issue, "IIS 6.0 ASP application bogs down despite free CPU & free memory" did changing AspProcessorThreadMax from 25 to 50 help ? did you manage to find a solution.. Thanks

Re: IIS 6.0 ASP application bogs down despite free CPU & free memory

lkevinl — Mon, 15 Sep 2008 13:01:22 GMT

Hmmm... not really sure I followed all of that. Are you saying I need to figure out which page requests are running slowly causing the queue to back up? If so, I can try that.

But again, it seems that on my system where I have quite a bit of spare memory and CPU cycles that things are getting choked off by some other limiting factor. That's why I raised the AspProcessorThreadMax from 25 to 50.

I've also created www2, www3, and www4 each as their own website in IIS to spread users across 3 extra w3wp.exe processes, kind of like a web garden for ASP where they're session will be maintained for that process. I've implemented some code to round robin users who first hit to be sent to www, www2, www3, or www4 and stay there. I'll be able to turn this on and off at will.

Thoughts??

Re: IIS 6.0 ASP application bogs down despite free CPU & free memory

qbernard — Mon, 15 Sep 2008 06:03:18 GMT

Well, asp queue is very high, while you are maxing on the asp executing, 25 x 4 = 100, and you getting 9X, so it is doing its job.

Next, I would look at long running query, the one that stuck in the executing queue for so long and force the request queue to jam up. Take a look at the time taken field in the log file, use log parser to help.

---Ch02Top10WebRequests.sql---
SELECT
TOP 10     STRCAT(EXTRACT_PATH(cs-uri-stem),'/') AS RequestPath,     EXTRACT_FILENAME(cs-uri-stem) AS RequestedFile,
COUNT(*) AS Hits,
MAX(time-taken) AS MaxTime,
AVG(time-taken) AS AvgTime,
AVG(sc-bytes) AS AvgBytesSent
FROM %source% TO %destination%
GROUP BY cs-uri-stem
ORDER BY MaxTime, TotalHits DESC
---Ch02Top10WebRequests.sql---

Damn... the IIS Insider archive screwed up, the Sep 2006 was missing. that was the last IIS insider column published. sigh. Run the above via

C:\LogParser\Samples>Logparser.exe file:Ch02Top10WebRequests.sql?source=”<//localhost/w3svc/1>”

+destination=”Top10WebRequests.txt” -o:NAT

check if you have high process time request, look inside the scripts and do time-stamping or each function or data call, etc.

next, you want wan to try iis debugdiag to see if it capture more valuable info to help you identify what's wrong.

Re: IIS 6.0 ASP application bogs down despite free CPU & free memory

lkevinl — Mon, 15 Sep 2008 01:03:28 GMT

dear god man... what a horrendous day....

I read an article about AspProcessorThreadMax in IIS 6.0. It menation about increasing the number of worker threads from the default of 25 to 50.

http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/d4ed262f-11d0-4a87-b15d-57a8c800539e.mspx?mfr=true

This sounds like it could be related to what I'm seeing. The CPU usage is not maxing out, there is free RAM available, yet the server is very unresponsive. I have seen some times where the request queue was high and the requests excuting was low. Doesn't this mean that IIS simply can't catch up? Even the numbers above for Active Server Pages show that.

I've bumped the AspProcessorThreadMax to 50 but my traffic has died down right now. This is horrible. Processing power to spare and I really don't know where things were getting choked. At least on my old server, it was clear that the CPU was the limiting factor.

Any thoughts?

Re: IIS 6.0 ASP application bogs down despite free CPU & free memory

lkevinl — Sun, 14 Sep 2008 16:45:52 GMT

Clearly I'm not benefitting from the availability of the Quad Core Xeon and splitting up the processes helps in this case. But I'm not sure why this is an issue. Should the one pocess just use more of the resources?

Here are some numbers:

Busy w3wp.exe process - 227MB mem usage, 283MB peak mem usage.

Perfmon numbers:

ASP Server Pages - Requests Executing Avg 91, Max 98 duration 1:40, Requests Queued Avg 408, Max 508, Request Rejected 0, Request/sec Avg 28, Max 152, Sessions Current 2319, Max 2339

Memory - Pages/sec Avg 1.1, Max 41.966

Physical Disk Total - Avg Disk Queue Length Avg 104, Max 294

Processor Total - % Processor Time Total - Avg 11%, Max 66%

SQL Server:Database - Active Transactions Avg 5, Max 9, Transactions/sec Avg 36, Max 170

SQL Server:General Stats - User Connections Avg 104, Max 107

Re: IIS 6.0 ASP application bogs down despite free CPU & free memory

lkevinl — Sun, 14 Sep 2008 16:08:50 GMT

OK, I can say definitively that this problem seems to be with IIS. Both the localhost AND the www2 subdomain perform nicely while the www subdomain is very unresponsive to requests for the exact same page.

I created another website in IIS for the www2 subdomain pointing to the exact same directory that the default www site points to and created it's own Application Pool. This way, I can test to see if it is IIS related only. Both of these are still pulling data from the same database and any disk I/O related issues will be shared since they are on the same box.

So one IIS process is slow and the other is not which seems to point to IIS getting bogged down despite having plenty of resources.

Any ideas? What can I check??

Re: IIS 6.0 ASP application bogs down despite free CPU & free memory

lkevinl — Sun, 14 Sep 2008 15:59:24 GMT

OK... I'm monitoring the busy traffic now. I have one browser with my site at the standard subdomain www, another browser at the subdomain www2 which should now get it's own IIS process, and a third browser which is opened to localhost with a copy of the website on it but connecting to the same, busy remote server.

This should show whether a different IIS process is as slow as the busy process and whether or not the slowdowns are coming from IIS or SQL Server.

I see slowness and so far, perfmon looks OK... I hope I'm tracking the right stats. I'll see how things going with these browser tests.

Re: IIS 6.0 ASP application bogs down despite free CPU & free memory

Rovastar — Wed, 10 Sep 2008 13:05:00 GMT

Maybe it is not the database but worth checking.

Nothing in the http error log? C:\WINDOWS\system32\LogFiles\HTTPERR

That is processing http requests before getting to IIS log files.

Here is a link where it explain about the TCP/IP connection limits in Windows 2003

http://forums.iis.net/p/1149707/1871069.aspx#1871069

See my post at the bottom - it is (5,000-1,024 nearly 4,000) maximum TCP connections on Windows 2003

And see here

http://blogs.msdn.com/david.wang/archive/2006/04/12/HOWTO-Maximize-the-Number-of-Concurrent-Connections-to-IIS6.aspx

I cannot remember what http status/error message you get but look for anything not a success and investigate that (so any 4xx or 5xx errors in that time)

When having a lot of concurrent connections.

Also use perfmon to see how many concurrent connections you have and use the max counter value as well this will tell you the max amout you have had since last reboot.

I always get a little concern when people have say 3,000+ connections on a single box there are a lot of others things to consider and more loops to jump through to get a smooth running system - hence why I (and others) recommend additional servers. ALthough WIndows/IIS can cope with more you have to tweak it some.

I am not saying that this is your problem but you should look at this especially as normally websites increase in traffic/concurrent connections over time and the higher numbers means more potential problems if you are not used to those sort of environments. Ask yourself/your clients is it better to spend money now on new kit/infrastructure or risk have it fall over and then try and fix problems when you are really busy/successful. A downed website is not good for business.

Re: IIS 6.0 ASP application bogs down despite free CPU & free memory

lkevinl — Tue, 09 Sep 2008 22:27:08 GMT

In reply to Rovastar:

Hmmm... I read articles about resource contention which made using several worker processes sound appealing, but I can't use web gardens anyway since this is Classic ASP... I found out the hard way!

It is just slow. Slow means several seconds, up to 10 seconds or more for a simple ASP page. I have one page where I show a flash "Processing" animation and then redirect to the actual page content. The page with the flash on it is very simple. I actually had a 10 second pause getting to that page rather than the actual page that had heavier processing.

The SQL Server has tables, stored procedures, views. During these updates, some stored procedures are running, some using table variables which consume memory. I actually wasn't so surprised by the memory consumption of SQL Server as I was by IIS. IIS had well over 200MB. On the old server, I never saw it get over 150MB.

Great suggestion about checking the SQL querys during these busy times. I didn't do it while it was happening because at the time, it really didn't seem like SQL was the bottle neck based on my test using the dev box running IIS and the same web site connecting to the busy box's SQL Server. Regardless, I'll give this a try.

Regarding the Mail Server, I was concerned about it, but it really wasn't taking up much of the CPU cycles.

I need to check the IIS logs, another good suggestion. I personally didn't see any Server Busy error pages but that doesn't mean other users didn't.

How will I know if I've reached the TCP connection limit? The HTTP status error messages will appear in the system event log or in the IIS logs? Where do I adjust the connection limit? Is this a good idea?

Did see anything else in the event logs. I'm looking through the IIS logs now. If I find anything I'll post. Thanks for the tips!

Re: IIS 6.0 ASP application bogs down despite free CPU & free memory

lkevinl — Tue, 09 Sep 2008 22:04:41 GMT

tomkmvp:

What does the application do? Are you following good ASP coding practices (i.e. not putting db objects into session vars)? Any chance you could offload SQl to another box, at least for testing?

Well, the application is a website... I call it an application because that's the way I think of it. I started coding it years ago and have hit most of the poor coding practice bumps and made corrections. No db objects in session vars, minimal use of session vars except to maintain a few user state variables consisting of integer values or short strings. I have clean-up code that runs at the end of every page to close DB connections and set record sets to nothing, etc.

I can't easily move SQL to another box but what I can do and tried to do a bit of during the heavy traffic time was connect my dev machine to the SQL Server on the busy box using the IIS and the copy of the website on the dev box. I need to do some more testing during the next rush, but I don't believe I saw slow downs on the dev box which means I had a lightly loaded IIS paired with a heavily loaded SQL Server 2000 which seemed to be OK. I'm going to try this config during the next big rush.

There is some pretty heavy disk activity going on. I have some pages with active content that updates periodically. To enhance performance, when and update occurs, I redraw the page and store the content to disk using a file system object. During the time inbetween updates, I just read from the disk file using a file system object and response.write instead of the other processing that would normally do to generate the page. Basically, I wrote a custom disk caching system for a couple of heavily hit pages on the site. These pages can be unique among groups of users, so I can have a several thousands of these pages being read from disk or being written during updates. This never was an issue on the old server. I have the ability to turn this feature off which would cause these pages to always be generated dynamically. This, in theory, should make the disk I/O go way down and the CPU usage go up, by how much, I don't know.

With this upgraded hardware maybe the benefit of using the disk on the old system is now outweighed by the power of the processor in the new system meaning that now the Disk I/O is the bottleneck where on the old system, the CPU was. Hmmmm. This is very easy for me to test by switching a flag. I'll definitely try this.

Re: IIS 6.0 ASP application bogs down despite free CPU & free memory

Rovastar — Tue, 09 Sep 2008 13:39:04 GMT

First off don't use web gardens. 99.99% of the time there is no need for them. Really they are only good for a short term fix for certain coding issues.

Is it just slow?

How slow is 'slow'?

The memory consumption of SQL looks high what is it doing there?

Do a trace on your SQL server box and see if there is a bottleneck there.

Run performance stats on the SQL server look out for locks, latches, etc. Compare this to your baseline stats for when the server was last know to be working well. What has changed?

If you are getting trafic like 3000+ concurrent users then I would look at least tiering your setup - so have your SQL on another box.

When it is slow can you perform an typical SQL query like in your app - compare the speed peak and off peak.

But depending on your setup it could well be your mail server (although I would expect to see an increase in your CPU) I had to move mail of one of our budget mass hosters to another box as this was affecting the websites.

Do you get any errors at all in your IIS logs in these busy times?

It could be that you rare running out of TCP/IP connections. Maybe there are other factors like you mail, etc eating up all those TCP connections. I think Windows 2003 has 6000 (or is it 8,000) by default but that is total connections for all apps not just the obvious ones in IIS. If it is this you will see server busy type HTTP staus error messages.

Also related to this do you have anything in the http sys error log?

It could be asp as otehr have said look out for queues, etc there.

Re: IIS 6.0 ASP application bogs down despite free CPU & free memory

tomkmvp — Tue, 09 Sep 2008 12:26:35 GMT

What does the application do? Are you following good ASP coding practices (i.e. not putting db objects into session vars)? Any chance you could offload SQl to another box, at least for testing?

Re: IIS 6.0 ASP application bogs down despite free CPU & free memory

qbernard — Tue, 09 Sep 2008 08:52:10 GMT

Looks like hardware bottlenecks to me, can you do some performance analysis to look at ASP - request executing, in queue, etc.

NLB is a good choice for you, say move out the SQL, then have 2 nodes NLB of IIS infront, with sticky session, no need to worry about the client sesion running around the two nodes, it will stick until the session end / expired.