IIS 5 & IIS 6
w3wp consuming over 1GB of memory
Last post Nov 13, 2008 06:54 AM by Rovastar
Nov 12, 2008 09:57 AM|hnugz|LINK
We have some problem with our server/app where one work process will all of a sudden start consuming memory until it hits about 1.2GB then freezes the app for all who are using it.
Normally the memory sits at around 200-300MB. This particular app pool has 2 worker processes. I will start to see one go up faster than the other. Also, the cpu time will start to go up faster as well. Using perfmon, the TCPv4 connections established
and Web Service current connections counters will rise. Normally these sit at about 150 and 5 respectively. When the problem happens they get in the hundreds for current connections.
I have been pawing through windbg on hang dumps I have created but I have a few questions. Here is the bottom result of a !dumpheap -stat:
79185948 16126 322520 System.Runtime.Serialization.SerializationFieldInfo
79101fe4 6124 342944 System.Collections.Hashtable
79104368 20592 494208 System.Collections.ArrayList
7912dd40 486 507960 System.Char
791319b0 105 890468 System.Int16
7912d9bc 6162 1195032 System.Collections.Hashtable+bucket
79109778 46462 2601872 System.Reflection.RuntimeMethodInfo
7912d8f8 26303 4406200 System.Object
7912dc14 29 4523452 System.Int64
7912dae8 3914 6518228 System.Byte
790fd8c4 131070 7835836 System.String
00125b78 151 40894008 Free
Total 494288 objects
Fragmented blocks larger than 0.5 MB:
Addr Size Followed by
039e7650 2.9MB 03cd4c44 System.String
03cd51b8 0.7MB 03d7e5d0 System.String
03dceeb0 2.1MB 03fe91dc System.Runtime.InteropServices.GCHandle
03fe9248 1.0MB 040ef320 System.Byte
07b4a69c 9.9MB 085253d4 System.String
0b9d6b04 6.3MB 0c027510 Oracle.DataAccess.Client.OracleConnection
0c027664 0.8MB 0c0fd788 System.String
0c0fdc4c 0.9MB 0c1edbcc System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6
0f99b570 1.6MB 0fb34330 IBM.Data.DB2.iSeries.MpDcData
0fb3442c 0.6MB 0fbd3760 System.String
0fbd3d04 8.0MB 103d5748 System.Byte
From all the posts I've read (Mostly Tess' blog) there is a total size at the bottom. Why doesn't mine have this total? There are so many items I can't tell what the size is however it doesn't look to me like it would be 1.2GB. I need to find out what
is taking up all that memory (40MB free, 7MB for string, etc... correct?)
Also, should I be concerned about the "Fragmented blocks larger than 0.5 MB" section? What exactly is this telling me?
I also did a !runaway command and saw one thread that has a ridiculous user time:
20:dc 0 days 0:54:02.968
The next highest is 0:10:26.437.
I switched to that thread and did a !clrstack but I don't get the detailed results the other pages I read are getting:
OS Thread Id: 0xdc (20)
02a1ec7c 7c8285ec [NDirectMethodFrameSlim: 02a1ec7c]
02a1f184 79e7c74b [HelperMethodFrame_PROTECTOBJ: 02a1f184]
02a1f8d8 79f047fd [ContextTransitionFrame: 02a1f8d8]
02a1f90c 79f047fd [GCFrame: 02a1f90c]
02a1fa68 79f047fd [ComMethodFrame: 02a1fa68]
Any suggestions on what I can do from here to find out what is going on with this worker process? Thanks.
Nov 12, 2008 01:43 PM|Rovastar|LINK
A lot of info there but what technologies are you using .net, php?
I am confused by
"This particular app pool has 2 worker processes. "
Have you 2 applications in an app pool?
Why not seperate the apps into seperate pools? It will be easier to troubleshoot. To narrow down what code is problematic.
Debugdiag might give you more clues.
Nov 12, 2008 02:25 PM|hnugz|LINK
In IIS there are 6 web sites. Default web site, app center 2000 admin site, and 4 other ones. In the application pools section there is a default apppool and 4 more, one for each website I mentioned. There is no app pool for the app center site.
One of the app pools uses web garden and has 2 for the maximum number of worker processes.
I have looked at debugdiag before but I do not remember seeing anything that useful from the report. I am trying to pull it up again but it takes forever on "preloading thread data." There are 136 threads in this dump.
Some other things I have tried in windbg are the !locks command. 3 items showed up but they all had a lock count of 0 which I read should be ignored.
I did a !syncblk which showed 2 threads. Monitor help was 9 on one, and 1 on the other. So far I have not seen anything concrete after exploring these threads that would be the problem.
I am still trying to find out what is taking up all the memory.
I feel like the answer is looking me in the face however I'm not sure what to ignore or pay attention to. There is an awful lot of info here :).
Nov 12, 2008 05:35 PM|hnugz|LINK
I think looking at lock information is sending me in the wrong direction. I did run the !eeheap –gc command and say that the total was only around 80MB. According to the website I was reading this means that it is a problem with the native heap. I setup
a leak rule in debugdiag on our server and will monitor it the next few days.
Just out of curiosity, does it only take dumps if I specify to generate one based upon private bytes/virtual bytes or does it automatically start generating them after a certain time period? I set it to create 5 dumps but left the private bytes/virtual
bytes unchecked. Is this not going to generate anything?
Nov 13, 2008 06:54 AM|Rovastar|LINK
To be honest I don't like web gardens and never see a need to use them. I never see a valid justification for using them and can't think of one.
First off do you get the issue without the web garden. I presume that is the app pool that is the problem. If you do not then why are you using a web garden?
Starting at the beginning troubleshooting I would look in the IIS logs, http err logs, event logs for anything that indicates what page is a problem. Look for http status, win32 codes and time taken for indications of what page it is on.
About your other troubleshooting of the problem then tbh you probably know more than me. That level of app support is not the day-to-day for IIS admin. Using !eeheap, !address, etc is something I would only do occasionally and often with devs that know a
little more about it. You might have more luck on www.asp.net (I presume you are using .net) for detailed troubleshooting.