IIS 5 & IIS 6
w3wp.exe memory usage is out of control
Last post Jul 23, 2010 01:13 PM by psh
Jul 17, 2008 01:35 PM|augiem|LINK
Update: I found a workaround that seems to work for now. See 1 post below...
I have set up a website with a webstore using classic ASP (because the cart system is VPASP, a classic ASP based cart system with full source code). I have the site hosted on a HostMySite.com Windows VPS running 384MB memory.
All my site is hosting is the website w/ shopping cart, Smarter Mail, and Smarter Stats.
My problem: I recently discovered serious memory issues. The server started going offline periodically and I traced the problem down to running out of memory. Doing some testing with task manager, I figured out what the problem was.
My Application Pool processes (w3wp.exe) grows tremendously with each and every seperate .ASP file accessed by a webuser. Once all my Commit Charge is used up (384megs), the website goes down with an error "HTTP/1.1 New Session Failed", and remote desktop
accesses no longer work. The only solution is to reboot the server.
With the shopping cart (VPASP), if I navigate around its various .ASP pages, the size of w3wp.exe builds to around 70MB. If I log in to administration and navigate around to all the different admin .ASP pages, the size goes even that much higher. Basically
with each and every .ASP page that is linked into VPASP's core files, I get a 2MB - 10MB increase in the size of w3wp.exe. So if a user is browsing the site and causes w3wp.exe to go up to like 70MB, and say I'm doing some adminstration on the back end adding
another 50MB, then someone opens Smarter Mail, POOF - OUT OF MEMORY. This problem is compounded by the fact that my site content pages are seperate .ASP files linked to VPASP's core files so I can use the system's user, cart, and session management on my content
pages. So, the more content pages I add, the more a user can browse, the bigger w3wp.exe will grow and then I'll run out of memory.
Additionally, even if two .ASP files are IDENTICAL in two seperate folder, each .ASP file will bloat w3wp.exe up by an equal amount. I was planning on running several iterations of the site for different regions, so the possibility exists to run out of memory
EXTREMELY fast -- if a webuser in Japan bloats w3wp.exe up to 70MB, then someone starts browsing the US site, we're going to run out of memory!
I've done a ton of research and digging around on this problem and here's what I think is really happening. (Please forgive my ignorance if I'm not totally correct as I am not an ASP or IIS expert):
I think I've boiled the problem down to Memory Template Caching (something ASP does to speed up processing) by setting up System Monitor and monitoring ALL Active Server Pages counters.
Template Caching explanation:
So here we go:
When an ASP page is accessed, IIS caches the whole of the source code, including ALL linked #include pages. The stupid thing is that even if the #include pages were already included by another ASP page previously accessed, THEY ARE CACHED AGAIN!
What does this mean? Every single ASP page that links to shop$db.asp (VPASP's core files) and all the other #included pages are cached REDUNDANTLY! The settings for changing the ASP Memory Template Caching go by # of pages, NOT memory size. (Default is 500
pages in memory). It's not smart enough to dump previously cached pages when it runs out of memory apparently -- but it will supposedly dump previous pages when you hit the # of pages limit (500). Now it's only taking a dozen or so pages to hit 70 megs, so
this is absolutely a big problem.
There also must be some CRAZY overhead. I created some test pages just to figure out how ASP was working. I created a 10MB asp file (just a ton of commented out code) just to see what would happen. Upon access, the App pool (w3wp.exe) explodes to 70MB. Seems
stupid the Memory Template Cache is 7x bigger than the original ASP file...
With all that said, I see no solution to the problem. VPASP uses very large ASP script files that are #included by each and every .ASP file you access. This is going to make it nearly impossible to host more than one VPASP store on an affordable VPS system
(HMS.com, 384MB memory). And my site design uses seperate .ASP pages for every content page linked to shop$db.asp so I can maintain login/session interface among all my pages. Once the number of pages increases in the future, the memory requirements to cache
the pages after accessing them will bring any server to its knees...
Also, I have seen the two articles below, but neither makes any difference to w3wp.exe's size:
Please give me some ideas.
Jul 17, 2008 03:45 PM|augiem|LINK
It was caching too many pages in memory, which wouldn't be a problem if each page wasn't so big.
Here's the workaround:
Right click on "Web Sites" -> Properties
Click "Home Directory" tab, click "Configuration"
Click "Cache Options" tab
Set "Cache limited ASP files in memory" to "5"
Set "Cache limited ASP files on disk" to whatever your disk has space for (default to 2000)
Now IIS will cache pages to disk, not to precious memory! Strangely, the cached pages on disk are WAYYYY smaller than the amount of memory they take to be cached in memory. Example: My 10MB test file full of commented stuff takes 70MB in memory cache. On disk
cache, it only takes like 800KB.
Now the 5 in-memory pages will be will dumped to disk as you navigate around and VPASP is staying within 35MB or so of memory!!!!
PHEW! This was a TOUGH nut to crack, I'll tell you. I hope this helps someone else out there!
I'm going to leave this thread as "unanswered" in case there's a better way to get the same result out there. (Like I said, I'm no ASP expert so there may be a better solution...)
Aug 21, 2008 06:21 PM|jgdean14|LINK
Thanks for your insight and response. I have a question: Do you notice any performance hit when navigating from page to page since the pages have been cached to disk?
Aug 22, 2008 12:50 PM|augiem|LINK
I didn't do any kind of server access speed testing per-se, but just browsing around it, it doesn't seem any slower at all. The site doesn't get much traffic, so I have no idea what kind of performance hit this would have on a site with hundreds or thousands
of users however.
I still haven't found any other workarounds for this issue.
Aug 29, 2008 09:31 AM|Lorddog|LINK
it sounds to me, without seeing any code, that the open source project shopping cart is probably storing the cart in a session or application array or object. if application then probably it is adding it onto an array or object.
To test that you could add some of your own asp pages the do nothing much but have some size to them and see if they do the same thing.
Again I suspect bad coding on that shopping cart program.
Aug 29, 2008 12:55 PM|augiem|LINK
I thought at first it was the fault of the cart, but later confirmed that it was not. I did mention in my original post that I created some of my own test pages which have nothing to do with the shopping cart, sessions, etc.
>> I created some test pages just to figure out how ASP was working. I created a
>> 10MB asp file (just a ton of commented out code) just to see what would happen.
>> Upon access, the App pool (w3wp.exe) explodes to 70MB. Seems stupid the
>> Memory Template Cache is 7x bigger than the original ASP file...
This file was nothing but a bunch of commented out code. When clicking through 4 different named copies of this file, memory skyrocketed out of control (each was 70MB in memory).
This seems to be flawed IIS caching behavior with ASP pages in general.
Aug 29, 2008 03:01 PM|Lorddog|LINK
I am sure you are right but does it have a global.asa file? if it does, check for a session on start and post any code there. That would fire from any test page you made unless that test page was in another virtual web site.
(hate to beat a dead horse but just to make sure)
Sep 06, 2008 12:14 PM|bp1212|LINK
I have an even worse memory problem with classic asp. We are just trying to migrate our application to new hardware, both the old server and new are w2k03 sp2, but we've noticed a dramatic increase in memory used by the w3wp.exes on the new harward. It
manifests itself when the very first ASP page is called. I set up a single website in a dedicated app pool, if you access an html page you can see a w3wp.exe process that grabs about 8Mb of memory, however if you access a single ASP page containing just a
single line <% response.write("test")%> with no global.asa for the site, the memory used by w3wp.exe innediately shoots up to approx 105 Mb. Any ideas why this would be? As far as I can tell we don't have any debugging enabled or anything. One thing to
note is that the webserver has 2 quad core processors.
Sep 06, 2008 03:31 PM|augiem|LINK
Try making the changes in this post. It seems to apply to servers with multiple processors. It seems to reduce the memory footprint for me as my server has 4 processors.
In response to Lorddog's latest email, there is no global.asa file.
Sep 08, 2008 10:51 AM|bp1212|LINK
Problem Solved! It turns out the virus checker we had installed (McAfee enterprise v8) had a new buffer overflow protection feature that causes problems with w3wp.exe. We downgraded to v7.1 and the problem was solved instantly
Sep 23, 2008 05:33 PM|KavumaIvan|LINK
The Solution is to Configure and application pool for the website as follows.
1. In the application pool folder right click on your application pool (Create one if non exists)
2. Select Properties.
3. In the recycling tab you can change all the setting to make sure that the w3wp.exe recycles when these conditions happen. I set mine to recycle every 10 mins, after every 1000 requests,
maximum virtual memory 50MB, Maximum memory used 50MB.
4 there are more setting on the other tabs. Remember to restart IIS after making these changes. Good luck.
This worked for me very efficiently.
Sep 23, 2008 05:48 PM|augiem|LINK
That was my 1st course of action when dealing with this problem. After trying everything with application pools, I determined it was not a viable solution to my problem. When the application pool resets, all sessions are deleted. This means anyone browsing
the site which uses sessions for login management will be suddenly logged out when the application pool reset happens. Unacceptable for my purposes.
Dec 21, 2008 09:07 PM|whitesites|LINK
Something you should know about the VPS boxes at hostmysite.com I have 4 VPS and about 20 shared accounts with them, so I have a pretty good understanding of how their setup works. I am assuming you are running on the Windows Base VPS. This unit only
has 256 Megs of memory. Even though it says 384. This is not memory, this is virtual memory. The second you start using the additional 128 Megs of memory, you are running off the disk. I did this and kept having problems with my system crashing. Event
logs became corrupt. ext.
You need to put a ceiling on the App Pools memory usage. This way it will automatically recycle when it gets full. If you are worried about loosing sessions. Turn on the ASP.NET State Server under services. Then configure your websites to use it. This
will preserve your sessions, when your app pool recycles.
If you are using caching I recommend you read up on a blog I wrote.
Using Caching to make your ASP.NET website more scalable
I would never cache entire pages. Just cache the reusable sections. Also a 256 Meg VPS is pretty worthless. If you are running MySQL , mail server, stats server, you need at least 512 Megs. I was able to get a 256 Meg VPS work, by moving all other servers
to a shared hosting account. So we have an $8 / month shared hosting account for MySQL, SmarterMail, Smarter Stats. And a 256 MEG VPS for the Website.
Hope this helps
Jul 23, 2010 01:13 PM|psh|LINK
By going through your workaround, what I found that we have other option selected (DO NOT CACHE ASP FILES) So asp caching might not be an issue for our asp application. Any other suggestions?