Unlock your iis7-aspnet performance [Answered]RSS

5 replies

Last post Nov 02, 2012 01:10 PM by foxontherock

  • Unlock your iis7-aspnet performance

    Feb 18, 2011 10:09 AM|foxontherock|LINK

    After 4 months of optimizing every line of code, every stored procedure, every index, adding a web farm, adding computed fields and triggers, cleaning every line of code, we found what causes headaches and performance issues, and I'll share it with you.

    We are using IIS to run 10 sites, with isolated application pools. 1 of these site is an API build with asp.net that get 15-20 calls every seconds for our main asp3 classic application. The other sites are both asp classic and aspnet sites.

    Once every 2-3 minutes, ALL our asp.net processes were locked for 10-15 seconds.

    We noticed recently that, each time we get the downtime, the "gen2 collection" performance counter increased just before the downtime.

    After reading a lot of articles and researches, I found this:

    http://msdn.microsoft.com/en-us/library/ff647787.aspx

    The magical answer is at the end of the article:

    Configuring the Workstation GC

    gcServer enabled="false"
    gcConcurrent enabled="false"

    That solves all our troubles. After adding this, we see that "gen2 collections" happen more often, once every 15-20 seconds instead of every 2 minutes, each process is isolated, if site1 is cleaning its memory, site 2 is not affected, and, overall we do not get any downtime anymore!

    Why is this setting not the default setting?

    gcserver is enabled by default on asp.net applications even if the documentation say that is it disabled by default. That causes the GC to be global to all applications. It waits longer before doing a gen2 cleanup, and when it does it, it takes a long time and lock every asp.net processes. for instance, before doing the change, we got around 1600 Gen2 collections in 5 days. Since the change, yesterday, we are now at 9000 Gen2 collections. It collects more often and more quickly without blocking other processes.

    I suggest to Microsoft to set these 2 setting as the DEFAULT SETTINGS for all servers. That's like night and day for our application. That simply saved our life, as customers were calling us to close their accounts because of bad performance issues.

    I post this article to maybe help someone who have the same issue.

    memory management garbage collection gcserver aspnet.config

  • Re: Unlock your iis7-aspnet performance

    Feb 18, 2011 10:23 PM|lextm|LINK

    If you have to use Workstation GC mode instead of the Server GC mode, the problem is not caused by Microsoft's mistake, but your developers of this application.

    Read this chapter, http://msdn.microsoft.com/en-us/library/ff647790.aspx and you will see why Server GC should be used.

    I guess there are some classes designed poorly in this application that finally have negative impact on GC. They may work acceptably in Workstation GC mode, but unacceptable in Server GC mode.

    Then you should not expect Microsoft to change a critical .NET setting simply for your application.

    Optimizing ASP.NET and IIS is not trivial, and please be patient and reasonable.

    Regards,

    Lex Li
    IIS Consulting Services at https://support.lextudio.com/services/consulting.html
    ---------------------------
    This posting is provided "AS IS" with no warranties, and confers no rights.
  • Re: Unlock your iis7-aspnet performance

    Feb 19, 2011 08:12 AM|foxontherock|LINK

    Your post seems right for someone who read mine for the first time. He may think that our whole code is bad. But, we searched for 4 months to solve this issue. We applied everything that your article suggests. And it didn't solved this issue. The quality of our code was increased, but this problem remains. Our threads were still freezing for 15 seconds every 2 minutes by the GC.

    If the article you post is right, the chapter 6 of this same article, maybe by the same author, should be right too.

    And, that's what we read:

    "If you host several isolated worker processes on Windows Server 2003 on a multiprocessor computer, use the workstation GC and nonconcurrent mode."

    Why are they suggesting to do that? I don't know. But it solved our problem. Now, our "maybe bad written" application is running lightning fast without any freezing problem. It never slow down even on the top of activity between 9 and 11 am when we get between 20 and 50 calls every second. All our api call are executed at an average time of 150 ms, and everything works fine.
  • Re: Unlock your iis7-aspnet performance

    Feb 19, 2011 09:14 PM|lextm|LINK

    I quote that chapter simply to emphasize a fact, that Server GC should be used for your production environment.

    I don't doubt that "the quality of your code was increased" or "you applied everything that article suggests". But I must confess the GC oriented optimization does not come easy, unless you have a deep enough understanding of GC/heaps runtime footprint well enough to identify the cause of the slowness. Even a long time .NET developer may feel difficult to identify or resolve such issues.

    The chapter 6 shows sometimes you can switch to Workstation GC, but IMHO that's only a dirty workaround (in case you cannot identify the root cause, this workaround at least makes you happy).

    I won't comment more, as it seems that you are very happy with the workaround. You can use it as far as nothing breaks any more. To summarize,

    1. .NET coding is not that simple. Playing nicely with the GC requires a lot of efforts.

    2. To debug GC related performance issues, you need special expertise, and sometimes involving Microsoft support via http://support.microsoft.com is a quicker way.

    3. It is not wrong to apply any workarounds mentioned by Microsoft articles, but make sure you understand the recommended ways.

    Regards,

    Lex Li
    IIS Consulting Services at https://support.lextudio.com/services/consulting.html
    ---------------------------
    This posting is provided "AS IS" with no warranties, and confers no rights.
  • Re: Unlock your iis7-aspnet performance

    Feb 19, 2011 10:11 PM|foxontherock|LINK

    Oh wow... so, as I can see in chapter 6, we're not using a "dirty workaround", we're using the "best practice we can do in our case" because our application is not alone on the server. We have several applications, that may be why we had trouble. Even SQL Server activity was locking our asp.net processes. And the article clearly say that if we have several hosted isolated processes, we should disable the gc server.

    That helps us and I hope it may help other people make their dirty code work fine. I understand that people writing perfect code will never need to use that dirty workaround.

    For those who need this, I'm happy to share it with you.

    That's all for this thread.
  • Re: Unlock your iis7-aspnet performance

    Nov 02, 2012 01:10 PM|foxontherock|LINK

    I knew I was right...

    "Starting with the .NET Framework 4.5, background server garbage collection is the default mode for server garbage collection. To choose this mode, set the enabled attribute of the element to true in the runtime configuration schema. This mode functions similarly to background workstation garbage collection"

    http://msdn.microsoft.com/en-us/library/ee787088.aspx#background_server_garbage_collection">link

    We had to use "workstation" garbage collection because in server mode, all our threads were stopped for several seconds every time we get a final level garbage collection. In workstation mode, the gc is done from a separate thread that does not block current running ones.