We are excited to announce that the IIS.NET Forums are moving to the new Microsoft Q&A experience. Learn more >

Locking Up / Not Responding / 500 ErrorRSS

45 replies

Last post Mar 28, 2014 02:58 PM by laurin1

  • Locking Up / Not Responding / 500 Error

    Mar 04, 2014 02:19 PM|laurin1|LINK

    Well, things went well for about 2 weeks and now we have a serious problem. After being up and running for around 3 days, IIS begins to lock up. Our Current Non-Anonymous user count (we are using Windows Authentication) goes from 10 to about 60. Eventually, we get timeout errors, all pointing to wincache_ucache_get() or wincache_ucache_set() lines in the code, and the 500 Internal Server errors. The only solution we have is iisreset!

    This is a huge problem and we're not even sure how to troubleshoot it.

  • DropPhone DropPhone

    360 Posts

    Microsoft

    Moderator

    Re: Locking Up / Not Responding / 500 Error

    Mar 04, 2014 04:30 PM|DropPhone|LINK

    Step one: Attach windbg.exe to the php-cgi.exe process and get a dump.  Then send me mail and we can work out how I can get that dump.

    I'll need to know which version of wincache & php you're using (like, where you picked up the bits), so I can find the matching symbols.

    Thx!

        --E.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 04, 2014 04:52 PM|laurin1|LINK

    At any time or when it's locked up?

    I've tried to do this before, but never succesfully. If it has to be done during a lockup, that will be very hard to do - we can't suffer an outage like this for more than a few minutes.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 05, 2014 11:41 AM|laurin1|LINK

    More info.

    The wincache.php file worked fine after the reset, but now the Summary and User Cache pages are blank! I assume that the problem is in it's early stage of failure:

    http://grab.by/jvaU

  • Re: Locking Up / Not Responding / 500 Error

    Mar 05, 2014 11:47 AM|laurin1|LINK

    So, if I do

    print_r(wincache_ucache_info(true));

    I get this:

    Array (    [total_cache_uptime] => 78052    [is_local_cache] =>     [total_item_count] => 126669    [total_hit_count] => 3993455    [total_miss_count] => 427399    [ucache_entries] => Array        (        ) )

    But if I do this:

    print_r(wincache_ucache_info(false));

    I get nothing!

  • Re: Locking Up / Not Responding / 500 Error

    Mar 05, 2014 11:52 AM|laurin1|LINK

    I'm thinking of downgrading to 1.3.5.0 to see if it solves the problem.

  • DropPhone DropPhone

    360 Posts

    Microsoft

    Moderator

    Re: Locking Up / Not Responding / 500 Error

    Mar 05, 2014 01:39 PM|DropPhone|LINK

    So, I have the following script:

    <?php
    
    $bar = 'BAR';
    wincache_ucache_set( 'foo', $bar );
    
    $colors_array = array('green' => '5', 'Blue' => '6', 'yellow' => '7', 'cyan' => '8');
    wincache_ucache_set($colors_array);
    
    echo( "----wincache_ucache_info(true))----\n" );
    print_r(wincache_ucache_info(true));
    echo( "----wincache_ucache_info(false)----\n" );
    print_r(wincache_ucache_info(false));
    
    ?>

    Which I ran against php 5.5.7 + Wincache 1.3.5.2 (the dev build I put on sourceforge in Feb).  The output I get is as follows:

    ----wincache_ucache_info(true))----
    Array
    (
        [total_cache_uptime] => 0
        [is_local_cache] =>
        [total_item_count] => 5
        [total_hit_count] => 0
        [total_miss_count] => 0
        [ucache_entries] => Array
            (
            )
    
    )
    ----wincache_ucache_info(false)----
    Array
    (
        [total_cache_uptime] => 0
        [is_local_cache] =>
        [total_item_count] => 5
        [total_hit_count] => 0
        [total_miss_count] => 0
        [ucache_entries] => Array
            (
                [1] => Array
                    (
                        [key_name] => green
                        [value_type] => string
                        [value_size] => 18
                        [ttl_seconds] => 0
                        [age_seconds] => 0
                        [hitcount] => 0
                    )
    
                [2] => Array
                    (
                        [key_name] => foo
                        [value_type] => string
                        [value_size] => 20
                        [ttl_seconds] => 0
                        [age_seconds] => 0
                        [hitcount] => 0
                    )
    
                [3] => Array
                    (
                        [key_name] => yellow
                        [value_type] => string
                        [value_size] => 18
                        [ttl_seconds] => 0
                        [age_seconds] => 0
                        [hitcount] => 0
                    )
    
                [4] => Array
                    (
                        [key_name] => cyan
                        [value_type] => string
                        [value_size] => 18
                        [ttl_seconds] => 0
                        [age_seconds] => 0
                        [hitcount] => 0
                    )
    
                [5] => Array
                    (
                        [key_name] => Blue
                        [value_type] => string
                        [value_size] => 18
                        [ttl_seconds] => 0
                        [age_seconds] => 0
                        [hitcount] => 0
                    )
    
            )
    
    )

    ...so, it looks like "It Works On My Box(tm)".  That said, I *am* using a way-old PHP 5.5.x build, so I'll see what happens after I sync & rebuild the world...

    Thx!

        --E.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 05, 2014 03:00 PM|laurin1|LINK

    Maybe I wasn't clear. Yes, that works fine for us too, until whatever is wrong with this begins to happen and then this appears to be the first symptom of a problem, but eventually it starts to lock up the php-cgi.exe processes and finally crashes.

  • DropPhone DropPhone

    360 Posts

    Microsoft

    Moderator

    Re: Locking Up / Not Responding / 500 Error

    Mar 05, 2014 04:53 PM|DropPhone|LINK

    Ah, okay, now i get it. 

    Yeah, get a crash dump when you get into the bad state, and send me a private message.  That's about the only way to figure out what's gone wrong here...

        --E.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 05, 2014 06:08 PM|laurin1|LINK

    Again, that may not be possible - I don't even know how to do that successfully (my previous attempts failed) and I cannot leave it in that state for more than a minute or two, as people need the site to do their jobs.

  • DropPhone DropPhone

    360 Posts

    Microsoft

    Moderator

    Re: Locking Up / Not Responding / 500 Error

    Mar 05, 2014 08:04 PM|DropPhone|LINK

    If you can figure out some kind of repro, then we can make progress.  If we don't have a repro, then we need a crash dump to investigate further.  I'm afraid those are our only two options.

    Thx!

        --E.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 06, 2014 12:03 AM|laurin1|LINK

    FYI, now when I run

    print_r(wincache_ucache_info(false));

    I am getting a 500 Internal Server Error

  • Re: Locking Up / Not Responding / 500 Error

    Mar 06, 2014 12:06 AM|laurin1|LINK

    And wincache_ucache_clear(); before it completely crashes does fix it (I can load wincache.php again.)

  • DropPhone DropPhone

    360 Posts

    Microsoft

    Moderator

    Re: Locking Up / Not Responding / 500 Error

    Mar 06, 2014 03:50 AM|DropPhone|LINK

    Hm.  If you can catch it when it returns 500 and then get a crash dump before you have to do a wincache_ucache_clear(), that would be the best way to make progress on the investigation.

    How long (and how many wincache_ucache_set() operations) before this thing starts to go bad?  Maybe we just need to run a simulation that hammers on this for several hours (or days).

    Thx!

        --E.

  • DropPhone DropPhone

    360 Posts

    Microsoft

    Moderator

    Re: Locking Up / Not Responding / 500 Error

    Mar 06, 2014 02:33 PM|DropPhone|LINK

    In your php.ini, what's the size of your wincache.ucachesize?  IIRC, you set it pretty large, like 930 MB, right?

    I ran into an issue when repro'ing where the call to wincache_ucache_info(false) failed because of an out-of-memory error...

    Thx!

        --E.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 06, 2014 07:44 PM|laurin1|LINK

    Yes, it's like 1000, but I've never seen it get over about 80MB. What happens if it goes over the max?

    I just checked and wincache.php already won't load the User Cache info and I just reset it last night.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 06, 2014 07:51 PM|laurin1|LINK

    sorry, yes 930.

    Ok, I was able to the memory info:

    Array (   

    [memory_total] => 975175680   

    [memory_free] => 908547916   

    [num_used_blks] => 446581   

    [num_free_blks] => 3768   

    [memory_overhead] => 7205616 )

    So, it looks like I am using 74MB.

    Ah! I just realized, that I am using the 1.3.5.0 version (to test this issue) and it's still happening. Great.

  • DropPhone DropPhone

    360 Posts

    Microsoft

    Moderator

    Re: Locking Up / Not Responding / 500 Error

    Mar 07, 2014 02:28 AM|DropPhone|LINK

    Okay, I've written a torture test for wincache_ucache_set/wincache_ucache_get.  I won't post the php code here (it's trivial really), but let's just say populated the cache with 100,000 entries, and I have ~400 php.exe processes hammering on those entries.   They are definitely going cross-process, shared memory.  I've been able to drive the CPU on the box above 97%.  So far, no failures, no AVs.

    Remembering your need to have things expire, I am also setting the elements up with a 120 second expiry.  Even with things getting purged out of the cache, I still haven't hit a problem.

    I'll devise more brutal tests to see if I can trip up the cross-process shared memory locking system.

    Thx!

        --E.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 07, 2014 07:18 AM|laurin1|LINK

    Yesterday, I reset Wincache User Cache at 6:49PM. Right-before I reset Wincache User Cache, I opened the wincache.php page and as i stated, it was already not displaying the User Cache information. I found this in the PHP - FastCGI_errors.log:

    [06-Mar-2014 18:45:31 America/Chicago] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 220 bytes) in C:\inetpub\wwwroot\wincache.php on line 669
    [06-Mar-2014 18:45:35 America/Chicago] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 263 bytes) in C:\inetpub\wwwroot\wincache.php on line 669
    [06-Mar-2014 18:45:36 America/Chicago] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 263 bytes) in C:\inetpub\wwwroot\wincache.php on line 669

    Line 669 is

    $ucache_info     = wincache_ucache_info();

    There are places in my code where I use this function to get the age of the entry or to get the time left of the entry (ttl - age.) I ran some other tests and after it begins to fail, I get the same error:

    [05-Mar-2014 23:07:17 America/Chicago] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 45 bytes) in C:\inetpub\wwwroot\test2.php on line 11

    For this code:

    echo "Age: ".wincache_ucache_info(false, "xxxx")["ucache_entries"][1]["age_seconds"]."<br />";

    So, my theory is that we are not storing too much data for use in any function, except for this one - and that's because wincache_ucache_info(false) returns the entire cache - which is causes a memory overrun!

    Sound right?

  • Re: Locking Up / Not Responding / 500 Error

    Mar 07, 2014 07:20 AM|laurin1|LINK

    Actually, my age and ttl code looks like this:

    44      /**
    45       * @param mixed $key
    46       * @return int
    47       */
    48      public static function getTimeLeftInSeconds($key){
    49 
    50           return (int) wincache_ucache_info(false, $key)["ucache_entries"][1]["ttl_seconds"];
    51 
    52 
    53      }
    54 
    55      /**
    56       * @param mixed $key
    57       * @return mixed
    58       */
    59      public static function getAgeOfEntryInSeconds($key){
    60 
    61           return (int) wincache_ucache_info(false, $key)["ucache_entries"][1]["age_seconds"];
    62 
    63      }
  • Re: Locking Up / Not Responding / 500 Error

    Mar 07, 2014 07:26 AM|laurin1|LINK

    I'm certain that is the cause of User Cache not displaying on wincache.php once a point is reached. I'm not 100% about the lock-up / failures, we are getting, but I'd suspect it's in the same vein.

  • DropPhone DropPhone

    360 Posts

    Microsoft

    Moderator

    Re: Locking Up / Not Responding / 500 Error

    Mar 07, 2014 11:39 AM|DropPhone|LINK

    From: laurin1

     I found this in the PHP - FastCGI_errors.log:

    [06-Mar-2014 18:45:31 America/Chicago] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 220 bytes) in C:\inetpub\wwwroot\wincache.php on line 669

    Yes, I was able to reproduce this error on my tests when I had >50K items in the ucache.  I had to stop using wincache_ucache_info(true) in my scripts, since it was complaining I was exceeding my memory consumption for the process.

    This makes sense, since the call is trying to return *all* of the items in the cache as an array.  So it has to alloc & make a copy  of the entire ucache to do this.

    Glad we ran down this part of the problem!

    I'll still see if I can replicate the lock up problem.  Just out of curiosity, how many concurrent php-cgi.exe's are running on your system?  I'm assuming you have at least two, and possibly 10.  If it's more than 10, let me know.

    Thx!

        --E.

       

  • Re: Locking Up / Not Responding / 500 Error

    Mar 07, 2014 11:50 AM|laurin1|LINK

    So, even though we can raise the limit to 930, that's really unusable if we need to do anything with wincache_ucache_info(false), correct? That sucks.

    Right now, we have 5 php-cgi.exe processes running and that's about normal. During these lock-up periods, I saw a few more, but not sure the total count (maybe 10.)

  • DropPhone DropPhone

    360 Posts

    Microsoft

    Moderator

    Re: Locking Up / Not Responding / 500 Error

    Mar 07, 2014 12:34 PM|DropPhone|LINK

    The 930MB limit is in Wincache....however, Wincache is still just a guest in PHP's process, which is just a guest of the OS.  I'm not sure whether this limit is imposed by PHP or by the OS, but either way, Wincache is making a call to alloc something big, and PHP is saying, "No."

    I just looked through the implementation of wincache_ucache_info, and if you supply a $key, then Wincache will only fetch that one element.  So you shouldn't be trying to alloc the whole array in that case.

    Looking at your code, if $key is null, it looks like the wincache_ucache_info() function will try to get the whole cache.  Is the code that's calling getTimeLeftInSeconds() and getAgeOfEntryInSeconds() guaranteeing that $key is non-null in all cases?  You might consider adding an assert that $key is non-null.  Otherwise you run the risk of unknowingly hitting this alloc failure.

    Thx!

        --E.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 07, 2014 12:45 PM|laurin1|LINK

    Actually, that 128MB error is in the php.ini setting memory_limit, which is set to 128MB. I could raise that, and my system can definitely handle more, but definitely not 930MB for multiple scripts!

    DropPhone

    I just looked through the implementation of wincache_ucache_info, and if you supply a $key, then Wincache will only fetch that one element.  So you shouldn't be trying to alloc the whole array in that case.

    Hmm, well, that's easy enough to work around. I must be grabbing the whole array at times.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 10, 2014 12:02 PM|laurin1|LINK

    I think I just found my problem. I misunderstood what the first parameter actually is for and so all my calls, even to get individual entries were like this:

    wincache_ucache_info(false, $key)["ucache_entries"][1]["age_seconds"];

    And should be this:

    wincache_ucache_info(true, $key)["ucache_entries"][1]["age_seconds"];

    Because if this is set to true, then even only a summary is returned, not the whole array:

    summaryonly

           Controls whether the returned array will contain information about individual cache entries along with the user cache summary.      

    key

           The key of an entry in the user cache. If specified then the returned array will contain information only about that cache entry. If not specified and summaryonly is set to FALSE then the returned array will contain information about all entries in the cache.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 10, 2014 12:24 PM|laurin1|LINK

    I also modifed this in wincache.php:

    668      if($user_cache_available && ($cache_data == SUMMARY_DATA || $cache_data == UCACHE_DATA)){
    669 
    670           $memory_limit = "256";
    671 
    672           ini_set("memory_limit", $memory_limit."M");
    673 
    674           $ucache_mem_info = wincache_ucache_meminfo();
    675 
    676           if(is_ucache_under_php_memory_limit($ucache_mem_info, $memory_limit))
    677                $ucache_info = wincache_ucache_info();
    678 
    679      }

    691 function is_ucache_under_php_memory_limit($ucache_mem_info, $memory_limit){
    692 
    693      return
    694           (($ucache_mem_info['memory_total'] - $ucache_mem_info['memory_free']) / 1024) <
    695           ($memory_limit - 12) * 1000;
    696 }

  • DropPhone DropPhone

    360 Posts

    Microsoft

    Moderator

    Re: Locking Up / Not Responding / 500 Error

    Mar 10, 2014 02:27 PM|DropPhone|LINK

    Great!

    After your changes, are you still seeing a problem with locking up?

    Thx!

         --E.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 10, 2014 06:05 PM|laurin1|LINK

    Usually takes a few days, but I'm fairly certain that is the problem.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 24, 2014 01:32 PM|laurin1|LINK

    Darn, just happened again.

  • DropPhone DropPhone

    360 Posts

    Microsoft

    Moderator

    Re: Locking Up / Not Responding / 500 Error

    Mar 24, 2014 01:42 PM|DropPhone|LINK

    Get a dump!  Get a dump!  Get a dump!

    Thx!

        --E.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 24, 2014 02:06 PM|laurin1|LINK

    I told you I don't know how - I've tried, but have been completely unsuccesful at this in the past. The documentation that I've read on PHP's website is either incomplete or incorrect.

  • DropPhone DropPhone

    360 Posts

    Microsoft

    Moderator

    Re: Locking Up / Not Responding / 500 Error

    Mar 24, 2014 05:18 PM|DropPhone|LINK

    In task manager (taskmgr), go to the tab that has all the process listed.

    rght-click on the php-cgi.exe process, select "create dump file".  There will be a dialog that comes up after the dump is written with a path to the dump.  It's usually in wherever the %TMP% or %TEMP% direcory is for the current user.  The file usually ends in .DMP.

    That's it!

    Thx!

        --E.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 24, 2014 05:38 PM|laurin1|LINK

    I'm confused. When do I do this? And which PHP-cgi.exe process do I use (when this is happening, there are about 30!)
  • Re: Locking Up / Not Responding / 500 Error

    Mar 24, 2014 05:39 PM|laurin1|LINK

    When it's locked up?
  • DropPhone DropPhone

    360 Posts

    Microsoft

    Moderator

    Re: Locking Up / Not Responding / 500 Error

    Mar 24, 2014 06:44 PM|DropPhone|LINK

    1. Yes, when it's locked up

    2. Yes, *all* of them.  We have to find out if *any* of them are holding the cross-process write lock.

    It's a tedious process, I know, but it's the only way to be sure....

        --E.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 25, 2014 06:09 PM|laurin1|LINK

    Happened again today.

    There is no way to do that using the manual means as there are way too many (too much downtime) and they keep spawning and hard to track which ones I've dumped.

    I did dump a few that were "active." Most of the processes showed 0 CPU usage during this period, but 1 was always spiking at around 25% CPU usage. I dumped several of those. However, we have another problem. I opened those dumps and all have this error message in Visual Studio:

    You cannot debug a 64-bit dump of a 32-bit process , you must collect  a 32 bit dump of a 32 process

    I did, however, watch the Work Set numbers for some of these processes spike to over 700MB!

  • Re: Locking Up / Not Responding / 500 Error

    Mar 25, 2014 06:15 PM|laurin1|LINK

    FYI, one of my staff turned me on to these, which I am going to try. The first one will get 32 bit dumps by default.

    http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

    ProcDump v6.0

    ProcDump is a command-line utility whose primary purpose is monitoring an application for CPU spikes and generating crash dumps during a spike that an administrator or developer can use to determine the cause of the spike. ProcDump also includes hung window monitoring (using the same definition of a window hang that Windows and Task Manager use), unhandled exception monitoring and can generate dumps based on the values of system performance counters. It also can serve as a general process dump utility that you can embed in other scripts.

    http://www.addictivetips.com/windows-tips/pzendump-create-memory-dump-for-multiple-processes-in-batch/

    PZenDump: Create Memory Dumps For Multiple Processes In Batch

  • Re: Locking Up / Not Responding / 500 Error

    Mar 26, 2014 10:03 AM|laurin1|LINK

    The situation is degrading rapidly. It crashed 5 days ago, then 2 days ago, then yesterday, then this morning and it's starting to fail again right now. We are in serious trouble.

    Now, for the really bad news. As it was crashing last time, I was able to pull up wincache.php and Wincache User Cache was only using about 38MB of RAM!!

    However, we got several timeouts that referrered to Wincache User Cache get and set lines of code. So we still believe the issue is Wincache, but we no longer believe it's due to high memory usage.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 26, 2014 10:13 AM|laurin1|LINK

    I've lowered our User Cache limit to 80 - I have to try something, as it's becoming unusable and our system relies heavily on this.

    Strange, once I lowered the value, phpinfo() displays the new value, but wincache.php still displays the old value.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 26, 2014 01:25 PM|laurin1|LINK

    All day today we are getting PHP script timeouts (referringt to Wincache User Cache gets and sets) and Working Set Maximums for php-cgi.exe processes are still spiking at ridiculous levels (some over 400MB!!). I've reset IIS several times, and the problem begins to occur again within minutes.

    I have at least confirmed that the issue is with Wincache. I have effectively disabled Wincache User Cache and performance is slow but stable.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 27, 2014 12:16 AM|laurin1|LINK

    I found an article (old) that states that Wincache User Cache does not behave the way the manual states:

    "If the total size of variables stored in the user cache exceeds the specified value, then the most stale variables will be removed from the cache."

    That instead of this limit is hit, the next set will fail!

  • DropPhone DropPhone

    360 Posts

    Microsoft

    Moderator

    Re: Locking Up / Not Responding / 500 Error

    Mar 27, 2014 02:04 PM|DropPhone|LINK

    laurin1

    I opened those dumps and all have this error message in Visual Studio:

    You cannot debug a 64-bit dump of a 32-bit process , you must collect  a 32 bit dump of a 32 process

    Yeah, you'll need windbg.exe to view these dumps.  I think there's a stand-alone part of the Win32 SDK that has just the debugger package (windbg/ntsd/cdb).

    In any case, message me privately as to where I can get these dumps, and I'll see what we can see.

    I'm assuming you're on the 1.3.5.2 php_wincache.dll, yes?

        --E.

  • Re: Locking Up / Not Responding / 500 Error

    Mar 27, 2014 03:58 PM|laurin1|LINK

    No, I verified that - you cannot use the Task Manager to save a dump from a  32 bit process, because Task Manager is 64 bit. I was able to get some good dumps using ProcMon.

    I'm using the version you sent me via PM yesterday (or 2 days ago, can't remember.)

  • Re: Locking Up / Not Responding / 500 Error

    Mar 28, 2014 02:57 PM|laurin1|LINK

    I figured this information might be useful to everyone, Eric, so I'm posting here instead of our private:

    So, I was attempting to get dumps from the php-cgi.exe processes when the issue occurred again, which it did today. Because there are so many processes, I wrote a script that grabs all of the PID's and runs procdump on all of these. My attempt today failed, which partially my fault. My first script ran for all instances concurrently - BAD IDEA. Completely locked up my server and was barely able to even restart Windows without having to do a hard restart (power off.) However, after it restarted, and while the server was performing normally, I fixed my script and ran it again, just to test it (ran procdump against one PID at a time.) This worked great....until it was complete. First using dumping these processes causes the memory Working Set Peak number to spike to what appears to be the user cache limit for wincache (ours is set at 1000MB) plus some overhead (the number is about 1.2GB.) This by itself should not be a problem for this machine (24GB of RAM, probably 20GB available.) This in turn creates dump files of about 1.2GB each, but this happens pretty quickly. But what happened next was a the thing that really suprised me.

    AFTER this was almost complete, as the last file was being written, the server almost came to a halt. Couldn't open any new windows and explorer.exe was locked up. The web application slowed down, but did not lock up completely.

    I was watching Resource Monitor and saw the disk was backing up big time - and it looked like the culprit was Windows Defender! I stopped and disabled Windows Defender service and whala! The server resumed normal behavior (processes dropped Worker Set numbers, everything.)

    For some reason, this made me thing about Data Execution Prevention and so I checked the setting for APP01 and for some reason, the DEP was set to "Turn on DEP for all programs and services except those I select: ", with nothing in the box below. This is not the default setting, right (not 100% about that, but I think so)? I checked our failover server (also used to run process scripts), APP02, and APP02 was configured for "Turn on DEP for essential Windows programs and services only."

    I configured APP01 for what I consider to the "correct" setting and disabled Windows Defender on both machine.

    I'm not saying that either of these settings are the problem, but it sure sounds like one or both of them are (or at least contributing factors.)

  • Re: Locking Up / Not Responding / 500 Error

    Mar 28, 2014 02:58 PM|laurin1|LINK

    I haven't been able to restart APP01 yet for the DEP setting, but the Windows Defender issue has not fixed it...because it's happening again right now!