Hanging problem and "Service unavailable"RSS

14 replies

Last post Feb 12, 2010 11:34 AM by don.raman

  • Hanging problem and "Service unavailable"

    Dec 04, 2009 11:23 AM|patlefort|LINK

    Hi, I was very pleased to see a "production ready" cache extension for PHP 5.3 on Windows which I happily tested on my dev server without problem. However, on our production server (an intranet with medium load), it randomly "hang". All PHP processes stop responding, sometimes one of them will be stuck on something using 50% of the cpu and at the base of the thread stack (with Process Explorer) is the php_wincache.dll. I have to manually kill all of them or the whole IIS stop responding. The browser try to load for a while and I get a "Service unavailable", even on static content. I kill them all, restart the service and it will work fine for an hour or 2 before it hang again. It does not coincides with app pool recycling as it happens even when it is turned off.

    My configuration is the following:

    • PHP 5.3.1 with a few extensions compiled myself with VC8. Compiled to use libmysql instead of mysqlnd. Non thread-safe.
    • FastCGI 1.0 with only one application pool
    • IIS 6 with only one application pool
    • Windows Server 2003 SP2
    • Wincache 1.0.1117.0 (PHP_WINCACHE_EXTVER value)
    Is it a problem to compile wincache with VC8?

    I ran into another problem: I would get 500 FastCGI error when compiling PHP to use libmysql instead of mysqlnd if I loaded the php_memcache.dll extension before php_wincache.dll. The error would go away if I moved php_memcache.dll after php_wincache.dll in my php.ini. I don't know if it has something to do with my current problem but there is clearly a bug somewhere.

    I don't have much more clues, is someone else having this problem?

    I have turned off wincache and all seems to be working for now, although slower. I cannot wait anymore longer for PHP 5.3, I need to slap some senses into my code with namespacing.
  • Re: Hanging problem and "Service unavailable"

    Dec 04, 2009 12:22 PM|don.raman|LINK

    Can you follow the post at http://forums.iis.net/t/1163114.aspx and see if this solves your problem. Ideally just reinstall WINCACHE from the location mentioned in the post and see if you are still getting the crash. If yes, please report it back. If this fixes your problem, it will be good to hear that too.

    Thanks,

    Don.

  • Re: Hanging problem and "Service unavailable"

    Dec 04, 2009 01:54 PM|patlefort|LINK

    I am compiling wincache myself and I don't see any difference in the source from the PECL site and the one I am using. Was the source also updated? I cannot use the pre builded binaries since I am compiling PHP with VC8, which I need to do to compile a PHP extension I made for our intranet.

     

  • Re: Hanging problem and "Service unavailable"

    Dec 04, 2009 01:57 PM|don.raman|LINK

    Yes, all the sources are also up-to-date. You should sync the sources and re-build.

    Thanks,

    Don.

  • Re: Hanging problem and "Service unavailable"

    Dec 10, 2009 04:16 PM|patlefort|LINK

    In any cases, I am going to wait for the next update of wincache to try it again in production use. Since the source from which I compiled is the same (I ran a script to compare them), I strongly doubt it will make a difference and our intranet is too important right now. I wasn't able to reproduce the problem on my dev server even running stress tests. It might be related to some specific script somewhere.

    I'll try some more tests on the production server when I have time.

    Thank you for your work, I hope most of the issues can be fixed.

  • Re: Hanging problem and "Service unavailable"

    Dec 18, 2009 04:53 PM|patlefort|LINK

    A little more information:

    It finally happened on my developement server. It happened when I tried sending a file through a POST request which is handled by some PHP scripts. I tried the same thing after restarting the service and it didn't happen that time, so it's very unpredictable.

    A screenshot of the state of one hanging PHP process:

    Notice that all the PHP process are all below despite restarting IIS.

    Here's the stack of the first thread:

     

    It seems to be stuck on WaitForMultipleObjects which conditions to return are never met.

  • Re: Hanging problem and "Service unavailable"

    Dec 18, 2009 05:57 PM|don.raman|LINK

    Hi,

    Seems some kind of deadlock. We are working on version 1.0.1 of WINCACHE. I will let you know once we have the release candidate ready so that you can verify it.

    Thanks for reporting this issue and helping us make WINCACHE a better caching solution.

    Don.

  • Re: Hanging problem and "Service unavailable"

    Jan 24, 2010 12:08 AM|don.raman|LINK

    Hi,

    We believe we have a solution for that. We are in the process of releasing WINCACHE 1.0.1 version with this fix. It will be good if you can test the private version of WINCACHE with the fix. This will be a win-win situation for both of us as you will get a chance to ensure that the fix is indeed working for you and if not we can address your concern too so that you can run WINCACHE smoothly. For us, it will be great to have a real customer validation. We have already got it validated by few customers. it would be great if you can help us in making WINCACHE a great caching solution on WINDOWS.

    For getting the privates, please mail me at DONRAMAN[@]MICROSOFT[dot]COM and let me know if you need the privates for PHP 5.2 or PHP 5.3.

    Again thanks in advance for helping us.

    Regards,

    Don.

  • Re: Hanging problem and "Service unavailable"

    Jan 26, 2010 01:11 PM|patlefort|LINK

     The problem seems to be solved so far, it passes a stress test that always caused the issue.

    I still have one little problem though: Sometimes, after a stress test, I am unable to restart IIS as IIS is unable to stop any of the php processes which I have to terminate manually. However, I can still navigate the site before restarting the service.

  • Re: Hanging problem and "Service unavailable"

    Jan 26, 2010 01:53 PM|don.raman|LINK

    Thank you a lot for providing feedback. We have released the WINCACHE 1.0.1 version and it can be found at http://sourceforge.net/projects/wincache/.

    I will dig into your other problem. I have never faced that. BTW, if you just leave the machine/site idle for sometime the PHP process should go away automatically. Just check if this is happening or not. This will happen based on what value has been set foe Idle Timeout.

    Thanks,

    Don.

  • Re: Hanging problem and "Service unavailable"

    Jan 26, 2010 04:03 PM|patlefort|LINK

    I have upgrade my FastCGI to version 1.5 and now it happens everytime with or without wincache. So it seems to be related to FastCGI.

  • Re: Hanging problem and "Service unavailable"

    Jan 26, 2010 04:46 PM|patlefort|LINK

    Sigh... it seems to be related to the Process Explorer application. Pretty ironic, the monitoring application causing the problem. I remembered that it did cause me some weird problems when I left it open in the past. It just seem to keep some IO handles open and it sometimes get in the way of other applications.

    We can consider the problem solved for now. Thank you for your work :)

  • Re: Hanging problem and "Service unavailable"

    Feb 11, 2010 03:26 PM|patlefort|LINK

     Hi again, I have been testing Wincache 1.0.1 some more and I have experienced a few 500 FastCGI errors. It is very random and it does on every subsequent requests from any connections until the opened connection or session from the client who first got the error is closed.

    The FastCGI error number is 0x80004005. A non specified error.

    The event viewer reports the following error (in french) when I try to load a page from the network:

    Type de l'événement :    Informations
    Source de l'événement :    Application Error
    Catégorie de l'événement :    (100)
    ID de l'événement :    1004
    Date :        2010-02-11
    Heure :        14:03:37
    Utilisateur :    N/A
    Ordinateur :    CRDI1-INTRATST
    Description :
    Erreur de file d'attente de rapport : application défaillante php-cgi.exe, version 5.3.1.0, module défaillant unknown, version 0.0.0.0, adresse de défaillance 0x00000000.

    Pour plus d'informations, consultez le centre Aide et support à l'adresse http://go.microsoft.com/fwlink/events.asp.
    Données :
    0000: 41 70 70 6c 69 63 61 74   Applicat
    0008: 69 6f 6e 20 46 61 69 6c   ion Fail
    0010: 75 72 65 20 20 70 68 70   ure  php
    0018: 2d 63 67 69 2e 65 78 65   -cgi.exe
    0020: 20 35 2e 33 2e 31 2e 30    5.3.1.0
    0028: 20 69 6e 20 75 6e 6b 6e    in unkn
    0030: 6f 77 6e 20 30 2e 30 2e   own 0.0.
    0038: 30 2e 30 20 61 74 20 6f   0.0 at o
    0040: 66 66 73 65 74 20 30 30   ffset 00
    0048: 30 30 30 30 30 30         000000  

    The following error is reported when I load a page from the server itself (through localhost):

    Type de l'événement :    Erreur
    Source de l'événement :    Application Error
    Catégorie de l'événement :    (100)
    ID de l'événement :    1000
    Date :        2010-02-11
    Heure :        14:39:57
    Utilisateur :    N/A
    Ordinateur :    CRDI1-INTRATST
    Description :
    Application défaillante php-cgi.exe, version 5.3.1.0, module défaillant unknown, version 0.0.0.0, adresse de défaillance 0x00000000.

    Pour plus d'informations, consultez le centre Aide et support à l'adresse http://go.microsoft.com/fwlink/events.asp.
    Données :
    0000: 41 70 70 6c 69 63 61 74   Applicat
    0008: 69 6f 6e 20 46 61 69 6c   ion Fail
    0010: 75 72 65 20 20 70 68 70   ure  php
    0018: 2d 63 67 69 2e 65 78 65   -cgi.exe
    0020: 20 35 2e 33 2e 31 2e 30    5.3.1.0
    0028: 20 69 6e 20 75 6e 6b 6e    in unkn
    0030: 6f 77 6e 20 30 2e 30 2e   own 0.0.
    0038: 30 2e 30 20 61 74 20 6f   0.0 at o
    0040: 66 66 73 65 74 20 30 30   ffset 00
    0048: 30 30 30 30 30 30         000000 
     

    The DEP (data execution protection) kicks in and stop PHP too. Something is trying to execute code at offset 0.

    If I remove wincache from php.ini, pages load correctly. If I put wincache back, I get the error again, that is until the connection to the webserver is closed.

    I have upgraded to FastCGI 1.5 lately, I am using the following configuration:

    [Types]

    *=PHP

    [PHP]

    ExePath=C:\php-5.3.1\php-cgi.exe
    ;IgnoreExistingFiles=1
    ActivityTimeout=36180
    InstanceMaxRequests=10000
    RequestTimeout=3600
    MaxInstances=0
    ResponseBufferLimit=0
    ;FlushNamedPipe=1
    ;SignalBeforeTerminateSeconds=6
    EnvironmentVars=PHP_FCGI_MAX_REQUESTS:11000,MAGICK_HOME:C:\ImageM~1

    The stack of php-cgi thread:

     

    The stack from MSVCR80.dll:

     

  • Re: Hanging problem and "Service unavailable"

    Feb 11, 2010 04:52 PM|patlefort|LINK

     Hi again, I have been testing Wincache 1.0.1 some more and I have experienced a few 500 FastCGI errors. It is very random and it does on every subsequent requests from any connections until the opened connection or session from the client who first got the error is closed.

    The FastCGI error number is 0x80004005. A non specified error.

    The event viewer reports the following error (in french) when I try to load a page from the network:

    Type de l'événement :    Informations
    Source de l'événement :    Application Error
    Catégorie de l'événement :    (100)
    ID de l'événement :    1004
    Date :        2010-02-11
    Heure :        14:03:37
    Utilisateur :    N/A
    Ordinateur :    CRDI1-INTRATST
    Description :
    Erreur de file d'attente de rapport : application défaillante php-cgi.exe, version 5.3.1.0, module défaillant unknown, version 0.0.0.0, adresse de défaillance 0x00000000.

    Pour plus d'informations, consultez le centre Aide et support à l'adresse http://go.microsoft.com/fwlink/events.asp.
    Données :
    0000: 41 70 70 6c 69 63 61 74   Applicat
    0008: 69 6f 6e 20 46 61 69 6c   ion Fail
    0010: 75 72 65 20 20 70 68 70   ure  php
    0018: 2d 63 67 69 2e 65 78 65   -cgi.exe
    0020: 20 35 2e 33 2e 31 2e 30    5.3.1.0
    0028: 20 69 6e 20 75 6e 6b 6e    in unkn
    0030: 6f 77 6e 20 30 2e 30 2e   own 0.0.
    0038: 30 2e 30 20 61 74 20 6f   0.0 at o
    0040: 66 66 73 65 74 20 30 30   ffset 00
    0048: 30 30 30 30 30 30         000000  

    The following error is reported when I load a page from the server itself (through localhost):

    Type de l'événement :    Erreur
    Source de l'événement :    Application Error
    Catégorie de l'événement :    (100)
    ID de l'événement :    1000
    Date :        2010-02-11
    Heure :        14:39:57
    Utilisateur :    N/A
    Ordinateur :    CRDI1-INTRATST
    Description :
    Application défaillante php-cgi.exe, version 5.3.1.0, module défaillant unknown, version 0.0.0.0, adresse de défaillance 0x00000000.

    Pour plus d'informations, consultez le centre Aide et support à l'adresse http://go.microsoft.com/fwlink/events.asp.
    Données :
    0000: 41 70 70 6c 69 63 61 74   Applicat
    0008: 69 6f 6e 20 46 61 69 6c   ion Fail
    0010: 75 72 65 20 20 70 68 70   ure  php
    0018: 2d 63 67 69 2e 65 78 65   -cgi.exe
    0020: 20 35 2e 33 2e 31 2e 30    5.3.1.0
    0028: 20 69 6e 20 75 6e 6b 6e    in unkn
    0030: 6f 77 6e 20 30 2e 30 2e   own 0.0.
    0038: 30 2e 30 20 61 74 20 6f   0.0 at o
    0040: 66 66 73 65 74 20 30 30   ffset 00
    0048: 30 30 30 30 30 30         000000
     

    The DEP (data execution protection) kicks in and stop PHP too. Something is trying to execute code at offset 0.

    If I remove wincache from php.ini, pages load correctly. If I put wincache back, I get the error again, that is until the connection to the webserver is closed.

    I have upgraded to FastCGI 1.5 lately, I am using the following configuration:

    [Types]

    *=PHP

    [PHP]

    ExePath=C:\php-5.3.1\php-cgi.exe
    ;IgnoreExistingFiles=1
    ActivityTimeout=36180
    InstanceMaxRequests=10000
    RequestTimeout=3600
    MaxInstances=0
    ResponseBufferLimit=0
    ;FlushNamedPipe=1
    ;SignalBeforeTerminateSeconds=6
    EnvironmentVars=PHP_FCGI_MAX_REQUESTS:11000,MAGICK_HOME:C:\ImageM~1

    The stack of php-cgi thread:

     

    The stack from MSVCR80.dll:

     
  • Re: Hanging problem and "Service unavailable"

    Feb 12, 2010 11:34 AM|don.raman|LINK

    Hi,

    This doesn't seem like a WINCACHE crash as the stack trace is not showing WINCACHE DLL. I understand that when you unload WINCACHE problem is gone. In that case we would like to get a crash dump in order to work on this. If you can get a dump please send it to us and file a bug at http://pecl.php.net/bugs/search.php?cmd=display&status=Open&package_name[]=WinCache.

    I would also advise you to switch to PHP5.2.12 and see if the crash is still there. This is all probability seems like PHP crash. I believe you are compiling PHP/WINCACHE yourself. Please compile 5.2 sources and let us know the result.

    Thanks,

    Don.