We are excited to announce that the IIS.NET Forums are moving to the new Microsoft Q&A experience. Learn more >

A strange freeze problemRSS

22 replies

Last post Aug 24, 2010 01:34 AM by qbernard

  • A strange freeze problem

    Jan 11, 2010 10:48 AM|stuartlodge|LINK

    Hi I've got an interesting issue at the moment. Basically my Server 2008 box is serving two .Net 3.5 sites from IIS7. Every few hours, one of these sites stops working - basically requests from external locations connect but no data is returned - so they just sit there with the browser waiting for data to be returned - regardless of which page is requested. During this "freeze" time: - I can use the other website fine from any browser. - If I look at the website using a browser opened on that server PC, then the website responds normally. The only way I've found so far is to resolve this is to recycle the application pool attached to that problem site. This has left me somewhat confused: - It feels like this must be a problem with the website application (since it's cured by the app pool recycling) - Also I believe this behaviour started just after we changed some ASP.Net code. - But I'm not sure why can I browse the website locally, but not remotely. Has anyone got any clues that might help me here? What sort of application problem could cause this? What sort of tools can I use to debug the issue? Thanks for any suggestions Stuart
  • Re: A strange freeze problem

    Jan 12, 2010 01:12 AM|qbernard|LINK

    Mmm.. it could be page access to some systems that take some time for remote request? Anyway, you can take a look at the IIS log file to see what's the time-taken for remote request. To debug you can first start with failed request tracing feature in IIS, and you can also try to get debugdiag to debug this.

    Cheers,
    Bernard Cheah
  • Re: A strange freeze problem

    Jan 12, 2010 06:01 AM|stuartlodge|LINK

    Thanks I'm really not sure what's going on at all. I've looked in the IIS logs and it's not clear what the problem is - I don't think IIS records the records of the requests until after they've completed - so whatever is getting "stuck" isn't showing up! There's nothing in failed request tracing either :/ (I have this switched on) I've just downloaded the debug diag toolkit - I'll see if it can help Thanks! Stuart
  • Re: A strange freeze problem

    Jan 13, 2010 05:06 AM|qbernard|LINK

    You can also take a look at the httperr log from http.sys.

    Cheers,
    Bernard Cheah
  • Re: A strange freeze problem

    Jan 13, 2010 07:31 AM|stuartlodge|LINK

    Thanks for the pointers. I've taken a look in that file... When the problem hits there are a whole queue of errors like Connection_Abandoned_By_ReqQueue - but I suspect these are symptoms rather than the cause - these errors seem to occur at the same exact time as a Windows Event Log message about the exe not shutting down within the time limits (so that process probably got killed causing the Connection_Abandoned_By_ReqQueue logs?) Still really have no clue what is happening! Still hunting for whatever is causing it... One question that might help - when using the diag tool, how do I get it to include my .pdb symbols in its call stack traces? Stuart
  • Re: A strange freeze problem

    Jan 15, 2010 01:13 AM|qbernard|LINK

    Mm.. the Connection_Abandoned_By_ReqQueue is new and it is only available for IIS7.
    For the symbols, you can either download it yourself, or you let DebugDiag download it automatically, just make sure the box has internal access.

    Cheers,
    Bernard Cheah
  • Re: A strange freeze problem

    Jan 15, 2010 09:34 AM|stuartlodge|LINK

    I've run several logs The DebugDiag analysis shows nothing. And I've now looked at the dumps in WinDbg - and can't spot anything (although I'm not really sure I know what I'm looking for!) Looking through the call stacks, there are only two threads actually doing any code execution: - the first is a DotNetNuke scheduler thread - sat in a Sleep - the second is the finalizer thread - which doesn't seem to be doing anything (it's not in a CLR stack) I'm still really lost in what's causing this... but will keep looking... Any ideas very much appreciated. What most confuses me is why I can browse the site on the server but can't remotely - but that it's definitely not a network issue - as the server is responding fine to other websites and as a forced recycle cures the problem. Thanks for the help so far - any clues anyone has much appreciated! Stuart
  • Re: A strange freeze problem

    Jan 15, 2010 12:59 PM|stuartlodge|LINK

    One thing I seem to be seeing is.... That just before each crash I seem to be seeing some hits from browsers with User-Agent GbPlugin From reading up, GbPlugin seems to be some sort of Brazilian anti-fraud thing. I haven't seen any reports of GbPlugin causing issues - but could it be that this browser is somehow blocking my process? Very puzzled Stuart
  • Re: A strange freeze problem

    Jan 15, 2010 01:50 PM|stuartlodge|LINK

    In particular I'm seeing some errors like: 2010-01-15 12:03:52 95.172.9.66 GET /MyDir/resource.ashx?u=3474 - 80 - 187.88.111.248 GbPlugin 400 0 2148734208 457 These come in immediately before the site disappears. I don't yet have an FREB file for this 400 (but I've upped my file limits - so next time...) - and there's nothing in the HTTPErr#.log files about this. What seems to be happening is that this GbPlugin User Agent requests are somehow jamming the HTTP connection of w3wp with the public IP address - the w3wp.exe is *not* hung - it can still answer requests from browsers locally on the server and if I attach windbg it just looks like a normally operating exe. As I've said recycling the application pool clears the problem. Does that make any sense at all? Is there any way of turning on super logging so that I can capture the requests as they arrive? I guess my next step might be to try blocking these GbPlugin requests... but I'm worried the fault here might not be in my client code.... :/
  • Re: A strange freeze problem

    Jan 18, 2010 05:23 AM|JonHawkins|LINK

    Hi, We are also experiencing a very similar problem to this. We have web two servers behind a load balancer, periodically either one will stop responding to external requests. In the hosts file of each machine there is a record to point the external URL to the local IP of the machine, (due to DNS issues inside the LAN). When the site is browsed from the local machine it will still respond, but if you try and view it from the web or from another machine within the LAN it will not respond, nothing comes back at all. To fix it we recycle the app pool and all is well again. We are at a loss as to why this is happening, any and all advice and help would be most gratefully received. We are an E-Commerce site running WS2003 SP1, on IIS6. We have multiple sites on IIS and all the other sites will continue to respond during this time. Thanks
  • Re: A strange freeze problem

    Jan 19, 2010 08:57 AM|stuartlodge|LINK

    I've gotten a bit further in diagnosing our problem. It is definitely in our case to do with two particular Brazilian users - if they access the site at work then their local network routes HTTP through a system called WEBWASHED which includes GBPlugIn - and this leads to our symptoms. If the same users perform the same requests on the same PCs on a different network, then we have no problems. It's still definite to me that the ASP.Net application is continuing to function perfectly - further debugging with WinDbg seems pointless :/ Anyone got any ideas? (I've already asked those two users not to login from work - but I'd rather have a solution)
  • Rovastar Rovastar

    5495 Posts

    MVP

    Moderator

    Re: A strange freeze problem

    Jan 19, 2010 10:24 AM|Rovastar|LINK

     "2010-01-15 12:03:52 95.172.9.66 GET /MyDir/resource.ashx?u=3474 - 80 - 187.88.111.248 GbPlugin 400 0 2148734208 457"

    Are you getting this 400 for the http status code in the IIS logs? What fields do you have logged?

    These should be stopped by the http.sys logs. 

    The win32 error code (which I presume it is in the logs)  I think is:

    2148734208 COR_E_EXCEPTION: Base class for all exceptions in the runtime

    But tbh that doesn't give me many clues but may help others.

    What other technologies are you using any COM or just pure .NET 3.0

    I would also get the raw data via a packet sniffer lliek netmon  / and web diagnotistc tools like fiddler of the perceived Brazilian problem request and subsequent problem requests.

    Troubleshoot IIS in style
    https://www.leansentry.com/
  • Re: A strange freeze problem

    Jan 20, 2010 11:27 AM|stuartlodge|LINK

    Thanks Rovastar Yes - those logs are in the IIS logs - nothing in the http.sys logs The app is pure .NET 3.5 - it uses a SQL2008 back end and some web services - but does not access any COM components Fiddler's of no use - apparently these requests might be going through something called WEBWASHED on the work network between the Brazillian user's PC and their local Internet connection. For now I've gotten those users to hold off accessing the site at work - and the result has been no new crashes... To be honest I don't want to install a winpcaps packet sniffer on my production server... but if I have no other choice then I guess I'll have to.
  • Re: A strange freeze problem

    Jan 20, 2010 11:06 PM|qbernard|LINK

    Sorry, am travelling at the moment. what mode did you set for debugdiag? i think this is hang case, and you should get the analysis + the dump file. To drill in further, you should engage microsoft support to help debug the app.
    Cheers,
    Bernard Cheah
  • Re: A strange freeze problem

    Feb 25, 2010 03:11 AM|stuartlodge|LINK

    Hmmmm..... After a month of quiet it looks like it came back to bite me last night :/ I think it's time to try Microsoft Support as you suggested.
  • Re: A strange freeze problem

    Mar 03, 2010 07:24 AM|qbernard|LINK

    LOL.. came back? any luck so far?
    Cheers,
    Bernard Cheah
  • Re: A strange freeze problem

    Mar 09, 2010 04:01 PM|caliel.costa|LINK

    Hello stuartlodge I have exactly the same problem, started yesterday (march 08). My problem is a little bigger, I'm from Brazil, and I can't block the IP of my customers. Any ideia about the solution??
  • Re: A strange freeze problem

    Mar 16, 2010 03:18 AM|qbernard|LINK

    No update from OP yet. You mentioned it started on the 8th March, so was it working before that or ?

    Cheers,
    Bernard Cheah
  • Re: A strange freeze problem

    Aug 11, 2010 06:48 PM|roniegh|LINK

    We have the same issues using our web ERP/WMS system on Internet Explorer when Gbplugin is installed. 

    Some info about Gbuster plugin (Gbplugin): http://www.gastecnologia.com.br/site/en/Product.aspx
    It installs as a device driver and is almost impossible to uninstall.

    Some of the banks in Brazil using Gbplugin are Banco do Brasil, Caixa Economica Federal, Sicredi and ABN/Real.
    At least with Banco do Brasil, users are required to install Gbuster, otherwise they cannot access their account.

    The problems with this Gbuster is that it makes Internet Explorer really slow, specially with websites which have javascript/ajax.

    Here are some links talking about problems with Internet Explorer caused by Gbuster (sorry, some of the info is in Portuguese):
    http://social.technet.microsoft.com/Forums/pt-BR/winxppt/thread/6bbe3f9f-7e3f-46dd-b192-d46ef83b4137
    http://www.guiadohardware.net/comunidade/programa-banco/922925/
    http://insanebits.blogspot.com/2007/04/g-buster-browser-defense-analysis-and.html

  • Re: A strange freeze problem

    Aug 16, 2010 07:40 AM|caliel.costa|LINK

    Hi, this problem has been my nightmare for a week. After many test and nothing, the solution was .... upgrade Windows Server 2008 that was 3 months out of date. Simply that.. :D But thanks a lot by this post, clarified some parts that had not understood.
  • Re: A strange freeze problem

    Aug 17, 2010 03:14 AM|qbernard|LINK

    Oohh. out of date? so just patch and reboot?
    Cheers,
    Bernard Cheah
  • Re: A strange freeze problem

    Aug 17, 2010 07:24 AM|caliel.costa|LINK

    LOL.... Update, reboot and problem solved.... cool huh? After this incident, when a customer has a problem, the first thing I look is Windows Update (Lesson Learned), even if problem is with Adobe Photoshop
  • Re: A strange freeze problem

    Aug 24, 2010 01:34 AM|qbernard|LINK

    Hehee... good key learning.
    Cheers,
    Bernard Cheah