« Previous Next »

• 12-20-2008, 3:15 PM

  Jemarah Joined on 12-20-2008, 4:34 AM Posts 4

  WebSite Deployment Issue Reply Contact Summary:  I have an inability to roll out ASP.NET assemblies to IIS 6.0 without website Slowing to a virtual halt.   (Framework 3.5/IIS 6.0/ Server 2003) Detail : Site works on individual servers (via IpAddress) but when running on Web Server farm through Domain, current anon users goes through the roof and the site becomes unusable.  Typical roll out summary: take 2 of our 4 webservers OOS leaving site running on 2 . Update OOS Servers with roll out 'Wake up'  (IIS recompile) these servers, all seems fine Put these servers back in service, all OK Take 2 original servers OOS, Current Anon goes ballistic and the site slows to a grind. (Anon per second is OK, processor is high) IF we roll back the 2 updated servers they work fine. We have been experiencing this problem for about 3 weeks now and, as you can imagine, the inability to update has caused a bit of a bottle neck.  We are a global site so there is always heavy traffic, and have always updated this way with out any problem. Interestingly the problem is not linked to any component.  I have tried updating different components on an individual basis and the same thing happens, rolling back is always fine.  I recently accdentally rolled out a debug version of a component (naughty) , the site worked fine, however updating that component with a realease version (no code change)  caused this problem.  It's like it's constantly re-compiling!. Potential problems: I know there is a component which was rolled out 3 weeks ago, that is erroring as it can't find a method on one of it's dependancies.  The site has worked fine with this errant component in place, could this be causing the update problem?. The bin is large and contains some renamed dlls.  I am intending on cleaning it up next week and the site has run fine with these in place. Any ideas?

  Tags: .netwebsite

• 12-20-2008, 11:41 PM

  In reply to

  Rovastar Joined on 03-13-2008, 10:00 AM London, UK Posts 1,744	Re: WebSite Deployment Issue Reply Contact I don't undersatnd what you mean by IIS recompile? COuld you explain. When you get anonymous users what are they looking at? What do tehy logs say? Multiple users? 1 user? To be honest it sound like a some sort of app issue and getting in a loop or maybe you load balancer/proxy is resending teh info again and again. http://www.iisportal.com

• 12-21-2008, 3:42 AM

  In reply to

  Jemarah Joined on 12-20-2008, 4:34 AM Posts 4

  Re: WebSite Deployment Issue Reply Contact IIS recompile : when IIS is reset it has to compile all the assemblies in the bin, I included this information to make people aware the web server site was 'awake' before being put back in service.  Please note I am primarily an ASP.NET developer my knowledge of the intracacies of IIS is limited. Current_Anonymoous_users goes through the roof, I'm not sure our connections are being served anything.  Haven't checked the IIS logs maybe I'll try that tomorrow, but our own Web logs (logged on app_endrequest) are showing nothing, not activity.  We are logging no application errors either. Like you, I thought it was an app issue, but we cannot roll out any thing at the moment.  Which ever component we update we get this problem, also it works fine on QA, and individual productions servers (admitedly without the load).  This might be a clue: usually if we roll out at dead quite periods we have success, so is it the sudden load?.  I have to admit I am now pretty certain it's not application changes that are causing the problem,

• 12-21-2008, 9:37 AM

  In reply to

  Rovastar Joined on 03-13-2008, 10:00 AM London, UK Posts 1,744

  Re: WebSite Deployment Issue Reply Contact If you have rollout in quite periods it is ok and always ok? but when it is busy it is always a problem and always a problem when it is. Or is it that you can depoly in a quite periods abut when it gets busy it is a problem? TBH you should be doing release deployments at off peak hours anyway, if possible. And major releases should be properly tested (stress tested) anyway. Have you tested this release on your test/staging/QA servers? I would work out what worker process w3wp.exe is the taking up the most CPU and link this to the you app pools. Use iisapp to help you work out what apps are linked to which worker process.  http://weblogs.asp.net/owscott/archive/2004/09/21/Which-w3wp.exe-process-belongs-to-which-App-Pool-in-IIS6.aspx Restructure your server and applications to make isolate the problem. Use IIS logs to work out what pages are taking up the CPU. If nothing is in the IIS logs then nothing is getting to IIS. Look at the long running 'timetaken' pages. This will indicate what pages you need to look at in detail to troubleshoot. You first need to narrow down what pages are problem - if any. Also what do the clients see? Does a simple hello world html or .net page have the same problems? For I am still looking at app problems - previously the server was ok before and the deployments were ok. Now this depolyment is causing a problem. So something has changed either your server has changed, your network config has changed or your app has changed. Now we know the app has changed. I am still a little confused what you mean by IIS recompile. Maybe the terminology is getting confused. Maybe this from a dev point of view will explain: http://forums.asp.net/p/1353153/2768560.aspx http://www.iisportal.com

• 12-21-2008, 9:59 AM

  In reply to

  Rovastar Joined on 03-13-2008, 10:00 AM London, UK Posts 1,744	Re: WebSite Deployment Issue Reply Contact Also follow these guides to pinpoint the problem pages. http://blogs.msdn.com/mike/archive/2007/12/06/troubleshooting-high-cpu-performance-issues.aspx http://blogs.msdn.com/mike/archive/2008/02/26/troubleshooting-high-cpu-performance-issues-part-2.aspx http://www.iisportal.com

• 12-22-2008, 6:18 AM

  In reply to

  Jemarah Joined on 12-20-2008, 4:34 AM Posts 4

  Re: WebSite Deployment Issue Reply Contact Firstly, many thanks for your help and good ideas.  By Recompile I mean compiling the .Net assemblies into the temporary ASp.Net folders (C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\).  We have tried clearing these, but still get the same problem.  I agree with you it is screaming app issue, but it is happening whatever ever component we roll out, even something completely unrelated to the original roll out.  As soon as we roll back all is well.  And we are able to run 4 servers in parrallel, 2 changed + 2 unchanged, however as soon as we take the unchanged OOS, Current Anon rockets. Having checked the IIS Logs we are getting page hits logged.  I will update when we have completed more investigations today

• 12-23-2008, 5:25 PM

  In reply to

  Jemarah Joined on 12-20-2008, 4:34 AM Posts 4	Re: WebSite Deployment Issue Reply Contact This has been resolved now.  The roll out expoxed a problem with a service the website relies on.  The good news is the whole thing is now much faster and more responsive, so it was all worthwhile.  Once again thanks for your help in the matter