Move IIS8 to IIS10 w3wp.exe 100% CPU usage!RSS

19 replies

Last post May 27, 2020 11:02 PM by Steven Quick

  • Move IIS8 to IIS10 w3wp.exe 100% CPU usage!

    Jun 11, 2019 10:37 PM|jakkrite.r|LINK

    Move the web application from MS Windows Server 2012 R2 to MS Windows Server 2019.
    The same , IIS .net SQL, but new server just starting to have a high Request 100% CPU, every Core is of w3wp.exe Apppool, although the new machine specification is almost double Try to use from User. Really use Online Load Test and JMeter. Same result. Old server normally 10-30% CPU. New server 100% CPU Hold and still.

    Old Server  [Stable used 7 years]
    DELL® PowerEdge® R420 Server , Intel® Xeon® E5-2420 Processor 1.90GHz X 2 (12 Core 24 Thread)
    8G ECC 1333MHz X 2 (16G), 300GB 15K RPM 6Gbps SAS 3.5" X 2 [Raid 1]
    MS Windows Server 2012 R2 Standard (64Bit) IIS8 ASP.net 4.0 , 32-Bit Disable ,SQL 2012 Express


    New Server 
    DELL®PowerEdge® R640 Server , Intel® Xeon® Gold 5120 Processor 2.2GHz X 2 (28 Core 56 Thread)
    32G RDIMM 2666MT/s X 2 (64G), 960GB SSD SAS 12Gbps SAS 2.5" X 2 [Raid 1]
    MS Windows Server 2019 Standard (64Bit) IIS10 ASP.net 4.0 , 32-Bit Disable ,SQL 2017 Express


    Old Server and New Server

    - FTP files from the old server to the new server The same directory placement structure, set the same permisson, solve only the IP as a new device.

    - IIS set up Apppool like an old server Only some of the settings for IIS8 and IIS10 are the same.

     

    - SQL database copy and then reattach (.mdf, .ldf) check rebuild index.

     

    - Change connection string Connect SQL on the old server w3wp.exe CPU 100% is the same.

     

    - The new server was first launched. There were 2,713 Threads at the time. About 1,600 old machines Threads

    - The web application uses stackify Retrace to capture Error/Log and fix Error.

     

    Server Location same rack ,same network workgroup

    I need your help as soon as possible.
    Thank you , Sorry about my English.

    Start [New Server]

    Start [New Server]

    Load test (JMeter) [New Server]
    Load test (JMeter) [New Server]

    Load test (JMeter)+ Real Production [Old Server]

    Load test (JMeter)+ Real Production [Old Server]

    Advanced Setting [Application Pools] Old and New server

    Advanced Setting [Application Pools] Old and New server

  • Lex Li
    IIS Consulting Services at https://support.lextudio.com/services/consulting.html
    ---------------------------
    This posting is provided "AS IS" with no warranties, and confers no rights.
  • Re: Move IIS8 to IIS10 w3wp.exe 100% CPU usage!

    Jun 12, 2019 06:08 AM|Able|LINK

    Hi jakkrite.r,

    According to your description,to find the reason , you should try some way to troubleshoot high CPU usage by a process.

    So I  suggest that you could use Debug Diagnostics Tool as lextm provided to see the details about how cpu usage are using.

    Best Regards

    Able

    Please remember to click "Mark as Answer" the responses that resolved your issue.
    If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.
  • Re: Move IIS8 to IIS10 w3wp.exe 100% CPU usage!

    Jun 12, 2019 06:52 AM|jakkrite.r|LINK

    Thanks lextm and Abel.

    I right click on w3wp.exe process and create drump file. 


    Add dum file to DebugDialog 2 Analysis



    Start analysis and wait.


    This Report

    3 days ago I try DebugDiag Analysis I see same this




    Please help !

    Best Regards
    Jakkrite.R

  • Re: Move IIS8 to IIS10 w3wp.exe 100% CPU usage!

    Jun 12, 2019 07:31 AM|Able|LINK

    Hi jakkrite.r,

    According to your images, you are using task manager for dump file.I suggest that you could use  windbg for  dump files because task manager dump is not full dump.

    Then have you captured all dump file?I suggest you could share all your dump files in onedrive and you could give the link to us by using private message. So it will be safe for your side. 

    Best Regards

    Able

    Please remember to click "Mark as Answer" the responses that resolved your issue.
    If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.
  • Re: Move IIS8 to IIS10 w3wp.exe 100% CPU usage!

    Jun 12, 2019 08:42 AM|jakkrite.r|LINK

    Ok I will try DebugDiag  Analysis again.

    Best Regards
    Jakkrite.R

  • Rovastar Rovastar

    5468 Posts

    MVP

    Moderator

    Re: Move IIS8 to IIS10 w3wp.exe 100% CPU usage!

    Jun 12, 2019 08:47 AM|Rovastar|LINK

    It is not clear to me if you are testing the same stuff here.

    Could it be that you are pushing more data through JMeter and it is maxing it out?

    Sometimes when testing you can have more data throughput through a similar setup.

    If you additional tools like stackify can you not see what the slow running pages are doing? Is it SQL? request queuing?

    Troubleshoot IIS in style
    https://www.leansentry.com/
  • Re: Move IIS8 to IIS10 w3wp.exe 100% CPU usage!

    Jun 12, 2019 09:12 AM|jakkrite.r|LINK

    Thanks Rosvastar for reply.

    Both servers are different in Spec / OS / IIS only. [Win 2012/2019 , IIS8,IIS10] same webApp and DB

    I test request via JMeter from my PC. I config  JMeter  Number of threads / Ramp-UP Peroid / Loop Count same value for both servers

    I use remote desktop connect to both Server show in my two desktop screen. 
    I set Old Server IP , Run JMeter , monitor Task Manager, Capture it.  switch New Server IP and Run again. 

    before install stackify or removed it . It hight CPU usage

    Thank you and Sorry about my English.  _/|\_

  • Re: Move IIS8 to IIS10 w3wp.exe 100% CPU usage!

    Jun 12, 2019 10:03 AM|jakkrite.r|LINK

    Old Server
    Loadtest 200 Clients per second




    NEW Server
    Loadtest 200 Clients per second





    Capture each  same time.

  • Re: Move IIS8 to IIS10 w3wp.exe 100% CPU usage!

    Jun 12, 2019 10:50 AM|jakkrite.r|LINK


    <br>
    &lt;p&gt;New Server Load test step. Loader.io&lt;br&gt;<br>
    &lt;br&gt;<br>
    &lt;img src=&quot;https://i.ibb.co/vQZ1txC/zzzz.png&quot; width=&quot;700&quot; height=&quot;1088&quot; alt=&quot;&quot;&gt;&lt;/p&gt;<br>
  • Re: Move IIS8 to IIS10 w3wp.exe 100% CPU usage!

    Jun 13, 2019 08:06 AM|jakkrite.r|LINK

    If can't fix Or can not be used on windows server 2019 ii10, I may have to go back to windows server 2012r2 ii8 because the new server, cpu and ram more than the old server Should be able to get more loads Or the same is still good

    But I was worried about support for Windows server 2012 lifecycle (10/10/2023)

    If there are more suggestions, please help. Thank you.
  • Re: Move IIS8 to IIS10 w3wp.exe 100% CPU usage!

    Sep 17, 2019 12:25 PM|brijeshbhandari|LINK

    We are facing exactly the same issue as described here. Just to give a brief background on the environment of the new servers -
    Dell PowerEdge R540, Intel Gold 6130 CPU (x2, total 32 cores) & 128 GB RAM (4*32GB DDR4 2666Mhz)
    OS: Windows Server 2016 Standard Ed.
    IIS: 10.0.14393.0
    .NET framework: 4.7.02053

    When app-pool (w3wp) is running in 64-bit mode, CPU stays significantly higher as compared to 32 bit. This change in utilisation is apparently visible and reflects almost immediately within few seconds of making the bit mode change. When app-pool is set to 32 bit, it settles to a stable pattern. Please note, other than this changing this value (64 to 32), no other changes are made, either in application / IIS / system level.


    However, running in 32-bit also raises OutOfMemory exceptions which needs frequent recycles and is not a permanent fix.The same applications are running fine on older servers (approx 3 yrs), in 64 bit mode. They are of the following configuration -
    Dell PowerEdge R530, Intel E5-2680 v4 (x2, total 28 cores) & 128 GB (4*32GB DDR4 2400Mhz)
    OS: Windows Server 2012 R2 Standard Ed.
    IIS: 8.5.9600.16384
    .Net framework: 4.7.02558

    We have raised a ticket with MS and following up on it. Came across this post recently and wanted to understand if there's any update on identifying the root cause and rectifying it.

  • Re: Move IIS8 to IIS10 w3wp.exe 100% CPU usage!

    Sep 18, 2019 01:43 AM|jakkrite.r|LINK

    My caused by the ASP.NET Apps.


    I try to load test one page and monitor CPU. I remove some line of code and test again to find a coding line of leak CPU.
    When I remove System.Globalization.CultureInfo("en-US") it works well.

    I changed my code System.Globalization.CultureInfo("en-US") to global cache function and call it. It works well.
    I don't know why System.Globalization in IIS10 is causing of leak CPU. 


    -----------------------------
    Public Shared Function en_US() As System.Globalization.CultureInfo
    Try
        Dim cu_us As System.Globalization.CultureInfo = TryCast(System.Web.HttpRuntime.Cache("globalization_cultureinfo_us"), CultureInfo)
        If cu_us Is Nothing Then
          cu_us = New System.Globalization.CultureInfo("en-US") 'en-US 0x0409 1033
    System.Web.HttpRuntime.Cache.Insert("globalization_cultureinfo_us", cu_us, Nothing, Now.AddDays(1), System.Web.Caching.Cache.NoSlidingExpiration)
        End If
    Return cu_us
    Catch ex As Exception
        Return New System.Globalization.CultureInfo("en-US")
    End Try
    End Function

    -----------------------------


    Sorry about my English.

  • Re: Move IIS8 to IIS10 w3wp.exe 100% CPU usage!

    Sep 19, 2019 01:16 PM|brijeshbhandari|LINK

    Thanks jakkrite.r

    The above solution is not working in my case. 

    If there is any other suggestions, please do help us.

    Thank you.

     

  • Rovastar Rovastar

    5468 Posts

    MVP

    Moderator

    Re: Move IIS8 to IIS10 w3wp.exe 100% CPU usage!

    Sep 19, 2019 02:28 PM|Rovastar|LINK

    It is always difficult to suggest things to things like this.

    Often I find that something else has changed code or something that wasn't noticed - no matter how many times I asked has something changed. So seriously double check that.

    Maybe something else has changed maybe the be people use you app changed the day you moved across. Look for any changes. Do like for like testing. ARe you testing again the prod db or a staging Db for example? Has that changed as well.

    Other things in migrations. The IIS server level parameters that the sites inherit these often get forgotten. So check your applicationhost.config (copy it directly or use webdeploy or something).

    I have other server level issues that were not replicated. For example to some sites talk internally to other APIs etc and this was in the local host host saying go directly to the servers/int load balancers but moving to new boxes this wasn't done and it was going to the outside world and back in again and everything ground to a halt.

    Or maybe it can't talk clearly now to the third-party providers you might use? Maybe when you make a payment to your payment provider it is getting a different route/confused. *shrug*

    Maybe is other things like OS, network, virtualization, Anti-virus that has changed.

    So it is really import to understand your environment.

    Other than that you need to go back to looking at general troubleshooting guides/tools/online analysis for looking at high CPU usage.

    Is it happening just on a certain page, etc? Are you have requests queued up?

    there are so many things and they will all be unique to you. It is about getting baby steps to get more and more clues.

    Hope that helps

    Troubleshoot IIS in style
    https://www.leansentry.com/
  • Re: Move IIS8 to IIS10 w3wp.exe 100% CPU usage!

    Sep 20, 2019 03:40 PM|brijeshbhandari|LINK

    Points noted and do concur with. Maybe I should have been a bit more verbose in drafting that post.

    My query to jakkrite was to seek a bit more clarity on the exact changes they made to resolve this issue and secondly, how did they identify it as the root cause in their case.

    We have tried checking all possibilities as noted above and pretty much exhausted the laundry list of options. Have even raised a ticket with MS for quite some time now & it's still open. So, any specific leads from folks who have encountered such scenarios and resolved would be much welcome.

  • Re: Move IIS8 to IIS10 w3wp.exe 100% CPU usage!

    Sep 21, 2019 01:09 AM|lextm|LINK

    brijeshbhandari

    Have even raised a ticket with MS for quite some time now & it's still open.

    Ask whoever works on that ticket to escalate. I don't expect every support person has the same level of expertise, but I don't believe a 100% CPU case should take longer than a week if you already collect the right data (like a series of hang dumps).

    Lex Li
    IIS Consulting Services at https://support.lextudio.com/services/consulting.html
    ---------------------------
    This posting is provided "AS IS" with no warranties, and confers no rights.
  • Re: Move IIS8 to IIS10 w3wp.exe 100% CPU usage!

    May 02, 2020 09:14 AM|greenpro|LINK

    jakkrite.r

    Move the web application from MS Windows Server 2012 R2 to MS Windows Server 2019.
    The same , IIS .net SQL, but new server just starting to have a high Request 100% CPU, every Core is of w3wp.exe Apppool, although the new machine specification is almost double Try to use from User. Really use Online Load Test and JMeter. Same result. Old server normally 10-30% CPU. New server 100% CPU Hold and still.

    Were you able to solve this problem? We are in exactly the same situation after moving from Windows Server 2008 R2 to Windows Server 2019. The new server machine has 4 times more memory and processing power and is generally much quicker, however under heavier load when the CPU usage is around 25-30% it goes directly to a flat line 100% CPU, stays there and the performance suddenly degrades to a halt. w3wp.exe is using all the CPU seeming doing nothing at all. Our application is based on classic ASP and is well optimized during the years and has worked flawlessly on the old IIS 7.5 for many years.

  • Re: Move IIS8 to IIS10 w3wp.exe 100% CPU usage!

    May 06, 2020 08:17 AM|greenpro|LINK

    greenpro

    however under heavier load when the CPU usage is around 25-30% it goes directly to a flat line 100% CPU, stays there and the performance suddenly degrades to a halt. w3wp.exe is using all the CPU seeming doing nothing at all.

    I managed to resolve the problem. It turned out not to be related to IIS or our code. It was related to overprovisioning of CPU resources to the server which degraded the performance instead of improving it.

  • Re: Move IIS8 to IIS10 w3wp.exe 100% CPU usage!

    May 27, 2020 11:02 PM|Steven Quick|LINK

    We are having a similar problem, couple of questions:

    - Was the vm on hyper-v or vmware or other?

    - What was the exact setting you needed to change?