App pool crashing due to security hardeningRSS

6 replies

Last post Dec 16, 2016 04:51 AM by michmike

  • App pool crashing due to security hardening

    Dec 13, 2016 04:21 PM|michmike|LINK

    One of our customers has hardened their IIS ARR server and our requests to that server are now crashing the app pool. we did a debug of w3wp to see how it fails, and we see an access denied error, but we can't pinpoint the resource it is trying to access.

    Can someone look at the code for Win2k12 R2 IIS server and tell us what line number 1735 for file wpipm.cxx is trying to access?

    here's the full trace.

    Trace from attaching WinDbg to the W3WP IIS worker process whenever it started and up to the point of the crash
    ----
    Microsoft (R) Windows Debugger Version 10.0.14321.1024 AMD64
    Copyright (c) Microsoft Corporation. All rights reserved.
    
    CommandLine: c:\windows\system32\inetsrv\w3wp.exe -ap "ExternalBindings" -v "v4.0" -l "webengine4.dll" -a \\.\pipe\iisipm5f1bad4f-514c-4d13-b3e0-59c49972c7c8 -h "C:\inetpub\temp\apppools\ExternalBindings\ExternalBindings.config" -w "" -m 0 -t 20 -ta 0
    Symbol search path is: srv*
    Executable search path is: 
    ModLoad: 00007ff7`63880000 00007ff7`6388a000   w3wp.exe
    ModLoad: 00007ffa`0a650000 00007ffa`0a7fd000   ntdll.dll
    ModLoad: 00007ffa`07d40000 00007ffa`07e7e000   C:\Windows\system32\KERNEL32.DLL
    ModLoad: 00007ffa`07b80000 00007ffa`07c95000   C:\Windows\system32\KERNELBASE.dll
    ModLoad: 00007ffa`09e90000 00007ffa`09f3a000   C:\Windows\system32\ADVAPI32.dll
    ModLoad: 00007ffa`0a280000 00007ffa`0a32a000   C:\Windows\system32\msvcrt.dll
    ModLoad: 00007ffa`08180000 00007ffa`08391000   C:\Windows\SYSTEM32\combase.dll
    ModLoad: 00007ffa`02af0000 00007ffa`02b3e000   c:\windows\system32\inetsrv\iisutil.dll
    ModLoad: 00007ffa`0a330000 00007ffa`0a389000   C:\Windows\SYSTEM32\sechost.dll
    ModLoad: 00007ffa`0a510000 00007ffa`0a650000   C:\Windows\system32\RPCRT4.dll
    ModLoad: 00007ffa`09de0000 00007ffa`09e3a000   C:\Windows\system32\WS2_32.dll
    ModLoad: 00007ffa`07ca0000 00007ffa`07cce000   C:\Windows\system32\SspiCli.dll
    ModLoad: 00007ffa`06c80000 00007ffa`06c8e000   C:\Windows\SYSTEM32\pcwum.DLL
    ModLoad: 00007ffa`09e40000 00007ffa`09e49000   C:\Windows\system32\NSI.dll
    ModLoad: 00007ffa`06430000 00007ffa`0643b000   C:\Windows\SYSTEM32\kernel.appcore.dll
    ModLoad: 00007ffa`075e0000 00007ffa`075eb000   C:\Windows\SYSTEM32\CRYPTBASE.dll
    ModLoad: 00007ffa`07570000 00007ffa`075d3000   C:\Windows\SYSTEM32\bcryptPrimitives.dll
    ModLoad: 00007ffa`03c70000 00007ffa`03ca2000   C:\Windows\SYSTEM32\ntmarta.dll
    ModLoad: 00007ffa`0a100000 00007ffa`0a277000   C:\Windows\SYSTEM32\user32.dll
    ModLoad: 00007ffa`0a390000 00007ffa`0a4df000   C:\Windows\system32\GDI32.dll
    ModLoad: 00007ff9`e9060000 00007ff9`e9076000   c:\windows\system32\inetsrv\w3wphost.dll
    ModLoad: 00007ffa`09a70000 00007ffa`09b36000   C:\Windows\system32\OLEAUT32.dll
    ModLoad: 00007ffa`02a50000 00007ffa`02ad2000   c:\windows\system32\inetsrv\nativerd.dll
    ModLoad: 00007ffa`04250000 00007ffa`04289000   C:\Windows\SYSTEM32\XmlLite.dll
    ModLoad: 00007ffa`04300000 00007ffa`0430b000   C:\Windows\SYSTEM32\ktmw32.dll
    ModLoad: 00007ffa`02850000 00007ffa`02889000   c:\windows\system32\inetsrv\IISRES.DLL
    ModLoad: 00007ffa`07100000 00007ffa`07120000   C:\Windows\SYSTEM32\CRYPTSP.dll
    ModLoad: 00007ffa`06c40000 00007ffa`06c76000   C:\Windows\system32\rsaenh.dll
    ModLoad: 00007ffa`07260000 00007ffa`07286000   C:\Windows\SYSTEM32\bcrypt.dll
    ModLoad: 00007ffa`09bb0000 00007ffa`09c66000   C:\Windows\SYSTEM32\clbcatq.dll
    ModLoad: 00007ffa`02790000 00007ffa`027ce000   C:\Windows\system32\mlang.dll
    ModLoad: 00007ff9`e8ed0000 00007ff9`e8f68000   C:\Windows\Microsoft.NET\Framework64\v4.0.30319\webengine4.dll
    ModLoad: 00007ffa`01b50000 00007ffa`01c26000   C:\Windows\SYSTEM32\MSVCR120_CLR0400.dll
    ModLoad: 00007ffa`06d20000 00007ffa`06d41000   C:\Windows\SYSTEM32\USERENV.dll
    ModLoad: 00007ffa`07cd0000 00007ffa`07cd7000   C:\Windows\system32\PSAPI.DLL
    ModLoad: 00007ffa`07770000 00007ffa`07785000   C:\Windows\SYSTEM32\profapi.dll
    ModLoad: 00007ffa`02670000 00007ffa`026d4000   C:\Windows\SYSTEM32\mscoree.dll
    ModLoad: 00007ffa`025d0000 00007ffa`0266d000   C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscoreei.dll
    ModLoad: 00007ffa`07ce0000 00007ffa`07d34000   C:\Windows\system32\SHLWAPI.dll
    ModLoad: 00007ffa`01c30000 00007ffa`025d0000   C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
    ModLoad: 00007ff9`e8e80000 00007ff9`e8ecf000   C:\Windows\system32\inetsrv\iiscore.dll
    ModLoad: 00007ff9`fb670000 00007ff9`fb67b000   c:\windows\system32\inetsrv\W3TP.dll
    ModLoad: 00007ff9`e9040000 00007ff9`e905e000   c:\windows\system32\inetsrv\w3dt.dll
    ModLoad: 00007ff9`fab20000 00007ff9`fab2e000   C:\Windows\SYSTEM32\HTTPAPI.dll
    ModLoad: 00007ffa`02fa0000 00007ffa`02fb5000   C:\Windows\system32\napinsp.dll
    ModLoad: 00007ffa`055e0000 00007ffa`055fb000   C:\Windows\system32\NLAapi.dll
    ModLoad: 00007ffa`070a0000 00007ffa`070f9000   C:\Windows\System32\mswsock.dll
    ModLoad: 00007ffa`06d90000 00007ffa`06e34000   C:\Windows\SYSTEM32\DNSAPI.dll
    ModLoad: 00007ffa`02fc0000 00007ffa`02fcd000   C:\Windows\System32\winrnr.dll
    ModLoad: 00007ffa`02fd0000 00007ffa`02fda000   C:\Windows\System32\rasadhlp.dll
    ModLoad: 00007ffa`04ca0000 00007ffa`04cca000   C:\Windows\SYSTEM32\IPHLPAPI.DLL
    ModLoad: 00007ffa`04ba0000 00007ffa`04baa000   C:\Windows\SYSTEM32\WINNSI.DLL
    ModLoad: 00007ffa`04a70000 00007ffa`04adb000   C:\Windows\System32\fwpuclnt.dll
    ModLoad: 00007ff9`e90f0000 00007ff9`e90f8000   C:\Windows\System32\inetsrv\cachuri.dll
    ModLoad: 00007ff9`e9030000 00007ff9`e903a000   C:\Windows\System32\inetsrv\cachfile.dll
    ModLoad: 00007ff9`e8e70000 00007ff9`e8e78000   C:\Windows\System32\inetsrv\cachtokn.dll
    ModLoad: 00007ff9`e8e60000 00007ff9`e8e70000   C:\Windows\System32\inetsrv\cachhttp.dll
    ModLoad: 00007ff9`e8e40000 00007ff9`e8e51000   C:\Windows\System32\inetsrv\compstat.dll
    ModLoad: 00007ff9`e8e30000 00007ff9`e8e39000   C:\Windows\System32\inetsrv\defdoc.dll
    ModLoad: 00007ff9`e8e20000 00007ff9`e8e2a000   C:\Windows\System32\inetsrv\dirlist.dll
    ModLoad: 00007ff9`e8e10000 00007ff9`e8e1a000   C:\Windows\System32\inetsrv\protsup.dll
    ModLoad: 00007ff9`e8e00000 00007ff9`e8e0d000   C:\Windows\System32\inetsrv\static.dll
    ModLoad: 00007ff9`e8df0000 00007ff9`e8dfe000   C:\Windows\System32\inetsrv\authanon.dll
    ModLoad: 00007ff9`e8de0000 00007ff9`e8dee000   C:\Windows\System32\inetsrv\modrqflt.dll
    ModLoad: 00007ff9`e8dd0000 00007ff9`e8ddf000   C:\Windows\System32\inetsrv\custerr.dll
    ModLoad: 00007ff9`e8dc0000 00007ff9`e8dcd000   C:\Windows\System32\inetsrv\loghttp.dll
    ModLoad: 00007ff9`e8db0000 00007ff9`e8db9000   C:\Windows\System32\inetsrv\validcfg.dll
    ModLoad: 00007ffa`05e50000 00007ffa`05f02000   C:\Windows\SYSTEM32\shcore.dll
    ModLoad: 00007ffa`083a0000 00007ffa`08534000   C:\Windows\system32\ole32.dll
    ModLoad: 00007ff9`e8da0000 00007ff9`e8dac000   C:\Windows\System32\inetsrv\warmup.dll
    ModLoad: 00007ff9`e8d80000 00007ff9`e8da0000   C:\Windows\System32\inetsrv\isapi.dll
    ModLoad: 00007ff9`e8d60000 00007ff9`e8d74000   C:\Windows\System32\inetsrv\filter.dll
    ModLoad: 00007ff9`e8cf0000 00007ff9`e8d54000   C:\Windows\system32\inetsrv\rewrite.dll
    ModLoad: 00007ff9`e8c90000 00007ff9`e8ce3000   C:\Program Files\IIS\Application Request Routing\requestRouter.dll
    ModLoad: 00007ff9`fd2e0000 00007ff9`fd3a9000   C:\Windows\SYSTEM32\WINHTTP.dll
    ModLoad: 00007ff9`f9480000 00007ff9`f94cf000   C:\Windows\SYSTEM32\pdh.dll
    ModLoad: 00007ffa`07890000 00007ffa`07a6f000   C:\Windows\system32\CRYPT32.dll
    ModLoad: 00007ff9`e8c70000 00007ff9`e8c85000   C:\Program Files\IIS\Application Request Routing\gzip.dll
    ModLoad: 00007ffa`07820000 00007ffa`07831000   C:\Windows\system32\MSASN1.dll
    ModLoad: 00007ff9`f9a30000 00007ff9`f9a38000   C:\Windows\System32\inetsrv\cachuri.dll
    ModLoad: 00007ff9`f9a20000 00007ff9`f9a2a000   C:\Windows\System32\inetsrv\cachfile.dll
    ModLoad: 00007ff9`f9a10000 00007ff9`f9a18000   C:\Windows\System32\inetsrv\cachtokn.dll
    ModLoad: 00007ff9`f9a00000 00007ff9`f9a10000   C:\Windows\System32\inetsrv\cachhttp.dll
    ModLoad: 00007ff9`f99e0000 00007ff9`f99f1000   C:\Windows\System32\inetsrv\compstat.dll
    ModLoad: 00007ff9`f99d0000 00007ff9`f99d9000   C:\Windows\System32\inetsrv\defdoc.dll
    ModLoad: 00007ff9`f99c0000 00007ff9`f99ca000   C:\Windows\System32\inetsrv\dirlist.dll
    ModLoad: 00007ff9`f99b0000 00007ff9`f99ba000   C:\Windows\System32\inetsrv\protsup.dll
    ModLoad: 00007ff9`f99a0000 00007ff9`f99ad000   C:\Windows\System32\inetsrv\static.dll
    ModLoad: 00007ff9`f9990000 00007ff9`f999e000   C:\Windows\System32\inetsrv\authanon.dll
    ModLoad: 00007ff9`f9980000 00007ff9`f998e000   C:\Windows\System32\inetsrv\modrqflt.dll
    ModLoad: 00007ff9`f9970000 00007ff9`f997f000   C:\Windows\System32\inetsrv\custerr.dll
    ModLoad: 00007ff9`f98b0000 00007ff9`f98bd000   C:\Windows\System32\inetsrv\loghttp.dll
    ModLoad: 00007ff9`f98a0000 00007ff9`f98a9000   C:\Windows\System32\inetsrv\validcfg.dll
    ModLoad: 00007ff9`f9890000 00007ff9`f989c000   C:\Windows\System32\inetsrv\warmup.dll
    ModLoad: 00007ff9`f7a00000 00007ff9`f7a20000   C:\Windows\System32\inetsrv\isapi.dll
    ModLoad: 00007ff9`f79e0000 00007ff9`f79f4000   C:\Windows\System32\inetsrv\filter.dll
    ModLoad: 00007ff9`e8e10000 00007ff9`e8e74000   C:\Windows\system32\inetsrv\rewrite.dll
    ModLoad: 00007ff9`e8db0000 00007ff9`e8e03000   C:\Program Files\IIS\Application Request Routing\requestRouter.dll
    ModLoad: 00007ff9`f9480000 00007ff9`f94cf000   C:\Windows\SYSTEM32\pdh.dll
    ModLoad: 00007ffa`07890000 00007ffa`07a6f000   C:\Windows\system32\CRYPT32.dll
    ModLoad: 00007ff9`f79c0000 00007ff9`f79d5000   C:\Program Files\IIS\Application Request Routing\gzip.dll
    ModLoad: 00007ffa`07820000 00007ffa`07831000   C:\Windows\system32\MSASN1.dll
    1740 w3wphost!WP_IPM::ReportListenerChannelStopped [wpipm.cxx @ 1735]:Report ListenerChannel stopped due to failure; ProtocolId:http, ListenerChannelId:0
                    Error(80070005): Access is denied.
    



  • Re: App pool crashing due to security hardening

    Dec 13, 2016 05:22 PM|mahamr|LINK

    An "Access denied" error is better troubleshooted first with Process Monitor.

    https://technet.microsoft.com/en-us/sysinternals/bb896645?f=255&MSPPError=-2147217396

    You can filter down to the w3wp.exe process name, then filter on access denieds and see what's there. This will be good for registry and NTFS permissions, but can be tough for policies.

    Here's a list of the default rights and policies that IIS needs. If ProcMon doesn't show you, then the next step is this resource:
    https://support.microsoft.com/en-us/kb/981949

  • Re: App pool crashing due to security hardening

    Dec 13, 2016 05:43 PM|michmike|LINK

    thanks @mahamr for the quick reply. if the process dies on the access denied exception, would process monitor still show the resource that failed the access request?

  • Re: App pool crashing due to security hardening

    Dec 13, 2016 05:46 PM|mahamr|LINK

    It depends on what specifically is being denied access. ProcMon is the fastest way to determine registry and/or NTFS access denied errors. If nothing problematic shows in ProcMon, then most likely it's a local or group policy issue. If w3wp.exe was trying to load a DLL, for example, and the OS denied access to it, then you would see that in ProcMon.

  • Rovastar Rovastar

    5458 Posts

    MVP

    Moderator

    Re: App pool crashing due to security hardening

    Dec 13, 2016 09:20 PM|Rovastar|LINK

    I agree use ProcMon for access denied issues.

    But also important is what hardening they did and does a rollback of that help?

    Do all app pools fail? straight away or when triggered by certain URLs?

    Troubleshoot IIS in style
    https://www.leansentry.com/
  • Re: App pool crashing due to security hardening

    Dec 15, 2016 04:49 AM|lextm|LINK

    There are too many hardening procedures that simply go against Microsoft's required settings,

    https://support.microsoft.com/en-us/kb/981949

    So please go through KB981949 or open a support case via http://support.microsoft.com 

    Process Monitor as others commented can help, but you do need some official explanation from Microsoft to convince your customer.

    Lex Li
    IIS Consulting Services at https://support.lextudio.com/services/consulting.html
    ---------------------------
    This posting is provided "AS IS" with no warranties, and confers no rights.
  • Re: App pool crashing due to security hardening

    Dec 16, 2016 04:51 AM|michmike|LINK

    thank you all for the repllies. we tried to use process monitor with all the monitoring capabilities enabled and i think we may have found the issue.

    there is an access denied for this resource.

    292 RegOpenKey HKLM\System\CurrentControlSet\Services\WinSock2\Parameters ACCESS DENIED Desired Access: All Access