Bluescreen and applicationHost.config overwritten with zeroes [Answered]RSS

12 replies

Last post Oct 15, 2019 07:27 AM by fw-flw

  • Bluescreen and applicationHost.config overwritten with zeroes

    Oct 11, 2019 11:26 AM|fw-flw|LINK

    I have an application which makes some changes to applicationHost.config. Every time I run this application, my system crashes with a bluescreen, and applicationHost.config is overwritten with zeroes.

    I have been able to recover a recent valid applicationHost.config file from c:\inetpub\history and reproduce the problem every time I run my application.

    This started happening after a Windows update that was installed yesterday (2019-10-10).

    What exactly my application does is quite complicated. For example, it creates applications in IIS by running appcmd. Some modifications to the config are done by a .NET application, which uses Microsoft.Web.Administration. I also cannot tell at which exact point the crash happens, but I am working on getting more details.

    System information:

    Windows 10 1903 (it is a development machine)

    Is this a known issue with the current Windows / IIS version?

    What more information can I provide to get help, and what debugging steps would you recommend for me to do?

  • Re: Bluescreen and applicationHost.config overwritten with zeroes

    Oct 11, 2019 11:54 AM|fw-flw|LINK

    More research shows that my application also runs DISM to install IIS and IIS features, and that (according to logs) this may have been the last operation before the crash.

    The DISM command-line is as follows:

    dism /online /norestart /enable-feature /featurename:IIS-ApplicationDevelopment /featurename:IIS-ASPNET /featurename:IIS-ASPNET45 /featurename:IIS-CGI /featurename:IIS-CommonHttpFeatures /featurename:IIS-HealthAndDiagnostics /featurename:IIS-HttpCompressionDynamic /featurename:IIS-HttpErrors /featurename:IIS-HttpLogging /featurename:IIS-HttpRedirect /featurename:IIS-HttpTracing /featurename:IIS-IPSecurity /featurename:IIS-ISAPIExtensions /featurename:IIS-ISAPIFilter /featurename:IIS-LoggingLibraries /featurename:IIS-ManagementScriptingTools /featurename:IIS-NetFxExtensibility /featurename:IIS-NetFxExtensibility45 /featurename:IIS-Performance /featurename:IIS-RequestMonitor /featurename:IIS-RequestFiltering /featurename:IIS-Security /featurename:IIS-URLAuthorization /featurename:IIS-WebServer /featurename:IIS-WebServerManagementTools /featurename:IIS-WebServerRole /all

    However, by running this command from the console, I cannot reproduce the crash.

  • Re: Bluescreen and applicationHost.config overwritten with zeroes

    Oct 11, 2019 12:53 PM|fw-flw|LINK

    DISM is innocent.

    The crash is caused by a .NET app using Microsoft.Web.Administration to set some ISAPI and CGI Restriction rules. Given the current configuration, this app would not make any actual changes. Removing and adding the rules manually in IIS Manager does not cause a crash. Other operations using the same .NET app and Microsoft.Web.Administration do not cause crashes.

  • Re: Bluescreen and applicationHost.config overwritten with zeroes

    Oct 11, 2019 01:45 PM|fw-flw|LINK

    More shit has come to light:

    The BSOD is caused by cldflt.sys, which according to various search results (often also related to BSOD situations) is a part of or used by Microsoft OneDrive.

    The crash is caused by the second CommitChanges() call in this sequence:

    1. 1. Remove ISAPI/CGI restriction
    2. 2. CommitChanges()
    3. 3. Add ISAPI/CGI restriction
    4. 4. CommitChanges()

    Another fun fact: When I ran this .NET application in the debugger in Visual Studio (which also reproduced the crash), the user settings file of the project was also overwritten with zeroes after the crash, causing Visual Studio to be unable to load the project until I deleted the .user file.

  • Re: Bluescreen and applicationHost.config overwritten with zeroes

    Oct 11, 2019 04:54 PM|lextm|LINK

    Where did you save your source code? It is strongly recommended that you don't save your code in OneDrive mapped folders as most of Microsoft development tools (IIS/Visual Studio) might not be tested to support such setup.

    If you really want to back up your code periodically, learn to use a source code control system like Git.

    Lex Li
    https://lextudio.com
    ---------------------------
    This posting is provided "AS IS" with no warranties, and confers no rights.
  • Re: Bluescreen and applicationHost.config overwritten with zeroes

    Oct 11, 2019 07:23 PM|fw-flw|LINK

    Neither my source code, nor my VS project files, nor anything related to IIS or my application is stored in OneDrive.
    This is why I don't understand why an API call to change the IIS configuration would cause a OneDrive driver to BSOD.
  • Re: Bluescreen and applicationHost.config overwritten with zeroes

    Oct 14, 2019 03:42 AM|Yuk Ding|LINK

    Hi fw-flw,

    Could you fix this issue by removing the update you installed in 10/10/2019? If this issue can be fixed by removing the update, We would know that some change applied in this update cause the crash.

    It is recommended to restore the update as a workaround.

    If you need to figure out the root cause. 

    1.Please find the dump file generated when the server become BSOD. The location would be  C:\Winows\memory.dmp

    2.We need to use dump analysis tool like WINDbg or Debug diagnostic tool to analyze the dump file

    If the exception come from managed code, we need to figure out this issue come from native code or managed code. It will help us find the root cause.

    If you are not expert in dump analysis, it is recommended to open a support ticket to https://support.microsoft.com/en-us.

    Professional support engineer will help you handle this. If it is proved to be bug, they will help you report this issue and ask for a workaround or solution.

    If the reply is helpful, it is appreciated if you could mark it as answer.

    Best Regards,

    Jokies Ding

    Yuk Ding

    MSDN Community Support
    Please remember to "Mark as Answer" the responses that resolved your issue.
  • Rovastar Rovastar

    5421 Posts

    MVP

    Moderator

    Re: Bluescreen and applicationHost.config overwritten with zeroes

    Oct 14, 2019 06:29 AM|Rovastar|LINK

    I'm not sure this is an IIS issue.

    Does changing the app host config by hand / appcmd result in this behavior?

    Troubleshoot IIS in style
    https://www.leansentry.com/
  • Re: Bluescreen and applicationHost.config overwritten with zeroes

    Oct 14, 2019 07:05 AM|fw-flw|LINK

    Hi,

    Changing the app host config by hand (using "IIS Manager" - the Microsoft one, not our app, which has the same name - see below) does not cause a crash.

    Here is the output of WinDBG. I can also provide the complete memory dump, if it is still needed

    (IISManager.exe is our app, which is a .NET 4.6.1 app that uses Microsoft.Web.Administration)

    Also note that (as mentioned earlier), our IISManager.exe makes several changes to app host config BEFORE the change to ISAPI/CGI restrictions that do NOT cause a crash.

    Microsoft (R) Windows Debugger Version 6.3.9600.17336 AMD64
    Copyright (c) Microsoft Corporation. All rights reserved.
    
    
    Loading Dump File [C:\Lab\MEMORY.DMP]
    Kernel Bitmap Dump File: Only kernel address space is available
    
    
    ************* Symbol Path validation summary **************
    Response                         Time (ms)     Location
    Deferred                                       https://msdl.microsoft.com/download/symbols
    Symbol search path is: https://msdl.microsoft.com/download/symbols
    Executable search path is: 
    Windows 8 Kernel Version 18362 MP (12 procs) Free x64
    Product: WinNt, suite: TerminalServer SingleUserTS
    Built by: 18362.1.amd64fre.19h1_release.190318-1202
    Machine Name:
    Kernel base = 0xfffff801`7dc00000 PsLoadedModuleList = 0xfffff801`7e048210
    Debug session time: Fri Oct 11 16:06:52.909 2019 (UTC + 2:00)
    System Uptime: 0 days 0:35:56.657
    Loading Kernel Symbols
    ......................................Page 20106bd67 too large to be in the dump file.
    .........................
    ................................................................
    ................................................................
    ........................................................
    Loading User Symbols
    PEB is paged out (Peb.Ldr = 000000e7`aa57c018).  Type ".hh dbgerr001" for details
    Loading unloaded module list
    ................................
    *******************************************************************************
    *                                                                             *
    *                        Bugcheck Analysis                                    *
    *                                                                             *
    *******************************************************************************
    
    Use !analyze -v to get detailed debugging information.
    
    BugCheck 3B, {c0000005, fffff8017afcfbfe, ffffab8c3d1dd800, 0}
    
    Probably caused by : cldflt.sys ( cldflt!HsmiFltPostECPCREATE+1da )
    
    Followup: MachineOwner
    ---------
    
    2: kd> !analyze -v
    *******************************************************************************
    *                                                                             *
    *                        Bugcheck Analysis                                    *
    *                                                                             *
    *******************************************************************************
    
    SYSTEM_SERVICE_EXCEPTION (3b)
    An exception happened while executing a system service routine.
    Arguments:
    Arg1: 00000000c0000005, Exception code that caused the bugcheck
    Arg2: fffff8017afcfbfe, Address of the instruction which caused the bugcheck
    Arg3: ffffab8c3d1dd800, Address of the context record for the exception that caused the bugcheck
    Arg4: 0000000000000000, zero.
    
    Debugging Details:
    ------------------
    
    
    EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.
    
    FAULTING_IP: 
    cldflt!HsmiFltPostECPCREATE+1da
    fffff801`7afcfbfe f60201          test    byte ptr [rdx],1
    
    CONTEXT:  ffffab8c3d1dd800 -- (.cxr 0xffffab8c3d1dd800;r)
    rax=ffffc18df636fae8 rbx=0000000000000000 rcx=0000000000000000
    rdx=0000000000000014 rsi=ffffab8c3d1ded60 rdi=ffffc18dde39b720
    rip=fffff8017afcfbfe rsp=ffffab8c3d1de1f0 rbp=ffffab8c3d1de240
     r8=0000000000000000  r9=7fffc18dde39b7a0 r10=fffff8017dc663d0
    r11=ffffab8c3d1de1e0 r12=0000000000000014 r13=0000000000000000
    r14=0000000000000000 r15=fffff8017af8d000
    iopl=0         nv up ei pl zr na po nc
    cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00050246
    cldflt!HsmiFltPostECPCREATE+0x1da:
    fffff801`7afcfbfe f60201          test    byte ptr [rdx],1 ds:002b:00000000`00000014=??
    Last set context:
    rax=ffffc18df636fae8 rbx=0000000000000000 rcx=0000000000000000
    rdx=0000000000000014 rsi=ffffab8c3d1ded60 rdi=ffffc18dde39b720
    rip=fffff8017afcfbfe rsp=ffffab8c3d1de1f0 rbp=ffffab8c3d1de240
     r8=0000000000000000  r9=7fffc18dde39b7a0 r10=fffff8017dc663d0
    r11=ffffab8c3d1de1e0 r12=0000000000000014 r13=0000000000000000
    r14=0000000000000000 r15=fffff8017af8d000
    iopl=0         nv up ei pl zr na po nc
    cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00050246
    cldflt!HsmiFltPostECPCREATE+0x1da:
    fffff801`7afcfbfe f60201          test    byte ptr [rdx],1 ds:002b:00000000`00000014=??
    Resetting default scope
    
    DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT
    
    BUGCHECK_STR:  0x3B
    
    PROCESS_NAME:  IISManager.exe
    
    CURRENT_IRQL:  0
    
    ANALYSIS_VERSION: 6.3.9600.17336 (debuggers(dbg).150226-1500) amd64fre
    
    LAST_CONTROL_TRANSFER:  from fffff8017afd0129 to fffff8017afcfbfe
    
    STACK_TEXT:  
    ffffab8c`3d1de1f0 fffff801`7afd0129 : ffffc18d`f636fae8 ffffab8c`3d1de360 00000000`00000000 00000000`00000000 : cldflt!HsmiFltPostECPCREATE+0x1da
    ffffab8c`3d1de280 fffff801`81d03c03 : ffffc18d`f636fae8 ffffab8c`3d1de360 ffffc18d`f636fa00 00000000`00016bf0 : cldflt!HsmFltPostQUERY_OPEN+0x29
    ffffab8c`3d1de310 fffff801`81d0243c : 00000000`00000000 ffffc18d`dae1dd00 ffffc18d`f4c87268 00000000`00000000 : FLTMGR!FltpPerformPostCallbacks+0x3e3
    ffffab8c`3d1de3e0 fffff801`7dc89aac : ffffab8c`3d1de480 ffffab8c`3d1ded0c ffffc18d`dac1e8f0 ffffc18d`ee3e6010 : FLTMGR!FltpPostFsFilterOperation+0x2c
    ffffab8c`3d1de410 fffff801`7e45010d : 00000000`00000000 ffffc18d`dae1ddc0 ffffab8c`3d1de540 fffff801`829ddda0 : nt!FsFilterPerformCompletionCallbacks+0x4c
    ffffab8c`3d1de440 fffff801`7e3ead94 : 00000000`6d4e6f49 fffff801`7df6f06d ffffab8c`00000003 00000000`00000000 : nt!FsRtlQueryOpen+0xd1
    ffffab8c`3d1de710 fffff801`7e1e62ba : fffff801`00000007 fffff801`7e1e5944 ffffab8c`3d1de950 00000000`00000000 : nt!IopQueryInformation+0x139ad4
    ffffab8c`3d1de770 fffff801`7e1ecfcf : ffffc18d`dac1e8f0 ffffc18d`dac1e844 ffffc18d`f17f3010 00000000`00000000 : nt!IopParseDevice+0x8ea
    ffffab8c`3d1de8e0 fffff801`7e1eb431 : ffffc18d`f17f3000 ffffab8c`3d1deb28 ffffc18d`00000240 ffffc18d`cfcfe640 : nt!ObpLookupObjectName+0x78f
    ffffab8c`3d1deaa0 fffff801`7e457ec3 : 00000000`00000001 00000000`00000000 ffffab8c`3d1df090 ffffab8c`3d1deef8 : nt!ObOpenObjectByNameEx+0x201
    ffffab8c`3d1debe0 fffff801`81d18063 : ffffab8c`3d1df000 ffffc18d`ed6ad9f0 ffffc18d`dd104a30 fffff801`81d076fb : nt!IoQueryInformationByName+0x263
    ffffab8c`3d1dee90 fffff801`7af85c99 : ffffab8c`3d1df088 00000000`00000000 ffffab8c`3d1df088 fffff801`7dc6b455 : FLTMGR!FltQueryInformationByName+0x153
    ffffab8c`3d1def40 fffff801`7af77924 : ffffab8c`3d1df088 00000000`00000000 00000000`00000000 00000000`00000000 : cldflt!FltQueryInformationByNameCallout+0x49
    ffffab8c`3d1def90 fffff801`7afcf77d : 00000000`00000000 ffffab8c`3d1e0000 ffffab8c`3d1d9000 ffffc18d`ed6ad9f0 : cldflt!HsmExpandKernelStackAndCallout+0x44
    ffffab8c`3d1defd0 fffff801`7afd0019 : ffffffff`0000ffff ffffc18d`f6666b38 ffffc18d`ed6adc80 ffffab8c`3d1df219 : cldflt!HsmiFltPreECPCREATE+0x34d
    ffffab8c`3d1df140 fffff801`81d04a5d : ffffc18d`f66669b0 00000000`00000000 00000000`00000000 00000000`00000000 : cldflt!HsmFltPreCREATE+0x9
    ffffab8c`3d1df170 fffff801`81d045a0 : ffffab8c`3d1df2f0 ffffab8c`3d1df300 00000000`00000000 00000000`00000000 : FLTMGR!FltpPerformPreCallbacks+0x2fd
    ffffab8c`3d1df280 fffff801`81d3cd13 : fffff801`81d29060 00000000`00000090 00000000`00000000 00000000`000003a4 : FLTMGR!FltpPassThroughInternal+0x90
    ffffab8c`3d1df2b0 fffff801`7dc31f39 : 00000000`00000000 fffff801`7e1e5905 00000000`00000000 00000000`00000000 : FLTMGR!FltpCreate+0x2f3
    ffffab8c`3d1df360 fffff801`7dc30fe4 : 00000000`00000003 00000000`00000000 00000000`00000000 fffff801`7dc317a3 : nt!IofCallDriver+0x59
    ffffab8c`3d1df3a0 fffff801`7e1e5ffb : ffffab8c`3d1df660 fffff801`7e1e5905 ffffab8c`3d1df5d0 ffffc18d`f63584e0 : nt!IoCallDriverWithTracing+0x34
    ffffab8c`3d1df3f0 fffff801`7e1ecfcf : ffffc18d`dac1e8f0 ffffc18d`dac1e805 ffffc18d`f3973260 00000000`00000001 : nt!IopParseDevice+0x62b
    ffffab8c`3d1df560 fffff801`7e1eb431 : ffffc18d`f3973200 ffffab8c`3d1df7a8 00000000`00000040 ffffc18d`cfcfe640 : nt!ObpLookupObjectName+0x78f
    ffffab8c`3d1df720 fffff801`7e230300 : 00000000`00000001 000000e7`aa3dd4c8 00000000`00000001 00000000`00000000 : nt!ObOpenObjectByNameEx+0x201
    ffffab8c`3d1df860 fffff801`7e22fac9 : 000000e7`aa3dd470 00000004`c0100080 000000e7`aa3dd4c8 000000e7`aa3dd488 : nt!IopCreateFile+0x820
    ffffab8c`3d1df900 fffff801`7ddd2b15 : 00000000`00000000 00000000`00000000 00000000`00000000 000000e7`aa3dcb98 : nt!NtCreateFile+0x79
    ffffab8c`3d1df990 00007ffe`7733cb64 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x25
    000000e7`aa3dd3f8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ffe`7733cb64
    
    
    FOLLOWUP_IP: 
    cldflt!HsmiFltPostECPCREATE+1da
    fffff801`7afcfbfe f60201          test    byte ptr [rdx],1
    
    SYMBOL_STACK_INDEX:  0
    
    SYMBOL_NAME:  cldflt!HsmiFltPostECPCREATE+1da
    
    FOLLOWUP_NAME:  MachineOwner
    
    MODULE_NAME: cldflt
    
    IMAGE_NAME:  cldflt.sys
    
    DEBUG_FLR_IMAGE_TIMESTAMP:  0
    
    STACK_COMMAND:  .cxr 0xffffab8c3d1dd800 ; kb
    
    BUCKET_ID_FUNC_OFFSET:  1da
    
    FAILURE_BUCKET_ID:  0x3B_cldflt!HsmiFltPostECPCREATE
    
    BUCKET_ID:  0x3B_cldflt!HsmiFltPostECPCREATE
    
    ANALYSIS_SOURCE:  KM
    
    FAILURE_ID_HASH_STRING:  km:0x3b_cldflt!hsmifltpostecpcreate
    
    FAILURE_ID_HASH:  {ff49d151-c405-fda0-3953-8b48357a52e0}
    
    Followup: MachineOwner
    ---------
    
    2: kd> lmvm cldflt
    start             end                 module name
    fffff801`7af70000 fffff801`7afe7000   cldflt     (pdb symbols)          C:\ProgramData\dbg\sym\cldflt.pdb\0698036E8827B2FF6ECB6676372B81FC1\cldflt.pdb
        Loaded symbol image file: cldflt.sys
        Image path: \SystemRoot\system32\drivers\cldflt.sys
        Image name: cldflt.sys
        Timestamp:        ***** Invalid (B7D0F1F2)
        CheckSum:         00079A82
        ImageSize:        00077000
        Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4
    

  • Re: Bluescreen and applicationHost.config overwritten with zeroes

    Oct 14, 2019 09:08 AM|Yuk Ding|LINK

    Hi fw-flw,

    cldflt.sys belong to cloud file mini filter driver. Since the error message is thrown from native code. Could you fix this issue by rolling back these updates? I believe this issue is a compatibility issue.

    I'm afraid you could only accept rolling back update as a workaround. 

    If you need to fix this, You may need to support a ticket to Microsoft since PG need to collect business impact.

    If the reply is helpful, it is appreciated if you could mark the reply as answer.

    Best Regards,

    Jokies Ding

    Yuk Ding

    MSDN Community Support
    Please remember to "Mark as Answer" the responses that resolved your issue.
  • Re: Bluescreen and applicationHost.config overwritten with zeroes

    Oct 14, 2019 09:33 AM|fw-flw|LINK

    Could you please give me a direct link to a website where I can submit a support ticket to Microsoft? I am finding it difficult to get past all the bullshit (such as knowledge bases, chatbots, etc.) Microsoft set up to keep people from submitting support tickets. This is obviously something that needs to be looked at by a human with some expertise and access to Microsoft developers.

    Otherwise, thanks for the help so far. As a workaround, we are now doing what we should have been doing for a while, which is replacing our home-brew IIS management application with appcmd.

  • Re: Bluescreen and applicationHost.config overwritten with zeroes

    Oct 15, 2019 01:42 AM|Yuk Ding|LINK

    Hi fw-flw,

    If you need to contact Professional Microsoft support engineer, you could create a business request ticket from here:

    https://support.microsoft.com/en-us/supportforbusiness/productselection

    Best Regards,

    Jokies Ding

    Yuk Ding

    MSDN Community Support
    Please remember to "Mark as Answer" the responses that resolved your issue.
  • Re: Bluescreen and applicationHost.config overwritten with zeroes

    Oct 15, 2019 07:27 AM|fw-flw|LINK

    Thank you for your help.

    I have marked your reply as answer. We will get in touch with Microsoft support.