WAS fails to start with larger Shared Configuration [Answered]RSS

3 replies

Last post Feb 01, 2020 07:00 AM by jchertudi

  • WAS fails to start with larger Shared Configuration

    Jan 03, 2020 06:00 AM|jchertudi|LINK

    My usual tricks aren't helping. I've been managing two Windows 2008R2 web servers for a 8 years, with a Shared configuration on UNC path, that host about 3000 small web sites.

    I am moving our sites to Windows 2019, UNC shared configuration, and roughly 2000 sites.

    But, WAS will fail to start (or restart) with more than ~70 sites configured.

    This occurs when I copy/paste valid XML into the shared applicationHost.config, or add sites via appcmd or New-WebSite

    The applicationHost.config is almost 40k lines in length, properly formatted XML.

    If this configuration is used LOCALLY, then WAS will restart nicely and all sites work!

    If I put this shared configured on a UNC path, it will work UNTIL WAS restarts. I can bulk add many sites, they function, up until a WAS restart.

    If I add comments blocks to disable applicationHost.config  between lines 1031 and and 38518, then WAS will start!

    But when I comment out lines 1649 to 38518, then WAS fails with

    net start was

    The Windows Process Activation Service service is starting.
    The Windows Process Activation Service service could not be started.

    A system error has occurred.

    System error 5 has occurred.

    Access is denied.

    Event viewer says...
    Event ID 5189
    The Windows Process Activation Service failed to generate an application pool config file for application pool '*'. The error type is '0'. To resolve this issue, please ensure that the applicationhost.config file is correct and recommit the last configuration changes made. The data field contains the error number.

    Procmon shows ACCESS DENIED on C:\inetpub\temp\appPools\APCD26B.tmp

    BUT I have allowed the IUSR specified to have Modify permission on this folder.

    Last weird clue... the intent is to have \\web1 and \\web2 used the Shared Config on \\unc

    When the shared config is on \\unc, both servers suffer this error with larger applicationHost.config, but work ok with smal shared config.

    BUT if I place the shared config on \\web1, and have \\web1 used shared config on its own share, WAS can start on \\web1 with all sites enabled!

    But WAS will fail on \\web2, pointing to shared config on \\web1

    It almost feels like an obscure XML parsing failure over UNC with a larger file. I am hoping there is some other avenue to be suggested. Ideas?

    Thanks!

  • Re: WAS fails to start with larger Shared Configuration

    Jan 03, 2020 08:45 AM|Yuk Ding|LINK

    Hi jchertudi,

    Have you ever tried the registry key ?

    HKLM\SOFTWARE\Microsoft\InetStp\Configuration\MaxWebConfigFileSizeInKB

    HKLM\SOFTWARE\Wow6432Node\Microsoft\InetStp\Configuration\MaxWebConfigFileSizeInKB

    https://forums.iis.net/t/1248702.aspx

    It seems to be an issue that we have never seen this before.  Since We don't have permission to identify this is a bug or not. It is recommended to open a support ticket.

    https://support.microsoft.com/en-us

    Best Regards,

    Jokies Ding

    Yuk Ding

    MSDN Community Support
    Please remember to "Mark as Answer" the responses that resolved your issue.
  • Re: WAS fails to start with larger Shared Configuration

    Jan 03, 2020 03:55 PM|jchertudi|LINK

    thanks for the hint, I had not needed to use MaxWebConfigFileSizeInKB under IIS 7.5. At peak our shared applicationHost.config was over 6000KB

    I applied MaxWebConfigFileSizeInKB to my \\web2 server in registry today, under both 64 and 32 bit. But still WAS will not start. I had to remove sites until the shared applicationHost.config  was below 150KB, that isn't very big.

    An possible work around ocurred to me. I can place the shared applicationHost.config in a local directory on each web server, that is replicated using DFS.

    Then, each web server uses its own \\webX\sharedconfig path as the Shared Config in IIS.

    Changes made to any web server should replicate via DFS to others.

    Plausible?

    I will open a ticket with Microsoft also, thanks again for reply.

  • Re: WAS fails to start with larger Shared Configuration

    Feb 01, 2020 07:00 AM|jchertudi|LINK

    Final answer... MS support directed me to adjust dynamicRegistrationThreshold to a number > sites.

    This is supposed to be a memory saving optimization, according to the docs, and introduced with IIS 8.5. However, I observed some show-stopping behavior by leaving dynamicRegistrationThreshold at default of 100, and having more than 100 sites under Shared Configuration

    • install Windows 2019 server with IIS
    • add ~ 120 sites, they can all point to c:\inetpub\wwwroot and use different ports (8001 - 8120)
    • everything works
    • export your IIS config with Shared Configuration tool to a UNC path
    • enabled the Shared Configuration pointing to UNC path above
    • Sites will work, everything is great!

    But I observed two deadly problems...

    1. on server reboot or service restart, WAS service won't start, citing "access denied"
    2. In some cases, making a change to Shared Config via IIS Manager causes all app pools to stop, and return 503 errors to clients

    Microsoft did not go as far as to confirm my bug, but they did refund my incident charge. I'm convinced there is some error with Shared Configuration and the dynamicRegistrationThreshold

    Hope this helps someone else!