We are excited to announce that the IIS.NET Forums are moving to the new Microsoft Q&A experience. Learn more >

IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentityRSS

18 replies

Last post Feb 17, 2012 12:34 AM by ccorkrum1979

  • IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentity

    Mar 04, 2011 03:03 PM|nub_340|LINK

    We have a small farm of 4 IIS 7.5 + Windows Server 2008 R2 machines. Each machine in the farm has only one website and is configured identically. All 4 website instances have a virtual directory configured to point to the same UNC share on the network (file server) so that all 4 machines are accessing the same content.

    Anonymous Authentication for the virtual directories on all 4 machines is set to use the AppPoolIdentity and we are using pass-through authentication. According to documentation, this should cause IIS to connect to the network resource with the machine account (MACHINENAME$) when a resource within the virtual directory is requested by an anonymous user. The machine accounts have read/write access to the UNC share.

    This setup has been working well so far. However every once in a while after we do a deployment, one or more of the machines will suddenly and mysteriously stop connecting to the network share with the machine account and will start connecting as anonymous. Anonymous does not have access so website users start receiving server 500 errors due to the internal Access Denied exception being thrown.

    Restarting the AppPool and/or website does not fix the problem; nor does restarting IIS entirely. Only a full reboot of the affected box seems to remedy the problem. After a reboot everything is back to normal and works fine again. The strange part is that it doesn't always happen when we deploy. Only sometimes, maybe once out of every 6-10 deployments. Obviously our network ops people aren't very happy that they need to reboot the web servers so often.

    We have tried to reproduce this issue in our test environment and have been somewhat successful. We theorized that the issue was occurring when the application recycled, so we sat there and recycled the app pool & website manually (via IIS) like 100 times and could not reproduce it. We then theorized that it might be related to how the deployment scripts stop & start the app pool / website, so we extracted those commands from the scripts and ran them by themselves and were able to reproduce the issue on our first try. However after that we could not reproduce it again for 4 days after running the same exact commands countless times. Below are the restart commands we are using in our deploy scripts:

    cscript.exe C:\windows\System32\iisweb.vbs /stop www.sitename.com /s SERVERNAME
    cscript.exe C:\Inetpub\AdminScripts\adsutil.vbs STOP_SERVER W3SVC/AppPools/APPPOOLNAME -s:SERVERNAME


    cscript.exe C:\Inetpub\AdminScripts\adsutil.vbs START_SERVER W3SVC/AppPools/APPPOOLNAME -s:SERVERNAME
    cscript.exe C:\windows\System32\iisweb.vbs /start www.sitename.com /s SERVERNAME


    At this point we still can't say for sure where the problem lies. We will continue troubleshooting going forward and will keep this post updated if we find any further information. If anyone has any thoughts, suggestions or has encountered this issue as well, please let us know. Thanks in advance!

    Application Pool Identity iis 7.5 Anonymous Authentication network resource Machine account

  • Re: IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentity

    Mar 04, 2011 08:57 PM|HCamper|LINK

    Hello,

    Since your looking for suggestion(s).

    I suggest you considered looking at State / Status as a problem.

    Your stop scripts might include a wait() for  individual services or servers.

    Your scripts might be include a status check.

    Your start scripts might include a wait()  time that would allow other services to catch up.

    Check the IIS Server log for status codes http://support.microsoft.com/kb/943891

    Martin



     

    Windows and Linux work Together IT-Pros
    Community Member Award 2011
  • Re: IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentity

    Mar 07, 2011 09:58 PM|Leo Tang - MSFT|LINK

    Hi,

    Thanks for posting. You may also consider specify the path credentials instead of use pass-through authentication(Basic Settings...->Connect as...). In this way, the a specific account will be used to connect to the network share.

    Please mark the replies as answers if they help or unmark if not.
    Feedback to us


  • Re: IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentity

    Mar 08, 2011 02:38 PM|nub_340|LINK

    @HCamper: Thanks, we will look into doing this.

    @Leo Tang: Yes, that method seems to work fine but we want to avoid using a specific domain account to access the network resource. According to all the MS documentation we've read we should be able to do this, and we are, most of the time. It just randomly stops working after several deployments. Not sure if it's a bug in IIS 7.5 or a problem w/our scripts or something else entirely...
  • Re: IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentity

    Mar 31, 2011 01:42 PM|nub_340|LINK

    Okay it's been a while but we have an update...

    It doesn't seem to matter if we use "path credentials" or "pass-through authentication" or AppPoolIdentity or specific user or any other combination. The issue occurs regardless. The connection to the network file share works fine for about 7-10 days, and then it randomly stops working after an application restart event.

    To clarify, you can recycle the application all day long for 7-10 days and everything will appear to be fine. However, sometime after this time period, it becomes a game of Russian roulette. Eventually one of the application restarts will trigger the issue on one or more of the servers. Once the issue occurs the only way to fix it is to reboot the entire box, at which point the 7-10 day window starts over again.

    We currently have the virtual directory on all servers in the farm configured to use path credentials using a specific domain account. Today we recycled the application (on all servers in farm) and one of the servers in the farm started experiencing the issue. We confirmed on the File Server via the Security Audit Event Log that IIS is trying to connect as Anonymous, even though it is configured to use path credentials.

    We have been able to consistently reproduce this issue now, although the exact time-frame varies somewhat. This makes us feel like it might be related to credential caching somewhere (IIS Boxes?, File Server?, other?).
  • Re: IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentity

    Mar 31, 2011 02:42 PM|HCamper|LINK

    Hello,

    Question did you create state() diagram for any of the server status?

    I will try with "Best Effort" to show what might be the important to look at. 

    Check the IIS Server log for status codes http://support.microsoft.com/kb/943891 .

    Use the Icacls command http://technet.microsoft.com/en-us/library/cc753525(WS.10).aspx

    MS Support http://support.microsoft.com/kb/919240 KB919240 information.

    Reference  http://en.wikipedia.org/wiki/Cacls .

    Martin

     

    Windows and Linux work Together IT-Pros
    Community Member Award 2011
  • Re: IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentity

    Jan 05, 2012 11:18 AM|AlanR54NC|LINK

    We to are having the same issue as described in this article, and so far no resolution. Was a solution determinded for the original issue ? Our environment is a IIS 7.5 shared hosting service for company internal websites. Each site runs in its own app pool and we have a mix of .Net 2.0 thru 4.0 hosted applications.

    ·      Where you were seeing the issue when connecting to a share, we are seeing it evidenced when making remote LDAP queries, remote web service calls (HTTPXML, etc) and connection to SSRS via ReportViewer component.·         The quantity of sites per server does not seem to correlate though older apps are less likely to do it while apps more newly added are more likely to do it.  We’ve even seen cases where, on the same server at the same time, some app pools work while some do not.  A reboot clears the issue temporarily but it comes back.·        Switching the App Pool to run as Network Service is the unsatisfactory workaround we have been using.·        We have tried all of the following (added a few more)

    o   IISReset

    o   Add [IIS AppPool\AppPoolAcct] to Local Administrators !o   Set App Pool ‘Load User Profile’ property = True.o   Hotfix http://support.microsoft.com/kb/2411938o   Restart of NetLogon serviceo   Restart of Application Information serviceo   Restart of Server serviceo   Restart of Workstation serviceo   Restart of WAS (Windows Process Activation) Service.

    o   GPUPDATE /force

    Thanks for any updates, clues, solutions

    ·     

  • Re: IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentity

    Jan 05, 2012 12:42 PM|nub_340|LINK

    @AlanR54NC - Sorry I haven't updated this thread in a while.

    We ended up opening a ticket with Microsoft and they have been working with us on it ever since (almost 1 year!). Unfortunately the issue still has not been resolved.

    We started out in India, and it has since been escalated to the IIS/Asp.net support team in Texas. They've been having us capture NetMon traces, Memory Dumps, XPERF, PerfMon, the list goes on... but they are essentially stumped. Most recently they have been working with our build manager to see if the issue is related to our deployment process.

    Anyhow, I would encourage you to open a ticket with Microsoft to at least add a little more pressure on them to get it fixed. They said they haven't heard about this issue from anyone else yet, which I find somewhat remarkable considering their customer base. We can't be the only ones experiencing this issue.. can we?
  • Re: IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentity

    Jan 05, 2012 01:27 PM|AlanR54NC|LINK

    Thanks for the update. We have opened a ticket with MS,but was hoping to save some time if you already had a "fix". We are just getting started with the data collection. We have the same website running on a server, but in different app pools.One works fine and the other is not passing the credentials so we can recreate the issue on demand (until we reboot). If we make any progress I will post back here. I will also provide this forum item to our MS resource in case they would like to compare notes with your contact.

  • Re: IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentity

    Jan 06, 2012 03:07 PM|AlanR54NC|LINK

    We have opened a problem ticket with MS and are in the data collection phase,no solution yet. However, if you are seeing this same problem and woudl like to reference our ticket to provide more examples; you may reference case #112010662771698. 

  • Re: IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentity

    Jan 25, 2012 01:06 PM|Foxer|LINK

    We've got the same issue as well; in fact I created a new thread this morning before I found this one. Looks like I'll be opening a ticket with MS as well.

    What have you guys been doing in the mean time? Are you just restarting the servers? We've been contemplating using a different account for the app pool identity, but haven't done anything yet.

  • Re: IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentity

    Jan 27, 2012 01:43 AM|kctt|LINK

     There should be some entry in Event log about the error.

    It could be some kind of permission messed up after server working for some time.

  • Re: IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentity

    Jan 31, 2012 03:43 PM|Foxer|LINK

    I talked to MS support and after a long phone call they directed me to a hotfix (KB2545850). It looks like there is an issue with the computer account changing, and then when you reset IIS it fails to authenticate with the machine account.

    http://support.microsoft.com/kb/2545850

    http://support.microsoft.com/kb/2411938

    After initial testing is seemed like this was working, however additional testing was inconclusive. I'm going to wait to see if there are any more errors in the next couple of weeks, and if not I'm going to assume that this fix did in fact work.

    I'd be interested to see if anyone else is able to solve this issue using this hotfix.

  • Re: IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentity

    Jan 31, 2012 09:19 PM|AlanR54NC|LINK

    Yes. We were directed to that same patch and so far, so good. we were able to recreate the issue on demand using a reg hack from MS. Issue has disappeared. We have applied the patch to several nonproduction servers, and while still a little early, no side effects or recurrances of the problem.

  • Re: IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentity

    Feb 01, 2012 09:59 AM|Foxer|LINK

    That's good to hear. How long has it been since you applied the patch?
  • Re: IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentity

    Feb 03, 2012 02:38 PM|nub_340|LINK

    We were also advised to install the hotfix mentioned above by @Foxer, which we just installed this week. We installed it on one of the 2 servers in our QA farm and then rebooted both so that the issue was not present. Then we manually ran the command on both servers:

    "nltest.exe /sc_change_pwd:domainname"

    ..which triggered the password reset for the machine accounts. We then restarted both IIS instances a la:

    "iisreset /restart"

    ..and the server without the hotfix started having the issue, while the server with the hotfix did not. We ran this test several times with the same result.

    So far it looks good! It's still a little too early to say for sure though. We still need to deploy the hotfix to our production environment and then play the waiting game to see if it happens again to be completely confident this has resolved the issue. I will update the thread in about a month to confirm if that is the case.
  • Rovastar Rovastar

    5495 Posts

    MVP

    Moderator

    Re: IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentity

    Feb 03, 2012 09:08 PM|Rovastar|LINK

     Looking at this thread it looks a little like an issue I got with apppoolidentity

    http://blogs.msdn.com/b/friis/archive/2010/06/09/system-security-securityexception-request-for-the-permission-of-type-system-web-aspnethostingpermission-failed.aspx

    but it depends on what 500 error you got.

     TBH those "undocumented" features it put me off using app pool identity and I just stick to network service or if I really need to have more security just create separate users for the identities.I really don't understand the true need for that account. In my experience it doesn't always work nicely with .net.

    Troubleshoot IIS in style
    https://www.leansentry.com/
  • Re: IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentity

    Feb 15, 2012 01:07 PM|nub_340|LINK

    @Rovastar - In our case however it did not matter if we used AppPoolIdentity, Network Service, or a specific domain account, the issue would still occur regardless of the identity and authentication method used. After some time, we would always start to get 500.19 server errors and would have to reboot to fix it. So far though the hotfix seems to be working.
  • Re: IIS 7.5 stops using machine account to connect to network resource when using AppPoolIdentity

    Feb 17, 2012 12:34 AM|ccorkrum1979|LINK

    I ran across this thread looking for something else but I am also using a web farm and haven't noticed any issues with authentication. I actually ran into this trying to compare my setup to others. I currently have 4 servers 2 IIS and 2 File/DFS servers all domain joined. Most of the content is static but as for permissions on the DFS NS and content links I am using Authenticated Users Full Control on the share, NTFS Authenticated Users with list data on this subfolder only and Administrators, SYSTEM Full Control, and WebFarmUser (Domain Account) Read. For the AppPools I'm using AppPoolIdentity, Authentication I am using the AppPoolIdentity, and using Path Credentials when connecting to the UNC path. The WebFarmUser was also added to IIS_IUSR group on each web node. I haven't brought into the mix FTP and WebDAV yet and that is how I stumbled on the thread seeing if my setup will work, Only thing I have noticed was people using the WebFarmUser as the AppPoolIdentity or Network Service, but not knowing much about share permissions and everything working as is maybe I'm already on the right track?