« Previous Next »

Answered Thread: 2008 R2, IIS 7 and UNC problems, managed pipeline modes ?

Last post 11-06-2009 10:55 AM by NicolaiS. 6 replies.

Average Rating Rate It (5)

RSS

Page 1 of 1 (7 items)

Sort Posts:

  • 10-30-2009, 9:40 AM

    • NicolaiS
    • Not Ranked
    • Joined on 10-20-2009, 2:57 AM
    • Posts 5

    2008 R2, IIS 7 and UNC problems, managed pipeline modes ?

    Hello,
    I am currently experiencing an interesting problem with IIS 7 on w2k8 r2 and 
    UNC content in a virtual directory.
    
    We have a front end web server running IIS7.5 off a W2K8 R2 box, on Active 
    Directoty. The application of interest is an ASP.NET 2.0 web application sourced from 
    the local c:\ disk. Within this WebSite Application, we have a virtual folder which is mapped to 
    a UNC file share on a back end server.
    
    This file server is running 2003 R2 server and is on the same domain.
    
    The UNC permissions are setup properly, and on the front end web 
    application, the Application Pool is running under a user which has the 
    proper permissions to source the content.
    
    By defaul, the application pool's Managed Pipeline mode is set to integrated.
    
    This works for a number of requests(not sure on exact number, but probably 
    under 50), after which point, we start loosing the ability to source content 
    from the file server (images, media, etc...) as mapped through the Virtual 
    Folder (ie from the UNC share).
    
    We get the following ERROR in the Event Log on the W2K8 Front end server
    Failed to start monitoring changes to '\\FILESERVER\CONTENTSHARE\YYYYYY' because the network BIOS command limit has been reached. For more information on this error, please 
    refer to Microsoft knowledge base article 810886. Hosting on a UNC share is 
    not supported for the Windows XP Platform.
    
    Now the interesting bit, in trying to debug this further, if we change the 
    Managed Pipeline mode of the application Pool to "Classic", I stop 
    experiencing the problem (though I have not had a chance to test this 
    extensively, but do not seem to be having the problem, no longer problem 
    content, nor any error messages in the event log).
    
    In referring to the error message, I know there are a number of articles 
    describing registry key changes which would allow for MaxCmds to be higher, 
    however these typically describe changes to both the back end server, as well 
    as the front end web server.
    http://learn.iis.net/page.aspx/198/smb-commands-registry-keys/
    http://weblogs.asp.net/hosterposter/archive/2006/10/30/Hosting-IIS-with-UNC-content-_2D00_-Network-BIOS-commands-and-other-errors.aspx
    
    
    Now the question, is WHY is this only happening in the integrated mode, and 
    not in classic and how can we remedy this.
    The backend file server cannot easily be tweaked, as is is currently serving 
    content to a production server runnning on W2003 IIS6 with the same 
    application (same domain user defined in the app pools on both front end 
    servers).
    
    Can someone tell me wheter we should schedule maintenance to do the registry 
    tweaks on the back end file server, and whether this would change anything 
    for the front end server running W2K8 R2.
    Why the difference in behavior between Classic VS Integrated Managed 
    Pipeline Mode in accessing the UNC content.
    Sorry for cross posting as I've put this message up in microsoft.public.inetserver.iis but think the community might be more active on the forums site here.
    Any insight would be appreciated.
    Sincerely,
    Nic
  • 10-30-2009, 9:16 PM In reply to

    • lextm
    • Top 10 Contributor
    • Joined on 10-22-2008, 12:18 AM
    • Shanghai, PRC
    • Posts 1,433

    Re: 2008 R2, IIS 7 and UNC problems, managed pipeline modes ?

    As the root cause is on SMB side, only SMB experts can explain to you. Though it appears to be related to integrated pipeline mode only, that's only an appearance. There must be something under the hood that triggers this SMB error.

    SMB is discussed in this forum,

    http://social.technet.microsoft.com/Forums/en/winserverPN/threads

    Lex Li
    Support Engineer at Microsoft
    ---------------------------
    This posting is provided "AS IS" with no warranties, and confers no rights.
  • 11-02-2009, 10:22 AM In reply to

    • NicolaiS
    • Not Ranked
    • Joined on 10-20-2009, 2:57 AM
    • Posts 5

    Re: 2008 R2, IIS 7 and UNC problems, managed pipeline modes ?

    I will go and ask in the forum you specified, however would like to know why the integrated pipeline mode changes the behavior in sourcing content from the SMB share.

    ? In "Integrated" managed pipeline mode: Are all individual files processes (think 25 images embedded in an aspx index page, which each of these pages being hit tousand of times per day) by an ASP.NET worker process and thus we are running into lanman worker process limits, whereas in the "Classic", the files are served up directly by http.sys ? I realize this is a oversimplification/poor explanation of what might be different between the managed pipeline mode of IIS7, but looking for possible reasons the web server can no longer access data from the back end file server.

    What are our best options ? From the articles on registry keys for LANMAN settings, these registry keys would have no effect in W2K8 based servers. Does this mean that is we want to run our front end servers on W2K8 sourcing content in a virtual folder from a SMB share on a back end server, the backend server also needs to be W2K8 in order to use integrated mode, or are we going to run into the same type of problem with both servers being 2008.

    In reference to the 2003 registry keys:
    http://learn.iis.net/page.aspx/196/smb-commands-registry-keys/

    Would the LANMANServer registry settings on the back end 2003 server (MaxMpxCt, MaxWorkItems), be the solution ? I worry about increasing the number of accesses and work items as the virtual folder we have mapped to this back end share maps thousands of folders, and don't want to end up in a situation with more thousand of WorkItems hangning onto files and possibly timing out.

    Having done more testing on the 2008 Front End, 2003 File server, the senarios described in the original post is definitively reproducible behavior.

    In "Integrated mode", as soon as we exceed a certain number of hits with content coming from the virtual folder (no code page there, simply media related content, mostly images), the front end server starts having 404s, and the error message start popping up in the event log of the 2008 web server.

    In Classic mode, this does not seem to happen, though I'm hard pressed to replicate the type of traffic we see in production on the existing 2003 server setup, so hesitant to move forward until we have a better graps on the source of the problem.

    Any further suggestions in order to get to the root cause of this or further debugging steps would be appreciated.

  • 11-02-2009, 8:38 PM In reply to

    • lextm
    • Top 10 Contributor
    • Joined on 10-22-2008, 12:18 AM
    • Shanghai, PRC
    • Posts 1,433

    Re: 2008 R2, IIS 7 and UNC problems, managed pipeline modes ?

    So when you mentioned classic mode is OK, you did not actually reproduce the same load?

    Then I think you drew a conclusion on integrated mode too early. Based on my experience such SMB error is closely related to load. 

    If you have a chance to check integrated and classic pipeline modes in details, you will see the differences are ASP.NET related, which should not have a significant impact on SMB related things.

    Yes, using Server 2008 as backend servers should have a better performance because of the SMB 2 version only Server 2008 supports.

    See if the SMB experts can provide more insights on the problem you faced.

    Regards,

    Lex Li
    Support Engineer at Microsoft
    ---------------------------
    This posting is provided "AS IS" with no warranties, and confers no rights.
  • 11-03-2009, 3:38 PM In reply to

    • NicolaiS
    • Not Ranked
    • Joined on 10-20-2009, 2:57 AM
    • Posts 5

    Re: 2008 R2, IIS 7 and UNC problems, managed pipeline modes ?

    RE>So when you mentioned classic mode is OK, you did not actually reproduce the same load?

    I am reproducing the same testing sequence, same type of load, by viewing the same pages in the same order... 

    I am the only one putting any load on the server at this time. (validating install and setup)
    In Integrated mode, start having problems within about 50 requests serving content from the virtual folder (accessing images from the Virtual folder mapped the SMB file share on a 2003 server)

    Classic mode, have not been able to reproduce problem at all.

    RE> If you have a chance to check integrated and classic pipeline modes in details, you will see the differences are ASP.NET related, which should not have a significant impact on SMB related things.

    I am quite sure this is exactely how it relates to "Integrated" mode in the managed pipeline, in that ASP.net is processing every request for all files including all JPG sourced on the virtual folder mapped the \\fileserver\SHARE

    How do we fix this ? We have Terabytes of data which lives on a shared backend file server and are server up by front end IIS servers, though a Virtual folder. for example

    http://ww1.com/CONTENT/ --> \\fileserver\CONTENTSHARE
    http://ww2.com/CONTENT/ --> \\fileserver\CONTENTSHARE 

    We where really wanting to migrate the existing web farms onto new IIS 7 servers running on 2008R2, but this seems like integrated pipeline is stopping this from happending properly. How can we get the virtual folder to keep sourcing content from the 2003 file server share.

    I understand that having W2K8 on both front end and back end server will help with SMB 2, however I need to know what is going on in terms of the managed pipeline mode in IIS7 as that is one of the reasons for the migrating to new front end servers.

    As an additional debugging information, we are getting the following "Warning" in the middle of all the ERROR messages in the 2008 Server Event Viewer.


    Event code: 3008 Event message: A configuration error has occurred.
    Event time: 11/3/2009 3:03:59 PM
    Event time (UTC): 11/3/2009 8:03:59 PM
    Event ID: 245d39a49f8540abbc119167e129aa9d
    Event sequence: 21
    Event occurrence: 1
    Event detail code: 0
    Application information:
    Application domain: /LM/W3SVC/3/ROOT-1-129017521922160712
    Trust level: Full Application
    Virtual Path: /
    Application Path: C:\inetpub\XYZ\
    Machine name: W2K8Server

    Process information: Process ID: 4516
    Process name: w3wp.exe
    Account name: XXX-AD-SERVICEACCOUNT
    Exception information:
    Exception type: ConfigurationErrorsException
    Exception message: An error occurred loading a configuration file: Failed to start monitoring changes to '\\fileserver\content\YYYY because the network BIOS command limit has been reached. For more information on this error, please refer to Microsoft knowledge base article 810886. Hosting on a UNC share is not supported for the Windows XP Platform. ('\\fileserver\content\YYY\web.config)

    Request information:
    Request URL: http://WWW/content/YYY/0005.jpg
    Request path: /content/YYY/0005.jpg
    User host address: XXX.YYY.ZZZ
    User: Is authenticated: False
    Authentication Type: Thread account name: XXX-AD-SERVICEACCOUNT

    Thread information:
    Thread ID: 21
    Thread account name: XXX-AD-SERVICEACCOUNT
    Is impersonating: False
    Stack trace: at System.Configuration.ConfigurationSchemaErrors.ThrowIfErrors(Boolean ignoreLocal) at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject) at System.Configuration.BaseConfigurationRecord.GetSection(String configKey, Boolean getLkg, Boolean checkPermission) at System.Configuration.BaseConfigurationRecord.GetSection(String configKey) at System.Web.Configuration.RuntimeConfig.GetSectionObject(String sectionName) at System.Web.Configuration.RuntimeConfig.GetSection(String sectionName, Type type, ResultsIndex index) at System.Web.Configuration.RuntimeConfig.get_Identity() at System.Web.HttpContext.SetImpersonationEnabled() at System.Web.HttpApplication.AssignContext(HttpContext context) at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)


    Seems like Integrated mode is causing the ?Worker-Processes threads? to stack up trying to access the UNC content ? and thus not able to server up the static content.

    Any suggestion on getting around this or than using classic mode ?

     

  • 11-05-2009, 12:18 AM In reply to

    Answered Re: 2008 R2, IIS 7 and UNC problems, managed pipeline modes ?

    I have very similar situation and problems and have spent a lot of days trying to solve the issue.

     Things that I am doing that are the same:

    1) High volume web servers running Windows 2008 and/or Windows 2008 R2 running sites in Integrated mode.

    2) Sites have virtual directories that point to UNC content on Windows 2003 server. Also have Terabytes of data (about 30 TB) on that UNC location.

    3) Active Directory domain account used for permissions to access shared content.

    Differences:

    1) From the errors I see in your it looks like you're using directory monitoring (File Change Notification) and I'm using the default mode which check the content periodically to see if it has been updated.

    I had similar problems and did have to tune LanManServer registry settings on the file server. Since my File Server is x64 Window 2003 I used large values to MaxMpxCT (16384) and MaxWorkItems (65536). 

    On My IIS Servers (Windows 2008 and Windows 2008 R2) I've got LanManWorkstation "MaxCmds" set to 16384.

    Now that tuning did *largely* solve my issues.

    However, under high load I get occasional times when the IIS Servers stop serving content from the Virtual Directories attached the the UNC content. "appcmd list requests" show a lot of requests in the "BeginRequest" or "AuthenticateRequest" state. Often times it take about 80 seconds for things to start moving again.

    So my advice, trying the LanManServer and LanManWorkstation registry setting and see how it work for you.

    *****UPDATE***** I've had great success with:

    Windows 2008 R2 (i've got serveral): lanmanworkstation\MaxCmds = 8192

    Windows 2003 File Server (All values decimal):

    lanmanserver\EnableOplocks = 0
    lanmanserver\MaxFreeConnections = 100
    lanmanserver\MaxMpxCT = 8192
    lanmanserver\MaxRawWorkItems = 512
    lanmanserver\MaxWorkItems = 65536
    lanmanserver\MinFreeConnections = 32

     

  • 11-06-2009, 10:55 AM In reply to

    • NicolaiS
    • Not Ranked
    • Joined on 10-20-2009, 2:57 AM
    • Posts 5

    Re: 2008 R2, IIS 7 and UNC problems, managed pipeline modes ?

    Thanks for the reply,

    Glad to see I am not the only one having lost sleep over this, appreciate the suggestions and your experience as we cannot easily afford to take down the main file server to do registry tunning as server is under heavy load 22+ hours a day.

    I also have an update in that we've setup for testing an alternate back end file server running 2008 R2 (\\FS2008) as a mirror of the older 2003 file server (\\FS2003) in the back end and in testing the 2008 R2 Web server against this new 2008 FS, we have not been able to reproduce the errors. These test are far from perfect, or close to our real world simulation, as we cannot replicate the same load on the back end servers, however based on the current testing, we have not been able to reproduce the errors.

    It would be great to have someone from MS chip in on exactly how the effect of the integrated managed pipeline in IIS7 is interacting with the UNC shares and why there problems show up between 2003 and 2008 servers; I understand there are important improvements in SMB between server OSes but for the sake of enterprise deployments where systems will need to co-exist with 2003 and 2008 servers, it would be good to know how to setup our servers to avoid such problems.

    Thanks again for the lanman registry tunning settings you've used, will report on final solution in the future.

Page 1 of 1 (7 items)
Microsoft Communities