We just recently installed WFF 2.1 and I noticed a couple of issues. First, the Application Provisioning relies on the application pools. If ANY application pool is recycled when an Application Provisioning check is taking place, it will error out, set all
servers to "Unavailable" and drop them out of Load balancing. You then have to 'Repair' all servers, and then go into 'Monitoring and Management' and set the servers as "Available."
This brings me to problem #2: Once a server is "Ready for load balancing" after a repair or any other action, it isn't being automatically updated as 'Available'. RobinHood707 posted the fix for this at least. I moved the WFFExtension.dll into the Extensions
directory and this at least allowed the servers to automatically be added back to load balancing. But of course, I still have downtime when the app pool gets recycled, sets both servers as 'Unhealthy', runs the repair, and then adds them back to the load balancing.
This issue is happening on ALL application pools, not just the Default app pool. I had to go through all of my app pools, set the 'Idle Time-out' to 0 and Recycling > 'Regular Time Interval (minutes)' to 0 as well. This is obviously not ideal since these
app pools will slowly eat up memory. I can set specific times to recycle (such as during the maintenance window), but I still run the risk of breaking the Application Provisioning during this set time.
Last is the Platform Provisioning. This just doesn't work. It ignores the timer set in the GUI and instead attempts to sync using the Application Provisioning timer. I just disabled this all together.
I'm surprised no one else has mentioned this issue with the app pools. This would explain a lot of problems people are seeing when their servers are all of a sudden going to 'unhealthy' or 'unavailable'. I wish a dev would comment on this, its crazy this release
has been out for 2 months with all these issues and no response from the Developers.
As I mentioned above, placing that DLL in the extension directory definitely helped, however, it still doesn't stop the application provisioning from temporarily breaking whenever an app pool is recycled. That DLL makes it so the servers will automatically
go back to 'healthy' and 'available' (thanks again for pointing that out), but it doesnt make up for the fact I still had my whole farm go down for a couple of minutes after an app pool recycled in the middle of an application provisioning check.
The actual error is: Failed to run operation 'KeepArrInSyncOperationProvider'
I'm able to recreate this at will: Set your application provisioning time to a low enough time to make sure you will recycle during a check (I used 15 seconds for testing), then recycle an app pool a few times. If you check the Server status, you will see
all of the servers are no longer 'Ready for Load Balancing' and have this error listed.
Here are the logs posted from the last time the farm went down. This happened after an app pool reached its default Recycling Regular Time Interval (1740 minutes) and recycled automatically:
2011-09-02T10:26:48.2986054-04:00 Error FARMNAME server02 Failed to run operation 'KeepArrInSyncOperationProvider'. The process terminated unexpectedly. (Exception from HRESULT: 0x8007042B). System.Runtime.InteropServices.COMException (0x8007042B): The process
terminated unexpectedly. (Exception from HRESULT: 0x8007042B) at Microsoft.Web.Farm.ServerOperationProvider.RunOperationAsyncResult.EndRunOperation() at Microsoft.Web.Farm.ServerOperationProvider.EndRunOperation(IAsyncResult result) at Microsoft.Web.Farm.RunOperationContext.FinalOperationCompleteCallback(IAsyncResult
result)
2011-09-02T10:26:48.3142058-04:00 Error FARMNAME server02 Server marked 'Unhealthy' due to failure to run operation 'KeepArrInSyncOperationProvider' System.Runtime.InteropServices.COMException (0x8007042B): The process terminated unexpectedly. (Exception
from HRESULT: 0x8007042B) at Microsoft.Web.Farm.ServerOperationProvider.RunOperationAsyncResult.EndRunOperation() at Microsoft.Web.Farm.ServerOperationProvider.EndRunOperation(IAsyncResult result) at Microsoft.Web.Farm.RunOperationContext.FinalOperationCompleteCallback(IAsyncResult
result)
2011-09-02T10:26:48.3142058-04:00 Error FARMNAME server01 Failed to run operation 'KeepArrInSyncOperationProvider'. The process terminated unexpectedly. (Exception from HRESULT: 0x8007042B). System.Runtime.InteropServices.COMException (0x8007042B): The process
terminated unexpectedly. (Exception from HRESULT: 0x8007042B) at Microsoft.Web.Farm.ServerOperationProvider.RunOperationAsyncResult.EndRunOperation() at Microsoft.Web.Farm.ServerOperationProvider.EndRunOperation(IAsyncResult result) at Microsoft.Web.Farm.RunOperationContext.FinalOperationCompleteCallback(IAsyncResult
result)
2011-09-02T10:26:48.3142058-04:00 Error FARMNAME server01 Server marked 'Unhealthy' due to failure to run operation 'KeepArrInSyncOperationProvider' System.Runtime.InteropServices.COMException (0x8007042B): The process terminated unexpectedly. (Exception
from HRESULT: 0x8007042B) at Microsoft.Web.Farm.ServerOperationProvider.RunOperationAsyncResult.EndRunOperation() at Microsoft.Web.Farm.ServerOperationProvider.EndRunOperation(IAsyncResult result) at Microsoft.Web.Farm.RunOperationContext.FinalOperationCompleteCallback(IAsyncResult
result)
chuck2384
19 Posts
WFF, Application Pools and other issues
Aug 30, 2011 08:55 PM|LINK
We just recently installed WFF 2.1 and I noticed a couple of issues. First, the Application Provisioning relies on the application pools. If ANY application pool is recycled when an Application Provisioning check is taking place, it will error out, set all servers to "Unavailable" and drop them out of Load balancing. You then have to 'Repair' all servers, and then go into 'Monitoring and Management' and set the servers as "Available."
This brings me to problem #2: Once a server is "Ready for load balancing" after a repair or any other action, it isn't being automatically updated as 'Available'. RobinHood707 posted the fix for this at least. I moved the WFFExtension.dll into the Extensions directory and this at least allowed the servers to automatically be added back to load balancing. But of course, I still have downtime when the app pool gets recycled, sets both servers as 'Unhealthy', runs the repair, and then adds them back to the load balancing.
This issue is happening on ALL application pools, not just the Default app pool. I had to go through all of my app pools, set the 'Idle Time-out' to 0 and Recycling > 'Regular Time Interval (minutes)' to 0 as well. This is obviously not ideal since these app pools will slowly eat up memory. I can set specific times to recycle (such as during the maintenance window), but I still run the risk of breaking the Application Provisioning during this set time.
Last is the Platform Provisioning. This just doesn't work. It ignores the timer set in the GUI and instead attempts to sync using the Application Provisioning timer. I just disabled this all together.
chuck2384
19 Posts
Re: WFF, Application Pools and other issues
Oct 27, 2011 09:29 PM|LINK
RobinHood707
40 Posts
Re: WFF, Application Pools and other issues
Oct 28, 2011 12:03 AM|LINK
Try this http://forums.iis.net/p/1180751/1992575.aspx#1992575
chuck2384
19 Posts
Re: WFF, Application Pools and other issues
Oct 28, 2011 01:24 PM|LINK
RobinHood707
40 Posts
Re: WFF, Application Pools and other issues
Oct 28, 2011 10:35 PM|LINK
chuck2384
19 Posts
Re: WFF, Application Pools and other issues
Nov 03, 2011 05:23 PM|LINK
The actual error is: Failed to run operation 'KeepArrInSyncOperationProvider'
I'm able to recreate this at will: Set your application provisioning time to a low enough time to make sure you will recycle during a check (I used 15 seconds for testing), then recycle an app pool a few times. If you check the Server status, you will see all of the servers are no longer 'Ready for Load Balancing' and have this error listed.
Here are the logs posted from the last time the farm went down. This happened after an app pool reached its default Recycling Regular Time Interval (1740 minutes) and recycled automatically:
2011-09-02T10:26:48.2986054-04:00 Error FARMNAME server02 Failed to run operation 'KeepArrInSyncOperationProvider'. The process terminated unexpectedly. (Exception from HRESULT: 0x8007042B). System.Runtime.InteropServices.COMException (0x8007042B): The process terminated unexpectedly. (Exception from HRESULT: 0x8007042B) at Microsoft.Web.Farm.ServerOperationProvider.RunOperationAsyncResult.EndRunOperation() at Microsoft.Web.Farm.ServerOperationProvider.EndRunOperation(IAsyncResult result) at Microsoft.Web.Farm.RunOperationContext.FinalOperationCompleteCallback(IAsyncResult result)
2011-09-02T10:26:48.3142058-04:00 Error FARMNAME server02 Server marked 'Unhealthy' due to failure to run operation 'KeepArrInSyncOperationProvider' System.Runtime.InteropServices.COMException (0x8007042B): The process terminated unexpectedly. (Exception from HRESULT: 0x8007042B) at Microsoft.Web.Farm.ServerOperationProvider.RunOperationAsyncResult.EndRunOperation() at Microsoft.Web.Farm.ServerOperationProvider.EndRunOperation(IAsyncResult result) at Microsoft.Web.Farm.RunOperationContext.FinalOperationCompleteCallback(IAsyncResult result)
2011-09-02T10:26:48.3142058-04:00 Error FARMNAME server01 Failed to run operation 'KeepArrInSyncOperationProvider'. The process terminated unexpectedly. (Exception from HRESULT: 0x8007042B). System.Runtime.InteropServices.COMException (0x8007042B): The process terminated unexpectedly. (Exception from HRESULT: 0x8007042B) at Microsoft.Web.Farm.ServerOperationProvider.RunOperationAsyncResult.EndRunOperation() at Microsoft.Web.Farm.ServerOperationProvider.EndRunOperation(IAsyncResult result) at Microsoft.Web.Farm.RunOperationContext.FinalOperationCompleteCallback(IAsyncResult result)
2011-09-02T10:26:48.3142058-04:00 Error FARMNAME server01 Server marked 'Unhealthy' due to failure to run operation 'KeepArrInSyncOperationProvider' System.Runtime.InteropServices.COMException (0x8007042B): The process terminated unexpectedly. (Exception from HRESULT: 0x8007042B) at Microsoft.Web.Farm.ServerOperationProvider.RunOperationAsyncResult.EndRunOperation() at Microsoft.Web.Farm.ServerOperationProvider.EndRunOperation(IAsyncResult result) at Microsoft.Web.Farm.RunOperationContext.FinalOperationCompleteCallback(IAsyncResult result)
chuck2384
19 Posts
Re: WFF, Application Pools and other issues
Feb 08, 2012 08:30 PM|LINK
This issue still exists with WFF 2.2, recycle an active app pool in the middle of a provisioning check and you just brought down your farm.