Sadly, I am prohibited from posting the logs in a public forum like this, but I took your advice and dug deeper into the logs rahter than just looking at the 200s reported by IIS. In the Win32 field I have mostly 0s which is good, but I also have a bunch of 2s, 3s and 64s (which, after researching, appears to be file, path and network resource not found, respectively). Those indicate that the code may be looking for stuff that's not there (or at least not anymore) but not breaking anything fatally so I've asked the developers to review to code from this persepctive, hopefully reducing this number will make the intermittent issue I'm facing stand out more. BUT, I did find 15 or 20 win32 status code 22s a day which appears to be "bad command". As this site is an upgrade from IIS 5 to IIS 6 and uses a completely revamped API to the content management system AND always crops up when running a particular order status aspx page that's where I think my intermittent demon lives so thanks very much for the suggestion. I'm having the developers do a deep dive into that code to determine the source. Is there anywhere that more detail about the source of those 22s (or, frankly, any of the non-zero Win32 Status returns) might be found or can be enabled in IIS6?
The HTTP error logs showing a number of Timer_MinBytesPerSecond errors (and 1236 Win32 Status erorrs in the IIS log) which I know are caused by my load balancer sending a FIN in once it's seen the string it wants, but before the server has finished sending it the page data during health checks so I'm working on rectifying that situation with them. Again, hopefully this will help the intermittent error stand out when it happens. I'd be generally inclined to consider the load balancer at fault, but when I watch the wire at that point I see a perfectly valid post get sent from the load balancer to the server followed immediately by a RST which the load balancer dutifully returns to the requesting client which is why I'm focused on the server itself.
Again, thanks for the guidance and if you might know where I can get any further data on the cause of the Win32 Status 22s I'd appreciate it.
Thanks,
Scott