IIS 7 and Above
minBytesPerSecond and HTTP/2 - does it drop the connection, or just t...
Last post Aug 02, 2020 03:43 AM by NedStark
Aug 18, 2017 08:11 PM|David Deutsch|LINK
When IIS cancels an HTTP/2 request due to a Timer_MinBytesPerSecond error, does it just cancel that one stream, or does it close the extire underlying TCP connection as well? In the IIS logs I see what appears to be stream IDs associated with the error,
as they are incrementing odd numbers, e.g.:
2017-08-17 14:53:35 XX.XX.XX.XX 60224 YY.YY.YY.YY 443 - - - 111 - - Timer_MinBytesPerSecond -
2017-08-17 14:53:35 XX.XX.XX.XX 60224 YY.YY.YY.YY 443 - - - 113 - - Timer_MinBytesPerSecond -
2017-08-17 14:53:35 XX.XX.XX.XX 60224 YY.YY.YY.YY 443 - - - 115 - - Timer_MinBytesPerSecond -
2017-08-17 14:53:35 XX.XX.XX.XX 60224 YY.YY.YY.YY 443 - - - 117 - - Timer_MinBytesPerSecond -
I'm curious to know if it is indeed the case that each of the above 4 requests did indeed go below the minBytesPerSecond threshold, or if it may be the case that just one did (say, stream 111), and that IIS responded by killing off the entire TCP connection
and logging one line for each remaining stream.
A related question I had was regarding how IIS measures the speed of individual stream on the same connection when they have different priorities. In the example above, each stream had the previous one as a parent, i.e. 111 was a parent of 113, which was
a parent of 115, etc. In addition, a long running, successful request which is not shown (109) was a parent of 111, yet IIS sent the browser a RST_STREAM frame (error code 8) for stream 111 before 109 was completed. While I know that a parent
stream is not required to complete before a child stream is processed, I was wondering if it is possible that IIS "started the clock" on 111 when its headers were first received, and since the TCP connection was monopolized by stream 109 for a long
duration, IIS determined that the traffic of stream 111 was not being processed by the client quickly enough? I would be surprised if that were the case (it would seem to be a pretty major bug), but I just wanted to confirm. It just seems odd that while stream
109 was obviously being processed by the client at a decent speed, IIS determined that 111 wasn't.
Aug 21, 2017 06:57 AM|Yuk Ding|LINK
The error message Timer_MinBytesPerSecond means 'The connection expired because the client was not receiving a response at a
reasonable speed. The response send rate was slower than the default of 240 bytes/sec. This can be controlled with the MinFileBytesPerSec metabase property.' So it should mean that the whole connection should be dropped more than specific stream.
Aug 21, 2017 02:44 PM|David Deutsch|LINK
Thanks Yuk. So am I correct in assuming that not only is it the entire connection that gets affected by the setting, but it is also the send rate of the entire connection that determines when it gets dropped? In other words, if an individual
stream's rate is under 240 bytes/second, but the sum of all streams in that connection is over 240 bytes/second, the connection should not be dropped, correct?
Aug 23, 2017 06:50 AM|Yuk Ding|LINK
Considering it is related to http.sys. I think maybe my understanding is wrong. the http.sys should terminate the connection based on per stream. If one individual stream's rate under 240 bytes/second it should be dropped.
Aug 23, 2017 08:38 PM|David Deutsch|LINK
the http.sys should terminate the connection based on per stream
To clarify, you meant to say that it should terminate the stream, not the connection, correct?
In any event, I did some more experimenting and indeed it seems that it is dropping streams, even though the connection is handling traffic at a high rate, e.g. I sent a bunch of simultaneous requests (streams), including one that retrieves 30MB of data,
to IIS, and even though I was getting back the 30MB of data at a rapid clip, that "starved" the other streams that shared the connection and caused IIS to terminate them. So I guess what I am wondering is, "should" IIS do that? I was under the impression that
minBytesPerSecond was to guard against misbehaving clients from using up resources, but would it matter if there were thousands of "monopolized" HTTP/2 streams if they were all confined to a single TCP connection? I was under the impression that they were
just logical divisions of frames, but I could easily be wrong.
Aug 31, 2017 07:10 AM|Yuk Ding|LINK
In HTTP1 or 1.1, it run as single thread to handle the stream. So in IIS, terminate connection equal to kill stream. However in HTTP/2, it run as multiplithreading.
I noticed that IIS restriction doesn't support HTTP/2 not yet. So maybe this function will not working.
It seems to be hard to find any answer from document but a test.
Aug 02, 2020 03:43 AM|NedStark|LINK
Based on the HTTP.SYS logs above, the root cause of the issue is the slow response throughput. The client is not receiving IIS response at a reasonable speed.
For Windows Server 2012 R2 (it was the server I worked on), IIS wants to send response at minimum 240 bytes per second. If the client is not able to receive it at this speed minimum, IIS terminates the connection. This logic
prevents the server against slow HTTP attacks that may result in denial-of-service (DoS) situations
Here is what I recommend if you come across this issue:
Check which clients are causing this issue
The client machines might be in a slow network. If the issue is caused by the same client(s), you may be able to improve their condition to solve the issue
Find out client IP addresses in the HTTPERR file and investigate their network as well as their OS
Important: The client IP addresses in the file are unlikely to be the actual client machines. They are probably load balancer, proxy, or firewall addresses. First, find out which device these IPs belong to. For example, If they are load balancer addresses,
check their logs to find out actual client IPs
Remove the threshold for minimum response rate
If the client IPs are legit (not malicious sources or possible attackers), set minBytesPerSecond to 0 and monitor the system for a while
If the issue doesn’t happen again (you don’t see Timer_MinBytesPerSecond records in HTTPERR file), go back to this setting and play around the value (Try 500). It’s not recommended to set it 0 because it may make your application vulnerable to DoS attacks
based on slow HTTP response
Steps to change minBytesPerSecond:
Source: Slow response times and Timer_MinBytesPerSecond in HTTP.SYS logs