minBytesPerSecond and HTTP/2 - does it drop the connection, or just the stream?RSS

6 replies

Last post Aug 02, 2020 03:43 AM by NedStark

  • minBytesPerSecond and HTTP/2 - does it drop the connection, or just the stream?

    Aug 18, 2017 08:11 PM|David Deutsch|LINK

    When IIS cancels an HTTP/2 request due to a Timer_MinBytesPerSecond error, does it just cancel that one stream, or does it close the extire underlying TCP connection as well? In the IIS logs I see what appears to be stream IDs associated with the error, as they are incrementing odd numbers, e.g.:

    2017-08-17 14:53:35 XX.XX.XX.XX 60224 YY.YY.YY.YY 443 - - - 111 - - Timer_MinBytesPerSecond -
    2017-08-17 14:53:35 XX.XX.XX.XX 60224 YY.YY.YY.YY 443 - - - 113 - - Timer_MinBytesPerSecond -
    2017-08-17 14:53:35 XX.XX.XX.XX 60224 YY.YY.YY.YY 443 - - - 115 - - Timer_MinBytesPerSecond -
    2017-08-17 14:53:35 XX.XX.XX.XX 60224 YY.YY.YY.YY 443 - - - 117 - - Timer_MinBytesPerSecond -


     I'm curious to know if it is indeed the case that each of the above 4 requests did indeed go below the minBytesPerSecond threshold, or if it may be the case that just one did (say, stream 111), and that IIS responded by killing off the entire TCP connection and logging one line for each remaining stream.

    A related question I had was regarding how IIS measures the speed of individual stream on the same connection when they have different priorities. In the example above, each stream had the previous one as a parent, i.e. 111 was a parent of 113, which was a parent of 115, etc. In addition, a long running, successful request which is not shown (109) was a parent of 111, yet IIS sent the browser a RST_STREAM frame (error code 8) for stream 111 before 109 was completed. While I know that a parent stream is not required to complete before a child stream is processed, I was wondering if it is possible that IIS "started the clock" on 111 when its headers were first received, and since the TCP connection was monopolized by stream 109 for a long duration, IIS determined that the traffic of stream 111 was not being processed by the client quickly enough? I would be surprised if that were the case (it would seem to be a pretty major bug), but I just wanted to confirm. It just seems odd that while stream 109 was obviously being processed by the client at a decent speed, IIS determined that 111 wasn't.

    Thanks,

      Dave

  • Re: minBytesPerSecond and HTTP/2 - does it drop the connection, or just the stream?

    Aug 21, 2017 06:57 AM|Yuk Ding|LINK

    Hi David,

    The error message  Timer_MinBytesPerSecond means 'The connection expired because the client was not receiving a response at a
    reasonable speed. The response send rate was slower than the default of 240 bytes/sec. This can be controlled with the MinFileBytesPerSec metabase property.' So it should mean that the whole connection should be dropped more than specific stream.

    Best Regards.

    Yuk Ding

    Yuk Ding

    MSDN Community Support
    Please remember to "Mark as Answer" the responses that resolved your issue.
  • Re: minBytesPerSecond and HTTP/2 - does it drop the connection, or just the stream?

    Aug 21, 2017 02:44 PM|David Deutsch|LINK

    Thanks Yuk. So am I correct in assuming that not only is it the entire connection that gets affected by the setting, but it is also the send rate of the entire connection that determines when it gets dropped? In other words, if an individual stream's rate is under 240 bytes/second, but the sum of all streams in that connection is over 240 bytes/second, the connection should not be dropped, correct?

  • Re: minBytesPerSecond and HTTP/2 - does it drop the connection, or just the stream?

    Aug 23, 2017 06:50 AM|Yuk Ding|LINK

    Hi David,

    Considering it is related to http.sys. I think maybe my understanding is wrong. the http.sys should terminate the connection based on per stream. If one individual stream's rate under 240 bytes/second it should be dropped.

    Best Regards,

    Yuk Ding

    Yuk Ding

    MSDN Community Support
    Please remember to "Mark as Answer" the responses that resolved your issue.
  • Re: minBytesPerSecond and HTTP/2 - does it drop the connection, or just the stream?

    Aug 23, 2017 08:38 PM|David Deutsch|LINK

    Yuk Ding

    the http.sys should terminate the connection based on per stream

    To clarify, you meant to say that it should terminate the stream, not the connection, correct?

    In any event, I did some more experimenting and indeed it seems that it is dropping streams, even though the connection is handling traffic at a high rate, e.g. I sent a bunch of simultaneous requests (streams), including one that retrieves 30MB of data, to IIS, and even though I was getting back the 30MB of data at a rapid clip, that "starved" the other streams that shared the connection and caused IIS to terminate them. So I guess what I am wondering is, "should" IIS do that? I was under the impression that minBytesPerSecond was to guard against misbehaving clients from using up resources, but would it matter if there were thousands of "monopolized" HTTP/2 streams if they were all confined to a single TCP connection? I was under the impression that they were just logical divisions of frames, but I could easily be wrong.

  • Re: minBytesPerSecond and HTTP/2 - does it drop the connection, or just the stream?

    Aug 31, 2017 07:10 AM|Yuk Ding|LINK

    Hi David,

    In HTTP1 or 1.1, it run as single thread to handle the stream. So in IIS, terminate connection equal to kill stream. However in HTTP/2, it run as multiplithreading.

    I noticed that IIS restriction doesn't support HTTP/2 not yet. So maybe this function will not working.

    https://blogs.iis.net/davidso/http2

    It seems to be hard to find any answer from document but a test.

    Best Regards,

    Yuk Ding

    Yuk Ding

    MSDN Community Support
    Please remember to "Mark as Answer" the responses that resolved your issue.
  • Re: minBytesPerSecond and HTTP/2 - does it drop the connection, or just the stream?

    Aug 02, 2020 03:43 AM|NedStark|LINK

    Based on the HTTP.SYS logs above, the root cause of the issue is the slow response throughput. The client is not receiving IIS response at a reasonable speed.

    For Windows Server 2012 R2 (it was the server I worked on), IIS wants to send response at minimum 240 bytes per second. If the client is not able to receive it at this speed minimum, IIS terminates the connection. This logic prevents the server against slow HTTP attacks that may result in denial-of-service (DoS) situations

    Here is what I recommend if you come across this issue:

    Check which clients are causing this issue

    The client machines might be in a slow network. If the issue is caused by the same client(s), you may be able to improve their condition to solve the issue

    Find out client IP addresses in the HTTPERR file and investigate their network as well as their OS

    Important: The client IP addresses in the file are unlikely to be the actual client machines. They are probably load balancer, proxy, or firewall addresses. First, find out which device these IPs belong to. For example, If they are load balancer addresses, check their logs to find out actual client IPs

    Remove the threshold for minimum response rate

    If the client IPs are legit (not malicious sources or possible attackers), set minBytesPerSecond to 0 and monitor the system for a while

    If the issue doesn’t happen again (you don’t see Timer_MinBytesPerSecond records in HTTPERR file), go back to this setting and play around the value (Try 500). It’s not recommended to set it 0 because it may make your application vulnerable to DoS attacks based on slow HTTP response

    Steps to change  minBytesPerSecond:

    1. Go to IIS Manager. Click on the server name
    2. Double click on Configuration Editor
    3. Go to “system.applicationHost/webLimits”
    4. Set minBytesPerSecond to 0
    5. Reset IIS

    Source: Slow response times and Timer_MinBytesPerSecond in HTTP.SYS logs