Recently I have discoved the Get-NetTCPConnection cmdlet while investigating an tcp port exhaustion issue we are having for one of our customer and it displayed a socket state that I have never seen before, the "Bound" state.
I cannot find a lot of information about it on the Microsoft website unfortunately and would like to get more information from this website if possible.
It seems that this particular state is not covered by the "TcpTimedWaitDelay" as they always seems to be alive and never close.
When running the cmdlet, I see a lot of bound state for the IIS worker process. It display a bit like this :
They increment because that's the way it works. An application issues an opensocket call and Windows selects a local port for the app. Typically its just the next one in numerical sequence.
Process monitor is another useful utility used to trace file, network, and registry calls. https://docs.microsoft.com/en-us/sysinternals/downloads/process-utilities
You should also check the IIS logs to what requests the application is making. You should also look at the current active requests. See https://stackoverflow.com/questions/15621509/how-do-i-see-currently-executing-web-request-on-iis-8
So based on what you are saying, the port exhaustion is caused by the fact that the application doesn't close properly the sockets so they increment indefinitely. That make sense.
Is Windows not supposed to close the idle socket when they are idle after 240 seconds by default ? Does this mean that the fact that theses are in the "bound" state make them invisible to the TCP garbage cleanup ?
When this problem occurs, you will want to run the "appcmd list requests" to see what IIS thinks is running. If nothing is running then maybe you have a low level socket problem. Typically what I have seen in the past is something like this: an application
calls a web site, the site code issues a database call, the database is in some state where it cannot process the request. From there all calls then back up. Maybe the original app times out and then issues more requests until it exhausts sockets. It's impossible
for me to guess what your system is doing.
You should use tcpview or process explorer to look at your IIS worker process and see what it has connections to. Any remote systems? Or is all local? What processes does IIS have connections to? If you recycle the IIS worker process, do all of the bound
connections go away? Check the IIS and HTTPERR logs to see what calls are being made. Are they successful calls, and is time-taken value for requests a second or 2 or are requests taking minutes to run?
10 Posts
IIS bounds socket and create ephemeral port exhaustion
Oct 04, 2018 02:03 PM|Philippe.Masse|LINK
Hi,
Recently I have discoved the Get-NetTCPConnection cmdlet while investigating an tcp port exhaustion issue we are having for one of our customer and it displayed a socket state that I have never seen before, the "Bound" state.
I cannot find a lot of information about it on the Microsoft website unfortunately and would like to get more information from this website if possible.
It seems that this particular state is not covered by the "TcpTimedWaitDelay" as they always seems to be alive and never close.
When running the cmdlet, I see a lot of bound state for the IIS worker process. It display a bit like this :
:: 65072 :: 0 Bound 2452 w3wp
:: 65074 :: 0 Bound 2452 w3wp
:: 65075 :: 0 Bound 2452 w3wp
:: 65125 :: 0 Bound 2452 w3wp
:: 65130 :: 0 Bound 2452 w3wp
:: 65184 :: 0 Bound 2452 w3wp
:: 65189 :: 0 Bound 2452 w3wp
:: 65194 :: 0 Bound 2452 w3wp
:: 65195 :: 0 Bound 2452 w3wp
:: 65290 :: 0 Bound 2452 w3wp
:: 65293 :: 0 Bound 2452 w3wp
:: 65295 :: 0 Bound 2452 w3wp
:: 65296 :: 0 Bound 2452 w3wp
:: 65378 :: 0 Bound 2452 w3wp
:: 65379 :: 0 Bound 2452 w3wp
:: 65380 :: 0 Bound 2452 w3wp
:: 65427 :: 0 Bound 2452 w3wp
:: 65428 :: 0 Bound 2452 w3wp
:: 65429 :: 0 Bound 2452 w3wp
:: 65430 :: 0 Bound 2452 w3wp
:: 65528 :: 0 Bound 2452 w3wp
:: 65529 :: 0 Bound 2452 w3wp
:: 65532 :: 0 Bound 2452 w3wp
Is this particular state can cause a port exhaustion issue ? Why do they always increment ?
Thanks
90 Posts
Re: IIS bounds socket and create ephemeral port exhaustion
Oct 04, 2018 04:35 PM|Madness80|LINK
They increment because that's the way it works. An application issues an opensocket call and Windows selects a local port for the app. Typically its just the next one in numerical sequence.
The real problem is that some application is in a loop causing your port exhaustion. Use Tcpview to identify the process. https://docs.microsoft.com/en-us/sysinternals/downloads/tcpview
Process monitor is another useful utility used to trace file, network, and registry calls. https://docs.microsoft.com/en-us/sysinternals/downloads/process-utilities
You should also check the IIS logs to what requests the application is making. You should also look at the current active requests. See https://stackoverflow.com/questions/15621509/how-do-i-see-currently-executing-web-request-on-iis-8
10 Posts
Re: IIS bounds socket and create ephemeral port exhaustion
Oct 04, 2018 05:47 PM|Philippe.Masse|LINK
Thanks for the reply,
So based on what you are saying, the port exhaustion is caused by the fact that the application doesn't close properly the sockets so they increment indefinitely. That make sense.
Is Windows not supposed to close the idle socket when they are idle after 240 seconds by default ? Does this mean that the fact that theses are in the "bound" state make them invisible to the TCP garbage cleanup ?
90 Posts
Re: IIS bounds socket and create ephemeral port exhaustion
Oct 04, 2018 06:51 PM|Madness80|LINK
When this problem occurs, you will want to run the "appcmd list requests" to see what IIS thinks is running. If nothing is running then maybe you have a low level socket problem. Typically what I have seen in the past is something like this: an application calls a web site, the site code issues a database call, the database is in some state where it cannot process the request. From there all calls then back up. Maybe the original app times out and then issues more requests until it exhausts sockets. It's impossible for me to guess what your system is doing.
You should use tcpview or process explorer to look at your IIS worker process and see what it has connections to. Any remote systems? Or is all local? What processes does IIS have connections to? If you recycle the IIS worker process, do all of the bound connections go away? Check the IIS and HTTPERR logs to see what calls are being made. Are they successful calls, and is time-taken value for requests a second or 2 or are requests taking minutes to run?