IIS 7 and Above
Stress Testing Issue When Adding A Second IIS Server
Last post Jan 09, 2018 09:45 PM by carehart
Jan 07, 2018 10:36 PM|lherbinaux|LINK
The diagram below describes our soon to be production environment, our user registration process, our async web API client, and our stress test application. I have successfully stress tested our User Registration process going to each of our IIS Servers
and I am able to run 110 concurrent User Registration tasks without receiving any errors. I was able to run 50,000 user registrations in about 53 minutes. The IIS Server's CPU Utilization averaged 78% and SQL Server's was 13%. Increasing the number of concurrent
tasks over 110 results in the following exceptions being thrown: "A task was canceled." or "The underlying connection was closed: A connection that was expected to be kept alive was closed by the server." My assumption was that IIS was running out of resources.
I was OK with this result.
I was hoping to be able to get 220 concurrent user registration tasks (110 for each server) when stress testing both IIS Servers simultaneously. I placed the stress test client on two separate development machines: 1 targeted the first IIS server and the
second one targeted the second one. I received the exceptions that I mentioned above right away. I had to reduce the concurrent user registration tasks to 55 for each server in order to successfully run my tests with without exceptions. My first thought
was that SQL was restricting the number of connections, but max concurrent connections is set to 0 which means it unlimited (32767 concurrent connections). From my research, I don't think there is a per database connection limit. It doesn't make sense that
the issue is on the IIS Servers because each one can separately run 110 tasks concurrently. I did try increasing the SQL Connection Pool Max Pool Size to 150 (I didn't expect it to solve this problem, but wanted to see if this could allow me to run more than
110 concurrent tasks on a single server). Any thoughts on where I can check next?
Stress Test Environment
Jan 08, 2018 05:30 PM|lherbinaux|LINK
I added a duplicate thread. I don't see a way of removing it. Can the moderator remove this thread? Thanks.
Jan 08, 2018 10:07 PM|lherbinaux|LINK
The other thread got canceled, so please reply to this one.
Jan 09, 2018 03:35 AM|Yuk Ding|LINK
I just see this image in msdn forum. Now that this issue occur in Azure cloud server, do they have any restriction for the concurrent connection or CPU memory usage? I would like to know where return the error task cancelled.
In addition, IIS should support the concurrent connection much more than 110. How did you set the queue length and max concurrent connection. Maybe you could try to set multiple worker process in application pool advanced setting to allow the IIS support
more concurrent connections.
Did you set the limit action for CPU usage or recycle action for memory limit?
Jan 09, 2018 05:40 PM|carehart|LINK
There was yet another, 2138534, which has not (yet) been removed/marked as a dupe of this.
Jan 09, 2018 05:52 PM|carehart|LINK
lherbinaux, there can be LOTS to explain the poor performance. You don't share nearly enough for any of us here to help you. For instance:
Those are just a few questions that may help get you going, or whose answers may help you help us. And the latter two apply to both the DB server AND the IIS server.
That said, there may be multiple issues, and there may be no better solution for you than to have a skilled troubleshooter join you in a remote session to look over things while doing this testing, to help you see what's amiss.
Jan 09, 2018 09:04 PM|lherbinaux|LINK
Hi Yuk Ding,
I should have mentioned that the server is hosting a Web API that is supporting User Registration. During the RegisterUser method, the service communicates with a 3rd party KYC service provider. We are supporting 47 countries right now and the all non-US
users are required to upload a passport photo. Yes, I should have mentioned this! My stress test client uniformly loops through the 47 test cases 46 which have images probably averaging around 200KB. It dawned on me that I was probably using up all my upload
bandwidth at home when running the stress test clients.
I repeated the test just with the US test case after increasing minIoThreads and minWorkerThreads to 1000 on the server and maxconnections to 1000 on both the server (for going out to 3rd party) and for on my client machine (to allow 1000 concurrent requests).
I ran a test case of 5000 total with 1000 concurrent and it comleted in 7 mintues and 32 seconds. For 5000 total and 500 concurrent it completed in 7 minutes and 24 seconds. Requests were be complete very fast on the server. When watching concurrent requests
on the server it never reached 50.
Besides minIoThreads, minWorkerThreads, and max concurrent connections, are there any other things I should be look at changing for optimization? I am pretty happy with how things are running now, but always wanting to learn the naunces.
Jan 09, 2018 09:11 PM|lherbinaux|LINK
Sorry about not providing more information. I provided more information about the application in my replay above. I am pretty sure that the issue was my bandwidth between my client machines running at home and the IIS servers because I was sending images
in the registration request. I was using Task Manager and Performance Monitor (ASP.Net and my own custom counters per method). I wasn't using Windows Resource Monitor which I definitely want to take a look into. I am pretty happy with the performance now,
but as I said above, I always want to learn a little more in terms of IIS / ASP.Net optimization. Any other thoughts on other optimizations I can perform?
Thanks for your reply,
Jan 09, 2018 09:45 PM|carehart|LINK
Yep, saw that previous reply you offered. You didn't say there what you do here, in passing, that you're using asp.net.
And yep, good deal that you figured out the problem being on the testing client. That was another reason why I had proposed watching things more closely from the server side: if you had seen that the server was not being stressed, perhaps that may have led
you consider what else it may be.
Indeed, you say in the other reply (now) that your app also involves integration with yet a 3rd party server. Well beware first that any such 3rd party server may have slowness of its own (which appears though to be due to "your server" if you don't realize
it's waiting on the 3rd party. This too is where better monitoring of request processing on your own server might help you see that a long-running request was itself waiting for that 3rd party.
But beware especially that some 3rd parties APIs could have throttling mechanisms, such that when you send only some number of calls they respond quickly, but if you send too many in a short time they may a) reject them or b) just slow them down. You may
not see that in light/functionality testing, but you could see that exacerbated by a load/stress test.
Along the same lines, another unexpected challenge with stress testing is that many such tools by default send no cookies, and that could lead your app (and ASP) to regard each request from such a client to create a new session. If you have code that does
something for new sessions, that code would now run on EVERY stress/load test request, whereas in the real world it may be only a small percent of requests that triggered such new session processing.
Then a question becomes where are sessions stored? By default in asp.net, they're stored in the app pool in IIS ("inproc"), but of course you can use alternative storage, such as in a database or external session storage service ("stateserver"), or third
party state servers. Those can themselves become an unexpected bottleneck.
But you conclude asking what else "other optimizations" you may want to consider. When it comes to asp.net app performance, there are still many more general-interest issues that you could consider, and indeed so many that one can waste time trying to find
"what works" with various "optimizations", something I liken to throwing darts at a board hoping to pop a balloon.
Better diagnostic tools will help you pinpoint where the problem really is, and there are still more than those mentioned so far, whether for better monitoring of .net apps specifically, or .net ORM (if you're using that), or the SQL Server(s) you're calling,
to the servers that both .net/iis and sql server run on, to the network between them, and then the network to 3rd party services (not to mention those 3rd party services themselves).
Troubleshooting poor performance in a web app can involve just so many things, but hopefully there are some ideas to get you going, if you prefer to try entirely on your own.