Hi All,
I have my spring boot services deployed to AKS cluster and had Istio installed.
While I am doing the load testing with JMeter am getting the below exception.
Non HTTP response code: org.apache.http.conn.HttpHostConnectException/Non HTTP response message: Connect to xxx.xxx.cloudapp.azure.com:80 [xxx.xxx.cloudapp.azure.com\/xxx.xx.xxx.xxx] failed: Cannot assign requested address (connect failed)
I have tried increased no:of pods for “istio-ingressgateway” & “istio-pilot” from https://github.com/istio/istio/issues/10904.
If the load is anything more than 1000 req/sec the above exception starts occurring and the remaining tests starts failing. Tried different versions of Istio as well.
Current confiruation is:
AKS - 1.12.8
Istio - 1.1.7
Thanks in advance,
Binoy
Hello,
The problem comes from TCP layer. You can figure out if the error is generated by client machine (where jmeter runs) by modifying the ip_local_port_range. Probably your client runs out of ports, because each jmeter thread can open several tcp connections.
What OS do you use for jmeter client?
Linux : http://www.linuxbrigade.com/reduce-time_wait-socket-connections/
Windows : https://support.microsoft.com/en-gb/help/929851/the-default-dynamic-port-range-for-tcp-ip-has-changed-in-windows-vista
You could play a bit and change the port range. First run your tests with a number of threads which doesn’t trow errors. Than decrease the port range and set only 1000 ephemeral ports for ex. and run the tests again and it should cause failures. If that happens, the problem is your jmeter client. In order to solve that set the range to the maximum number of ports, enable port reuse, reduce tcp reuse wait_time (They are all described in the links above). If that is not enough, you should go for distributed testing.
Cheers,
Mihai
1 Like
Thanks a lot for your feedback @mihai_barta. Much appreciated 
I will follow the steps mentioned in the above link (Linux).
Another way for supporting more TCP connections is to allocate multiple IPs for client machine and configure jmeter to send requests from all the IPs ( the number of ephemeral ports are per IP) , but I haven’t tested it yet. Distributed testing ( or just run a few jmeter clients in master mode on different machines) is still a better approach for this purpose. Good luck.
1 Like
I am facing the same issue. But the strange thing is i am facing it only for first couple thousand requests only. After that it is not giving me any error.
So when jmeter starts spawning threads, for first 3 or 4 seconds, i get following error.
Non HTTP response code: org.apache.http.conn.HttpHostConnectException,Non HTTP response message: Connect to mercury-app:80 [mercury-app/172.16.53.247] failed: Connection refused (Connection refused)
But after that it works fine.
Here is my Jmeter log shows.
calling JMX test apply
Nov 21, 2019 11:30:04 AM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
Creating summariser
Created the tree successfully using /home/jmx/apply/main.jmx
Starting the test @ Thu Nov 21 11:30:05 PST 2019 (1574364605340)
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
#0 Threads: 1/100 Samples: 1 Latency: 0 Resp.Time: 43 Errors: 1
#1 Threads: 3/100 Samples: 1218 Latency: 0 Resp.Time: 0 Errors: 1218
#2 Threads: 5/100 Samples: 1609 Latency: 1 Resp.Time: 1 Errors: 1429
#3 Threads: 7/100 Samples: 402 Latency: 13 Resp.Time: 13 Errors: 0
#4 Threads: 9/100 Samples: 371 Latency: 18 Resp.Time: 19 Errors: 0
#5 Threads: 11/100 Samples: 329 Latency: 28 Resp.Time: 28 Errors: 0
#6 Threads: 13/100 Samples: 410 Latency: 27 Resp.Time: 27 Errors: 0
#7 Threads: 15/100 Samples: 347 Latency: 37 Resp.Time: 37 Errors: 0
Any pointers.
Thanks
Found the issue. Istio proxy is not up, when jmeter starts firing the request.
Put some delay before jmeter starts , it fixed the issue.