Istio 1.3.3 bad performance when using HTTPS

1

Anyone else knows how to improve the performance of Istio when using HTTPS?

I’m getting the following performance when comparing HTTP vs HTTPS:

siege -c128 -d1 -t 10S https://url/
** SIEGE 4.0.4
** Preparing 128 concurrent users for battle.
The server is now under siege…
Lifting the server siege…
Transactions: 680 hits
Availability: 100.00 %
Elapsed time: 9.24 secs
Data transferred: 1.71 MB
Response time: 0.16 secs
Transaction rate: 73.59 trans/sec
Throughput: 0.18 MB/sec
Concurrency: 11.65
Successful transactions: 769
Failed transactions: 0
Longest transaction: 1.13
Shortest transaction: 0.05

siege -c128 -d1 -t 10S http://url/
** SIEGE 4.0.4
** Preparing 128 concurrent users for battle.
The server is now under siege…
Lifting the server siege…
Transactions: 2596 hits
Availability: 100.00 %
Elapsed time: 9.92 secs
Data transferred: 5.76 MB
Response time: 0.02 secs
Transaction rate: 261.69 trans/sec
Throughput: 0.58 MB/sec
Concurrency: 4.39
Successful transactions: 2596
Failed transactions: 0
Longest transaction: 0.19
Shortest transaction: 0.00

At the same time, the performance when using the default ingress with HTTPS is 4 times better:

siege -c128 -d1 -t 10S https://url/
** SIEGE 4.0.4
** Preparing 128 concurrent users for battle.
The server is now under siege…
Lifting the server siege…
Transactions: 2163 hits
Availability: 100.00 %
Elapsed time: 9.14 secs
Data transferred: 3.50 MB
Response time: 0.06 secs
Transaction rate: 236.65 trans/sec
Throughput: 0.38 MB/sec
Concurrency: 15.34
Successful transactions: 2163
Failed transactions: 0
Longest transaction: 1.07
Shortest transaction: 0.02

Followed the guide from Istio / Secure Gateways to setup the certificates using SDS, but the performance is similar to using the file mount.

2

This sounds like a bug. Some quick questions to clarify:

  1. Which application was used for testing?
  2. You mentioned that the 3rd siege request was sending request to default ingress (istio ingress gateway?) How about the first two? Are they using other ingress?

If possible, can you try the latest 1.4.2 to reproduce the same issue? or let us know the brief steps to reproduce it.

3

How did you enable HTTPS in this scenario? Is it on the load balancer (EG: AWS ELB)?

4

@taohe Answers to your questions:

  1. It seems to be happening with any application the same. The simplest one I tested was a “Hello World” web server written in Golang.
  2. Sorry for the confusion, the third results is the siege sending requests to the nginx ingress. The first two results are using the istio ingress gateway (one result for http and one for https). For http the performance is similar between istio and nginx, the problem seems to be when using https.

I have a Rancher installation with two ingress controllers (https://rancher.com/docs/rancher/v2.x/en/cluster-admin/tools/istio/#two-ingresses)

I will try to reproduce to see if I can reproduce the issue with 1.4.2. I believe the following steps should help to reproduce the problem:

@polothy The HTTPS is enabled on the nginx ingress. It uses the certificates from the tls secret. The load balancer doesn’t do any SSL termination. The ingress definition contains this tls block:

tls:
- hosts:
  - www.example.com
  secretName: example-tls
5

@taohe seems like the issue still persists when using Istio 1.4.3.

Pasted the test results here: https://github.com/liquid-cloud/hello-go/blob/781af1a7cf0b183923b32b091ec51377e2aaa987/istio_results.md

The manifests for the application are here: https://github.com/liquid-cloud/hello-go/tree/master/kube-manifests

Note that it’s one single application exposed with both Istio and Nginx using HTTP/HTTPS.

You should also be able to reproduce the issue by deploying the yamls in any cluster.

Is this expected from Istio when using HTTPS?