How to terminate SSL at ELB and at istio ingressgateway

Hello.
I am running istio 1.0.5 in kubernetes 1.11 (EKS). I am trying to configure the istio ingressgateway service ELB to terminate SSL and also have the istio ingressgateway pod terminate SSL as well.

My thought is that in AWS we have wildcard certs for our domain that I want to use with the ELB. And then the ingressgateway pod would be configured to use a self signed cert. When I try this configuration it doesn’t look like the connection makes it from the ELB to the istio ingressgateway pod.

Here is how I have configured the istio ingressgateway service annotations to control the configuration of the associated ELB

      annotations:
        service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-west-2:XXXXXXXXXXXX:certificate/XXXX-XXXXX-XXXXXX-XXXXXX"
        service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "https"
        service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"

The ELB listener for https is configured like this

The ingress gateway pod has the self signed certs in this secret

kubectl -n istio-system get secret istio-ingressgateway-certs
NAME                         TYPE                DATA      AGE
istio-ingressgateway-certs   kubernetes.io/tls   3         21h

The istio gateway object

apiVersion: v1
items:
- apiVersion: networking.istio.io/v1alpha3
  kind: Gateway
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.istio.io/v1alpha3","kind":"Gateway","metadata":{"annotations":{},"name":"nginx-ingress-gateway","namespace":"nossl"},"spec":{"selector":{"istio":"ingressgateway"},"servers":[{"hosts":["hostname.sub.domainname"],"port":{"name":"http-nginx-ingress-gateway","number":443,"protocol":"HTTPS"},"tls":{"mode":"SIMPLE","privateKey":"/etc/istio/ingressgateway-certs/tls.key","serverCertificate":"/etc/istio/ingressgateway-certs/tls.crt"}}]}}
    creationTimestamp: 2019-02-05T20:57:25Z
    generation: 1
    name: nginx-ingress-gateway
    namespace: nossl
    resourceVersion: "812558"
    selfLink: /apis/networking.istio.io/v1alpha3/namespaces/nossl/gateways/nginx-ingress-gateway
    uid: a3f1efca-2988-11e9-8300-0a2813d7dab0
  spec:
    selector:
      istio: ingressgateway
    servers:
    - hosts:
      - hostname.sub.domainname
      port:
        name: http-nginx-ingress-gateway
        number: 443
        protocol: HTTPS
      tls:
        mode: SIMPLE
        privateKey: /etc/istio/ingressgateway-certs/tls.key
        serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

The istio virtualservice object

apiVersion: v1
items:
- apiVersion: networking.istio.io/v1alpha3
  kind: VirtualService
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"nginx-ingress","namespace":"nossl"},"spec":{"gateways":["nginx-ingress-gateway"],"hosts":["hostname.sub.domainname"],"http":[{"route":[{"destination":{"host":"nginx-ingress.nossl.svc.cluster.local"}}]}]}}
    creationTimestamp: 2019-02-05T20:57:41Z
    generation: 1
    name: nginx-ingress
    namespace: nossl
    resourceVersion: "807380"
    selfLink: /apis/networking.istio.io/v1alpha3/namespaces/nossl/virtualservices/nginx-ingress
    uid: ad8adc94-2988-11e9-8300-0a2813d7dab0
  spec:
    gateways:
    - nginx-ingress-gateway
    hosts:
    - hostname.sub.domainname
    http:
    - route:
      - destination:
          host: nginx-ingress.nossl.svc.cluster.local
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

When I try to curl the fqdn aliased to the ELB i get the following

❯ curl -v https://hostname.sub.domainname
* Rebuilt URL to: https://hostname.sub.domainname/
*   Trying 10.52.25.245...
* TCP_NODELAY set
* Connected to hostname.sub.domainname (10.52.25.245) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.sub.domainname
*  start date: Jun 14 00:00:00 2018 GMT
*  expire date: Jul 14 12:00:00 2019 GMT
*  subjectAltName: host "hostname.sub.domainname" matched cert's "*.sub.domainname"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: hostname.sub.domainname
> User-Agent: curl/7.54.0
> Accept: */*
>

< HTTP/1.1 408 REQUEST_TIMEOUT
< Content-Length:0
< Connection: Close
<
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):

When I just terminate SSL at the ELB and do not have the ingress gatway it works fine.
When I do not perform SSL at the ELB and only do it at the ingress gateway it works fine.

Combining the two configurations does not work though and i’m hoping someone can point out the error of my ways as I would like to use our wildcard cert from AWS.

Thanks. G

When i change the gateway object to use a wild card hosts entry the connection works. Not sure how to debug this further.

So, generally speaking, you can only terminate a particular TLS/SSL session once, where by “terminate” we mean authenticate and decrypt. When an ELB is set to terminate TLS, it means that it will forward plaintext traffic to the upstream target.

Note that the Istio Ingress Gateway can terminate the end user TLS connection, decrypt the traffic, but then re-encrypt it using mutually authenticated TLS before delivering it to the backends.

It’s not clear to me whether AWS ELBs support terminating the incoming TLS connection and then re-encrypting in this way. The APIs seem to allow it, by setting a target group of type HTTPS, but I haven’t tried it.

@spikecurtis

If the ELB is configured to terminate TLS/SSL that is the cert that the end users sees. But the ELB does not have to send the traffic to the next hop in plain text. That is configurable. This annotation on the istio-ingressgateway service how we tell the ELB that the next hop is going to be using https
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "https"

It’s a pretty common pattern to terminate TLS/SSL at the load balancer using a cert signed by a well known CA and then have the ELB proxy the traffic to the backend using https and have that next hop be terminated using a self signed CA. This all works and is pretty common practice.

And this works with istio-ingressgateway when the gateway object has the following configuration

hosts:
  - "*"

But not with a fully qualified domainname like

hosts:
  - "hostname.domainname.net"

I was hoping to find some way to see the decision making process in the logs where one of these scenarios works but the other does not.

Thanks. G

@Guido_Pepper, I’m having the exact same issue (eks 1. 12, istio 1.1.0), did you manage to come up with any solutions other than making ‘*’ for hosts entry?

@Guido_Pepper @arek.chedoszko

same problem here, any update and what’s the CN you are specifying on the self signed certificate?

I set the hostname to the actual FQDN and sent some requests and I couldn’t see any of those hitting the gateway from the debug logs but as soon as I changed the hostname to “*” I immediately saw those requests coming through in the logs. Maybe this is something on the ELB? I wonder if this is because the hostname on ELB and the gateway is the same so the ELB goes into some recursion but no solution so far

This impacts the usage of the gateway though as we cannot use more than one host for the gateway to process. As a workaround for multiple hosts maybe just set the CN to “*” and just apply one gateway resource with all virtual services pointing to this gateway.