503 errors when using TLS on gateway

Networking
1

Hi there,

I am new to istio, and am having some trouble with TLS on an istio gateway resource. Any and all help greatly appreciated!

The following setup works as expected:

I am using AWS, and have an ELB (classic) load balancer which was created with defaults by istioctl. The istio/istio-ingressgateway service has annotations which terminate TLS at the load balancer, and send the plaintext request to the gateway. The annotations are:

service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:<redacted>"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"

My gateway and virtual service are configured with just port 80, and all is working correctly. However, I want communication between the load balancer and the gateway to also be encrypted.

This is where things stop working…

I change the
service.beta.kubernetes.io/aws-load-balancer-backend-protocol annotation above from http -> https. I then follow the instructions with the documentation (for istio 1.5) but now persistently receive HTTP 503’s from my service.

The changes I made were to add an HTTPS/443 section to my gateway definition and create a self signed certificate (using cert-manager). These are detailed below:

gateway.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: default-gateway
  namespace: default
spec:
  selector:
    istio: ingressgateway 
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: default-gateway-self-signed-cert
    hosts:
    - '*.<redacted>.net'
  - port:
      number: 80
      name: http
      protocol: HTTP
    tls:
      httpsRedirect: true 
    hosts:
      '- '*.<redacted>.net'

secret

The secret (default-gateway-self-signed-cert) was created using the command:

kubectl -n istio-system \
    create secret generic default-gateway-self-signed-cert \
    --from-file=key=tls.key \
    --from-file=cert=tls.cert

(where tls.crt is a self signed certificates from cert-manager.)

virtual-service.yaml (unchanged)

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: my-service
spec:
  hosts:
  - <redacted>.net
  gateways:
  - default-gateway
  http:
  - route:
    - destination:
        port:
          number: 80
        host: my-service.default.svc.cluster.local

I have tried to look at logs for the istio ingress service pod and my service pod, but I cannot see any activity when I make my request on port 443. I simply receive an HTTP 503.

Thanks so much in advance for any pointers in the right direction!

2

Hi there, :slight_smile:
could you please turn on logs on ingressgateway? You can do so by using pod name and istioctl dashboard envoy pod-name.istio-namespace. It will forward the port for you and then use POST /logging?level=debug.

After that a lot of logs will spawn and you could try that call again.

Btw. do not forget to turn that logging to at least info. :slight_smile:

1 Like
3

@Tomas_Kohout thanks for your reply.

I have done, but now have realised that the failure happens much higher up the chain.

If I leave set the annotation service.beta.kubernetes.io/aws-load-balancer-backend-protocol to http, then the EC2 instances pass the ELB healthchecks and are “InService”. The healthcheck is configured in AWS as TCP:31919.

But as soon as I change the annotation to https, the EC2 instances fail their healthchecks and are “OutOfService”. I notice the health check gets changed to SSL:31919. Because the instances are failing their health checks, my gateway object is never even hit because the ELB has taken the EKS nodes out of service.

Is there something else I should be configuring when I set this annotation to https? Also, is there a way in which I can see healthcheck logs from the load balancer? Turning on debug logs provides much more information on the istio-ingressgateway pod, but I can’t see any requests from the load balancer for the TCP health checks?

4

As an update to this, it seems if I manually override the health check on the ELB to be TCP instead of SSL, then everything works as expected. I.e. the load balancer terminates TLS at the edge (using an ACM certificate), and then proxies the request upstream using HTTPS (which uses the self signed certificate in the istio gateway).

However overriding the healthcheck configuration makes me uneasy, as the automatic configuration is obviously there by design. So I am curious to know why the healthcheck fails using SSL, which is applied when configuring the service.beta.kubernetes.io/aws-load-balancer-backend-protocol annotation to https.

6

Hey @celitcfunk

I just created an account to answer your question as I’d seen the same behaviour at work when setting up some Istio Ingress Gateways on AWS (we already had gateways configured in our On-Prem data centre).

The SSL ping health check will fail from the ELBs as it’s hitting the cluster IP directly rather than via a specific host - this causes the Envoy Proxy in your Pod to not recognise the traffic and fails to serve a correct response back to the health check.

Another way to product the intended behaviour would be to add the wildcard * value to your hosts list in your Gateway resource type. This causes the envoy proxy to correctly respond the the ELB health check, and traffic works as you’d expect.

This also probably isn’t the best way to solve the problem, but it does mean you can use the SSL health check type for your load balancer serving HTTPS traffic. I haven’t had a further play around beyond this to see if there is a way to correctly isolate/serve ELB health check connections.

7

In legacy Helm charts there is a mention that first port defined on ingress gateway service will be pinged from ELB. Maybe it could be worth of a try.