ISTIO fails to initiate HTTPS request to backend-service (via destination-rule)

I have just started playing around with the Istio on my EKS cluster.

With my current setup, I have AWS ELB sitting at the front, which will then route the request toward istio-ingressgateway via K8s-ingress.

I’m trying to do something like this below:

AWS ALB ==(HTTP 80)==> istio-ingressgateway ==(HTTPS-5601)==> kibana-backend-service

My Kibana is currently exposed on HTTPS port 5601 (xpack-security enabled)

Here is what I try:

  1. I set up the AWS Ingress to point to istio-ingressgateway:80
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  namespace: istio-system
  annotations:
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":{
      "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-west-1:12345678910:certificate/26229a3b-26a9-4dce-ba02-95274978cac9
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    #alb.ingress.kubernetes.io/backend-protocol: HTTPS
    kubernetes.io/ingress.class: alb
  labels:
    app.kubernetes.io/instance: ingress-sandboxa
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress
    helm.sh/chart: ingress-0.1.0
  name: istio-ingress-sandboxa
spec:
  rules:
  - host: errbit.eu-west-1.sandbox.test.com
    http:
      paths:
      - backend:
          serviceName: ssl-redirect
          servicePort: use-annotation
        path: /*
      - backend:
          serviceName: istio-ingressgateway
          servicePort: 80
        path: /*
  - host: grafana.eu-west-1.sandbox.test.com
    http:
      paths:
      - backend:
          serviceName: ssl-redirect
          servicePort: use-annotation
        path: /*
      - backend:
          serviceName: istio-ingressgateway
          servicePort: 80
        path: /*
  - host: kibana.eu-west-1.sandbox.test.com
    http:
      paths:
      - backend:
          serviceName: ssl-redirect
          servicePort: use-annotation
        path: /*
      - backend:
          serviceName: istio-ingressgateway
          servicePort: 80
        path: /*
  1. I set up a Gateway listening on HTTP:80.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  namespace: istio-system
  name: kibana
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - kibana.eu-west-1.sandbox.test.com
  1. Then I create a VirtualService pointing to that Gateway, and setting a DestinationRule, aiming to initiate the HTTPS call to the backend-service-pod via tls:SIMPLE.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  namespace: istio-system
  name: kibana
spec:
  hosts:
  - kibana.eu-west-1.sandbox.test.com
  gateways:
  - istio-system/kibana
  http:
  - match:
    - uri:
        prefix: /
    rewrite:
       uri: /
    route:
    - destination:
        host: monpack-sandboxa-kibana.monitoring.svc.cluster.local
        subset: v1
        port:
          number: 5601
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  namespace: istio-system
  name: kibana
spec:
  host: monpack-sandboxa-kibana.monitoring.svc.cluster.local
  subsets:
  - name: v1
    labels:
      release: monpack-sandboxa
    trafficPolicy:
      portLevelSettings:
      - port:
          number: 5601
        loadBalancer:
          simple: ROUND_ROBIN
      tls:
        mode: SIMPLE # initiates HTTPS
        privateKey: /etc/istio/ingressgateway-certs/tls.key
        caCertificates: /etc/istio/ingressgateway-certs/tls.crt        

However, things doesn’t seem to work well this time.

When I try to do the CURL: curl -v https://kibana.eu-west-1.sandbox.test.com
I got the 503 with this error : upstream connect error or disconnect/reset before headers

* Rebuilt URL to: https://kibana.eu-west-1.sandbox.test.com/
*   Trying 54.76.109.48...
* TCP_NODELAY set
* Connected to kibana.eu-west-1.sandbox.test.com (54.76.109.48) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.eu-west-1.sandbox.test.com
*  start date: Aug 21 00:00:00 2020 GMT
*  expire date: Sep 20 12:00:00 2021 GMT
*  subjectAltName: host "kibana.eu-west-1.sandbox.test.com" matched cert's "*.eu-west-1.sandbox.test.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fffcdebc580)
> GET / HTTP/2
> Host: kibana.eu-west-1.sandbox.test.com
> User-Agent: curl/7.58.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 503
< date: Mon, 24 Aug 2020 12:47:27 GMT
< content-type: text/plain
< content-length: 95
< server: istio-envoy
< x-envoy-upstream-service-time: 62
<
* Connection #0 to host kibana.eu-west-1.sandbox.test.com left intact
upstream connect error or disconnect/reset before headers. reset reason: connection termination

When I check the log in the istio-ingressgateway’s pod, I found this one :

"GET / HTTP/1.1" 503 URX "-" "-" 0 95 62 62 "59.102.86.159,10.199.64.125" "curl/7.58.0" "16085fc0-823b-95cb-a571-0d43e1cf117f" "kibana.eu-west-1.sandbox.test.com" "10.199.64.26:5601" outbound|5601|v1|monpack-sandboxa-kibana.monitoring.svc.cluster.local 10.199.64.175:44888 10.199.64.175:8080 10.199.64.125:24894 - -

And then on the istio-proxy container within Kibana’s pod, I found this log:

"GET / HTTP/1.1" 503 UC "-" "-" 0 95 4 - "59.102.86.159,10.199.64.125" "curl/7.58.0" "16085fc0-823b-95cb-a571-0d43e1cf117f" "kibana.eu-west-1.sandbox.test.com" "127.0.0.1:5601" inbound|5601|http|monpack-sandboxa-kibana.monitoring.svc.cluster.local 127.0.0.1:38532 10.199.64.26:5601 10.199.64.125:0 outbound_.5601_.v1_.monpack-sandboxa-kibana.monitoring.svc.cluster.local default

Furthermore, on the kibana container within Kibana’s pod, I found this log :

{"type":"error","@timestamp":"2020-08-24T12:47:27Z","tags":["connection","client","error"],"pid":6,"level":"error","error":{"message":"140484513027968:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n","name":"Error","stack":"Error: 140484513027968:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"},"message":"140484513027968:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"}

Correct me if I’m wrong, baving a quick look on the log above, it seems that the request has gone through as expected :
AWS ALB => istio-ingress-gateway => pod’s istio(envoy) proxy => pod

Despite it was eventually being rejected by the backend-pod - as can be seen from the SSL3 error log.

In a nutshell, what I was trying to do is letting AWS-ALB to capture external-https traffic, in which then it terminates the SSL there, and forward it to istio-ingressgateway via HTTP, and then after that let istio forward this traffic as HTTPS to the backend-service.

Hence, I’m just wondering if I miss any step here?
Or, perhaps if there is another better way to achieve it? :slight_smile:

PS :

  • I am on Istio 1.6.8
  • I install the istio via istioctl below:
istioctl install \
	--set profile=demo \
	--set values.grafana.enabled=false \
	--set values.prometheus.enabled=false \
	--set values.gateways.istio-ingressgateway.type=NodePort
  • I haven’t touched the mTLS setting yet
  • I have populated the istio-ingressgateway-certs K8s secret

Many thanks in advance