I have just started playing around with the Istio on my EKS cluster.
With my current setup, I have AWS ELB sitting at the front, which will then route the request toward istio-ingressgateway via K8s-ingress.
I’m trying to do something like this below:
AWS ALB ==(HTTP 80)==> istio-ingressgateway ==(HTTPS-5601)==> kibana-backend-service
My Kibana is currently exposed on HTTPS port 5601 (xpack-security enabled)
Here is what I try:
- I set up the AWS Ingress to point to istio-ingressgateway:80
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
namespace: istio-system
annotations:
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":{
"Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-west-1:12345678910:certificate/26229a3b-26a9-4dce-ba02-95274978cac9
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
alb.ingress.kubernetes.io/scheme: internet-facing
#alb.ingress.kubernetes.io/backend-protocol: HTTPS
kubernetes.io/ingress.class: alb
labels:
app.kubernetes.io/instance: ingress-sandboxa
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: ingress
helm.sh/chart: ingress-0.1.0
name: istio-ingress-sandboxa
spec:
rules:
- host: errbit.eu-west-1.sandbox.test.com
http:
paths:
- backend:
serviceName: ssl-redirect
servicePort: use-annotation
path: /*
- backend:
serviceName: istio-ingressgateway
servicePort: 80
path: /*
- host: grafana.eu-west-1.sandbox.test.com
http:
paths:
- backend:
serviceName: ssl-redirect
servicePort: use-annotation
path: /*
- backend:
serviceName: istio-ingressgateway
servicePort: 80
path: /*
- host: kibana.eu-west-1.sandbox.test.com
http:
paths:
- backend:
serviceName: ssl-redirect
servicePort: use-annotation
path: /*
- backend:
serviceName: istio-ingressgateway
servicePort: 80
path: /*
- I set up a Gateway listening on HTTP:80.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
namespace: istio-system
name: kibana
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- kibana.eu-west-1.sandbox.test.com
- Then I create a VirtualService pointing to that Gateway, and setting a DestinationRule, aiming to initiate the HTTPS call to the backend-service-pod via tls:SIMPLE.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
namespace: istio-system
name: kibana
spec:
hosts:
- kibana.eu-west-1.sandbox.test.com
gateways:
- istio-system/kibana
http:
- match:
- uri:
prefix: /
rewrite:
uri: /
route:
- destination:
host: monpack-sandboxa-kibana.monitoring.svc.cluster.local
subset: v1
port:
number: 5601
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
namespace: istio-system
name: kibana
spec:
host: monpack-sandboxa-kibana.monitoring.svc.cluster.local
subsets:
- name: v1
labels:
release: monpack-sandboxa
trafficPolicy:
portLevelSettings:
- port:
number: 5601
loadBalancer:
simple: ROUND_ROBIN
tls:
mode: SIMPLE # initiates HTTPS
privateKey: /etc/istio/ingressgateway-certs/tls.key
caCertificates: /etc/istio/ingressgateway-certs/tls.crt
However, things doesn’t seem to work well this time.
When I try to do the CURL: curl -v https://kibana.eu-west-1.sandbox.test.com
I got the 503 with this error : upstream connect error or disconnect/reset before headers
* Rebuilt URL to: https://kibana.eu-west-1.sandbox.test.com/
* Trying 54.76.109.48...
* TCP_NODELAY set
* Connected to kibana.eu-west-1.sandbox.test.com (54.76.109.48) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.eu-west-1.sandbox.test.com
* start date: Aug 21 00:00:00 2020 GMT
* expire date: Sep 20 12:00:00 2021 GMT
* subjectAltName: host "kibana.eu-west-1.sandbox.test.com" matched cert's "*.eu-west-1.sandbox.test.com"
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fffcdebc580)
> GET / HTTP/2
> Host: kibana.eu-west-1.sandbox.test.com
> User-Agent: curl/7.58.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 503
< date: Mon, 24 Aug 2020 12:47:27 GMT
< content-type: text/plain
< content-length: 95
< server: istio-envoy
< x-envoy-upstream-service-time: 62
<
* Connection #0 to host kibana.eu-west-1.sandbox.test.com left intact
upstream connect error or disconnect/reset before headers. reset reason: connection termination
When I check the log in the istio-ingressgateway’s pod, I found this one :
"GET / HTTP/1.1" 503 URX "-" "-" 0 95 62 62 "59.102.86.159,10.199.64.125" "curl/7.58.0" "16085fc0-823b-95cb-a571-0d43e1cf117f" "kibana.eu-west-1.sandbox.test.com" "10.199.64.26:5601" outbound|5601|v1|monpack-sandboxa-kibana.monitoring.svc.cluster.local 10.199.64.175:44888 10.199.64.175:8080 10.199.64.125:24894 - -
And then on the istio-proxy container within Kibana’s pod, I found this log:
"GET / HTTP/1.1" 503 UC "-" "-" 0 95 4 - "59.102.86.159,10.199.64.125" "curl/7.58.0" "16085fc0-823b-95cb-a571-0d43e1cf117f" "kibana.eu-west-1.sandbox.test.com" "127.0.0.1:5601" inbound|5601|http|monpack-sandboxa-kibana.monitoring.svc.cluster.local 127.0.0.1:38532 10.199.64.26:5601 10.199.64.125:0 outbound_.5601_.v1_.monpack-sandboxa-kibana.monitoring.svc.cluster.local default
Furthermore, on the kibana container within Kibana’s pod, I found this log :
{"type":"error","@timestamp":"2020-08-24T12:47:27Z","tags":["connection","client","error"],"pid":6,"level":"error","error":{"message":"140484513027968:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n","name":"Error","stack":"Error: 140484513027968:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"},"message":"140484513027968:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"}
Correct me if I’m wrong, baving a quick look on the log above, it seems that the request has gone through as expected :
AWS ALB => istio-ingress-gateway => pod’s istio(envoy) proxy => pod
Despite it was eventually being rejected by the backend-pod - as can be seen from the SSL3 error log.
In a nutshell, what I was trying to do is letting AWS-ALB to capture external-https traffic, in which then it terminates the SSL there, and forward it to istio-ingressgateway via HTTP, and then after that let istio forward this traffic as HTTPS to the backend-service.
Hence, I’m just wondering if I miss any step here?
Or, perhaps if there is another better way to achieve it? 
PS :
- I am on Istio 1.6.8
- I install the istio via istioctl below:
istioctl install \
--set profile=demo \
--set values.grafana.enabled=false \
--set values.prometheus.enabled=false \
--set values.gateways.istio-ingressgateway.type=NodePort
- I haven’t touched the mTLS setting yet
- I have populated the istio-ingressgateway-certs K8s secret
Many thanks in advance