How to terminate SSL at NLB and at istio ingressgateway to provide end to end encryption on EKS?

1

istio version used: 1.12.1.
Istio operator deployed using helm chart that is downloaded from here:
curl -L https://git.io/getLatestIstio | ISTIO_VERSION=1.12.1 sh -

I want to create a setup where the client creates a request that goes to NLB. HTTPS termination happens at NLB using a certificate created by ACM. Then we start another HTTPS request that goes to istio ingress gateway where I have a self-signed certificate to terminate HTTPS started by NLB. I am seeing an issue with this setup.

What is working is : client -https->NLB(HTTPS terminate here and HTTP request goes further) -http-> istio-ingressgateway. I want to use NLB for https so that i can use ACM created certificate and don’t want to do direct HTTPS termination at gateway level.

I am using istio operator to create istio control plane using the below file

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: demo-istiocontrolplane
spec:
  profile: demo
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          service:
            type: LoadBalancer
          serviceAnnotations:
            external-dns.alpha.kubernetes.io/hostname: "istio.mydomain.com"
            external-dns.alpha.kubernetes.io/ttl: "60"
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
            service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "https" # starting backend as HTTPS, tried by changing this as SSL, but didn't work
            service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
            service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "REDACTED_ACM_CERT_ARN"
            service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https

My gateway and virtual service file are below:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-ingressgateway
  namespace: istio-system
  labels:
    app: istio-ingressgateway
    istio: ingressgateway  # use istio default controller
    release: istio
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      protocol: HTTP
      name: http
    hosts:
    - "*"
    tls:
      httpsRedirect: true
  - port:
      number: 443
      protocol: HTTPS
      name: https-default
    tls:
      mode: SIMPLE
      credentialName: istio-cred
    hosts:
    - "istio.mydomain.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: bookinfo
spec:
  hosts:
   - "istio.mydomain.com"
  gateways:
   - istio-ingressgateway
  http:
    - route:
      - destination:
          host: productpage.bookinfo.svc.cluster.local
          port:
            number: 9080

A certificate is created using the below commands:

Create CA certificate: `openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=mydomain.com Inc./CN=mydomain.com -keyout mydomain.com.key -out mydomain.com.crt`

Create domain certificate: `openssl req -out istio.mydomain.com.csr -newkey rsa:2048 -nodes -keyout istio.mydomain.com.key -subj "/CN=istio.mydomain.com/O=istio.mydomain.com app"`

Sign certificate: `openssl x509 -req -days 30 -CA mydomain.com.crt -CAkey mydomain.com.key -set_serial 0 -in istio.mydomain.com.csr -out istio.mydomain.com.crt`

Create secret in istio-system namespace: `kubectl create secret tls istio-cred --key=istio.mydomain.com.key --cert=istio.mydomain.com.crt -n istio-system`

Curl command response:

curl -v https://istio.mydomain.com                          
*   Trying x.x.x.x...
* TCP_NODELAY set
* Connected to istio.mydomain.com (x.x.x.x) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.mydomain.com
*  start date: Oct  7 00:00:00 2021 GMT
*  expire date: Nov  5 23:59:59 2022 GMT
*  subjectAltName: host "istio.mydomain.com" matched cert's "*.mydomain.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: istio.mydomain.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
* TLSv1.2 (IN), TLS alert, close notify (256):
* Empty reply from server
* Connection #0 to host istio.mydomain.com left intact
curl: (52) Empty reply from server
* Closing connection 0

Final intended behaviour:
Client -HTTPS->NLB(Terminate, start https for backend) -HTTPS->istio-ingressgateway

I am running a book info service in bookinfo namespace and sending requests to productpage (productpage.bookinfo.svc.cluster.local) service as per virtual service.

can you guys guide what needs to be changed?

I have verified that by doing port forward directly to istio-ingressgateway https is working fine with self-sign certificate, issue is happening when a request comes from NLB.

2

Hi Mitesh,

I know this thread is pretty old, but I am running into the same thing that you are, and am not sure what the solution is.

Were you able to figure out what needed to be done here?

3

One key insight that helped me here is that NLB or ALB works with a self-signed certificate and they don’t need to trust them as all of them are internal.

Create CA certificate: openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=[dev.example.com](http://dev.example.com/) Inc./CN=[dev.example.com](http://dev.example.com.com/)' -keyout dev.example.com.key -out dev.example.com.crt

Create domain certificate: openssl req -out istio.dev.example.com.csr -newkey rsa:2048 -nodes -keyout istio.dev.example.com.key -subj "/CN=[istio.dev.example.com/O=istio.dev.example.com](http://istio.dev.example.com/O=istio.dev.example.com) app"

Sign certificate: openssl x509 -req -days 30 -CA dev.example.com.crt -CAkey dev.example.com.key -set_serial 0 -in istio.dev.example.com.csr -out istio.dev.example.com.crt

Create secret in istio-system namespace: kubectl create secret tls istio-cred --key=istio.dev.example.com.key --cert=istio.dev.example.com.crt -n istio-system

Once this is created, update the gateway:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-ingressgateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: istio-cred
    hosts:
    - istio.dev.example.com

Keep SSL termination at NLB, so now internet → NLB, certificate issues by ACM or some authority everyone trusts while NLB to the gateway is a self-signed certificate.

To verify do:
Do port forward: kubectl port-forward -n istio-system service/istio-ingressgateway 8000:443

Curl command to test certificate: curl -v -HHost:istio.dev.example.com --resolve "istio.dev.example.com:8000:127.0.0.1" --cacert dev.example.com.crt "[https://istio.dev.example.com:8000](https://istio.dev.example.com:8000/)"

1 Like
4

@Mitesh_Sharma Hello, Did you solve the problems? I got the same issue.