Istio Secure Gateway TLS 404 NR Error

Hi,

We have an Istio 1.8.0 running our on-prem kubernetes cluster v.19. Before securing gateway with TLS everything works fine where I can access the frontend out of our cluster from internet…

We have F5 Load Balancer in front of our k8s cluster where all certs ext installed on this LB… So my issue is after I secure gateway via TLS I recieved an ERR_EMPTY_RESPONSE error. Here are what I did so far and how resources looks like…

I created a secret in istio-system namespace according to the organization’s certs with key and crt.

$ kubectl get secrets -n istio-system | grep dev
dev-credential                                 kubernetes.io/tls                     2      14h

Here is the log from istio ingressgateway… Seems NR route but ı could not able to see any fault for my virtual service.

[2021-02-02T06:48:51.884Z] "- - -" 0 NR "-" 0 0 0 - "-" "-" "-" "-" "-" - - 10.6.5.216:8443 10.6.0.10:19712 - -

I checked the http://localhost:15000/config_dump and here is how look like where I am not sure why I have kubernetes in front of crendential that I created…

        "tls_certificate_sds_secret_configs": [
         {
          "name": "kubernetes://hub-dev-credential",
          "sds_config": {
           "ads": {},
           "resource_api_version": "V3"
          }
         }
        ]

Lastly I believe our F5 LB is configured correctly cause when I curl the addresses seems it can be passed from our F5 LB.

curl -vv -x http://<proxy> -I https://foo-example.net

*   Trying 53.xxxx..
* TCP_NODELAY set
* Connected to <proxy> ... port 3128 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to foo-example.com:443
> CONNECT  foo-example.com:443 HTTP/1.1
> Host:  foo-example.com:443

And istio ingressgateway service

  externalTrafficPolicy: Cluster
  loadBalancerSourceRanges:
  - 0.0.0.0/0
  ports:
  - name: status-port
    nodePort: 35553
    port: 15021
    protocol: TCP
    targetPort: 15021
  - name: http2
    nodePort: 41979
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    nodePort: 31376
    port: 443
    protocol: TCP
    targetPort: 8443
  - name: tcp-istiod
    nodePort: 39282
    port: 15012
    protocol: TCP
    targetPort: 15012
  - name: tls
    nodePort: 31373
    port: 15443
    protocol: TCP
    targetPort: 15443
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 53..x.x.x.

Here is Gateway and VirtualService

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: dev-gateway
  namespace: dev
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: dev-credential
      hosts:
        - "foo-example.net"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: hub-dev
  namespace: dev
spec:
  hosts:
    - foo-example.net
  gateways:
    - dev-gateway
  http:
    - route:
        - destination:
            host: frontend.dev.svc.cluster.local
            port:
              number: 80
      rewrite:
        authority: backend.dev.svc.cluster.local