HTTPS Healthcheck Fails on GKE

1

Hi,

I’m working on setting up an Istio instance using Istioctl on GKE. For some reason the GCP HTTPS (443) healthcheck is failing but is passing on port 80. Has anyone encountered this problem and what is the fix? I’ve confirmed firewall rules are in place to allow GCP health checks for all TCP ports.

I’ve even used tcpdump and netstat to confirm connectivity is available on the ingress gateway pod for port 443.

Here are my Gateway and VirtualService configurations.

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: health-istio-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - "*"
      port:
        name: healthcheck-http
        number: 80
        protocol: HTTP
    - hosts:
        - "*"
      port:
        name: healthcheck-https
        number: 443
        protocol: HTTPS
      tls:
        mode: SIMPLE
        privateKey: /etc/istio/ingressgateway-certs/tls.key
        serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: istio-health
  namespace: istio-system
spec:
  gateways:
    - health-istio-gateway
  hosts:
    - "*"
  http:
    - match:
        - headers:
            user-agent:
              prefix: GoogleHC
          method:
            exact: GET
          uri:
            exact: /
      rewrite:
        authority: istio-ingressgateway.istio-system.svc.cluster.local:15020
        uri: /healthz/ready
      route:
        - destination:
            host: istio-ingressgateway.istio-system.svc.cluster.local
            port:
              number: 15020
2

Eventually I got past this. I’ll post my code snippet down below on what fixed the issue. Ultimately, the Istio gateway Service needed the cloud.google.com/app-protocols annotation for this to work. Really annoying that this isn’t posted in Google’s documentation - the majority of everything I’ve seen is for plain HTTP.

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/app-protocols: '{"https":"HTTPS","http2":"HTTP"}'
    cloud.google.com/neg: '{"ingress": true}'
  labels:
    app: istio-ingressgateway
    istio: ingressgateway
    release: istio
  name: istio-ingressgateway
  namespace: istio-system
spec:
  ports:
  - name: status-port
    port: 15020
    targetPort: 15020
  - name: http2
    port: 80
    targetPort: 80
  - name: https
    port: 443
  - name: kiali
    port: 15029
    targetPort: 15029
  - name: prometheus
    port: 15030
    targetPort: 15030
  - name: grafana
    port: 15031
    targetPort: 15031
  - name: tracing
    port: 15032
    targetPort: 15032
  - name: tcp
    port: 31400
    targetPort: 31400
  - name: tls
    port: 15443
    targetPort: 15443
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  type: NodePort
3

thanks for sharing. I’ve tried it but still ‘Unknown State’ how did you add the annotation? By patching the JSON file?

4

I first generated a manifest with Istioctl, ripped out the Service definition and then used Kustomize to apply the annotations going forward. You could patch with Kubectl as well but I feel using another method to manage it via code is more scalable.

5

@Stefano_Pirrello Can you share how you are doing istio installation?
Specifically , How you have configured istio-ingressgateway with google loadbancer.
I tried to use service type Loadbanacer and NodePort on istio-ingressgate service but lost in the healtcheck and annotation

6

@Pramod_Kumar Did you manage to get this working? Im running in to a similar issue.
@Stefano_Pirrello Are those annotations still relevant? Im finding documentation suggestion using

cloud.google.com/neg='{"exposed_ports": {"80":{}}}'

rather than

cloud.google.com/neg='{"ingress": true}'

not sure what the difference is or if its just changed?