GKE ManagedCertificate + Istio Gateways on Istio 1.6

Networking
1

Hi all,

Decided to upgrade to Istio 1.6 yesterday on a GKE cluster, and attempt to get HTTPS via managed certificate while using Istio Gateways.

I have been trying to implement this for the last 24 hours, but am unable to achieve any progress achieving it.

service:

apiVersion: v1
kind: Service
metadata:
  name: monolith-backend-v1
  namespace: monolith
  labels:
    app: monolith-backend
    service: monolith-backend
    version: v1
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: monolith-backend
    version: v1

virtualservice:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: monolith-backend
  namespace: monolith
spec:
  hosts:
  - "*"
  gateways:
  - istio-system/api-gateway
  http:
  - match:
    - uri:
        prefix: /managed
    - uri:
        prefix: /callbacks
    - uri:
        prefix: /legacy
    route:
    - destination:
        host: monolith-backend-v1
        port:
          number: 8080

ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: istio-ingress
  namespace: istio-system
  annotations:
    kubernetes.io/ingress.global-static-ip-name: gke-ip-address
    networking.gke.io/managed-certificates: managed-certificate
spec:
  rules:
  - host: 'xxxxxxxxx.com'
    http:
      paths:
      - paths:
        backend:
          serviceName: istio-ingressgateway
          servicePort: 80

gateway:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: api-gateway
  namespace: istio-system
  annotations:
    networking.gke.io/managed-certificates: production-certificate
    kubernetes.io/ingress.global-static-ip-name: gke-ip-address
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
    tls:
      httpsRedirect: true
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - "*"
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/managed-certificate/tls.crt
      privateKey: /etc/istio/managed-certificate/tls.key

I have patched the istio-ingress service with this:

[
  {
    "op": "add",
    "path": "/metadata/annotations/cloud.google.com~1neg",
    "value": "{\"ingress\": true}"
  },
  {
    "op": "replace",
    "path": "/spec/type",
    "value": "NodePort"
  },
  {
    "op": "remove",
    "path": "/status"
  }
]

istioctl analyze is verified
istioctl proxy-status shows everything as ready

connecting to the loadbalancer url results in a 502

Any help on how to proceed debugging / solve this?

2

Hello,

First of all, you setup ingress to port 80 of istio ingress gateway configured with a redirect to https.

If a user call http://your.url/ then GCP LB will redirect the request to istio port 80 that will redirect the user to https. Ok
Now you user is redirected to https://your.url, the GCP LB get your request do the SSL offloading part and call your istio ingress with plain http, which will again redirect to https. You are now in an infinite loop.

The only way to prevent that: Disabling http entry point OR wait for Google to release the GKE option to enable the automatic redirect on the GCP LB itself OR setup an envoy filter to catch “X-Forwarded-Scheme” (http/1.1) or “scheme” (http/2) and redirect if the value is “http”.

On your configuration,

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: api-gateway
  namespace: istio-system
  annotations:
    networking.gke.io/managed-certificates: production-certificate
    kubernetes.io/ingress.global-static-ip-name: gke-ip-address

those annotations are not needed on the gateway part, let them on the ingress part that all.

For the patch on the ingress it seems ok (well remove /status not sure why …) , even if I prefer the operator clean way to do it

For instance:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: minimal
  components:
    ingressGateways:
      - name: istio-gateway
        enabled: true
        k8s:
          serviceAnnotations:
            cloud.google.com/neg: '{"ingress": true}'
          service:
            type: NodePort
            ports:
              - name: http
                port: 80

Now the debugging part. It is stated in the Google LB / GKE documentation that the HTTP LB Health Check call the “/” endpoint with the host set the the pod IP, and the pod MUST return a 200 status.

Here, it cannot be as it return a redirect to https.

Again, you may have some choice to make here. An EnvoyFilter? set a default virtual service ?

Just a few word about the debug of such issue: Check the LB status on the console / with gcloud. Your LB is unhealthy I guess.
You can enable the access log on Istio gateway :

  meshConfig:
    accessLogFile: "/dev/stdout"

You will see the request from the health check, and the return code.

Note: If you keep those option the gateway can reference port 80 only and you can remove the port 443.

You can however want to use http/2 between the GCP LB and your istio gateway. If so, the link will be HTTPS. You can create a simple self signed certificate and put it in a secret (here named self-signed) (if sds enabled) and have:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: minimal
  components:
    ingressGateways:
      - name: istio-gateway
        enabled: true
        k8s:
          serviceAnnotations:
            cloud.google.com/neg: '{"ingress": true}'
            cloud.google.com/app-protocols: '{"https":"HTTP2"}'
          service:
            type: NodePort
            ports:
              - name: https
                port: 443

and

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: api-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: http2
        protocol: HTTPS
      hosts:
        - "*"
      tls:
        mode: SIMPLE
        credentialName: self-signed

NOTE2: Even if you are in http/2 (so https between LB and Istio GW) the scheme header will be “http” if you client connect with http.

3

Hi Gregoire,

Thank you ever so much, for taking your time explaining what went wrong, and how I could proceed to investigate.

After disabling the httpsRedirect (can’t believe I missed that myself), and adding the “uri: prefix: /” to the virtualservice, everything is now working as intended.

4

Hi @Gregoire.

First of all, thanks for investing your time helping here on the forums and being so explicit with your answer.

I’ve been trying to accomplish the same thing that Martin was. And everything works correctly If I leave both HTTP (80) and HTTPS (443) ports on the istio gateway.

Basically, removing httpsRedirect: true allows traffic to the pod, and everything works as expected, except that now I also have HTTP traffic that I wouldn’t like to allow as my service is not supposed to allow it. But once I add this rule back, health checks starts failing.

I’m kind of lost on the behavior of the “workflow”. This setup works:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: domain-global-ingress
  namespace: istio-system
  annotations:
    kubernetes.io/ingress.class: gce
    kubernetes.io/ingress.global-static-ip-name: global
    ingress.gcp.kubernetes.io/pre-shared-cert: "global-certificate"
spec:
  rules:
  - host: "global.domain.ca"
    http:
      paths:
      - path: /*
        backend:
          serviceName: istio-ingressgateway
          servicePort: 80

Then the gateway with:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-autogenerated-k8s-ingress
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"
    - port:
        number: 443
        name: https
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: "ingress-cert"
      hosts:
        - "*"

And adding the following to the istio-ingressgateway:

  annotations:
    cloud.google.com/app-protocols: '{"https":"HTTPS","http2":"HTTP"}'
    cloud.google.com/neg: '{"ingress": true}'

This allows both HTTP and HTTPS traffic. But I can’t see to restrict it to only HTTPS, or maybe rewrite HTTP to HTTPS while keeping the health check working.

5

I got this working creating another HTTP Load Balancer redirecting to the HTTPS one (created by the gce-ingress). This wasn’t possible before:

https://cloud.google.com/load-balancing/docs/https/setting-up-http-https-redirect