Questions about Gateways and Let's Encrypt

Hello everyone,

I’m trying to get Istio up an running into a new project we’re building.
Our app consists right now on an angular frontend, a backend app and an internal service.

First things I did was installing Istio, I’ve dumped a profile and customized it to use our existing grafana and prometheus servers. Kaili and Jaeger are both running correctly.

Then I’ve added the sidecar injection to the namespace, created a new subdomain and created the necessary Gateway and Virtual Service.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: myapp-gateway
  namespace: myapp
spec:
  servers:
    - port:
        number: 80
        name: http-myapp
        protocol: HTTP
      hosts:
        - myapp.company.com
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: myapp
  namespace: myapp
spec:
  hosts:
    - "*"
  gateways:
    - production-gateway
  http:
    - match:
      - uri:
          prefix: /api
      route:
        - destination:
            host: backend
            port:
              number: 80
    - route:
        - destination:
            host: frontend
            port:
              number: 80

at this point everything was working properly, I pointed the DNS to the loadbalancer created by ISTIO and everything was ok.

I’ve then tried to enable HTTP using let’s encrypt. To do so I’ve changed my istio profile to include these changes:

  --set values.gateways.istio-ingressgateway.sds.enabled=true \
  --set values.global.k8sIngress.enabled=true \
  --set values.global.k8sIngress.enableHttps=true \
  --set values.global.k8sIngress.gatewayName=ingressgateway

as mentioned in https://istio.io/docs/tasks/traffic-management/ingress/ingress-certmgr/

At this point I saw that the guide only mentions having a single certificate for the main k8sIngress just created and it also uses default k8s ingresses but I want to use gateways and VirtualServices.
So I’ve then changed:

  • turned off values.global.k8sIngress.enableHttps because it was constantly logging this error:
[Envoy (Epoch 0)] [2020-04-19 03:28:38.123][19][warning][config] [external/envoy/source/common/config/grpc_subscription_impl.cc:87] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) 0.0.0.0_443: Invalid path: /etc/istio/ingressgateway-certs/tls.crt
  • created a new cert manager cluster issuer that uses istio ingress class instead of the default nginx one
  • created the certificates I needed into the istio-system:
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: myapp-certificate
  namespace: istio-system
spec:
  secretName: myapp-tls
  issuerRef:
    name: letsencrypt-prod-istio
    kind: ClusterIssuer
  dnsNames:
    - myapp.company.com
  • changed the gateway to include the https post with the new certificate:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: myapp-gateway
  namespace: myapp
spec:
  servers:
    - port:
        number: 80
        name: http-myapp
        protocol: HTTP
      hosts:
        - myapp.company.com
      tls:
        httpsRedirect: true
    - port:
        number: 443
        name: https-myapp
        protocol: HTTPS
      hosts:
        - myapp.company.com
      tls:
        credentialName: myapp-tls
        mode: SIMPLE
        privateKey: sds
        serverCertificate: sds

at this point the ingress controller logged:

2020-04-19T04:34:14.808156Z	warn	secretfetcher	failed load server cert/key pair from secret myapp-tls: server cert or private key is empty
2020-04-19T04:34:18.409583Z	info	secretfetcher	scrtUpdated is called on kubernetes secret myapp-tls
2020-04-19T04:34:18.409632Z	warn	secretfetcher	failed load server cert/key pair from secret myapp-tls: server cert or private key is empty

and the https endpoint was just returning a timeout. Cert manager already saved the secret but nothing happened on the ingress controller side.

After restarting the ingress controller instead, everything worked:

2020-04-19T04:36:39.395275Z	info	sds	node:router~100.106.199.214~istio-ingressgateway-85576dff98-qt7pz.istio-system~istio-system.svc.cluster.local-3 resource:myapp-tls new connection
2020-04-19T04:36:39.395578Z	info	sds	node:router~100.106.199.214~istio-ingressgateway-85576dff98-qt7pz.istio-system~istio-system.svc.cluster.local-3 resource:myapp-tls pushed key/cert pair to proxy
2020-04-19T04:36:39.395600Z	info	sds	node:router~100.106.199.214~istio-ingressgateway-85576dff98-qt7pz.istio-system~istio-system.svc.cluster.local-3 resource:myapp-tls pushed secret

Now, I’ve many questions but I’ll try to summarize them:

  • what’s the point of having values.global.k8sIngress.enabled set to true?
  • do I need both values.global.k8sIngress.enabled and the regular ingress gateway? maybe I need k8sIngress for cert manager to get the certs via http validation?
  • is there an obvious reason why I was getting the timeouts? (could it be because of https://istio.io/docs/ops/common-problems/network-issues/#port-conflict-when-configuring-multiple-tls-hosts-in-a-gateway ?)
  • should I keep the default ingressgateway and istio-autogenerated-k8s-ingress gateway? Because opening my gateway config with Kiali I get KIA0301 More than one Gateway for the same host port combination I think because the two gateways above has host *

Thank you in advance and sorry for the long post

1 Like

Hello,

we use only istio-ingressgateway with cert manager and it works great. Only think that you have to do is to enable SDS for gateway.

  1. I’ve never use k8singress from Istio so I do not know.
  2. No, you don’t need to have it enabled.
  3. Use selectors for selecting the right gateway. If you use istio-ingressgateway then the selectors field should contain Istio: ingressgateway.
  4. I’d go with gateway here. Also be careful for wildcard Gateways. :slight_smile:
1 Like

as @Tomas_Kohout says, it should work.
no need to use k8singress.

1 Like

@Tomas_Kohout @bethmage thank you for your answers.

So I’ve tried to remove the "k8singress’ and that makes cert-manager not being able to solve http challenges so it’s not able to get the certificate from let’s encrypt. After re-enabling it the provisioning worked.
This seems something that’s not needed anymore in 1.6 https://github.com/istio/istio/issues/22306#issuecomment-601228808 so I’ll try to remove it again after 1.6 is released.

Regarding wildcard gateways, excluding the one generated from k8sIngress, istio manifest always generate a default gateway:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: ingressgateway
  namespace: istio-system
  labels:
    release: istio
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
      - "*"
    # Additional ports in gateaway for the ingressPorts - apps using dedicated port instead of hostname

what is that for? Can I delete it?

I’m trying to imagine what I’ll need in the future, let’s say I have 2+ apps, each one with its own domain.
E.g. a.com, b.com and c.com and so on.
I will need 2+ gateways, where each port need to have its own unique combination of number and name because of this so gateway for a.com will be:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: a-gateway
  namespace: a
spec:
  servers:
    - port:
        number: 80
        name: http-a
        protocol: HTTP
      hosts:
        - a.com
      tls:
        httpsRedirect: true
    - port:
        number: 443
        name: https-a
        protocol: HTTPS
      hosts:
        - a.com
      tls:
        credentialName: a-com-tls
        mode: SIMPLE
        privateKey: sds
        serverCertificate: sds

gateway for b.com will be:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: b-gateway
  namespace: b
spec:
  servers:
    - port:
        number: 80
        name: http-b
        protocol: HTTP
      hosts:
        - b.com
      tls:
        httpsRedirect: true
    - port:
        number: 443
        name: https-b
        protocol: HTTPS
      hosts:
        - b.com
      tls:
        credentialName: b-com-tls
        mode: SIMPLE
        privateKey: sds
        serverCertificate: sds

right? if each app has its own gateway, what is the default one for?

Btw: docs here say about the selector:

One or more labels that indicate a specific set of pods/VMs on which this gateway configuration should be applied. The scope of label search is restricted to the configuration namespace in which the the resource is present. In other words, the Gateway resource must reside in the same namespace as the gateway workload instance.

however I have gateways (without any selector) in the app namespace (so not in istio-system) working just fine, is that expected?

Thank you

Seems like cert manager don’t know how to work with Istio’s VirtualService. I’m sorry, but I don’t think I can help you. :frowning_face:

About the Gateway. IMHO you can delete that default I don’t know why it’s generated in the first place. Is it from demo profile?

It works now, but AFAIK it’s a bug so it can stop working when it’s fixed.

1 Like

Cert manager only generates an ingress to get traffic to the acme pod so maybe without the k8singress a regular ingress won’t work.

It’s a profile generated from the default one and modified to use custom grafana/prometheus basically.
The helm template is this one which doesn’t seem to have any condition on when to generate the default gateway or not https://github.com/istio/istio/blob/d7c82cf727ec9d8d24143907a12bf1fd405e0c23/manifests/gateways/istio-ingress/templates/gateway.yaml
I’ll just keep it for now because what I do usually is just modify the profile template and apply that, so if I do some changes I need to remove the default gateway every time. Plus for now I also need to keep the k8singress because of cert-manager.

If it’s a bug, then the Gateway must be in the istio-system namespace and the VirtualService in the app namespace?

Apologize for late answer.

I think that VS can be in different namespace. At least I use it that way for egress traffic. :slight_smile:

If you can’t use DNS challenge and you don’t want to renew certs manually then it looks like only way now. Of course you could volunteer and contribute to cert-manager. :slight_smile:

Oh ok so I can probably:

  • use dns challenge and remove k8singress
  • use a single gateway with multiple hosts and multiple cert secrets and remove the default ingress, putting mine in istio-system
  • use only VS in the application namespace

Thank you very much for your answers, I feel like this would be a good setup for now!
Regarding contributing to cert-manager, I wish my go level would be good enough to do so :slight_smile: that would mean contribute in a way cert-manager could use VS to forward the requests to the acme pod instead of ingresses?

2 Likes

You’re welcome. :slight_smile: