How to enable k8s ingress

Networking
1

I am trying to use cert-manager with Let’s Encrypt to issue certificates using HTTP-01 challenge.

It looks like routing problem - certificate is stuck on domain control validation with error: Waiting for HTTP-01 challenge propagation: wrong status code '304', expected '200'. Route to application (oauth2-proxy) is working so it responds with 403 - standard for oauth2-proxy.

Software stack:

  • Istio installed using helm version 1.19.3 (base + istiod in namespace istio-system, gateway in separate one istio-ingress-public - just like in Istio docs)
  • cert-manager installed using helm version 1.13.2 in namespace cert-manager
  • application itself (oauth2-proxy) in namespace oauth2-proxy
  • hello-kubernetes for testing purposes
  • k8s version 1.26.4

For both oauth2-proxy and hello-kubernetes I have native Istio Gateway and VirtualService for each one of them in their namespaces and it works.

For cert-manager I have created Issuer.

I have created Certificate in namespace oauth2-proxy and cert-manager created CertificateRequest, Order, Challenge and Ingress, Service and Pod to handle verification.

I have created IngressClass for Istio:

kubectl get IngressClass -o yaml
apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: IngressClass
  metadata:
    creationTimestamp: "2023-11-03T12:12:09Z"
    generation: 1
    name: istio
    resourceVersion: "4756892"
    uid: cd6205b9-e818-48dd-8882-59c6d038e28e
  spec:
    controller: istio.io/ingress-controller
kind: List
metadata:
  resourceVersion: ""

Ingress has .spec.ingressClassName set to istio:

kubectl get Ingress -o yaml -n oauth2-proxy
apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
    creationTimestamp: "2023-11-03T14:03:42Z"
    generateName: cm-acme-http-solver-
    generation: 1
    labels:
      acme.cert-manager.io/http-domain: "1390741491"
      acme.cert-manager.io/http-token: "1315376821"
      acme.cert-manager.io/http01-solver: "true"
    name: cm-acme-http-solver-4z6kk
    namespace: oauth2-proxy
    ownerReferences:
    - apiVersion: acme.cert-manager.io/v1
      blockOwnerDeletion: true
      controller: true
      kind: Challenge
      name: oauth2-proxy-cert-staging-1-2829224423-1571648249
      uid: 95daf075-ee7e-465e-b129-9d99218addf1
    resourceVersion: "4774848"
    uid: e6a15d62-e010-4b16-8ee0-86bf809e5365
  spec:
    ingressClassName: istio
    rules:
    - host: apps.mydomain.com
      http:
        paths:
        - backend:
            service:
              name: cm-acme-http-solver-spkrv
              port:
                number: 8089
          path: /.well-known/acme-challenge/************
          pathType: ImplementationSpecific
  status:
    loadBalancer: {}
kind: List
metadata:
  resourceVersion: ""

When I list Istio routes config for both oauth2-proxy and hello-kubernetes is visible, but there are no route to HTTP-01 solver Service:

$ istioctl proxy-config routes -n istio-ingress-public istio-ingress-public-c86949ccb-8qx22.istio-ingress-public
NAME        VHOST NAME                  DOMAINS                  MATCH                  VIRTUAL SERVICE
http.80     *:80                        *                        /*                     hello-world-public-virtual.hello-kubernetes
http.80     apps.mydomain.com:80        apps.mydomain.com        /*                     oauth2-proxy-public-virtual.oauth2-proxy
            backend                     *                        /healthz/ready*
            backend                     *                        /stats/prometheus*

I followed docs for integrating Istio with cert-manager: Istio / cert-manager and how to deal with k8s Ingress: Istio / Kubernetes Ingress.

I’ve also tried to use kubernetes.io/ingress.class annotation with same result - no new routes.

What am I doing wrong? From docs it seems that Ingress with .spec.ingressClassName set to istio should be handled without any other configuration. Is there any flag to be set like in previous Istio versions? I didn’t find it in helm values…

What should happen after creating Ingress? I assume that new route should be created and visible using istioctl. Should Istio create new VirtualService?

2

I figured that out.

This issue helped me: Certificate Challenges failing with Istio + Istio Gateway when installed via Helm · Issue #37329 · istio/istio (github.com)

You need to configure few values when installing using helm. Of course this may change depending on your environment - but without this config Istio does not recognize Ingress resources.

  set {
    name  = "meshConfig.ingressService"
    value = "istio-ingress"
  }
  set {
    name  = "meshConfig.ingressSelector"
    value = "ingress"
  }

When properly configured, Istio will create VirtualService to handle challenge and you should be able to debug Istio config using istioctl:

// list ingress gateways
$ istioctl proxy-status

// show routes
$ istioctl proxy-config routes -n <ingress_gateway_namespace> <ingress_gateway_name_from_previous_command>