Ingress for ACME-challenge not working (404)

Networking
1

Hello,

We are running Istio 1.1.3 on 1.12.5-gke.10 cluster-nodes.

We use certmanager for managing our let’s encrypt certificates.

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: certs.ourdomain.nl
  namespace: istio-system
spec:
  secretName: certs.ourdomain.nl
  newBefore: 360h # 15d
  commonName: operations.ourdomain.nl
  dnsNames:
    - operations.ourdomain.nl
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
  acme:
    config:
      - http01:
          ingressClass: istio
        domains:
        - operations.ourdomain.nl

Next thing we see the acme backend, service (nodeport and ingress) deployed. The ingress (auto-generated) looks like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: istio
  generateName: cm-acme-http-solver-
  generation: 1
  labels:
    certmanager.k8s.io/acme-http-domain: "1734084804"
    certmanager.k8s.io/acme-http-token: "1476005735"
  name: cm-acme-http-solver-69vzw
  namespace: istio-system
  ownerReferences:
  - apiVersion: certmanager.k8s.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Certificate
    name: certs.ourdomain.nl
    uid: 751011d2-4fc8-11e9-b20e-42010aa40101
spec:
  rules:
  - host: operations.ourdomain.nl
    http:
      paths:
      - backend:
          serviceName: cm-acme-http-solver-fzk8q
          servicePort: 8089
        path: /.well-known/acme-challenge/dnrcr-LRRMdXhBaUefjqpHQx8ytYuk-feEfXu9gW-Ck
status:
  loadBalancer: {}

However, when we try to access the url operations.ourdomain.nl /.well-known/acme-challenge/dnrcr-LRRMdXhBaUefjqpHQx8ytYuk-feEfXu9gW-Ck we get a 404.

We do have a loadbalancer for istio:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  labels:
    app: istio-ingress
    chart: gateways-1.1.0
    heritage: Tiller
    istio: ingress
    release: istio
  name: istio-ingress
  namespace: istio-system
spec:
  selector:
    app: istio-ingress
  servers:
  - hosts:
    - operations.ourdomain.nl
    port:
      name: http
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true
  - hosts:
    - operations.ourdomain.nl
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: certs.ourdomain.nl
      mode: SIMPLE
      privateKey: sds
     serverCertificate: sds

What’s going on? Why isn’t the ingress routing the inbound trafic? Any help would be more than welcom!

1 Like
2

Hello,

Are the 404’s showing up in the ingress-gateway logs?

I assume you are doing something like in the example: https://istio.io/docs/examples/advanced-gateways/ingress-certmgr/. If so, did HTTP work?

3

In the istio ingress (docker.io/istio/node-agent-k8s:1.1.3) I do get some interesting logs:

warn Secret object: letsencrypt has empty field, skip update
warn Secret object: letsencrypt-staging has empty field, skip update
“GET /.well-known/acme-challenge/WWVTqAHf89uqumxI02ti1I_GGhBE7B2QbfoZJIltZGI HTTP/1.1” 301 - “-” 0 0 0 - “10.16.20.1” “Go-http-client/1.1” “ff8e9880-c017-4f39-a893-d4c30b5da0b2” “operations.ourdomain.nl” “-” - - 10.16.20.167:80 10.16.20.1:43254 -
“GET /.well-known/acme-challenge/WWVTqAHf89uqumxI02ti1I_GGhBE7B2QbfoZJIltZGI HTTP/2” 404 NR “-” 0 0 0 - “10.16.20.1” “Go-http-client/2.0” “e6b2cf8e-ed8e-49b0-8dfa-ff407d3b366f” “operations.ourdomain.nl” “-” - - 10.16.20.167:443 10.16.20.1:54376 operations.ourdomain.nl

What is the empty field all about? It’s a line that comes around every 20 seconds or so.
The permanent redirect… what does it redirect to? Is it the permanent redirect to https? And what about the 404 with a http/2? More information, but also more questions :slight_smile:

4

When I do a curl the acme challenge url does a permanent redirect to https. So I guess my problem is that the ingressclass istio does not come before the istio-gateway… the gateway catches the request and because there is no corresponding virtualservice it throws a 404 back at you. More people should have a similar issue if this is the case.

1 Like
5

Hi all,

if ones uses the gateway in ISTIO, the routing is managed by the virtual services, which extend the gateway.
The certmanager implements ingress.extension in the case of using the ISTIO as ingressclass.

Does anyone know how to get the Certmanage included in ISTIO working with the gateway?

Thanks in advance.
Greetings,
Jan

P.S.: Using ISTIO-1.1.3 in the IBM Cloud Kubernetes Service with 1.13.5-IKS

1 Like
6

Hi again,

helped me to get certmanager issuing a cert for four domains successfully using gateway and virtualservice againt letsencrypt-stage.

Best,
Jan

7

Hello, How about new version (1.1.7). In document (https://istio.io/docs/examples/advanced-gateways/ingress-certmgr/) the are using ingress resource but how about we want to use Gateway + VirtualService + http01.

Thanks!

2 Likes
8

Hey @spekr - I have this exact same issue. Since doing a HELM upgrade on my installation of ISTIO to leverage Cert-Manager, I receive:

warn Secret object: ingress-cert has empty field, skip update
warn Secret object: letsencrypt-staging has empty field, skip update
warn Secret object: kiali has empty field, skip update
warn Secret object: letsencrypt has empty field, skip update
warn Secret object: ingress-cert has empty field, skip update

Did you make any progress on getting around this in the end? If i do a clean istio install with cert manager on a new cluster, works a treat…

9

I’m afraid we did not get the cert-manager to work. We went for the ugly manual solution, which is better than no solution at all. The issue we had, has to do with the Mutual TLS option. Because the cert-manager has auto-injection disabled (hardcoded), you can’t get it to work when mtls is enabled.

10

Hi there,

I am afraid, that I am late. Well, we work that way, that we let Certmanager work, until the 404 come,
and I extend the virtual service to provide the way for LE to get to its pod for the verification.
At least ist there a conversion from INGRESS to Gateway/virtualservice configuration, so one may
extract the INGRESS configuration with kubectl get -o yaml and convert it to a running virtual service.

Best,
Jan

P.S.: Does anyone know, whether it is fixed or not?

12

Solution at Using Gateway + VirtualService + http01 + SDS - #12 by dhavlev

13

Today with Istio 1.15 and httpsRedirect, the solution linked to by @dhavlev , needs a little more work if you want regular traffic redirected to https. I worked it out here: Istio Gateway and Kubernetes Ingress on same hostname because of cert-manager HTTP01 ACME challenge: can this work? - Stack Overflow

I would be really happy if someone knowledgable to say if the solution makes sense. In summary:

  • add a * host to the servers on port 80 in each Gateway,
  • drop the use of the built-in httpsRedirect: true function,
  • and write a slightly more manual redirect rule for http to https which only matches the non-ACME paths.