Using Gateway + VirtualService + http01 + SDS

Security
12

Finally nailed it
Waiting for HTTP-01 challenge propagation: wrong status code ‘404’, expected '200’

Short Answer
Make your all gateways to receive traffic on “*” host.

My Scenario:

  1. I had Gateway set up on a particular host like “subdomain.domain.com”
  2. When Cert-Manager starts the cert renewal process for domain “subdomain.domain.com” after 60 days, using HTTP01 challenge, I got Waiting for HTTP-01 challenge propagation: wrong status code ‘404’, expected '200’ in the challenge description.

Debug:

  1. I tried analyzing the istio configuration of istiod container. Exec into the container and hit
    curl localhost:8080/debug/configz
  2. I notice, when Cert-Manager creates K8s Ingress, istio creates Gateway and VirtualService to resolve the challenge.
  3. This results in 2 Gateways in your cluster with 2 different hosts. (You can not see it in the cluster but in localhost:8080/debug/configz)
    a. subdomain.domain.com
    b. “*”
  4. I guess this confuses istio when handling the incoming requests.

Solution:

  1. In order not to confuse the istio make sure all the Gateway resources have the same host that is “*”
  2. Your Virtual Service should be able to handle the host based routing so no change in VS.
  3. I guess, when istio has multiple Gateways with the same host, it treats them as one and challenge gets resolved.