In the document there is an example about Securing Kubernetes Ingress with Cert-Manager which is not using Gateway + VirtualService. I have tried to make it work with acme http01 but the certificate can not be issued as in log challenge I have 404 error. Is there any example that I can follow with the specifications I mentioned?
I want to use Istio Gateway with SDS option for TLS and secure that by using cert-manager with http-01.
I need an instruction which including Istio Gateway with SDS option for TLS and secure that by using cert-manager with http-01. Istio gateway give me ability to use VirtualService .
I have done like editing istio-autogenerated-k8s-ingress. I deleted TLS and HTTPS part of this filem so the cert-manager can issue certificate. And I created another Gateway with TLS and HTTPS options and everything is working well.
However is not clear to me why I can not only make my gateway to issue certificate by cert-manager and also use VirtualService refer to that Gateway!
It is a bit tricky to use http-01 due to 2 things:
cert-manager create a K8s ingress to reach the workload that serve http challenge. However, this could be shadowed by istio ingress-gateway hence not reachable. My way to get around this is to update virtualservice to make the ingress-gateway able to route to challenge server. May be there is a better way
The workload that cert-manager created doesn’t have sidecar, so if you enable mTLS globally (i.e via mesh-policy and match-all destination rules), ingress gateway may still unable to make connection with it. To fix this, you may need to add destination rule to disable mTLS for those services.
@diemtvu Could you, please, add a bit more details how to update virtualservice to make the ingress-gateway able to route to challenge server?
We have the similar issues with http-01 challenge for cert-manager running in EKS v 1.12 with Istio 1.1.9
Let me correct myself first. my ingress rule was matching all paths, so the ingress rule that cert manager adds (for the path well-known/acme-challenge/<some-hash>) is not reachable. If you follow the example in istio.io, you may not have to do (1).
If you want to have ingress rules to match all paths like I did (I was using ingress-gateway, btw), you can add a match rule for .well-known path to route to the challenge service that cert manager created. However, as you see, you’ll need to find the service name at that time, so this solution doesn’t works for key rotation. I don’t know other better solution, except try to keep the ingress match rule more conservative.
Number (2) is the problem when mTLS is enabled (globally) for your mesh. Istio installation may add a default destination rule that turn on mTLS for all service. However, as mentioned above, the acme solver doesn’t have sidecar so that rule will also make the service unreachable (503). You may add a destination rule to disable mTLS for that service specifically (or even the whole istio-system namespace, and enable for those that needed, e.g telemetry and policy)
Thank you @diemtvu.
We have mTLS active. Manually created destination rule for specific service solved the issue. Certificate is happily created.
Indeed, as you mentioned, the major obstacle is still left open - how to make automatic updates of the certificate work. In this case we cannot manually intervene with handcrafted destination rule for generated service name (cm-acme-http-solver-xxxx).
Not sure is the second option is a way to go, I mean disable mTLS for the whole Istio-system namespace but a few services (telemetry and policy). In this case upgrades of the Istio versions could became a pain. Still thinking is it a way to go.
One of the possible workaround for case with mTLS enabled globally (number 2 described by @diemtvu). Certmanager Ingress is running on 8089. We can try to disable mTLS for this port inside istio-system. It looks no other services are running in this port
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: cert-htt01-challenge
namespace: istio-system
spec:
host: "*.istio-system.svc.cluster.local"
trafficPolicy:
tls:
# keeping the same mTLS mode as `default` DestinationRule in istio-system
mode: ISTIO_MUTUAL
portLevelSettings:
- port:
# CertManager generate services to perform the challenge on port 8089
# it looks so far no other services in istio-system use this port
number: 8089
tls:
mode: DISABLE
When Cert-Manager starts the cert renewal process for domain “subdomain.domain.com” after 60 days, using HTTP01 challenge, I got Waiting for HTTP-01 challenge propagation: wrong status code ‘404’, expected '200’ in the challenge description.
Debug:
I tried analyzing the istio configuration of istiod container. Exec into the container and hit
curl localhost:8080/debug/configz
I notice, when Cert-Manager creates K8s Ingress, istio creates Gateway and VirtualService to resolve the challenge.
This results in 2 Gateways in your cluster with 2 different hosts. (You can not see it in the cluster but in localhost:8080/debug/configz)
a. subdomain.domain.com
b. “*”
I guess this confuses istio when handling the incoming requests.
Solution:
In order not to confuse the istio make sure all the Gateway resources have the same host that is “*”
Your Virtual Service should be able to handle the host based routing so no change in VS.
I guess, when istio has multiple Gateways with the same host, it treats them as one and challenge gets resolved.