Problems with Istio Ingress and Cert-Manager

Hi, I’m trying to follow the instructions from your documentation:

But I’m getting the following error:
error: unable to recognize “STDIN”: no matches for kind “Certificate” in version “certmanager.k8s.io/v1alpha1

Any help would be greatly appreciated.

Hi,

Istio installation contain older (0.6.2?) version of cert-manager. I think that you installed CRDs for the newer version.

It turns out the problem is not with the cert-manager version at all. I got the same error with version v0.6.2 as with the latest v0.13.0 of cert-manager. The problems are with the manifests that follow. I’ll try to figure it out and post what I come up with here.

1 Like

It would appear the issue is with the api-group that the CRD is a member of. According to cert-manager documentation, you should use:
apiVersion: cert-manager.io/v1alpha2

This worked for me. Perhaps the documentation needs to be updated on Istio side.

With the latest version of cert-manager (0.13.0) I am now running into this open cert-manager issue where the ClusterIssuer is not found. https://github.com/jetstack/cert-manager/issues/2487

Maybe an older version of cert-manager will work properly, so maybe Tomas was right after all. Thanks for your assistance.

Hi,

we use EKS 1.14 and cert-manager 0.13.0 and everything works for us. :frowning: :slight_smile:

Well, I got it to work once. I want to make sure I have all the steps right, so I deleted the cluster and tried again. Now, it doesn’t work, and I tried a few times and it’s a solid failure. I skipped the self-signed and staging issuers, so maybe something about creating those makes it work. I dunno. Below are all the steps, culminating with a 503 error. I’m using Digital Ocean.

# Digital Ocean Kubernetes v1.16.6 cluster - 4 nodes x 2vCPUs 4GB RAM each - total 6vCPUs 12GB RAM.
# (Set it to autoscale under "Nodes" to min 4, max 8 nodes just in case)

# For the commands below, omit the $.  That's a bash prompt.  I use Git Bash on Windows
# I'm using kubectl client v1.17.0

# Install Istio per the instructions here https://istio.io/docs/tasks/traffic-management/ingress/ingress-certmgr/
$ istioctl manifest apply \
  --set values.gateways.istio-ingressgateway.sds.enabled=true \
  --set values.global.k8sIngress.enabled=true \
  --set values.global.k8sIngress.enableHttps=true \
  --set values.global.k8sIngress.gatewayName=ingressgateway
  
# Wait until all Istio pods are ready...
$ kubectl get pods -n istio-system

# Wait a few minutes for the LoadBalancer which was created by the 
# Istio install to be ready and show 3/3 pods.
# Create a DNS A record that points to the load balancer external IP - like host.example.com

# Set the $INGRESS_DOMAIN environment variable equal to the hostname of your cluster you 
# created the DNS A record earlier.
$ INGRESS_DOMAIN=host.example.com

# Make sure you can resolve the host name by pinging the server...

# Install Cert-Manager 0.13.0
$ kubectl create namespace cert-manager
$ kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.13.0/cert-manager.yaml

# Wait for all 3 cert-manager pods to be running
$ kubectl get pods -n cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-6f578f4565-qc9mk              1/1     Running   0          14m
cert-manager-cainjector-75b6bc7b8b-wmtf4   1/1     Running   0          14m
cert-manager-webhook-8444c4bc77-d4bfq      1/1     Running   0          14m

# Patch the Istio Gateway per the instructions.
$ kubectl -n istio-system \
  patch gateway istio-autogenerated-k8s-ingress --type=json \
  -p='[{"op": "replace", "path": "/spec/servers/1/tls", "value": {"credentialName": "ingress-cert", "mode": "SIMPLE", "privateKey": "sds", "serverCertificate": "sds"}}]'

# Create the cluster Issuer.  Update the email address to your email address.
$ cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: admin@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: istio-ingressgateway-certs
    solvers:
      - http01:
          ingress:
            class: istio
---
EOF

# Setup the helloworld application. Note this uses the $INGRESS_DOMAIN variable.
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: helloworld
  labels:
    app: helloworld
spec:
  ports:
  - port: 5000
    name: http
  selector:
    app: helloworld
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld
spec:
  selector:
    matchLabels:
      app: helloworld
  template:
    metadata:
      labels:
        app: helloworld
    spec:
      containers:
      - name: helloworld
        image: istio/examples-helloworld-v1
        resources:
          requests:
            cpu: "100m"
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 5000
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: istio
  name: helloworld-ingress
spec:
  rules:
    - host: "$INGRESS_DOMAIN"
      http:
        paths:
          - path: /hello
            backend:
              serviceName: helloworld
              servicePort: 5000
---
EOF

# Wait for helloworld to be ready
$ kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
helloworld-6b489b9859-bvxcw   1/1     Running   0          6m10s

# Confirm that hello world is working non-https:
$ curl http://$INGRESS_DOMAIN/hello
Hello version: v1, instance: helloworld-6b489b9859-9pqc5 

# Finally, create the certificate.
$ cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: ingress-cert
  namespace: istio-system
spec:
  secretName: ingress-cert
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: 'host.example.com'
  dnsNames:
    - 'host.example.com'
---
EOF

# Wait for the certificate to be ready:
$ kubectl describe certificate ingress-cert -n istio-system

# Wait for the Certificate Request (the cert id comes from the output of the above command)
$ kubectl describe certificaterequest ingress-cert-4051514424 -n istio-system

# Wait for the Certificate Order (The order id comes from output of the above command)
$ kubectl describe order ingress-cert-4051514424-3229718444 -n istio-system

# Wait for the challenge (the challenge id comes from the output of the above command)
$kubectl describe challenge ingress-cert-4051514424-3229718444-3565931193 -n istio-system

  Reason:      Waiting for http-01 challenge propagation: wrong status code '503', expected '200'
  State:       pending

Ok, I found a reference to someone having the same problem here https://community.letsencrypt.org/t/error-503-using-cert-manager/107930/3 , and they solved it by deleting and re-creating the ingress. I tried this and it worked for me too. Not clear why this is required, but it is. Note that just deleting the pod and letting Kubernetes recreate it for you doesn’t solve the issue. You have to do “kubectl delete -f ingress.yml”, and then “kubectl apply -f ingress.yml” to solve the problem.

Hi,
Wanted to ask how you got the ingress to serve outside traffic?
I’m having issues with the ingress part at https://istio.io/docs/tasks/traffic-management/ingress/ingress-certmgr/#setting-up-a-demo-application

I’m installing as described, using k8s 1.17 with istio 1.4.4, NodePort type, with no success.
I’m able to get traffic using Gateway and VirtualService resources, but not using regular kubernetes ingress resource.

getting Connection refused on port 80.
Other ingresses work fine (nginx traefik etc…), so what am i missing regarding the kubernetes.io/ingress.class: istio annotation?

I think your question is sufficiently unique that it warrants a new thread on its own. I can see why you might not want to pay extra for a Load Balancer. If it can be done via NodePort safely, I don’t want to pay extra either. The LoadBalancer from the example above was created automatically at the first step, so I went with it. This issue, with Cert-Manager took me about 3 days of trial, error and investigation to resolve. In the process, I did come across some references that might help you.

From here: https://istio.io/docs/setup/getting-started/
“If your cluster is running in an environment that does not support an external load balancer (e.g., minikube), the EXTERNAL-IP of istio-ingressgateway will say <pending> . To access the gateway, use the service’s NodePort , or use port-forwarding instead.”

In the above reference to “the service” I assume the author must mean the “istio-ingressgateway”. I also assume that by “use” they mean “expose”, which you can find instructions for here:

https://kubernetes.io/docs/tutorials/stateless-application/expose-external-ip-address/
Which lists the command:
kubectl expose deployment hello-world --type=LoadBalancer --name=my-service
Where of course you would use NodePort instead of LoadBalancer and istio-ingressgateway as required for the other two parameters, not hello-world and/or my-service, although I’m not exactly sure which here.

Good luck in your journey.

Just in case I didn’t answer your question directly enough… The load balance was created automatically when I installed Istio. A few minutes after you kick-off the Istio installation, the external address will appear, and it will show that it is balancing requests to all the nodes in your cluster. I don’t think Istio supports using a Node Port for the istio-ingressgateway service, someone from the Istio team might be able to comment on that.

To get the external IP of the Load Balancer, you can just look in the Digital Ocean interface at the definition of the Load Balancer. Again, it takes a few minutes to appear after it is created. You can also run the following command and look at the External IP:
From: https://istio.io/docs/tasks/traffic-management/ingress/ingress-certmgr/
kubectl -n istio-system get service istio-ingressgateway

Hi, I found that if you create the Ingress last, there is no need to delete and re-create it. The complete, working instructions are below.

# Digital Ocean Kubernetes v1.16.6 cluster - 3 nodes x 2vCPUs 4GB RAM each - total 6vCPUs 12GB RAM.
# (Set it to autoscale under "Nodes" to min 3, max 8 nodes just in case)

# Install Istio per the instructions here https://istio.io/docs/tasks/traffic-management/ingress/ingress-certmgr/
$ istioctl manifest apply \
  --set values.gateways.istio-ingressgateway.sds.enabled=true \
  --set values.global.k8sIngress.enabled=true \
  --set values.global.k8sIngress.enableHttps=true \
  --set values.global.k8sIngress.gatewayName=ingressgateway
  
# Wait until all Istio pods are ready...
$ kubectl get pods -n istio-system

# Wait a few minutes for the LoadBalancer which was created by the 
# Istio install to be ready and show 3/3 pods.
# Create a DNS A record that points to the load balancer external IP - like host.example.com

# Set the $INGRESS_DOMAIN environment variable equal to the hostname of your cluster you 
# created the DNS A record earlier.
$ INGRESS_DOMAIN=host.example.com

# Install Cert-Manager 0.13.0
$ kubectl create namespace cert-manager
$ kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.13.0/cert-manager.yaml

# Wait for all 3 cert-manager pods to be running
$ kubectl get pods -n cert-manager

# Patch the Istio Gateway per the instructions.
$ kubectl -n istio-system \
  patch gateway istio-autogenerated-k8s-ingress --type=json \
  -p='[{"op": "replace", "path": "/spec/servers/1/tls", "value": {"credentialName": "ingress-cert", "mode": "SIMPLE", "privateKey": "sds", "serverCertificate": "sds"}}]'

# Now RESTART ISTIO and CERT-MANAGER or else they won't be able to resolve your new DNS entry.
# First, restart Istio

$ kubectl delete pods --all -n istio-system

# Wait and confirm that Istio is all up and running 
$ kubectl get pods -n istio-system

# Restart Cert Manager
$ kubectl delete pods --all -n cert-manager

# Wait until Cert Manager is all back up and running again
$ kubectl get pods -n cert-manager

# Setup the helloworld application. Note this uses the $INGRESS_DOMAIN variable.
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: helloworld
  labels:
    app: helloworld
spec:
  ports:
  - port: 5000
    name: http
  selector:
    app: helloworld
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld
spec:
  selector:
    matchLabels:
      app: helloworld
  template:
    metadata:
      labels:
        app: helloworld
    spec:
      containers:
      - name: helloworld
        image: istio/examples-helloworld-v1
        resources:
          requests:
            cpu: "100m"
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 5000
---
EOF

# Create the Cluster Issuer.  Update the email address to your email address.
$ cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: admin@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: istio-ingressgateway-certs
    solvers:
      - http01:
          ingress:
            class: istio
---
EOF

# Wait for helloworld to be ready
$ kubectl get pods

# Create the certificate.
$ cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: ingress-cert
  namespace: istio-system
spec:
  secretName: ingress-cert
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: 'host.example.com'
  dnsNames:
    - 'host.example.com'
---
EOF

# Create the Ingress.
$ cat <<EOF | kubectl apply -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: istio
  name: helloworld-ingress
spec:
  rules:
    - host: "$INGRESS_DOMAIN"
      http:
        paths:
          - path: /hello
            backend:
              serviceName: helloworld
              servicePort: 5000
---
EOF

# Wait for the certificate to be ready:
$ kubectl -n istio-system describe certificate ingress-cert 

# Wait for the Certificate Request (the cert id comes from the output of the above command)
$ kubectl -n istio-system describe certificaterequest ingress-cert-4051514424 

# Wait for the Certificate Order (The order id comes from output of the above command)
$ kubectl -n istio-system describe order ingress-cert-4051514424-3229718444 

# Wait for the challenge (the challenge id comes from the output of the above command)
$ kubectl -n istio-system describe challenge ingress-cert-4051514424-3229718444-3565931193