Citadel generated certificates don't match the ones returned by the openssl command

Hi folks,

Before asking my question, here is the setup: automatic side injection is enabled for the default namespace and mtls is enabled for all the applications in it. The applications running in this namespace have been created thanks to this manifest and I’ve created another deployment/serviceaccount/service named bookinfo-reviews-hps which is a copy of bookinfo-reviews to perform some tests.

To check the validity of the certificate generated by Citadel, I executed the following command:

kubectl exec -it reviews-v1-85c474d9b8-f9q7q -c istio-proxy -- openssl s_client -showcerts -connect 
reviews-hps:9080

The result is the following:

---
Certificate chain
 0 s:
   i:O = cluster.local
-----BEGIN CERTIFICATE-----
MIIDLDCCAhSgAwIBAgIQIsJjESDlLg3BcE+wMfe9ZjANBgkqhkiG9w0BAQsFADAY
MRYwFAYDVQQKEw1jbHVzdGVyLmxvY2FsMB4XDTIwMDMxOTE5MzkxNFoXDTIwMDMy
MDE5MzkxNFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANEQrpOa
AsiUtIzqLWczazyk2PbildptPgVC1UF3x3Y055/XV+nALUgFFILcDrnvOgaw9FGS
phTTjlbVI7p47ZA0HF9P9rO6yw26hi1/T4ODTjBkE1e2YX+Rw3dLB6Vk29aVN+4u
9gXmUgx/w612tns9eC7NkHkyQTu2mP6V3r2TZ1ejCEKHJcGhfQq0ux9dAlqP2JqV
2eR9J6/Fu8eUukayElzt4YOEAbxX3M8ap4PLmdJFnoPkxBaAml+fJ43hwpv3ysOC
RcekBBd/6xTvFdUL1n0J0d6n8yzn1ZRs6z2eLymu1QBz4XZ8hn+PgA8K5oTQ1LR3
A4ylAJ/MuxTlMJsCAwEAAaOBiTCBhjAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYw
FAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwRwYDVR0RAQH/BD0w
O4Y5c3BpZmZlOi8vY2x1c3Rlci5sb2NhbC9ucy9kZWZhdWx0L3NhL2Jvb2tpbmZv
LXJldmlld3MtaHBzMA0GCSqGSIb3DQEBCwUAA4IBAQA6pYsz3sZyDFCiIjCD1g56
77eYoGdH7jhQ/4ZNopIrDiqDzlFM6nmxX/vGZmjhXgRStQp8Xjz0M+KHu5J3kqPp
hD72/j3x5lkke0C2HaDF2JWguwnTAQVLf9TCQZU/iGW7+ZV7C1MPDwWpNq+zsN1q
A5q/4FhhTXoeDnUX5WFnutjs8Uh5eXWEg2SSbdGlaQMvdt16dVMytR/Qy0N7zg6t
6uKMO2l4vpeYja3vXHxAltd+2X4hNkoT0NUAzOi/OE0Q16hQbUM5vZAXWoxpyUZo
2mO1/VNrYx0YfiHgKZV3YLq51l4tCJ5arX8qoKJCp2Q9/VteZ3A6JU8ZZxfWpQu9
-----END CERTIFICATE-----
 1 s:O = cluster.local
i:O = cluster.local
-----BEGIN CERTIFICATE-----
MIIC3TCCAcWgAwIBAgIQd3TlDZz5vZqGPIUWqa2VnTANBgkqhkiG9w0BAQsFADAY
MRYwFAYDVQQKEw1jbHVzdGVyLmxvY2FsMB4XDTIwMDMxOTEwMzUzMloXDTMwMDMx
NzEwMzUzMlowGDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDCCASIwDQYJKoZIhvcN
AQEBBQADggEPADCCAQoCggEBANHe8sn3d9fuJuFA8I/65AEP0Aw+fajui08/nkBc
YZLh8TAVLcMmBODHfDVbkyuWD9eRB0LGEEixq5tBu0BazUcOzUsi9VypDOBF/2WU
+W3HBxtB3+vwWGhGRS0TuyXEgTxXusd+uTVzIXFZsGDEl79U4fkz+a71UPelkg56
KyxtFvayn55ZRtVsjuzmuuIGU0g7M2Q7ohDEz+51wzzSl6mtJVSHtncr56JeD5EQ
n/q+1m8XuOa+hwTbilomPXhCTWCR+5KaDgocz39qe9a6SaS1vwi7TdHpQAPTb46X
tBuQYhZsHRA4+lK6DJwQpnfMEJMJf82Vle2Sb540OkqksoMCAwEAAaMjMCEwDgYD
VR0PAQH/BAQDAgIEMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEB
AIibRH9RFsNDrnfptlL9U8YnnMBSp2kRlXf7iFJ4bCr0gBoavkCBlZ0tb6qmXiYK
ls5HLJaz583BOyZ9cd76S0VNAMsiESjx1RTt1lfuFsk8YJ3Hmj6htTHyk08O2VT2
ylCkmdfF2djzjQysNaivxOQM+2DLqww+/Hy4q0TYQFTR16RUAG+ofOqm4GaPO40x
PYAGMazrrmWewkI8UNO0o0bfM/s7Hr9X07Sg+Td0209bXw2dvDJGDsILP4jCOJFZ
Qwwr8JxGWRh4+qRzk9GKlFHiPWu1qNoSfGJBRKVMq4dFUtcDJhI1fQzi+NaPzl4S
70WEgQSGzD24LltfJkMgMG0=
-----END CERTIFICATE-----
---

The first certificate is for the SAN URI:spiffe://cluster.local/ns/default/sa/bookinfo-reviews-hps and is valid until the 20th of March 2020.

In the documentation, I could find the following explanations:

  1. Citadel watches the Kubernetes apiserver , creates a SPIFFE certificate and key pair for each of the existing and new service accounts. Citadel stores the certificate and key pairs as Kubernetes secrets.
  2. When you create a pod, Kubernetes mounts the certificate and key pair to the pod according to its service account via Kubernetes secret volume.
  3. Citadel watches the lifetime of each certificate, and automatically rotates the certificates by rewriting the Kubernetes secrets.
  4. Pilot generates the secure naming information, which defines what service account or accounts can run a certain service. Pilot then passes the secure naming information to the sidecar Envoy.
  1. I do have a secret for bookinfo-reviews-hps containing a certificate:

    kubectl get secret istio.bookinfo-reviews-hps -o yaml | yq r - "data.[cert-chain.pem]" | base64 -D
    
     -----BEGIN CERTIFICATE-----
     MIIDLDCCAhSgAwIBAgIQZmi2IFxUpeSSQOmb0hNx3TANBgkqhkiG9w0BAQsFADAY
     MRYwFAYDVQQKEw1jbHVzdGVyLmxvY2FsMB4XDTIwMDMxOTIxNDMxOVoXDTIwMDMx
     OTIyNDMxOVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMY2DIiN
     +qPpy8BGJ3oWh4GQZdn+vJ7qvGPtWV6niotvYil0R0H87fVR2avzr3UN+ZsehNwo
     RC4CPmVcEGWza9nOYQs7B/UH/Oiy2uQ07oqJJGSZ+Xo8+CkQiWaBWprG5wVAoK+b
     29w/7RUtSrwyJG9UrchQwxG1mIbCb9nQ12UAHAfPx1p79ZdfutuY67TKBxM7yL8T
     BYWq44H41USezHkd3Q3X2OvXf6TvZxBmeyqG+26h2z50EVVVOdg/1ZqI9rMA6BMA
     dm8rOwqyY2/JSTjbln/OOU5S/XGad1tCV2EuWFVFc7Pal716VIdHQ/rpgc8IbDZk
     EfHhRPulT6X5S7cCAwEAAaOBiTCBhjAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYw
     FAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwRwYDVR0RAQH/BD0w
     O4Y5c3BpZmZlOi8vY2x1c3Rlci5sb2NhbC9ucy9kZWZhdWx0L3NhL2Jvb2tpbmZv
     LXJldmlld3MtaHBzMA0GCSqGSIb3DQEBCwUAA4IBAQABnkDU8qPYDPzZhi7gpXr8
     xgvsWMx3bfvhmkVonhjacxevxeZ4chYUrJPhBxsLn/1oiMvCYbf6wEQ9wG/AIt+t
     PE09UJoifVOXyWHSJkFJddaXXlh1GihpeKb79feowHgnyGF4PLth9qk6J4/VQXeZ
     5UuPi8rjfPkFoUa8/n4Jg5j9S9MUerRWlOXl9TWLIz65BptntcggqVInRBBbDhya
     lc8dmXCQiqIq6u3IpLu4LUkVZVhlRIdTPPIJ2FL3d08JQFwGhqjPesN6ddkp6SVi
     pQy+7I0x8f8hg4N2sJiOZAe9OQ2EexcGRn1SssvuTxXc/IDYmpf4VpbEkx6PduOi
     -----END CERTIFICATE-----
    

The certificate isn’t the same as the one returned by openssl and it’s valid for one hour (due to my modification of the parameter --workload-cert-ttl set to 1h)

  1. I can’t see the secret related to the certificate mounted in the pod

     kubectl get pod reviews-hps-558df495c8-qx6g9 -o yaml | yq r - spec.containers.1.volumeMounts
    
     - mountPath: /var/run/secrets/istio
       name: istiod-ca-cert
     - mountPath: /etc/istio/proxy
       name: istio-envoy
     - mountPath: /var/run/secrets/tokens
       name: istio-token
     - mountPath: /etc/istio/pod
       name: podinfo
     - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
       name: bookinfo-reviews-hps-token-tkg47
       readOnly: true
    
  2. Citadel does rewrite the secret to update the certificate

     2020-03-19T19:42:59.164333Z info secretcontroller Re-creating deleted secret default/istio.bookinfo-reviews-hps.
     2020-03-19T19:42:59.493370Z info secretcontroller Secret default/istio.bookinfo-reviews-hps is created successfully
     2020-03-19T20:13:19.172929Z info secretcontroller Refreshing about to expire secret default/istio.istio.bookinfo-reviews-hps: got a certificate that should be renewed now
     2020-03-19T20:13:19.384501Z info secretcontroller Secret default/istio.istio.bookinfo-reviews-hps refreshed successfully.
     2020-03-19T20:33:15.138652Z info rootcertrotator Jitter complete, start rotator.
     2020-03-19T20:43:19.204772Z info secretcontroller Refreshing about to expire secret default/istio.istio.bookinfo-reviews-hps: got a certificate that should be renewed now
     2020-03-19T20:43:19.548527Z info secretcontroller Secret default/istio.istio.bookinfo-reviews-hps refreshed successfully.
     2020-03-19T21:13:19.219090Z info secretcontroller Refreshing about to expire secret default/istio.istio.bookinfo-reviews-hps: got a certificate that should be renewed now
     2020-03-19T21:13:19.655003Z info secretcontroller Secret default/istio.istio.bookinfo-reviews-hps refreshed successfully.
     2020-03-19T21:33:15.141369Z info rootcertrotator Check and rotate root cert.
     2020-03-19T21:33:15.176671Z info rootcertrotator Root cert is not about to expire, skipping root cert rotation.
     2020-03-19T21:43:19.311745Z info secretcontroller Refreshing about to expire secret default/istio.istio.bookinfo-reviews-hps: got a certificate that should be renewed now
     2020-03-19T21:43:19.999175Z info secretcontroller Secret default/istio.istio.bookinfo-reviews-hps refreshed successfully.
    

My questions are:

  • Why is the certificate returned by the openssl command different from the certificate in the secret?
  • Which certificate is used by the sidecar? What is the purpose of the other one?
  • What is the frequency used by Citadel to check the certificates?

Thanks a lot for your help,

Hugo

Hello,

I still don’t understand when I check the certificates in Envoy with the command kubectl exec -it <pod_name> -c istio-proxy -- curl http://localhost:15000/certs, I see a “cert_chain” which expires in 24 hours like the one returned by the openssl command but the serial number is different.

Any ideas?

Thanks,
Hugo

Hello,

I understood why the certificates returned by the openssl command and the one shown in the admin interface aren’t the same. I installed istio with a custom configuration and citadel was enabled so I think there were some kind of double generations of certificates since istiod generates certificates as well.

I re installed istio with the default profile and everything is ok.

For those who want to tune the certificate TTL, setting the SECRET_TTL parameter (https://istio.io/docs/reference/commands/pilot-agent/) in the configmap istio-sidecar-injector (in the config section look for the env section under template) does the trick. Unfortunately, I didn’t find any templating to add it from the IstioOperator.

By the way, I’m using the version 1.5.

Thanks,

Hugo

Hi @Hugo, I have enabled mTLS for mesh globally, sometimes for pod to pod communications fails with the below error. I have not enabled istio_cni. This gets fixed if I restart the istio-pilot deployment. Getting this issue very frequently. Is this something related to your issue. you could be able to help me?

upstream_transport_failure_reason":"TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER