Trouble using existing CA Certs

Hello,

I’m having some trouble using an existing CA with Istio. I followed the instructions here and installed my root cert, ca-key, ca-cert and cert-chain which is identical to the ca-key into the ‘cacerts’ secret in the ‘Istio-system’ namespace. I then used istioctl to install Istio… istiod starts with no errors but the ingress and egress gateways both fail to become ready.

In the ingress/egress controller logs, I see this:

2020-10-01T20:21:47.847540Z info PilotSAN string{“istiod.istio-system.svc”}
2020-10-01T20:21:47.847557Z info MixerSAN string{“spiffe://cluster.local/ns/istio-system/sa/istio-mixer-service-account”}
2020-10-01T20:21:47.847597Z info sa.serverOptions.CAEndpoint == istiod.istio-system.svc:15012
2020-10-01T20:21:47.847607Z info Using user-configured CA istiod.istio-system.svc:15012
2020-10-01T20:21:47.847613Z info istiod uses self-issued certificate
2020-10-01T20:21:47.847664Z info the CA cert of istiod is: -----BEGIN CERTIFICATE-----
.
the contents of the ROOT cert, not the CA cert
.
.
-----END CERTIFICATE-----

further down in the logs, I see this:

2020-10-02T16:37:48.807639Z error sds resource:default received error: code:13 message:"Failed to load certificate chain from ". Will not respond until next secret update

and this…

2020-10-01T20:21:48.342382Z error sds resource:default received error: code:13 message:"Failed to load certificate chain from ". Will not respond until next secret update
2020-10-01T20:21:48.705111Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2020-10-01T20:21:50.642372Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2020-10-01T20:22:04.898628Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2020-10-01T20:22:12.427720Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2020-10-01T20:22:20.285804Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-10-01T20:22:22.285230Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-10-01T20:22:24.285176Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-10-01T20:22:26.285161Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-10-01T20:22:28.285068Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-10-01T20:22:30.285358Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-10-01T20:22:32.285160Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected

and those error continue forever.

I logged into the ingress/egress container and I don’t see any certs generated.

when I remove my cacerts secret, Istio starts normally.

here’s my cacerts secret:

kubectl describe secrets -n istio-system cacerts
Name: cacerts
Namespace: istio-system
Labels:
Annotations:

Type: Opaque

Data

root-cert.pem: 1447 bytes
ca-cert.pem: 1402 bytes
ca-key.pem: 1674 bytes
cert-chain.pem: 1402 bytes

in my case, I have only the Istio cacert so ca-cert.pem and cert-chain.pem are identical.

i have a Makefile copied and updated from istio’s git, it will create self signed CA, try if certs generated with this help, comments in the Makefile has instructions.

# Create self signed root and intermediate certs for testing etc.
#
# export CITADEL_ORG=<example.com> # this is Org in intermediate ssl cert
# export ROOTCA_ORG=<example.com> # this is Org for root cert
# make intermediate-cluster-certs # intermediate-cluster will be the Location in intermediate cert
# cd intermediate-ctnp
# k create secret generic cacerts -n istio-system --from-file=ca-cert.pem --from-file=ca-key.pem --from-file=root-cert.pem --from-file=cert-chain.pem

.SUFFIXES: .csr .pem .conf
.PRECIOUS: %/ca-key.pem %/ca-cert.pem %/cert-chain.pem
.PRECIOUS: root-cert.csr root-ca.conf %/cluster-ca.csr %/intermediate.conf
.SECONDARY: root-cert.csr root-ca.conf %/cluster-ca.csr %/intermediate.conf

.DEFAULT_GOAL := help

#------------------------------------------------------------------------
# variables: root CA
ROOTCA_DAYS ?= 3650
ROOTCA_KEYSZ ?= 4096
ROOTCA_ORG ?= Istio
ROOTCA_CN ?= Root CA
# Additional variables are defined in root-ca.conf target below.

#------------------------------------------------------------------------
# variables: intermediate CA (Citadel)
CITADEL_SERIAL ?= $(shell echo $$PPID) 	# certificate serial number (uses current PID)
CITADEL_DAYS ?= 730
CITADEL_KEYSZ ?= 4096
CITADEL_ORG ?= Istio
CITADEL_CN ?= Intermediate CA
CITADEL_SAN_DNS ?= localhost
# Additional variables are defined in %/intermediate.conf target below.

#------------------------------------------------------------------------
##help:		print this help message
.PHONY: help

help: Makefile
	@sed -n 's/^##//p' $<

#------------------------------------------------------------------------
##root-ca:	generate root CA files (key and certifcate) in current directory
.PHONY: root-ca

root-ca: root-key.pem root-cert.pem

root-cert.pem: root-cert.csr root-key.pem
	@echo "generating $@"
	@openssl x509 -req -days $(ROOTCA_DAYS) -signkey root-key.pem \
		-extensions req_ext -extfile root-ca.conf \
		-in $< -out $@

root-cert.csr: root-key.pem root-ca.conf
	@echo "generating $@"
	@openssl req -new -key $< -config root-ca.conf -out $@ 

root-ca.conf: 
	@echo "[ req ]" > $@
	@echo "encrypt_key = no" >> $@
	@echo "prompt = no" >> $@
	@echo "utf8 = yes" >> $@
	@echo "default_md = sha256" >> $@
	@echo "default_bits = $(ROOTCA_KEYSZ)" >> $@
	@echo "req_extensions = req_ext" >> $@
	@echo "x509_extensions = req_ext" >> $@
	@echo "distinguished_name = req_dn" >> $@
	@echo "[ req_ext ]" >> $@
	@echo "subjectKeyIdentifier = hash" >> $@
	@echo "basicConstraints = critical, CA:true" >> $@
	@echo "keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign" >> $@
	@echo "[ req_dn ]" >> $@
	@echo "O = $(ROOTCA_ORG)" >> $@
	@echo "CN = $(ROOTCA_CN)" >> $@

root-key.pem:
	@echo "generating $@"
	@openssl genrsa -out $@ 4096

#------------------------------------------------------------------------
##<name>-certs:	generate Citadel certificates for <name>. Includes all PEM files needed.
.PHONY: %-certs

%-certs: %/cert-chain.pem root-cert.pem
	@echo "Citadel inputs stored in $(dir $<)"
	@cp root-cert.pem $(dir $<)

%/cert-chain.pem: %/ca-cert.pem root-cert.pem
	@echo "generating $@"
	@cat $^ > $@

%/ca-cert.pem: %/cluster-ca.csr root-key.pem root-cert.pem
	@echo "generating $@"
	@openssl x509 -req -days $(CITADEL_DAYS) \
		-CA root-cert.pem -CAkey root-key.pem -set_serial $(CITADEL_SERIAL) \
		-extensions req_ext -extfile $(dir $<)/intermediate.conf \
		-in $< -out $@

%/cluster-ca.csr: L=$(dir $@)
%/cluster-ca.csr: %/ca-key.pem %/intermediate.conf
	@echo "generating $@"
	@openssl req -new -config $(L)/intermediate.conf -key $< -out $@ 

%/ca-key.pem:
	@echo "generating $@"
	@mkdir -p $(dir $@)
	@openssl genrsa -out $@ 4096

%/intermediate.conf: L=$(dir $@)
%/intermediate.conf:
	@echo "[ req ]" > $@
	@echo "encrypt_key = no" >> $@
	@echo "prompt = no" >> $@
	@echo "utf8 = yes" >> $@
	@echo "default_md = sha256" >> $@
	@echo "default_bits = $(CITADEL_KEYSZ)" >> $@
	@echo "req_extensions = req_ext" >> $@
	@echo "x509_extensions = req_ext" >> $@
	@echo "distinguished_name = req_dn" >> $@
	@echo "[ req_ext ]" >> $@
	@echo "subjectKeyIdentifier = hash" >> $@
	@echo "basicConstraints = critical, CA:true, pathlen:0" >> $@
	@echo "keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign" >> $@
	@echo "subjectAltName=@san" >> $@
	@echo "[ san ]" >> $@
	@echo "URI.1 = spiffe://cluster.local/ns/istio-system/sa/citadel" >> $@
	@echo "URI.2 = spiffe://$(L:/=)/ns/istio-system/sa/citadel" >> $@
	@echo "DNS.1 = $(CITADEL_SAN_DNS)" >> $@
	@echo "[ req_dn ]" >> $@
	@echo "O = $(CITADEL_ORG)" >> $@
	@echo "CN = $(L:/=)" >> $@
	@echo "L = $(L:/=)" >> $@

Thank you. I had found that Makefile earlier and it does indeed produce working ca certs. Using that Makefile, I eventually traced my specific problem to the fact that my openssl commands were not adding a newline character to certs and keys that were being generated. Once I added a ‘\n’ to my ca secrets, Istio began to work normally.

1 Like

Glad you fixed this, would you mind to clarify a little bit what do you mean I added a ‘\n’ to my ca secrets (maybe a diff of the change)?

@Oliver I wonder if this can be improved so that it’s clearly logged that the ca secrets is malformed?

certainly… openssl creates certs and keys with no trailing newline character. the end of the file is: “-----END CERTIFICATE-----”. Istio would not accept my CA certs and keys until I added a newline to those files like this: "-----END CERTIFICATE----- <----- hit enter here
".

I suspect there’s a regex somewhere that’s looking for “-----END CERTIFICATE-----\n” i.e. there’s a newline after ‘-----END CERTIFICATE-----’ and since openssl doesn’t add that be default, the match fails.

see also: https://github.com/istio/istio/issues/24609