Trouble using existing CA Certs

Security
1

Hello,

I’m having some trouble using an existing CA with Istio. I followed the instructions here and installed my root cert, ca-key, ca-cert and cert-chain which is identical to the ca-key into the ‘cacerts’ secret in the ‘Istio-system’ namespace. I then used istioctl to install Istio… istiod starts with no errors but the ingress and egress gateways both fail to become ready.

In the ingress/egress controller logs, I see this:

2020-10-01T20:21:47.847540Z info PilotSAN string{“istiod.istio-system.svc”}
2020-10-01T20:21:47.847557Z info MixerSAN string{“spiffe://cluster.local/ns/istio-system/sa/istio-mixer-service-account”}
2020-10-01T20:21:47.847597Z info sa.serverOptions.CAEndpoint == istiod.istio-system.svc:15012
2020-10-01T20:21:47.847607Z info Using user-configured CA istiod.istio-system.svc:15012
2020-10-01T20:21:47.847613Z info istiod uses self-issued certificate
2020-10-01T20:21:47.847664Z info the CA cert of istiod is: -----BEGIN CERTIFICATE-----
.
the contents of the ROOT cert, not the CA cert
.
.
-----END CERTIFICATE-----

further down in the logs, I see this:

2020-10-02T16:37:48.807639Z error sds resource:default received error: code:13 message:“Failed to load certificate chain from ”. Will not respond until next secret update

and this…

2020-10-01T20:21:48.342382Z error sds resource:default received error: code:13 message:“Failed to load certificate chain from ”. Will not respond until next secret update
2020-10-01T20:21:48.705111Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2020-10-01T20:21:50.642372Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2020-10-01T20:22:04.898628Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2020-10-01T20:22:12.427720Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2020-10-01T20:22:20.285804Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-10-01T20:22:22.285230Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-10-01T20:22:24.285176Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-10-01T20:22:26.285161Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-10-01T20:22:28.285068Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-10-01T20:22:30.285358Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-10-01T20:22:32.285160Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected

and those error continue forever.

I logged into the ingress/egress container and I don’t see any certs generated.

when I remove my cacerts secret, Istio starts normally.

here’s my cacerts secret:

kubectl describe secrets -n istio-system cacerts
Name: cacerts
Namespace: istio-system
Labels:
Annotations:

Type: Opaque

Data

root-cert.pem: 1447 bytes
ca-cert.pem: 1402 bytes
ca-key.pem: 1674 bytes
cert-chain.pem: 1402 bytes

in my case, I have only the Istio cacert so ca-cert.pem and cert-chain.pem are identical.

2

i have a Makefile copied and updated from istio’s git, it will create self signed CA, try if certs generated with this help, comments in the Makefile has instructions.

# Create self signed root and intermediate certs for testing etc.
#
# export CITADEL_ORG=<example.com> # this is Org in intermediate ssl cert
# export ROOTCA_ORG=<example.com> # this is Org for root cert
# make intermediate-cluster-certs # intermediate-cluster will be the Location in intermediate cert
# cd intermediate-ctnp
# k create secret generic cacerts -n istio-system --from-file=ca-cert.pem --from-file=ca-key.pem --from-file=root-cert.pem --from-file=cert-chain.pem

.SUFFIXES: .csr .pem .conf
.PRECIOUS: %/ca-key.pem %/ca-cert.pem %/cert-chain.pem
.PRECIOUS: root-cert.csr root-ca.conf %/cluster-ca.csr %/intermediate.conf
.SECONDARY: root-cert.csr root-ca.conf %/cluster-ca.csr %/intermediate.conf

.DEFAULT_GOAL := help

#------------------------------------------------------------------------
# variables: root CA
ROOTCA_DAYS ?= 3650
ROOTCA_KEYSZ ?= 4096
ROOTCA_ORG ?= Istio
ROOTCA_CN ?= Root CA
# Additional variables are defined in root-ca.conf target below.

#------------------------------------------------------------------------
# variables: intermediate CA (Citadel)
CITADEL_SERIAL ?= $(shell echo $$PPID) 	# certificate serial number (uses current PID)
CITADEL_DAYS ?= 730
CITADEL_KEYSZ ?= 4096
CITADEL_ORG ?= Istio
CITADEL_CN ?= Intermediate CA
CITADEL_SAN_DNS ?= localhost
# Additional variables are defined in %/intermediate.conf target below.

#------------------------------------------------------------------------
##help:		print this help message
.PHONY: help

help: Makefile
	@sed -n 's/^##//p' $<

#------------------------------------------------------------------------
##root-ca:	generate root CA files (key and certifcate) in current directory
.PHONY: root-ca

root-ca: root-key.pem root-cert.pem

root-cert.pem: root-cert.csr root-key.pem
	@echo "generating $@"
	@openssl x509 -req -days $(ROOTCA_DAYS) -signkey root-key.pem \
		-extensions req_ext -extfile root-ca.conf \
		-in $< -out $@

root-cert.csr: root-key.pem root-ca.conf
	@echo "generating $@"
	@openssl req -new -key $< -config root-ca.conf -out $@ 

root-ca.conf: 
	@echo "[ req ]" > $@
	@echo "encrypt_key = no" >> $@
	@echo "prompt = no" >> $@
	@echo "utf8 = yes" >> $@
	@echo "default_md = sha256" >> $@
	@echo "default_bits = $(ROOTCA_KEYSZ)" >> $@
	@echo "req_extensions = req_ext" >> $@
	@echo "x509_extensions = req_ext" >> $@
	@echo "distinguished_name = req_dn" >> $@
	@echo "[ req_ext ]" >> $@
	@echo "subjectKeyIdentifier = hash" >> $@
	@echo "basicConstraints = critical, CA:true" >> $@
	@echo "keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign" >> $@
	@echo "[ req_dn ]" >> $@
	@echo "O = $(ROOTCA_ORG)" >> $@
	@echo "CN = $(ROOTCA_CN)" >> $@

root-key.pem:
	@echo "generating $@"
	@openssl genrsa -out $@ 4096

#------------------------------------------------------------------------
##<name>-certs:	generate Citadel certificates for <name>. Includes all PEM files needed.
.PHONY: %-certs

%-certs: %/cert-chain.pem root-cert.pem
	@echo "Citadel inputs stored in $(dir $<)"
	@cp root-cert.pem $(dir $<)

%/cert-chain.pem: %/ca-cert.pem root-cert.pem
	@echo "generating $@"
	@cat $^ > $@

%/ca-cert.pem: %/cluster-ca.csr root-key.pem root-cert.pem
	@echo "generating $@"
	@openssl x509 -req -days $(CITADEL_DAYS) \
		-CA root-cert.pem -CAkey root-key.pem -set_serial $(CITADEL_SERIAL) \
		-extensions req_ext -extfile $(dir $<)/intermediate.conf \
		-in $< -out $@

%/cluster-ca.csr: L=$(dir $@)
%/cluster-ca.csr: %/ca-key.pem %/intermediate.conf
	@echo "generating $@"
	@openssl req -new -config $(L)/intermediate.conf -key $< -out $@ 

%/ca-key.pem:
	@echo "generating $@"
	@mkdir -p $(dir $@)
	@openssl genrsa -out $@ 4096

%/intermediate.conf: L=$(dir $@)
%/intermediate.conf:
	@echo "[ req ]" > $@
	@echo "encrypt_key = no" >> $@
	@echo "prompt = no" >> $@
	@echo "utf8 = yes" >> $@
	@echo "default_md = sha256" >> $@
	@echo "default_bits = $(CITADEL_KEYSZ)" >> $@
	@echo "req_extensions = req_ext" >> $@
	@echo "x509_extensions = req_ext" >> $@
	@echo "distinguished_name = req_dn" >> $@
	@echo "[ req_ext ]" >> $@
	@echo "subjectKeyIdentifier = hash" >> $@
	@echo "basicConstraints = critical, CA:true, pathlen:0" >> $@
	@echo "keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign" >> $@
	@echo "subjectAltName=@san" >> $@
	@echo "[ san ]" >> $@
	@echo "URI.1 = spiffe://cluster.local/ns/istio-system/sa/citadel" >> $@
	@echo "URI.2 = spiffe://$(L:/=)/ns/istio-system/sa/citadel" >> $@
	@echo "DNS.1 = $(CITADEL_SAN_DNS)" >> $@
	@echo "[ req_dn ]" >> $@
	@echo "O = $(CITADEL_ORG)" >> $@
	@echo "CN = $(L:/=)" >> $@
	@echo "L = $(L:/=)" >> $@
3

Thank you. I had found that Makefile earlier and it does indeed produce working ca certs. Using that Makefile, I eventually traced my specific problem to the fact that my openssl commands were not adding a newline character to certs and keys that were being generated. Once I added a ‘\n’ to my ca secrets, Istio began to work normally.

1 Like
4

Glad you fixed this, would you mind to clarify a little bit what do you mean I added a ‘\n’ to my ca secrets (maybe a diff of the change)?

@Oliver I wonder if this can be improved so that it’s clearly logged that the ca secrets is malformed?

5

certainly… openssl creates certs and keys with no trailing newline character. the end of the file is: “-----END CERTIFICATE-----”. Istio would not accept my CA certs and keys until I added a newline to those files like this: "-----END CERTIFICATE----- <----- hit enter here
".

I suspect there’s a regex somewhere that’s looking for “-----END CERTIFICATE-----\n” i.e. there’s a newline after ‘-----END CERTIFICATE-----’ and since openssl doesn’t add that be default, the match fails.

6

see also: https://github.com/istio/istio/issues/24609