Custom CA on istio 1.6.2

Wilson_Chuah · June 18, 2020, 5:11pm

Hi, I’ve been trying to get the replicated control plane to work on Istio 1.6.2 with a custom CA by following the guide located at https://istio.io/latest/docs/setup/install/multicluster/gateways/. Some details on what I’ve done:

Cluster: Google Kubernetes Engine
Type: Private Cluster

Deployed a private CA using https://github.com/smallstep/certificates
Generated a certificate, exported the root and intermediate certificates for the CA
Validated that all certificates can be linked up their chain (actually, even used it for a hello-world webhost just to be absolutely sure they work)
Renamed all the files to the correct naming and created cacerts: ca-cert.pem, ca-key.pem, cert-chain.pem and root-cert.pem
Tried to deploy istio using the manifests/examples/multicluster/values-istio-multicluster-gateways.yaml as per the guide

However, istiod continuously fails to work with the certificate and reports the following error:
Error: failed to create discovery service: failed to create CA: failed to create an istiod CA: certificate is not authorized to sign other certificates

I’ve tried swapping certificates to a legitimate wildcard certificate from GoDaddy but that failed as well. If I delete cacerts or use Istio’s sample certs, the deployment works. I did some digging and found that the sample ca-cert.pem provided with istioctl has the following which my custom CA cert and GoDaddy certificate does not have:

X509v3 Key Usage:
Certificate Sign

Could someone verify if this is a required condition for certificates to work with istiod? Is there an easier way to go about getting a custom CA to work with 1.6.2? My apologies if I missed this in the documentation somewhere, but if anyone could point it out to me, it would help alot, thanks!

Wilson_Chuah · June 19, 2020, 5:58pm

I’ve resolved this, here is what I did, hope it helps someone:

Switched to cfssl as the CA, you need the ability to modify key usage and extended key usage of the certs
Generated a CA Cert (root-cert.pem) & Intermediate Cert according to this guide: https://www.mikenewswanger.com/posts/2018/kubernetes-pki/
Generated a Cert for Istio’s usage (ca-cert.pem & ca-key.pem). This was the extremely tricky part. The cert should have the Key Usage “Certificate Signing” and if you are using the same cfssl version that I am, add the Extended Key Usage “any”*
Combine the above 3 certs into a cert-chain.pem file
Use the 4 files to deploy cacerts as per the Istio guide

*I lack the expertise to explain the necessity behind the “any” Extended Key Usage but I got the clue from here: https://github.com/jetstack/cert-manager/issues/2407 on jessedearing’s comment about RFC 5280. If any Extended Key Usage exists, it must conform to the exact usage of the Key. Therefore, without knowing more about Istio’s usage of the Key, I added the “any” value. Please note, this could be insecure.

Zufar_Dhiyaulhaq · June 22, 2020, 1:29pm

try generating root ca, intermediate ca, and cert chain using step certificate, it is more easier.

you can do something like:

step certificate create zufar-root-ca root-cert.pem root-key.pem --profile root-ca  --kty RSA --no-password --insecure --not-after 87600h --san zufardhiyaulhaq.com
step certificate create zufar-intermediate-ca ca-cert.pem ca-key.pem --profile intermediate-ca --kty RSA --ca ./root-cert.pem --ca-key ./root-key.pem --no-password --insecure --not-after 43800h --san zufardhiyaulhaq.com
step certificate bundle ca-cert.pem root-cert.pem cert-chain.pem

Wilson_Chuah · June 22, 2020, 2:31pm

Hey Zufar,

Thanks for the reply, I actually tried smallstep first, but I could not modify the Key Usage and Key Extended Usage values.

I believe this is tracked here: https://github.com/smallstep/cli/issues/110

kutsav · July 15, 2020, 9:13am

Hi,
I also used cfssl to create certs and add in istio, now I am not getting this error: “certificate is not authorized to sign other certificates”, but I see this error: “grpc: Server.Serve failed to complete security handshake tls: unknown certificate authority”
Were you able to solve this?

deepak_deore · July 16, 2020, 7:25am

I have a modified version of Makefile to create self signed certs

Follow the commands in comment to create self signed certs. I have tested this in 1.4 and 1.5 istio versions.

# Create self signed root and intermediate certs for testing etc.
#
# export CITADEL_ORG=company.com # this is Org in intermediate ssl cert
# export ROOTCA_ORG=company.com # this is Org for root cert
# make intermediate-cluster1-certs # intermediate-cluster1 will be the Location in intermediate cert
# cd intermediate-cluster1
# kubectl create secret generic cacerts -n istio-system --from-file=ca-cert.pem --from-file=ca-key.pem --from-file=root-cert.pem --from-file=cert-chain.pem

.SUFFIXES: .csr .pem .conf
.PRECIOUS: %/ca-key.pem %/ca-cert.pem %/cert-chain.pem
.PRECIOUS: root-cert.csr root-ca.conf %/cluster-ca.csr %/intermediate.conf
.SECONDARY: root-cert.csr root-ca.conf %/cluster-ca.csr %/intermediate.conf

.DEFAULT_GOAL := help

#------------------------------------------------------------------------
# variables: root CA
ROOTCA_DAYS ?= 3650
ROOTCA_KEYSZ ?= 4096
ROOTCA_ORG ?= Istio
ROOTCA_CN ?= Root CA
# Additional variables are defined in root-ca.conf target below.

#------------------------------------------------------------------------
# variables: intermediate CA (Citadel)
CITADEL_SERIAL ?= $(shell echo $$PPID) 	# certificate serial number (uses current PID)
CITADEL_DAYS ?= 730
CITADEL_KEYSZ ?= 4096
CITADEL_ORG ?= Istio
CITADEL_CN ?= Intermediate CA
CITADEL_SAN_DNS ?= localhost
# Additional variables are defined in %/intermediate.conf target below.

#------------------------------------------------------------------------
##help:		print this help message
.PHONY: help

help: Makefile
	@sed -n 's/^##//p' $<

#------------------------------------------------------------------------
##root-ca:	generate root CA files (key and certifcate) in current directory
.PHONY: root-ca

root-ca: root-key.pem root-cert.pem

root-cert.pem: root-cert.csr root-key.pem
	@echo "generating $@"
	@openssl x509 -req -days $(ROOTCA_DAYS) -signkey root-key.pem \
		-extensions req_ext -extfile root-ca.conf \
		-in $< -out $@

root-cert.csr: root-key.pem root-ca.conf
	@echo "generating $@"
	@openssl req -new -key $< -config root-ca.conf -out $@ 

root-ca.conf: 
	@echo "[ req ]" > $@
	@echo "encrypt_key = no" >> $@
	@echo "prompt = no" >> $@
	@echo "utf8 = yes" >> $@
	@echo "default_md = sha256" >> $@
	@echo "default_bits = $(ROOTCA_KEYSZ)" >> $@
	@echo "req_extensions = req_ext" >> $@
	@echo "x509_extensions = req_ext" >> $@
	@echo "distinguished_name = req_dn" >> $@
	@echo "[ req_ext ]" >> $@
	@echo "subjectKeyIdentifier = hash" >> $@
	@echo "basicConstraints = critical, CA:true" >> $@
	@echo "keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign" >> $@
	@echo "[ req_dn ]" >> $@
	@echo "O = $(ROOTCA_ORG)" >> $@
	@echo "CN = $(ROOTCA_CN)" >> $@

root-key.pem:
	@echo "generating $@"
	@openssl genrsa -out $@ 4096

#------------------------------------------------------------------------
##<name>-certs:	generate Citadel certificates for <name>. Includes all PEM files needed.
.PHONY: %-certs

%-certs: %/cert-chain.pem root-cert.pem
	@echo "Citadel inputs stored in $(dir $<)"
	@cp root-cert.pem $(dir $<)

%/cert-chain.pem: %/ca-cert.pem root-cert.pem
	@echo "generating $@"
	@cat $^ > $@

%/ca-cert.pem: %/cluster-ca.csr root-key.pem root-cert.pem
	@echo "generating $@"
	@openssl x509 -req -days $(CITADEL_DAYS) \
		-CA root-cert.pem -CAkey root-key.pem -set_serial $(CITADEL_SERIAL) \
		-extensions req_ext -extfile $(dir $<)/intermediate.conf \
		-in $< -out $@

%/cluster-ca.csr: L=$(dir $@)
%/cluster-ca.csr: %/ca-key.pem %/intermediate.conf
	@echo "generating $@"
	@openssl req -new -config $(L)/intermediate.conf -key $< -out $@ 

%/ca-key.pem:
	@echo "generating $@"
	@mkdir -p $(dir $@)
	@openssl genrsa -out $@ 4096

%/intermediate.conf: L=$(dir $@)
%/intermediate.conf:
	@echo "[ req ]" > $@
	@echo "encrypt_key = no" >> $@
	@echo "prompt = no" >> $@
	@echo "utf8 = yes" >> $@
	@echo "default_md = sha256" >> $@
	@echo "default_bits = $(CITADEL_KEYSZ)" >> $@
	@echo "req_extensions = req_ext" >> $@
	@echo "x509_extensions = req_ext" >> $@
	@echo "distinguished_name = req_dn" >> $@
	@echo "[ req_ext ]" >> $@
	@echo "subjectKeyIdentifier = hash" >> $@
	@echo "basicConstraints = critical, CA:true, pathlen:0" >> $@
	@echo "keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign" >> $@
	@echo "subjectAltName=@san" >> $@
	@echo "[ san ]" >> $@
	@echo "URI.1 = spiffe://cluster.local/ns/istio-system/sa/citadel" >> $@
	@echo "URI.2 = spiffe://$(L:/=)/ns/istio-system/sa/citadel" >> $@
	@echo "DNS.1 = $(CITADEL_SAN_DNS)" >> $@
	@echo "[ req_dn ]" >> $@
	@echo "O = $(CITADEL_ORG)" >> $@
	@echo "CN = $(L:/=)" >> $@
	@echo "L = $(L:/=)" >> $@

Madhusudhan_G_Revank · February 3, 2022, 9:15pm

Hi,

I have started working on istio implementation and able to see some progress. Have got few doubts and I see myself missing something in the documentation…

Currently, I am using a gateway, virtual service, destination rules, authorization services for 2 deployments and I am able to see the graph with mTLS enabled in the kiali dashboard. and it’s all working fine.

As I have to use now GoDaddy CA certs , gone through the documentation regarding " Plug in certificates and key into the cluster ", looks like during the fresh installation of the istio it’s picking custom CA certs but with below errors

2022-02-03T16:23:31.337162Z info initializing mesh networks from mesh config watcher
2022-02-03T16:23:31.337167Z info initializing mesh handlers
2022-02-03T16:23:31.337178Z info creating CA and initializing public key
2022-02-03T16:23:31.337214Z info Use local CA certificate
Error: failed to create discovery service: failed to create CA: failed to create an istiod CA: certificate is not authorized to sign other certificates
2022-02-03T16:23:31.338090Z error failed to create discovery service: failed to create CA: failed to create an istiod CA: certificate is not authorized to sign other certificates

I am a bit stuck in this and exploring for any other approaches if I have to look into anything if I am missing.

Can you please advise with some of the inputs regarding this

Topic		Replies	Views
Istio Multicluster Custom CA Issues Security	9	2052	April 20, 2020
Istio 1.5.1 - Problems with selfSigned: false	1	2069	June 25, 2020
Unable to insert custom certificates	0	343	September 24, 2020
Plugging in a new (ephemeral) CA certificate Security	7	1434	July 9, 2020
Sds failed to warm certificate: failed to generate workload certificate: create certificate Security security , istiocon	0	1122	October 18, 2023

Custom CA on istio 1.6.2

Related Topics