Plugging in existing CA Certificates in istio 1.7

Security
1

Hi Team,

we are trying to install the ca certs - we followed the documentation at Istioldie 1.7 / Plugging in existing CA Certificates - we encountered the following error

warning envoy config gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) 0.0.0.0_8443: Invalid path: /etc/istio/ingressgateway-certs/tls.crt

we cleaned up everything and added certs manually to Gateway but no luck - faced below error

added values in Gateway file and
tls:
mode: SIMPLE #enables HTTPS on this port
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
privateKey: /etc/istio/ingressgateway-certs/tls.key

created secret

apiVersion: v1
kind: Secret
metadata:
name: istio-ingressgateway-certs
namespace: istio-system
type: kubernetes.io/tls
data:
tls.crt:
tls.key:

error: 2021-01-19T10:21:18.284754Z warning envoy config gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) 0.0.0.0_80: cannot bind ‘0.0.0.0:80’: Permission denied
0.0.0.0_443: Failed to load certificate chain from /etc/istio/ingressgateway-certs/tls.crt

finally, we cleaned up certs and redeployed istio but we are still getting below error

error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"

can you assist us - What did we miss?

Thanks,
Phani

2

These error message seem refer to different issues.

The permission one could be the change that we disable the gateway to bound to ports in lower range so it does not have to be run as root?

Not sure about the last one. @shankgan might have better suggestion.

3

When you redeployed Istio, did you also restart the workloads? It appears that the Istio-agent is unable to communicate with Istiod. Can you check if “istiod-ca-cert” configMap has been created in the workload namespaces? Istio-agent needs this to communicate with Istiod

4

Hi @shankgan

have found the secrets certs in istio-system namespace, deleted the certs and restarted all the workloads transport: authentication handshake failed: x509 has been cleared but i am unable to access the application when try to access using ingress ip(external IP) getting the below error

curl -v http://ingress external IP/productpage

  • Trying ingress external IP…
  • TCP_NODELAY set
  • connect to ingress external IP port 80 failed: Connection timed out
  • Failed to connect to ingress external IP port 80: Connection timed out
  • Closing connection 0
    curl: (7) Failed to connect to ingress external IP port 80: Connection timed out

tried to access the application within the cluster, getting following error’s

istio-proxy to product svc
kubectl exec -it ratings-v1-c6cdf8d98-tjdgf -c istio-proxy -n istio-phani – bash

istio-proxy@ratings-v1-c6cdf8d98-tjdgf:/$ curl -v http://ip:9080/productpage

  • Trying ip…
  • TCP_NODELAY set
  • Connected to ip (ip) port 9080 (#0)

GET /productpage HTTP/1.1
Host: ip:9080
User-Agent: curl/7.58.0
Accept: /

  • Recv failure: Connection reset by peer
  • stopped the pause stream!
  • Closing connection 0
    curl: (56) Recv failure: Connection reset by peer

rating pod to product svc
kubectl exec -it ratings-v1-c6cdf8d98-tjdgf -c ratings -n istio-phani – bash
root@ratings-v1-c6cdf8d98-tjdgf:/opt/microservices# curl -I http://ip:9080/productpage
HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
content-length: 5183
server: envoy
date: Mon, 01 Feb 2021 18:56:41 GMT
x-envoy-upstream-service-time: 67

istio-proxy log:

“GET /productpage HTTP/1.1” 503 LR “-” “-” 0 84 10001 - “-” “curl/7.52.1” “2a495f69-35cd-4b63-919f-621b355ff5fb” “ip” “ip:80” PassthroughCluster - ip:80 ratingpodip:35274 - allow_any

[2021-02-01T18:55:11.634Z] “GET /ratings/0 HTTP/1.1” 200 - “-” “-” 0 48 15 15 “-” “curl/7.52.1” “ed87a03f-b1ed-4f47-9ab2-ebbabc09f910” “ratings:9080” “127.0.0.1:9080” inbound|9080|http|ratings.istio-phani.svc.cluster.local 127.0.0.1:58606 ratingpodip:9080 reviewspodIP:39364 outbound_.9080_._.ratings.istio-phani.svc.cluster.local default

istio-ingressgateway pod logs

2021-02-01T19:32:36.782538Z warn Envoy proxy is NOT ready: failed to get readiness stats: listener_manager.workers_started is not yet updated: server.state: 0

2021-02-01T19:32:38.639280Z warn Envoy proxy is NOT ready: failed to get readiness stats: listener_manager.workers_started is not yet updated: server.state: 0

2021-02-01T19:32:40.653061Z info Envoy proxy is ready
2021-02-01T19:32:45.232047Z warning envoy config StreamAggregatedResources gRPC config stream closed: 13,
2021-02-01T19:32:45.259739Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2021-02-01T19:32:45.446071Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2021-02-01T19:32:47.347843Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2021-02-01T19:32:47.664101Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2021-02-01T19:32:52.309180Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: local reset
2021-02-01T19:33:04.770628Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: local reset

fyi, ip—>productpage svc ip

have been debugging the issue but i am unable to find it out - have verified all the config looks good.

can you assist me?