Istio upgrade from 1.4.6 -> 1.5.0 throws istiod errors : remote error: tls: error decrypting message

Security
1

Just upgraded istio from 1.4.6 (helm) to istio 1.5.0 (istioctl) [Purged istio and installed from istioctl] but it appears the istiod logs keep throwing the following :

2020-03-16T18:25:45.209055Z info    grpc: Server.Serve failed to complete security handshake from "10.150.56.111:56870": remote error: tls: error decrypting message
2020-03-16T18:25:46.792447Z info    grpc: Server.Serve failed to complete security handshake from "10.150.57.112:49162": remote error: tls: error decrypting message
2020-03-16T18:25:46.930483Z info    grpc: Server.Serve failed to complete security handshake from "10.150.56.160:36878": remote error: tls: error decrypting message
2020-03-16T18:25:48.284122Z info    grpc: Server.Serve failed to complete security handshake from "10.150.52.230:44758": remote error: tls: error decrypting message
2020-03-16T18:25:48.288180Z info    grpc: Server.Serve failed to complete security handshake from "10.150.57.149:56756": remote error: tls: error decrypting message
2020-03-16T18:25:49.108515Z info    grpc: Server.Serve failed to complete security handshake from "10.150.57.151:53970": remote error: tls: error decrypting message
2020-03-16T18:25:49.111874Z info    Handling event update for pod contentgatewayaidest-7f4694d87-qmq8z in namespace djin-content -> 10.150.53.50
2020-03-16T18:25:49.519861Z info    grpc: Server.Serve failed to complete security handshake from "10.150.57.91:59510": remote error: tls: error decrypting message
2020-03-16T18:25:50.133664Z info    grpc: Server.Serve failed to complete security handshake from "10.150.57.203:59726": remote error: tls: error decrypting message
2020-03-16T18:25:50.331020Z info    grpc: Server.Serve failed to complete security handshake from "10.150.57.195:59970": remote error: tls: error decrypting message
2020-03-16T18:25:52.110695Z info    Handling event update for pod contentgateway-d74b44c7-dtdxs in namespace djin-content -> 10.150.56.215
2020-03-16T18:25:53.312761Z info    Handling event update for pod dysonpriority-b6dbc589b-mk628 in namespace djin-content -> 10.150.52.91
2020-03-16T18:25:53.496524Z info    grpc: Server.Serve failed to complete security handshake from "10.150.56.111:57276": remote error: tls: error decrypting message

This also leads to no sidecars successfully launching and failing with :

2020-03-16T18:32:17.265394Z info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 16 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-03-16T18:32:19.269334Z info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 16 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-03-16T18:32:21.265214Z info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 16 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-03-16T18:32:23.266159Z info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 16 successful, 0 rejected; lds updates: 0 successful,

Weirdly other clusters that I upgraded go through fine. Any idea where this error might be popping up from ? istioctl analyze works fine.

2

error goes away after killing the nodes (recreating) but istio-proxies still fail with :

info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 0 rejected

3

Hi, I have the same issue. After upgrading from 1.4.6 to 1.5.0, my ingressgateway istio-proxy says:

info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 4 rejected; lds updates: 4 successful, 0 rejected

Istiod is running fine, but I have not purged my cluser, I’ve just killed the old Pilot.

Any ideas?

4

istio 1.5.1

Have you solved this problem? :喜悦: Same problem

5

yes, this was due to a duplicate service entry rule. I saw the duplication and failure in pilot logs and on removal of the duplicate service entry this was resolved.

1 Like
6

Can you help me？ i install istio 1.5.1

7

thanks, your suggestion fix my problem.

8

@Kuber_Kaul @Joaquin_Philippi
For my case, upgrading from 1.4.10 to 1.5.8, I have istio-ingressgateway pods are in NOT READY state even though they are running.

I checked logs on istiod and istio-ingressgateway but they are not helpful

istio-ingressgateway-564dd9c995-g8jqt istio-proxy [Envoy (Epoch 0)] [2020-07-20 00:29:10.967][19][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure istio-ingressgateway-564dd9c995-g8jqt istio-proxy 2020-07-20T00:29:12.147575Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected

istiod-678b7fb6dc-4h9nd discovery 2020-07-20T00:29:49.908147Z info grpc: Server.Serve failed to complete security handshake from "10.88.10.14:42890": EOF istiod-678b7fb6dc-4h9nd discovery 2020-07-20T00:29:49.961482Z info Handling event update for pod harbor-harbor-jobservice-7f85c99df4-k4bxk in namespace federation-lab-sandbox-us-east1 -> 10.88.10.12

I also increased the cpu and memory of the istio-ingressgateway but no luck either on this.

Any helps is appreciated.