our cluster has Istio 1.3 and k8s 1.15 , we recently updated the CNI and launched new nodes with latest ami. After the update istio-policy went unstable.
Below are the logs of mixer container.
2020-12-29T19:57:46.803810Z info pickfirstBalancer: HandleSubConnStateChange: 0xc00001b060, CONNECTING
2020-12-29T19:57:46.806789Z info grpc: addrConn.createTransport failed to connect to {istio-galley.istio-system.svc:9901 0 }. Err :connection error: desc = “transport: authentication handshake failed: tls: first record does not look like a TLS handshake”. Reconnecting…
2020-12-29T19:57:46.806835Z info pickfirstBalancer: HandleSubConnStateChange: 0xc00001b060, TRANSIENT_FAILURE
2020-12-29T19:57:47.366819Z info mcp (re)trying to establish new MCP sink stream
2020-12-29T19:57:47.366887Z error mcp Failed to create a new MCP sink stream: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = “transport: authentication handshake failed: tls: first record does not look like a TLS handshake”
Below are the logs of istio-proxy container
[2020-12-29 19:58:37.641][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 2, failed to get root cert
[2020-12-29 19:58:41.180][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 16, request authenticate failure
[2020-12-29 19:58:42.059][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 16, request authenticate failure
[2020-12-29 19:58:51.925][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 16, request authenticate failure
[2020-12-29 19:58:56.910][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 16, request authenticate failure
[2020-12-29 19:59:04.895][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 2, failed to get root cert
Below are the logs from galley
xe9\x03\x03\x93\xca\t\x8cc\xa2\x9f\x96*(\x1a|G""
2020-12-29T15:17:20.781278Z info grpc: Server.Serve failed to create ServerTransport: connection error: desc = “transport: http2Server.HandleStreams received bogus greeting from client: "\x16\x03\x01\x00\xed\x01\x00\x00\xe9\x03\x03~\x1e/\xff{M\x84\xe6\x11v\xfd\xb5\x13"”
2020-12-29T15:17:43.873279Z error mcp MCP: connection {addr=127.0.0.1:40768 id=293}: TERMINATED with errors: rpc error: code = Canceled desc = context canceled
2020-12-29T15:17:43.873280Z error mcp MCP: connection {addr=127.0.0.1:40768 id=294}: TERMINATED with errors: rpc error: code = Canceled desc = context canceled
2020-12-29T15:17:43.873320Z info mcp MCP: connection {addr=127.0.0.1:40768 id=293}: CLOSED
2020-12-29T15:17:43.873325Z info mcp MCP: connection {addr=127.0.0.1:40768 id=294}: CLOSED
2020-12-29T15:17:44.874053Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295}: NEW (ResourceSource), supported collections: string{“istio/config/v1alpha2/legacy/metrics”, “istio/policy/v1beta1/attributemanifests”, “istio/config/v1alpha2/legacy/listcheckers”, “istio/config/v1alpha2/legacy/rbacs”, “istio/networking/v1alpha3/virtualservices”, “istio/config/v1alpha2/httpapispecbindings”, “istio/config/v1alpha2/legacy/memquotas”, “istio/authentication/v1alpha1/policies”, “istio/config/v1alpha2/legacy/redisquotas”, “istio/config/v1alpha2/legacy/signalfxs”, “istio/config/v1alpha2/legacy/bypasses”, “istio/networking/v1alpha3/envoyfilters”, “istio/config/v1alpha2/legacy/checknothings”, “istio/networking/v1alpha3/serviceentries”, “istio/config/v1alpha2/legacy/logentries”, “istio/rbac/v1alpha1/servicerolebindings”, “istio/config/v1alpha2/legacy/fluentds”, “istio/networking/v1alpha3/destinationrules”, “istio/config/v1alpha2/legacy/prometheuses”, “istio/config/v1alpha2/legacy/opas”, “istio/config/v1alpha2/legacy/dogstatsds”, “istio/config/v1alpha2/legacy/authorizations”, “istio/config/v1alpha2/legacy/tracespans”, “istio/config/v1alpha2/legacy/statsds”, “istio/config/v1alpha2/httpapispecs”, “istio/config/v1alpha2/legacy/deniers”, “istio/config/v1alpha2/legacy/edges”, “istio/mixer/v1/config/client/quotaspecbindings”, “istio/config/v1alpha2/legacy/listentries”, “istio/config/v1alpha2/legacy/apikeys”, “istio/networking/v1alpha3/gateways”, “istio/rbac/v1alpha1/clusterrbacconfigs”, “istio/config/v1alpha2/legacy/circonuses”, “istio/policy/v1beta1/handlers”, “istio/rbac/v1alpha1/serviceroles”, “k8s/extensions/v1beta1/ingresses”, “istio/config/v1alpha2/legacy/kuberneteses”, “istio/config/v1alpha2/legacy/stdios”, “istio/authentication/v1alpha1/meshpolicies”, “istio/config/v1alpha2/legacy/reportnothings”, “istio/config/v1alpha2/legacy/solarwindses”, “istio/config/v1alpha2/adapters”, “istio/config/v1alpha2/legacy/cloudwatches”, “istio/policy/v1beta1/rules”, “istio/config/v1alpha2/legacy/stackdrivers”, “istio/config/v1alpha2/legacy/zipkins”, “istio/networking/v1alpha3/synthetic/serviceentries”, “istio/policy/v1beta1/instances”, “istio/networking/v1alpha3/sidecars”, “istio/config/v1alpha2/legacy/kubernetesenvs”, “istio/config/v1alpha2/templates”, “istio/mixer/v1/config/client/quotaspecs”, “istio/config/v1alpha2/legacy/quotas”, “istio/rbac/v1alpha1/rbacconfigs”, “istio/mesh/v1alpha1/MeshConfig”, “istio/config/v1alpha2/legacy/noops”}
2020-12-29T15:17:44.874078Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296}: NEW (ResourceSource), supported collections: string{“istio/config/v1alpha2/legacy/metrics”, “istio/policy/v1beta1/attributemanifests”, “istio/config/v1alpha2/legacy/listcheckers”, “istio/config/v1alpha2/legacy/rbacs”, “istio/networking/v1alpha3/virtualservices”, “istio/config/v1alpha2/httpapispecbindings”, “istio/config/v1alpha2/legacy/memquotas”, “istio/authentication/v1alpha1/policies”, “istio/config/v1alpha2/legacy/redisquotas”, “istio/config/v1alpha2/legacy/signalfxs”, “istio/config/v1alpha2/legacy/bypasses”, “istio/networking/v1alpha3/envoyfilters”, “istio/config/v1alpha2/legacy/checknothings”, “istio/networking/v1alpha3/serviceentries”, “istio/config/v1alpha2/legacy/logentries”, “istio/rbac/v1alpha1/servicerolebindings”, “istio/config/v1alpha2/legacy/fluentds”, “istio/networking/v1alpha3/destinationrules”, “istio/config/v1alpha2/legacy/prometheuses”, “istio/config/v1alpha2/legacy/opas”, “istio/config/v1alpha2/legacy/dogstatsds”, “istio/config/v1alpha2/legacy/authorizations”, “istio/config/v1alpha2/legacy/tracespans”, “istio/config/v1alpha2/legacy/statsds”, “istio/config/v1alpha2/httpapispecs”, “istio/config/v1alpha2/legacy/deniers”, “istio/config/v1alpha2/legacy/edges”, “istio/mixer/v1/config/client/quotaspecbindings”, “istio/config/v1alpha2/legacy/listentries”, “istio/config/v1alpha2/legacy/apikeys”, “istio/networking/v1alpha3/gateways”, “istio/rbac/v1alpha1/clusterrbacconfigs”, “istio/config/v1alpha2/legacy/circonuses”, “istio/policy/v1beta1/handlers”, “istio/rbac/v1alpha1/serviceroles”, “k8s/extensions/v1beta1/ingresses”, “istio/config/v1alpha2/legacy/kuberneteses”, “istio/config/v1alpha2/legacy/stdios”, “istio/authentication/v1alpha1/meshpolicies”, “istio/config/v1alpha2/legacy/reportnothings”, “istio/config/v1alpha2/legacy/solarwindses”, “istio/config/v1alpha2/adapters”, “istio/config/v1alpha2/legacy/cloudwatches”, “istio/policy/v1beta1/rules”, “istio/config/v1alpha2/legacy/stackdrivers”, “istio/config/v1alpha2/legacy/zipkins”, “istio/networking/v1alpha3/synthetic/serviceentries”, “istio/policy/v1beta1/instances”, “istio/networking/v1alpha3/sidecars”, “istio/config/v1alpha2/legacy/kubernetesenvs”, “istio/config/v1alpha2/templates”, “istio/mixer/v1/config/client/quotaspecs”, “istio/config/v1alpha2/legacy/quotas”, “istio/rbac/v1alpha1/rbacconfigs”, “istio/mesh/v1alpha1/MeshConfig”, “istio/config/v1alpha2/legacy/noops”}
2020-12-29T15:17:44.874117Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295}: inc=false WATCH for istio/authentication/v1alpha1/meshpolicies
2020-12-29T15:17:44.874126Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296}: inc=false WATCH for istio/networking/v1alpha3/virtualservices
2020-12-29T15:17:44.874146Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295}: inc=false WATCH for istio/rbac/v1alpha1/rbacconfigs
2020-12-29T15:17:44.874166Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295}: inc=false WATCH for istio/rbac/v1alpha1/servicerolebindings
2020-12-29T15:17:44.874179Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295}: inc=false WATCH for istio/networking/v1alpha3/destinationrules
2020-12-29T15:17:44.874237Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296}: inc=false WATCH for istio/mixer/v1/config/client/quotaspecbindings
2020-12-29T15:17:44.874296Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295}: inc=false WATCH for istio/config/v1alpha2/httpapispecs
2020-12-29T15:17:44.874288Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296}: inc=false WATCH for istio/rbac/v1alpha1/serviceroles
2020-12-29T15:17:44.874327Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296}: inc=false WATCH for istio/rbac/v1alpha1/servicerolebindings
2020-12-29T15:17:44.874390Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295}: inc=false WATCH for istio/networking/v1alpha3/serviceentries
2020-12-29T15:17:44.874408Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295}: inc=false WATCH for istio/mixer/v1/config/client/quotaspecbindings
2020-12-29T15:17:44.874423Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295}: inc=false WATCH for istio/networking/v1alpha3/envoyfilters
2020-12-29T15:17:44.874434Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295}: inc=false WATCH for istio/networking/v1alpha3/sidecars
2020-12-29T15:17:44.874450Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295}: inc=false WATCH for istio/config/v1alpha2/httpapispecbindings
2020-12-29T15:17:44.874469Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295}: inc=false WATCH for istio/mixer/v1/config/client/quotaspecs
2020-12-29T15:17:44.874480Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295}: inc=false WATCH for istio/authentication/v1alpha1/policies
2020-12-29T15:17:44.874502Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295}: inc=false WATCH for istio/rbac/v1alpha1/serviceroles
2020-12-29T15:17:44.874511Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296}: inc=false WATCH for istio/rbac/v1alpha1/clusterrbacconfigs
2020-12-29T15:17:44.874539Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296}: inc=false WATCH for istio/networking/v1alpha3/gateways
2020-12-29T15:17:44.874553Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296}: inc=false WATCH for istio/networking/v1alpha3/sidecars
2020-12-29T15:17:44.874579Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295}: inc=false WATCH for istio/networking/v1alpha3/virtualservices
2020-12-29T15:17:44.874612Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296}: inc=false WATCH for istio/authentication/v1alpha1/meshpolicies
2020-12-29T15:17:44.874625Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296}: inc=false WATCH for istio/rbac/v1alpha1/rbacconfigs
2020-12-29T15:17:44.874637Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296}: inc=false WATCH for istio/config/v1alpha2/httpapispecbindings
2020-12-29T15:17:44.874647Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295}: inc=false WATCH for istio/networking/v1alpha3/gateways
2020-12-29T15:17:44.874663Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296}: inc=false WATCH for istio/mixer/v1/config/client/quotaspecs
2020-12-29T15:17:44.874674Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295}: inc=false WATCH for istio/rbac/v1alpha1/clusterrbacconfigs
2020-12-29T15:17:44.874676Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296}: inc=false WATCH for istio/authentication/v1alpha1/policies
2020-12-29T15:17:44.874687Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295} ACK collection=istio/authentication/v1alpha1/meshpolicies with version=“168” nonce=“1” inc=false
2020-12-29T15:17:44.874695Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296}: inc=false WATCH for istio/networking/v1alpha3/serviceentries
2020-12-29T15:17:44.874700Z info mcp Watch(): created watch 4705 for istio/authentication/v1alpha1/meshpolicies from group “default”, version “168”
2020-12-29T15:17:44.874715Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295} ACK collection=istio/rbac/v1alpha1/rbacconfigs with version=“0” nonce=“2” inc=false
2020-12-29T15:17:44.874721Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296}: inc=false WATCH for istio/networking/v1alpha3/destinationrules
2020-12-29T15:17:44.874725Z info mcp Watch(): created watch 4706 for istio/rbac/v1alpha1/rbacconfigs from group “default”, version “0”
2020-12-29T15:17:44.874739Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296}: inc=false WATCH for istio/networking/v1alpha3/envoyfilters
2020-12-29T15:17:44.874785Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296}: inc=false WATCH for istio/config/v1alpha2/httpapispecs
2020-12-29T15:17:44.874964Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295} ACK collection=istio/rbac/v1alpha1/servicerolebindings with version=“68” nonce=“3” inc=false
2020-12-29T15:17:44.874994Z info mcp Watch(): created watch 4707 for istio/rbac/v1alpha1/servicerolebindings from group “default”, version “68”
2020-12-29T15:17:44.875135Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295} ACK collection=istio/networking/v1alpha3/destinationrules with version=“167” nonce=“4” inc=false
2020-12-29T15:17:44.875167Z info mcp Watch(): created watch 4708 for istio/networking/v1alpha3/destinationrules from group “default”, version “167”
2020-12-29T15:17:44.875181Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295} ACK collection=istio/config/v1alpha2/httpapispecs with version=“0” nonce=“5” inc=false
2020-12-29T15:17:44.875191Z info mcp Watch(): created watch 4709 for istio/config/v1alpha2/httpapispecs from group “default”, version “0”
2020-12-29T15:17:44.875261Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295} ACK collection=istio/networking/v1alpha3/serviceentries with version=“6” nonce=“6” inc=false
2020-12-29T15:17:44.875274Z info mcp Watch(): created watch 4710 for istio/networking/v1alpha3/serviceentries from group “default”, version “6”
2020-12-29T15:17:44.875283Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295} ACK collection=istio/mixer/v1/config/client/quotaspecbindings with version=“0” nonce=“7” inc=false
2020-12-29T15:17:44.875296Z info mcp Watch(): created watch 4711 for istio/mixer/v1/config/client/quotaspecbindings from group “default”, version “0”
2020-12-29T15:17:44.875296Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296} ACK collection=istio/networking/v1alpha3/virtualservices with version=“134” nonce=“1” inc=false
2020-12-29T15:17:44.875310Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295} ACK collection=istio/networking/v1alpha3/envoyfilters with version=“135” nonce=“8” inc=false
2020-12-29T15:17:44.875323Z info mcp Watch(): created watch 4712 for istio/networking/v1alpha3/envoyfilters from group “default”, version “135”
2020-12-29T15:17:44.875334Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295} ACK collection=istio/networking/v1alpha3/sidecars with version=“231” nonce=“9” inc=false
2020-12-29T15:17:44.875342Z info mcp Watch(): created watch 4713 for istio/networking/v1alpha3/virtualservices from group “default”, version “134”
2020-12-29T15:17:44.875362Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296} ACK collection=istio/mixer/v1/config/client/quotaspecbindings with version=“0” nonce=“2” inc=false
2020-12-29T15:17:44.875372Z info mcp Watch(): created watch 4714 for istio/mixer/v1/config/client/quotaspecbindings from group “default”, version “0”
2020-12-29T15:17:44.875387Z info mcp Watch(): created watch 4715 for istio/networking/v1alpha3/sidecars from group “default”, version “231”
2020-12-29T15:17:44.875397Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295} ACK collection=istio/config/v1alpha2/httpapispecbindings with version=“0” nonce=“10” inc=false
2020-12-29T15:17:44.875406Z info mcp Watch(): created watch 4716 for istio/config/v1alpha2/httpapispecbindings from group “default”, version “0”
2020-12-29T15:17:44.875418Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295} ACK collection=istio/mixer/v1/config/client/quotaspecs with version=“0” nonce=“11” inc=false
2020-12-29T15:17:44.875429Z info mcp Watch(): created watch 4717 for istio/mixer/v1/config/client/quotaspecs from group “default”, version “0”
2020-12-29T15:17:44.875441Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295} ACK collection=istio/authentication/v1alpha1/policies with version=“0” nonce=“12” inc=false
2020-12-29T15:17:44.875456Z info mcp Watch(): created watch 4718 for istio/authentication/v1alpha1/policies from group “default”, version “0”
2020-12-29T15:17:44.875501Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296} ACK collection=istio/rbac/v1alpha1/serviceroles with version=“230” nonce=“3” inc=false
2020-12-29T15:17:44.875529Z info mcp Watch(): created watch 4719 for istio/rbac/v1alpha1/serviceroles from group “default”, version “230”
2020-12-29T15:17:44.875534Z info mcp MCP: connection {addr=127.0.0.1:40768 id=295} ACK collection=istio/rbac/v1alpha1/serviceroles with version=“230” nonce=“13” inc=false
2020-12-29T15:17:44.875559Z info mcp Watch(): created watch 4720 for istio/rbac/v1alpha1/serviceroles from group “default”, version “230”
2020-12-29T15:17:44.875732Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296} ACK collection=istio/rbac/v1alpha1/servicerolebindings with version=“68” nonce=“4” inc=false
2020-12-29T15:17:44.875757Z info mcp Watch(): created watch 4721 for istio/rbac/v1alpha1/servicerolebindings from group “default”, version “68”
2020-12-29T15:17:44.875769Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296} ACK collection=istio/rbac/v1alpha1/clusterrbacconfigs with version=“4” nonce=“5” inc=false
2020-12-29T15:17:44.875779Z info mcp Watch(): created watch 4722 for istio/rbac/v1alpha1/clusterrbacconfigs from group “default”, version “4”
2020-12-29T15:17:44.875789Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296} ACK collection=istio/networking/v1alpha3/gateways with version=“7” nonce=“6” inc=false
2020-12-29T15:17:44.875800Z info mcp Watch(): created watch 4723 for istio/networking/v1alpha3/gateways from group “default”, version “7”
2020-12-29T15:17:44.875810Z info mcp MCP: connection {addr=127.0.0.1:40768 id=296} ACK collection=istio/networking/v1alpha3/sidecars with version=“231” nonce=“7” inc=false
2020-12-29T15:17:44.875818Z info mcp Watch(): created watch 4724 for istio/networking/v1alpha3/sidecars from group “default”, version “231”
Can someone help us with the issue .