Envoy sidecar fails to standup (K8S / AKS / mtls / Vault CA)

My env is Istio 1.2.7 on AKS (1.4.8). With mTLS enabled. We have configured Vault as the CA using a self signed cert. The envoy container does not start … the log from the envoy proxy is below …

The application container exposes port 80. Interestingly this happens only when I associate a clusterip service with the deployment. The deployment (pods, replicaset) on its own stands up.

Any pointers would be great help.

2019-11-13T14:53:32.369872Z info FLAG: --applicationPorts="[80]"
2019-11-13T14:53:32.369901Z info FLAG: --binaryPath="/usr/local/bin/envoy"
2019-11-13T14:53:32.369905Z info FLAG: --concurrency=“2”
2019-11-13T14:53:32.369909Z info FLAG: --configPath="/etc/istio/proxy"
2019-11-13T14:53:32.369913Z info FLAG: --connectTimeout=“10s”
2019-11-13T14:53:32.369917Z info FLAG: --controlPlaneAuthPolicy=“NONE”
2019-11-13T14:53:32.369921Z info FLAG: --controlPlaneBootstrap=“true”
2019-11-13T14:53:32.369924Z info FLAG: --customConfigFile=""
2019-11-13T14:53:32.369927Z info FLAG: --datadogAgentAddress=""
2019-11-13T14:53:32.369930Z info FLAG: --disableInternalTelemetry=“false”
2019-11-13T14:53:32.369933Z info FLAG: --discoveryAddress=“istio-pilot.istio-system:15010”
2019-11-13T14:53:32.369936Z info FLAG: --dnsRefreshRate=“300s”
2019-11-13T14:53:32.369940Z info FLAG: --domain=“sw-system.svc.cluster.local”
2019-11-13T14:53:32.369943Z info FLAG: --drainDuration=“45s”
2019-11-13T14:53:32.369946Z info FLAG: --envoyMetricsServiceAddress=""
2019-11-13T14:53:32.369949Z info FLAG: --help=“false”
2019-11-13T14:53:32.369952Z info FLAG: --id=""
2019-11-13T14:53:32.369955Z info FLAG: --ip=""
2019-11-13T14:53:32.369958Z info FLAG: --lightstepAccessToken=""
2019-11-13T14:53:32.369961Z info FLAG: --lightstepAddress=""
2019-11-13T14:53:32.369964Z info FLAG: --lightstepCacertPath=""
2019-11-13T14:53:32.369967Z info FLAG: --lightstepSecure=“false”
2019-11-13T14:53:32.369970Z info FLAG: --log_as_json=“false”
2019-11-13T14:53:32.369972Z info FLAG: --log_caller=""
2019-11-13T14:53:32.369976Z info FLAG: --log_output_level=“default:info”
2019-11-13T14:53:32.369979Z info FLAG: --log_rotate=""
2019-11-13T14:53:32.369982Z info FLAG: --log_rotate_max_age=“30”
2019-11-13T14:53:32.369985Z info FLAG: --log_rotate_max_backups=“1000”
2019-11-13T14:53:32.369989Z info FLAG: --log_rotate_max_size=“104857600”
2019-11-13T14:53:32.369992Z info FLAG: --log_stacktrace_level=“default:none”
2019-11-13T14:53:32.369998Z info FLAG: --log_target="[stdout]"
2019-11-13T14:53:32.370001Z info FLAG: --mixerIdentity=""
2019-11-13T14:53:32.370004Z info FLAG: --parentShutdownDuration=“1m0s”
2019-11-13T14:53:32.370007Z info FLAG: --pilotIdentity=""
2019-11-13T14:53:32.370023Z info FLAG: --proxyAdminPort=“15000”
2019-11-13T14:53:32.370027Z info FLAG: --proxyComponentLogLevel=“misc:error”
2019-11-13T14:53:32.370030Z info FLAG: --proxyLogLevel=“warning”
2019-11-13T14:53:32.370033Z info FLAG: --serviceCluster=“canreg.sw-system”
2019-11-13T14:53:32.370036Z info FLAG: --serviceregistry=“Kubernetes”
2019-11-13T14:53:32.370039Z info FLAG: --statsdUdpAddress=""
2019-11-13T14:53:32.370042Z info FLAG: --statusPort=“15020”
2019-11-13T14:53:32.370045Z info FLAG: --templateFile=""
2019-11-13T14:53:32.370048Z info FLAG: --trust-domain=""
2019-11-13T14:53:32.370051Z info FLAG: --zipkinAddress=“zipkin.istio-system:9411”
2019-11-13T14:53:32.370067Z info Version root@2058c745b46d-docker.io/istio-1.2.7-323094605c9c4afead13b46eb8da7d96673d9c6e-dirty-Modified
2019-11-13T14:53:32.370194Z info Obtained private IP [10.0.0.107]
2019-11-13T14:53:32.370246Z info Proxy role: &model.Proxy{ClusterID:"", Type:“sidecar”, IPAddresses:string{“10.0.0.107”, “10.0.0.107”}, ID:“canreg-c79f85864-p4s8g.sw-system”, Locality:(*core.Locality)(nil), DNSDomain:“sw-system.svc.cluster.local”, TrustDomain:“cluster.local”, PilotIdentity:"", MixerIdentity:"", ConfigNamespace:"", Metadata:map[string]string{}, SidecarScope:(*model.SidecarScope)(nil), ServiceInstances:*model.ServiceInstance(nil), WorkloadLabels:model.LabelsCollection(nil)}
2019-11-13T14:53:32.370254Z info PilotSAN string(nil)
2019-11-13T14:53:32.370589Z info Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: /etc/istio/proxy
connectTimeout: 10s
discoveryAddress: istio-pilot.istio-system:15010
drainDuration: 45s
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: canreg.sw-system
statNameLength: 189
tracing:
zipkin:
address: zipkin.istio-system:9411

2019-11-13T14:53:32.370599Z info Monitored certs: string{"/etc/certs/cert-chain.pem", “/etc/certs/key.pem”, “/etc/certs/root-cert.pem”}
2019-11-13T14:53:32.370609Z info PilotSAN string(nil)
2019-11-13T14:53:32.370711Z info Opening status port 15020

2019-11-13T14:53:32.370796Z info Starting proxy agent
2019-11-13T14:53:32.370881Z warn watching /etc/certs encountered an error no such file or directory
2019-11-13T14:53:32.370889Z info Received new config, resetting budget
2019-11-13T14:53:32.370893Z info Reconciling retry (budget 10)
2019-11-13T14:53:32.370901Z info Epoch 0 starting
2019-11-13T14:53:32.376163Z info Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster canreg.sw-system --service-node sidecar~10.0.0.107~canreg-c79f85864-p4s8g.sw-system~sw-system.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --allow-unknown-fields -l warning --component-log-level misc:error --concurrency 2]
[2019-11-13 14:53:32.398][13][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:86] gRPC config stream closed: 14, no healthy upstream
[2019-11-13 14:53:32.398][13][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:49] Unable to establish new stream
[2019-11-13 14:53:33.526][13][critical][assert] [external/envoy/source/extensions/transport_sockets/tls/context_impl.cc:838] assert failure: cn_index >= 0.
[2019-11-13 14:53:33.527][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:81] Caught Aborted, suspect faulting address 0x5390000000d
[2019-11-13 14:53:33.527][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:69] Backtrace (use tools/stack_decode.py to get line numbers):
[2019-11-13 14:53:33.527][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #0: __restore_rt [0x7fb56b847390]
[2019-11-13 14:53:33.532][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #1: Envoy::Extensions::TransportSockets::Tls::ServerContextImpl::ServerContextImpl() [0xc6b31d]
[2019-11-13 14:53:33.537][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #2: Envoy::Extensions::TransportSockets::Tls::ContextManagerImpl::createSslServerContext() [0xc71620]
[2019-11-13 14:53:33.541][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #3: Envoy::Extensions::TransportSockets::Tls::ServerSslSocketFactory::ServerSslSocketFactory() [0x8f57b6]
[2019-11-13 14:53:33.546][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #4: Envoy::Extensions::TransportSockets::Tls::DownstreamSslSocketFactory::createTransportSocketFactory() [0x8f1a8c]
[2019-11-13 14:53:33.550][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #5: Envoy::Server::ListenerImpl::ListenerImpl() [0xc4431b]
[2019-11-13 14:53:33.555][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #6: Envoy::Server::ListenerManagerImpl::addOrUpdateListener() [0xc48e1a]
[2019-11-13 14:53:33.559][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #7: Envoy::Server::LdsApiImpl::onConfigUpdate() [0xc5dffa]
[2019-11-13 14:53:33.564][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #8: Envoy::Config::GrpcMuxSubscriptionImpl::onConfigUpdate() [0xdd8f81]
[2019-11-13 14:53:33.568][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #9: Envoy::Config::GrpcMuxImpl::onDiscoveryResponse() [0xdd5f69]
[2019-11-13 14:53:33.573][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #10: Envoy::Grpc::TypedAsyncStreamCallbacks<>::onReceiveMessageUntyped() [0xdd6dfe]
[2019-11-13 14:53:33.578][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #11: Envoy::Grpc::AsyncStreamImpl::onData() [0xe013ed]
[2019-11-13 14:53:33.582][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #12: Envoy::Http::AsyncStreamImpl::encodeData() [0xe04ca4]
[2019-11-13 14:53:33.586][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #13: Envoy::Http::Http2::ConnectionImpl::onFrameReceived() [0xe74f09]
[2019-11-13 14:53:33.591][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #14: nghttp2_session_on_data_received [0xe84aaf]
[2019-11-13 14:53:33.596][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #15: nghttp2_session_mem_recv [0xe86787]
[2019-11-13 14:53:33.600][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #16: Envoy::Http::Http2::ConnectionImpl::dispatch() [0xe745e9]
[2019-11-13 14:53:33.605][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #17: Envoy::Http::CodecClient::onData() [0xdf85f6]
[2019-11-13 14:53:33.610][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #18: Envoy::Http::CodecClient::CodecReadFilter::onData() [0xdf901d]
[2019-11-13 14:53:33.614][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #19: Envoy::Network::FilterManagerImpl::onRead() [0xc82b69]
[2019-11-13 14:53:33.619][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #20: Envoy::Network::ConnectionImpl::onReadReady() [0xc7fc4c]
[2019-11-13 14:53:33.624][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #21: Envoy::Network::ConnectionImpl::onFileEvent() [0xc7f711]
[2019-11-13 14:53:33.628][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #22: Envoy::Event::FileEventImpl::assignEvents()::$_0::__invoke() [0xc7a180]
[2019-11-13 14:53:33.633][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #23: event_process_active_single_queue [0x101313d]
[2019-11-13 14:53:33.637][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #24: event_base_loop [0x10116e0]
[2019-11-13 14:53:33.642][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #25: Envoy::Event::DispatcherImpl::run() [0xc791af]
[2019-11-13 14:53:33.646][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #26: Envoy::Server::InstanceImpl::run() [0xc3a783]
[2019-11-13 14:53:33.650][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #27: Envoy::MainCommonBase::run() [0x8dc17d]
[2019-11-13 14:53:33.655][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #28: main [0x8db3f8]
[2019-11-13 14:53:33.655][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #29: [0x7fb56aa88830]
2019-11-13T14:53:33.811958Z warn Epoch 0 terminated with an error: signal: aborted (core dumped)
2019-11-13T14:53:33.811987Z warn Aborted all epochs
2019-11-13T14:53:33.812025Z info Epoch 0: set retry delay to 200ms, budget to 9
2019-11-13T14:53:33.873332Z info Envoy proxy is NOT ready: failed retrieving Envoy stats: Get http://127.0.0.1:15000/stats?usedonly: dial tcp 127.0.0.1:15000: connect: connection refused
2019-11-13T14:53:34.012140Z info Reconciling retry (budget 9)
2019-11-13T14:53:34.012333Z info Epoch 0 starting
2019-11-13T14:53:34.013192Z info Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster canreg.sw-system --service-node sidecar~10.0.0.107~canreg-c79f85864-p4s8g.sw-system~sw-system.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --allow-unknown-fields -l warning --component-log-level misc:error --concurrency 2]
[2019-11-13 14:53:34.035][32][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:86] gRPC config stream closed: 14, no healthy upstream
[2019-11-13 14:53:34.035][32][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:49] Unable to establish new stream

1 Like

@leitang

is Vault supported in 1.2? I remembered in some release we disable the support due to some reason?

Vault is supported in Istio 1.2, but not supported for Istio 1.3 due to the switch to new k8s service account format.