My env is Istio 1.2.7 on AKS (1.4.8). With mTLS enabled. We have configured Vault as the CA using a self signed cert. The envoy container does not start … the log from the envoy proxy is below …
The application container exposes port 80. Interestingly this happens only when I associate a clusterip service with the deployment. The deployment (pods, replicaset) on its own stands up.
Any pointers would be great help.
2019-11-13T14:53:32.369872Z info FLAG: --applicationPorts="[80]"
2019-11-13T14:53:32.369901Z info FLAG: --binaryPath="/usr/local/bin/envoy"
2019-11-13T14:53:32.369905Z info FLAG: --concurrency=“2”
2019-11-13T14:53:32.369909Z info FLAG: --configPath="/etc/istio/proxy"
2019-11-13T14:53:32.369913Z info FLAG: --connectTimeout=“10s”
2019-11-13T14:53:32.369917Z info FLAG: --controlPlaneAuthPolicy=“NONE”
2019-11-13T14:53:32.369921Z info FLAG: --controlPlaneBootstrap=“true”
2019-11-13T14:53:32.369924Z info FLAG: --customConfigFile=""
2019-11-13T14:53:32.369927Z info FLAG: --datadogAgentAddress=""
2019-11-13T14:53:32.369930Z info FLAG: --disableInternalTelemetry=“false”
2019-11-13T14:53:32.369933Z info FLAG: --discoveryAddress=“istio-pilot.istio-system:15010”
2019-11-13T14:53:32.369936Z info FLAG: --dnsRefreshRate=“300s”
2019-11-13T14:53:32.369940Z info FLAG: --domain=“sw-system.svc.cluster.local”
2019-11-13T14:53:32.369943Z info FLAG: --drainDuration=“45s”
2019-11-13T14:53:32.369946Z info FLAG: --envoyMetricsServiceAddress=""
2019-11-13T14:53:32.369949Z info FLAG: --help=“false”
2019-11-13T14:53:32.369952Z info FLAG: --id=""
2019-11-13T14:53:32.369955Z info FLAG: --ip=""
2019-11-13T14:53:32.369958Z info FLAG: --lightstepAccessToken=""
2019-11-13T14:53:32.369961Z info FLAG: --lightstepAddress=""
2019-11-13T14:53:32.369964Z info FLAG: --lightstepCacertPath=""
2019-11-13T14:53:32.369967Z info FLAG: --lightstepSecure=“false”
2019-11-13T14:53:32.369970Z info FLAG: --log_as_json=“false”
2019-11-13T14:53:32.369972Z info FLAG: --log_caller=""
2019-11-13T14:53:32.369976Z info FLAG: --log_output_level=“default:info”
2019-11-13T14:53:32.369979Z info FLAG: --log_rotate=""
2019-11-13T14:53:32.369982Z info FLAG: --log_rotate_max_age=“30”
2019-11-13T14:53:32.369985Z info FLAG: --log_rotate_max_backups=“1000”
2019-11-13T14:53:32.369989Z info FLAG: --log_rotate_max_size=“104857600”
2019-11-13T14:53:32.369992Z info FLAG: --log_stacktrace_level=“default:none”
2019-11-13T14:53:32.369998Z info FLAG: --log_target="[stdout]"
2019-11-13T14:53:32.370001Z info FLAG: --mixerIdentity=""
2019-11-13T14:53:32.370004Z info FLAG: --parentShutdownDuration=“1m0s”
2019-11-13T14:53:32.370007Z info FLAG: --pilotIdentity=""
2019-11-13T14:53:32.370023Z info FLAG: --proxyAdminPort=“15000”
2019-11-13T14:53:32.370027Z info FLAG: --proxyComponentLogLevel=“misc:error”
2019-11-13T14:53:32.370030Z info FLAG: --proxyLogLevel=“warning”
2019-11-13T14:53:32.370033Z info FLAG: --serviceCluster=“canreg.sw-system”
2019-11-13T14:53:32.370036Z info FLAG: --serviceregistry=“Kubernetes”
2019-11-13T14:53:32.370039Z info FLAG: --statsdUdpAddress=""
2019-11-13T14:53:32.370042Z info FLAG: --statusPort=“15020”
2019-11-13T14:53:32.370045Z info FLAG: --templateFile=""
2019-11-13T14:53:32.370048Z info FLAG: --trust-domain=""
2019-11-13T14:53:32.370051Z info FLAG: --zipkinAddress=“zipkin.istio-system:9411”
2019-11-13T14:53:32.370067Z info Version root@2058c745b46d-docker.io/istio-1.2.7-323094605c9c4afead13b46eb8da7d96673d9c6e-dirty-Modified
2019-11-13T14:53:32.370194Z info Obtained private IP [10.0.0.107]
2019-11-13T14:53:32.370246Z info Proxy role: &model.Proxy{ClusterID:"", Type:“sidecar”, IPAddresses:string{“10.0.0.107”, “10.0.0.107”}, ID:“canreg-c79f85864-p4s8g.sw-system”, Locality:(*core.Locality)(nil), DNSDomain:“sw-system.svc.cluster.local”, TrustDomain:“cluster.local”, PilotIdentity:"", MixerIdentity:"", ConfigNamespace:"", Metadata:map[string]string{}, SidecarScope:(*model.SidecarScope)(nil), ServiceInstances:*model.ServiceInstance(nil), WorkloadLabels:model.LabelsCollection(nil)}
2019-11-13T14:53:32.370254Z info PilotSAN string(nil)
2019-11-13T14:53:32.370589Z info Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: /etc/istio/proxy
connectTimeout: 10s
discoveryAddress: istio-pilot.istio-system:15010
drainDuration: 45s
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: canreg.sw-system
statNameLength: 189
tracing:
zipkin:
address: zipkin.istio-system:9411
2019-11-13T14:53:32.370599Z info Monitored certs: string{"/etc/certs/cert-chain.pem", “/etc/certs/key.pem”, “/etc/certs/root-cert.pem”}
2019-11-13T14:53:32.370609Z info PilotSAN string(nil)
2019-11-13T14:53:32.370711Z info Opening status port 15020
2019-11-13T14:53:32.370796Z info Starting proxy agent
2019-11-13T14:53:32.370881Z warn watching /etc/certs encountered an error no such file or directory
2019-11-13T14:53:32.370889Z info Received new config, resetting budget
2019-11-13T14:53:32.370893Z info Reconciling retry (budget 10)
2019-11-13T14:53:32.370901Z info Epoch 0 starting
2019-11-13T14:53:32.376163Z info Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster canreg.sw-system --service-node sidecar~10.0.0.107~canreg-c79f85864-p4s8g.sw-system~sw-system.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --allow-unknown-fields -l warning --component-log-level misc:error --concurrency 2]
[2019-11-13 14:53:32.398][13][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:86] gRPC config stream closed: 14, no healthy upstream
[2019-11-13 14:53:32.398][13][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:49] Unable to establish new stream
[2019-11-13 14:53:33.526][13][critical][assert] [external/envoy/source/extensions/transport_sockets/tls/context_impl.cc:838] assert failure: cn_index >= 0.
[2019-11-13 14:53:33.527][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:81] Caught Aborted, suspect faulting address 0x5390000000d
[2019-11-13 14:53:33.527][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:69] Backtrace (use tools/stack_decode.py to get line numbers):
[2019-11-13 14:53:33.527][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #0: __restore_rt [0x7fb56b847390]
[2019-11-13 14:53:33.532][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #1: Envoy::Extensions::TransportSockets::Tls::ServerContextImpl::ServerContextImpl() [0xc6b31d]
[2019-11-13 14:53:33.537][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #2: Envoy::Extensions::TransportSockets::Tls::ContextManagerImpl::createSslServerContext() [0xc71620]
[2019-11-13 14:53:33.541][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #3: Envoy::Extensions::TransportSockets::Tls::ServerSslSocketFactory::ServerSslSocketFactory() [0x8f57b6]
[2019-11-13 14:53:33.546][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #4: Envoy::Extensions::TransportSockets::Tls::DownstreamSslSocketFactory::createTransportSocketFactory() [0x8f1a8c]
[2019-11-13 14:53:33.550][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #5: Envoy::Server::ListenerImpl::ListenerImpl() [0xc4431b]
[2019-11-13 14:53:33.555][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #6: Envoy::Server::ListenerManagerImpl::addOrUpdateListener() [0xc48e1a]
[2019-11-13 14:53:33.559][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #7: Envoy::Server::LdsApiImpl::onConfigUpdate() [0xc5dffa]
[2019-11-13 14:53:33.564][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #8: Envoy::Config::GrpcMuxSubscriptionImpl::onConfigUpdate() [0xdd8f81]
[2019-11-13 14:53:33.568][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #9: Envoy::Config::GrpcMuxImpl::onDiscoveryResponse() [0xdd5f69]
[2019-11-13 14:53:33.573][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #10: Envoy::Grpc::TypedAsyncStreamCallbacks<>::onReceiveMessageUntyped() [0xdd6dfe]
[2019-11-13 14:53:33.578][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #11: Envoy::Grpc::AsyncStreamImpl::onData() [0xe013ed]
[2019-11-13 14:53:33.582][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #12: Envoy::Http::AsyncStreamImpl::encodeData() [0xe04ca4]
[2019-11-13 14:53:33.586][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #13: Envoy::Http::Http2::ConnectionImpl::onFrameReceived() [0xe74f09]
[2019-11-13 14:53:33.591][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #14: nghttp2_session_on_data_received [0xe84aaf]
[2019-11-13 14:53:33.596][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #15: nghttp2_session_mem_recv [0xe86787]
[2019-11-13 14:53:33.600][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #16: Envoy::Http::Http2::ConnectionImpl::dispatch() [0xe745e9]
[2019-11-13 14:53:33.605][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #17: Envoy::Http::CodecClient::onData() [0xdf85f6]
[2019-11-13 14:53:33.610][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #18: Envoy::Http::CodecClient::CodecReadFilter::onData() [0xdf901d]
[2019-11-13 14:53:33.614][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #19: Envoy::Network::FilterManagerImpl::onRead() [0xc82b69]
[2019-11-13 14:53:33.619][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #20: Envoy::Network::ConnectionImpl::onReadReady() [0xc7fc4c]
[2019-11-13 14:53:33.624][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #21: Envoy::Network::ConnectionImpl::onFileEvent() [0xc7f711]
[2019-11-13 14:53:33.628][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #22: Envoy::Event::FileEventImpl::assignEvents()::$_0::__invoke() [0xc7a180]
[2019-11-13 14:53:33.633][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #23: event_process_active_single_queue [0x101313d]
[2019-11-13 14:53:33.637][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #24: event_base_loop [0x10116e0]
[2019-11-13 14:53:33.642][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #25: Envoy::Event::DispatcherImpl::run() [0xc791af]
[2019-11-13 14:53:33.646][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #26: Envoy::Server::InstanceImpl::run() [0xc3a783]
[2019-11-13 14:53:33.650][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #27: Envoy::MainCommonBase::run() [0x8dc17d]
[2019-11-13 14:53:33.655][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #28: main [0x8db3f8]
[2019-11-13 14:53:33.655][13][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #29: [0x7fb56aa88830]
2019-11-13T14:53:33.811958Z warn Epoch 0 terminated with an error: signal: aborted (core dumped)
2019-11-13T14:53:33.811987Z warn Aborted all epochs
2019-11-13T14:53:33.812025Z info Epoch 0: set retry delay to 200ms, budget to 9
2019-11-13T14:53:33.873332Z info Envoy proxy is NOT ready: failed retrieving Envoy stats: Get http://127.0.0.1:15000/stats?usedonly: dial tcp 127.0.0.1:15000: connect: connection refused
2019-11-13T14:53:34.012140Z info Reconciling retry (budget 9)
2019-11-13T14:53:34.012333Z info Epoch 0 starting
2019-11-13T14:53:34.013192Z info Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster canreg.sw-system --service-node sidecar~10.0.0.107~canreg-c79f85864-p4s8g.sw-system~sw-system.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --allow-unknown-fields -l warning --component-log-level misc:error --concurrency 2]
[2019-11-13 14:53:34.035][32][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:86] gRPC config stream closed: 14, no healthy upstream
[2019-11-13 14:53:34.035][32][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:49] Unable to establish new stream