Hi all, we’re running Istio 1.1.8 and we’re experiencing weird problems with the istio ingress gateway after we restart it to test survivability of the system.
After the pod comes backup it just sits there like below (Though both pods we had was passing traffic before the restart.)
2019-06-11T22:18:30.284306Z info FLAG: --applicationPorts=“”
2019-06-11T22:18:30.284358Z info FLAG: --binaryPath=“/usr/local/bin/envoy”
2019-06-11T22:18:30.284366Z info FLAG: --concurrency=“0”
2019-06-11T22:18:30.284371Z info FLAG: --configPath=“/etc/istio/proxy”
2019-06-11T22:18:30.284377Z info FLAG: --connectTimeout=“10s”
2019-06-11T22:18:30.284381Z info FLAG: --controlPlaneAuthPolicy=“NONE”
2019-06-11T22:18:30.284387Z info FLAG: --controlPlaneBootstrap=“true”
2019-06-11T22:18:30.284392Z info FLAG: --customConfigFile=“”
2019-06-11T22:18:30.284396Z info FLAG: --datadogAgentAddress=“”
2019-06-11T22:18:30.284400Z info FLAG: --disableInternalTelemetry=“false”
2019-06-11T22:18:30.284405Z info FLAG: --discoveryAddress=“istio-pilot:15010”
2019-06-11T22:18:30.284410Z info FLAG: --domain=“istio-system.svc.cluster.local”
2019-06-11T22:18:30.284415Z info FLAG: --drainDuration=“45s”
2019-06-11T22:18:30.284419Z info FLAG: --envoyMetricsServiceAddress=“”
2019-06-11T22:18:30.284424Z info FLAG: --help=“false”
2019-06-11T22:18:30.284438Z info FLAG: --id=“”
2019-06-11T22:18:30.284443Z info FLAG: --ip=“”
2019-06-11T22:18:30.284447Z info FLAG: --lightstepAccessToken=“”
2019-06-11T22:18:30.284451Z info FLAG: --lightstepAddress=“”
2019-06-11T22:18:30.284456Z info FLAG: --lightstepCacertPath=“”
2019-06-11T22:18:30.284460Z info FLAG: --lightstepSecure=“false”
2019-06-11T22:18:30.284464Z info FLAG: --log_as_json=“false”
2019-06-11T22:18:30.284469Z info FLAG: --log_caller=“”
2019-06-11T22:18:30.284476Z info FLAG: --log_output_level=“default:info”
2019-06-11T22:18:30.284480Z info FLAG: --log_rotate=“”
2019-06-11T22:18:30.284485Z info FLAG: --log_rotate_max_age=“30”
2019-06-11T22:18:30.284490Z info FLAG: --log_rotate_max_backups=“1000”
2019-06-11T22:18:30.284494Z info FLAG: --log_rotate_max_size=“104857600”
2019-06-11T22:18:30.284499Z info FLAG: --log_stacktrace_level=“default:none”
2019-06-11T22:18:30.284510Z info FLAG: --log_target=“[stdout]”
2019-06-11T22:18:30.284516Z info FLAG: --parentShutdownDuration=“1m0s”
2019-06-11T22:18:30.284522Z info FLAG: --proxyAdminPort=“15000”
2019-06-11T22:18:30.284526Z info FLAG: --proxyLogLevel=“warning”
2019-06-11T22:18:30.284531Z info FLAG: --serviceCluster=“istio-ingressgateway”
2019-06-11T22:18:30.284536Z info FLAG: --serviceregistry=“Kubernetes”
2019-06-11T22:18:30.284540Z info FLAG: --statsdUdpAddress=“”
2019-06-11T22:18:30.284545Z info FLAG: --statusPort=“15020”
2019-06-11T22:18:30.284549Z info FLAG: --templateFile=“”
2019-06-11T22:18:30.284553Z info FLAG: --trust-domain=“”
2019-06-11T22:18:30.284559Z info FLAG: --zipkinAddress=“jaeger-collector.tracing.svc.cluster.local:9411”
2019-06-11T22:18:30.284575Z info Version root@11387264-87af-11e9-b00d-0a580a2c0205-docker.io/istio-1.1.8-145b18a441045d6fad33d7916380d8642c7bf21d-Clean
2019-06-11T22:18:30.284823Z info Obtained private IP [10.200.2.100]
2019-06-11T22:18:30.284895Z info Proxy role: &model.Proxy{ClusterID:“”, Type:“router”, IPAddresses:string{“10.200.2.100”, “10.200.2.100”}, ID:“istio-ingressgateway-6875c68d8b-dqmkh.istio-system”, Locality:(*core.Locality)(nil), DNSDomain:“istio-system.svc.cluster.local”, ConfigNamespace:“”, TrustDomain:“cluster.local”, Metadata:map[string]string{}, SidecarScope:(*model.SidecarScope)(nil), ServiceInstances:*model.ServiceInstance(nil), WorkloadLabels:model.LabelsCollection(nil)}
2019-06-11T22:18:30.284913Z info PilotSAN string(nil)
2019-06-11T22:18:30.285446Z info Effective config: binaryPath: /usr/local/bin/envoy
configPath: /etc/istio/proxy
connectTimeout: 10s
discoveryAddress: istio-pilot:15010
drainDuration: 45s
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: istio-ingressgateway
statNameLength: 189
tracing:
zipkin:
address: jaeger-collector.tracing.svc.cluster.local:9411
2019-06-11T22:18:30.285468Z info Monitored certs: string{“/etc/certs/root-cert.pem”, “/etc/certs/cert-chain.pem”, “/etc/certs/key.pem”}
2019-06-11T22:18:30.285497Z info PilotSAN string(nil)
2019-06-11T22:18:30.285723Z info Starting proxy agent
2019-06-11T22:18:30.285805Z info Opening status port 15020
2019-06-11T22:18:30.286245Z info Received new config, resetting budget
2019-06-11T22:18:30.286295Z info Reconciling retry (budget 10)
2019-06-11T22:18:30.286318Z info watching /etc/certs for changes
2019-06-11T22:18:30.286367Z info Epoch 0 starting
2019-06-11T22:18:30.287899Z info Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingressgateway --service-node router~10.200.2.100~istio-ingressgateway-6875c68d8b-dqmkh.istio-system~istio-system.svc.cluster.local --max-obj-name-len 189 --allow-unknown-fields -l warning]
[2019-06-11 22:18:30.309][19][warning][misc] [external/envoy/source/common/protobuf/utility.cc:174] Using deprecated option ‘envoy.api.v2.Cluster.hosts’ from file cds.proto. This configuration will be removed from Envoy soon. Please see Deprecated — envoy 1.28.0-dev-c29b4a documentation for details.
[2019-06-11 22:18:30.310][19][warning][misc] [external/envoy/source/common/protobuf/utility.cc:174] Using deprecated option ‘envoy.api.v2.Cluster.hosts’ from file cds.proto. This configuration will be removed from Envoy soon. Please see Deprecated — envoy 1.28.0-dev-c29b4a documentation for details.
[2019-06-11 22:18:30.310][19][warning][misc] [external/envoy/source/common/protobuf/utility.cc:174] Using deprecated option ‘envoy.api.v2.Cluster.hosts’ from file cds.proto. This configuration will be removed from Envoy soon. Please see Deprecated — envoy 1.28.0-dev-c29b4a documentation for details.
[2019-06-11 22:18:30.314][19][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:86] gRPC config stream closed: 14, no healthy upstream
[2019-06-11 22:18:30.314][19][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:49] Unable to establish new stream
2019-06-11T22:18:31.650343Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-06-11T22:18:33.650096Z info Envoy proxy is ready