I’m running on GCP’s GKE and I’m blocking access to metadata server (169.254.169.254) by using GlobalNetworkPolicy as I don’t want pods to be able to interact with it.
Looking at the logs of the istio-proxy sidecar I see that it is trying to interact with the metadata server to discover on which platform it is running on and because metadata server access is blocked it delays the time it takes the istio-proxy container to be ready significantly (over 2 minutes):
2022-04-26T08:44:07.103224Z info FLAG: --concurrency="2"
2022-04-26T08:44:07.103255Z info FLAG: --domain="test.svc.cluster.local"
2022-04-26T08:44:07.103264Z info FLAG: --help="false"
2022-04-26T08:44:07.103269Z info FLAG: --log_as_json="false"
2022-04-26T08:44:07.103273Z info FLAG: --log_caller=""
2022-04-26T08:44:07.103278Z info FLAG: --log_output_level="default:info"
2022-04-26T08:44:07.103282Z info FLAG: --log_rotate=""
2022-04-26T08:44:07.103287Z info FLAG: --log_rotate_max_age="30"
2022-04-26T08:44:07.103292Z info FLAG: --log_rotate_max_backups="1000"
2022-04-26T08:44:07.103297Z info FLAG: --log_rotate_max_size="104857600"
2022-04-26T08:44:07.103302Z info FLAG: --log_stacktrace_level="default:none"
2022-04-26T08:44:07.103329Z info FLAG: --log_target="[stdout]"
2022-04-26T08:44:07.103335Z info FLAG: --meshConfig="./etc/istio/config/mesh"
2022-04-26T08:44:07.103340Z info FLAG: --outlierLogPath=""
2022-04-26T08:44:07.103366Z info FLAG: --proxyComponentLogLevel="misc:error"
2022-04-26T08:44:07.103372Z info FLAG: --proxyLogLevel="warning"
2022-04-26T08:44:07.103377Z info FLAG: --serviceCluster="istio-proxy"
2022-04-26T08:44:07.103381Z info FLAG: --stsPort="0"
2022-04-26T08:44:07.103385Z info FLAG: --templateFile=""
2022-04-26T08:44:07.103390Z info FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2022-04-26T08:44:07.103397Z info FLAG: --vklog="0"
2022-04-26T08:44:07.103403Z info Version 1.13.3-b28579cb30c12c428ea58279b7c06f3302abe924-Clean
2022-04-26T08:44:07.103627Z info Proxy role ips=[10.36.0.28] type=sidecar id=test-0.test domain=test.svc.cluster.local
2022-04-26T08:44:07.103790Z info Apply proxy config from env {}
2022-04-26T08:44:07.105562Z info Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: istio-proxy
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
zipkin:
address: zipkin.istio-system:9411
2022-04-26T08:44:07.105588Z info JWT policy is third-party-jwt
2022-04-26T08:44:07.124377Z info platform detected is GCP
2022-04-26T08:44:07.124575Z info CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2022-04-26T08:44:07.124617Z info Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2022-04-26T08:44:07.124802Z info citadelclient Citadel client using custom root cert: var/run/secrets/istio/root-cert.pem
2022-04-26T08:44:07.127847Z info Opening status port 15020
2022-04-26T08:44:07.162681Z info ads All caches have been synced up in 68.84646ms, marking server ready
2022-04-26T08:44:07.163031Z info sds SDS server for workload certificates started, listening on "etc/istio/proxy/SDS"
2022-04-26T08:44:07.163076Z info xdsproxy Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2022-04-26T08:44:07.168975Z info sds Starting SDS grpc server
2022-04-26T08:44:07.169103Z info starting Http service at 127.0.0.1:15004
2022-04-26T08:44:07.206685Z error error in getting aws info for iam/info : Get "http://169.254.169.254/latest/meta-data/iam/info": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-04-26T08:44:07.445829Z info cache generated new workload certificate latency=281.597604ms ttl=23h59m59.554188773s
2022-04-26T08:44:07.445877Z info cache Root cert has changed, start rotating root cert
2022-04-26T08:44:07.445902Z info ads XDS: Incremental Pushing:0 ConnectedEndpoints:0 Version:
2022-04-26T08:44:07.445970Z info cache returned workload trust anchor from cache ttl=23h59m59.554034003s
2022-04-26T08:46:21.950914Z warn Error fetching GCP zone: Get "http://169.254.169.254/computeMetadata/v1/instance/zone": dial tcp 169.254.169.254:80: i/o timeout
2022-04-26T08:46:35.671510Z warn Error fetching GCP zone: Get "http://169.254.169.254/computeMetadata/v1/instance/zone": dial tcp 169.254.169.254:80: i/o timeout
2022-04-26T08:46:35.671560Z info Pilot SAN: [istiod.istio-system.svc]
2022-04-26T08:46:35.673749Z info Starting proxy agent
2022-04-26T08:46:35.673945Z info Epoch 0 starting
2022-04-26T08:46:35.674101Z info Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --drain-strategy immediate --parent-shutdown-time-s 60 --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --log-format %Y-%m-%dT%T.%fZ %l envoy %n %v -l warning --component-log-level misc:error --concurrency 2]
2022-04-26T08:46:35.768721Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2022-04-26T08:46:35.815383Z info ads ADS: new connection for node:test-0.test-1
2022-04-26T08:46:35.815527Z info cache returned workload trust anchor from cache ttl=23h57m31.184482382s
2022-04-26T08:46:35.816077Z info ads SDS: PUSH request for node:test-0.test resources:1 size:1.1kB resource:ROOTCA
2022-04-26T08:46:35.855430Z info ads ADS: new connection for node:test-0.test-2
2022-04-26T08:46:35.856000Z info cache returned workload certificate from cache ttl=23h57m31.144015711s
2022-04-26T08:46:35.856481Z info ads SDS: PUSH request for node:test-0.test resources:1 size:4.0kB resource:default
2022-04-26T08:46:36.155029Z info Readiness succeeded in 2m29.059840508s
2022-04-26T08:46:36.155640Z info Envoy proxy is ready
How can I instruct the istio-proxy sidecar to skip the cloud platform discovery step?
And generally asking - why is this step required to begin with? (why does it needs to know on which cloud platform it is running on?)
I tried to set the CLOUD_PLATFORM environment variable on the istiod pod to none but it didn’t seem to have any affect.
Please advise.