istio service backend seems to be sending traffic to the wrong ports for ingress proxy.
Environment Information
-
I am running Istio on Docker for Desktop with Kubernetes with settings as specified in
Istio / Docker Desktop -
Docker for Desktop: 3.6.0 (37651)
Engine: 20.10.8
Kubernetes: 1.21.3
OS: MacOS Big Sur 11.5.2 -
istioctl version (Installed using Brew)
client version: 1.11.0
control plane version: 1.11.0
data plane version: 1.11.0 (8 proxies)
install command:istioctl install --set profile=demo
Application Information
Installed book info sample application with ingress routes
kubectl label namespace default istio-injection=enabled
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.11/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.11/samples/bookinfo/networking/bookinfo-gateway.yaml
Application deployments and resources are created successfully
What is happening
I am unable to access application using http://localhost or using node port. TCP connection is established but gets terminated immediately.
The ingress gateway load balancer has been assigned localhost address.
istio-ingressgateway LoadBalancer 10.108.237.13 localhost 15021:31861/TCP,80:30171/TCP,443:31007/TCP,31400:31460/TCP,15443:31510/TCP 18h
Possible Cause
It looks like the service load balancer is sending request to backend port 8080
Port: http2 80/TCP
TargetPort: 8080/TCP
NodePort: http2 30171/TCP
Endpoints: 10.1.0.71:8080
However, when I do nestat on envoy ingress proxy container, I do not see any thing listening on port 8080
kubectl exec -it istio-ingressgateway-5dc645f586-2wtxc -n istio-system – /bin/bash
istio-proxy@istio-ingressgateway-5dc645f586-2wtxc:/$ netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:15021 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:http-alt 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:15090 0.0.0.0:* LISTEN
tcp 0 0 localhost:15000 0.0.0.0:* LISTEN
tcp 0 0 localhost:15004 0.0.0.0:* LISTEN
tcp6 0 0 [::]:15020 [::]:* LISTEN
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 46382 etc/istio/proxy/SDS
unix 2 [ ACC ] STREAM LISTENING 46383 etc/istio/proxy/XDS
I am new to Istio, but have used other service meshes, and something does not seem to write on port mapping for service.
Logs:
`
2021-08-24T18:27:49.108647Z info FLAG: --concurrency=“0”
2021-08-24T18:27:49.108675Z info FLAG: --domain=“istio-system.svc.cluster.local”
2021-08-24T18:27:49.108679Z info FLAG: --help=“false”
2021-08-24T18:27:49.108862Z info FLAG: --log_as_json=“false”
2021-08-24T18:27:49.108866Z info FLAG: --log_caller=""
2021-08-24T18:27:49.108869Z info FLAG: --log_output_level=“default:info”
2021-08-24T18:27:49.108870Z info FLAG: --log_rotate=""
2021-08-24T18:27:49.108872Z info FLAG: --log_rotate_max_age=“30”
2021-08-24T18:27:49.108874Z info FLAG: --log_rotate_max_backups=“1000”
2021-08-24T18:27:49.108876Z info FLAG: --log_rotate_max_size=“104857600”
2021-08-24T18:27:49.108878Z info FLAG: --log_stacktrace_level=“default:none”
2021-08-24T18:27:49.108939Z info FLAG: --log_target="[stdout]"
2021-08-24T18:27:49.108949Z info FLAG: --meshConfig="./etc/istio/config/mesh"
2021-08-24T18:27:49.108951Z info FLAG: --outlierLogPath=""
2021-08-24T18:27:49.108953Z info FLAG: --proxyComponentLogLevel=“misc:error”
2021-08-24T18:27:49.108955Z info FLAG: --proxyLogLevel=“warning”
2021-08-24T18:27:49.108957Z info FLAG: --serviceCluster=“istio-proxy”
2021-08-24T18:27:49.108959Z info FLAG: --stsPort=“0”
2021-08-24T18:27:49.108961Z info FLAG: --templateFile=""
2021-08-24T18:27:49.108963Z info FLAG: --tokenManagerPlugin=“GoogleTokenExchange”
2021-08-24T18:27:49.108965Z info Version 1.11.0-57d639a4fd19ee8c3559b9a4032f91e4d23c6f14-Clean
2021-08-24T18:27:49.109206Z info Proxy role ips=[10.1.0.71] type=router id=istio-ingressgateway-5dc645f586-2wtxc.istio-system domain=istio-system.svc.cluster.local
2021-08-24T18:27:49.109332Z info Apply mesh config from file accessLogFile: /dev/stdout
defaultConfig:
discoveryAddress: istiod.istio-system.svc:15012
proxyMetadata: {}
tracing:
zipkin:
address: zipkin.istio-system:9411
enablePrometheusMerge: true
rootNamespace: istio-system
trustDomain: cluster.local
2021-08-24T18:27:49.110410Z info Effective config: binaryPath: /usr/local/bin/envoy
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
parentShutdownDuration: 60s
proxyAdminPort: 15000
proxyMetadata: {}
serviceCluster: istio-proxy
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
zipkin:
address: zipkin.istio-system:9411
2021-08-24T18:27:49.110421Z info JWT policy is third-party-jwt
2021-08-24T18:27:49.111851Z info CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2021-08-24T18:27:49.111901Z info Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2021-08-24T18:27:49.111936Z info Opening status port 15020
2021-08-24T18:27:49.112058Z info citadelclient Citadel client using custom root cert: istiod.istio-system.svc:15012
2021-08-24T18:27:49.162855Z info ads All caches have been synced up in 59.018202ms, marking server ready
2021-08-24T18:27:49.173926Z info sds SDS server for workload certificates started, listening on “etc/istio/proxy/SDS”
2021-08-24T18:27:49.174168Z info xdsproxy Initializing with upstream address “istiod.istio-system.svc:15012” and cluster “Kubernetes”
2021-08-24T18:27:49.174460Z info sds Starting SDS grpc server
2021-08-24T18:27:49.175143Z info Pilot SAN: [istiod.istio-system.svc]
2021-08-24T18:27:49.175563Z info starting Http service at 127.0.0.1:15004
2021-08-24T18:27:49.177997Z info Starting proxy agent
2021-08-24T18:27:49.179684Z info Epoch 0 starting
2021-08-24T18:27:49.179930Z info Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --drain-strategy immediate --parent-shutdown-time-s 60 --local-address-ip-version v4 --bootstrap-version 3 --file-flush-interval-msec 1000 --disable-hot-restart --log-format %Y-%m-%dT%T.%fZ %l envoy %n %v -l warning --component-log-level misc:error]
2021-08-24T18:28:04.116432Z warn ca ca request failed, starting attempt 1 in 102.093205ms
2021-08-24T18:28:04.118520Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, connection error: desc = “transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.1.0.71:55112->10.96.0.10:53: i/o timeout”
2021-08-24T18:28:04.219787Z warn ca ca request failed, starting attempt 2 in 217.620363ms
2021-08-24T18:28:04.438523Z warn ca ca request failed, starting attempt 3 in 413.164804ms
2021-08-24T18:28:04.831093Z warn ca ca request failed, starting attempt 4 in 790.034269ms
2021-08-24T18:28:09.552227Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 10.110.121.41:15012: connect: connection refused”
2021-08-24T18:28:09.552752Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, connection error: desc = “transport: Error while dialing dial tcp 10.110.121.41:15012: connect: connection refused”
2021-08-24T18:28:10.135926Z warn ca ca request failed, starting attempt 1 in 103.736461ms
2021-08-24T18:28:10.191087Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, connection error: desc = “transport: Error while dialing dial tcp 10.110.121.41:15012: connect: connection refused”
2021-08-24T18:28:10.240920Z warn ca ca request failed, starting attempt 2 in 182.62548ms
2021-08-24T18:28:10.424554Z warn ca ca request failed, starting attempt 3 in 372.52154ms
2021-08-24T18:28:10.798262Z warn ca ca request failed, starting attempt 4 in 735.515123ms
2021-08-24T18:28:11.224944Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, connection error: desc = “transport: Error while dialing dial tcp 10.110.121.41:15012: connect: connection refused”
2021-08-24T18:28:11.548250Z info cache generated new workload certificate latency=1.532884282s ttl=23h59m59.451761964s
2021-08-24T18:28:11.548298Z info cache Root cert has changed, start rotating root cert
2021-08-24T18:28:11.548312Z info ads XDS: Incremental Pushing:0 ConnectedEndpoints:0 Version:
2021-08-24T18:28:11.548353Z info cache returned workload trust anchor from cache ttl=23h59m59.451649043s
2021-08-24T18:28:15.184642Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-08-24T18:28:15.200213Z info ads ADS: new connection for node:istio-ingressgateway-5dc645f586-2wtxc.istio-system-1
2021-08-24T18:28:15.200291Z info ads ADS: new connection for node:istio-ingressgateway-5dc645f586-2wtxc.istio-system-2
2021-08-24T18:28:15.200378Z info cache returned workload trust anchor from cache ttl=23h59m55.799627562s
2021-08-24T18:28:15.200520Z info cache returned workload certificate from cache ttl=23h59m55.799483891s
2021-08-24T18:28:15.200843Z info ads SDS: PUSH for node:istio-ingressgateway-5dc645f586-2wtxc.istio-system resources:1 size:1.1kB resource:ROOTCA
2021-08-24T18:28:15.200977Z info ads SDS: PUSH for node:istio-ingressgateway-5dc645f586-2wtxc.istio-system resources:1 size:4.0kB resource:default
2021-08-24T18:28:17.218320Z info Initialization took 28.134930928s
2021-08-24T18:28:17.218351Z info Envoy proxy is ready
`