Proxies never get ready

Config
1

Hello, I am trying to get Istio (built locally from master branch) running on Openshift 3.10. I have resolved all the child diseases, allowing the containers run privileged, but when I’ve deployed a basic routing rules the sidecar never gets ready. I’ve added just one gateway, one virtualservice and one destination rule (see below).

The istio-proxy never becomes ready, and in the log I can see:

2019-01-28T15:48:26.261191Z     info    Envoy proxy is NOT ready: 3 errors occurred:

* failed checking application ports. listeners="0.0.0.0:15090","172.30.39.89:5432","172.30.42.115:9779","172.30.248.217:3306","172.30.68.182:8080","172.30.134.206:443","172.30.74.133:9306","172.30.185.78:8080","172.30.227.252:15032","172.30.219.171:9779","172.30.165.246:11211","172.30.185.78:8090","172.30.0.1:443","172.30.134.167:9000","172.30.51.209:8080","172.30.227.252:443","172.30.229.137:8080","172.30.181.5:8778","172.30.131.100:5432","172.30.121.89:443","172.30.227.252:15443","172.30.112.189:1936","172.30.36.3:9306","172.30.214.35:8080","172.30.227.252:31400","172.30.0.1:53","172.30.121.27:3306","172.30.42.115:8778","172.30.22.73:5432","172.30.142.44:42422","172.30.9.89:11211","172.30.239.175:8080","172.30.112.189:80","172.30.227.252:15031","172.30.61.2:8090","172.30.54.214:6379","172.30.96.112:15011","172.30.61.2:8080","172.30.25.6:443","172.30.1.224:5000","172.30.131.49:8090","172.30.82.123:6379","172.30.243.106:443","172.30.181.5:9779","172.30.86.241:6379","172.30.128.32:6379","172.30.227.252:15029","172.30.112.189:443","172.30.219.171:8778","0.0.0.0:9100","172.30.104.69:443","172.30.229.137:8090","172.30.226.110:27017","172.30.166.34:443","172.30.121.89:15443","172.30.227.252:15030","172.30.81.233:443","172.30.131.49:8080","0.0.0.0:15004","0.0.0.0:3000","0.0.0.0:80","0.0.0.0:9090","0.0.0.0:15010","0.0.0.0:9091","0.0.0.0:8080","0.0.0.0:9093","0.0.0.0:9901","0.0.0.0:8060","[fe80::78d9:56ff:fed5:739d]:3333","[fe80::78d9:56ff:fed5:739d]:9999","0.0.0.0:15001","10.130.0.78:15020"
* envoy missing listener for inbound application port: 0
* envoy missing listener for inbound application port: 8080

I would expect that the application port matching works on 0.0.0.0:8080 which is in the list above. I am also not sure how exactly is the list above generated, but when I exec into the node, the ifconfig gives me different address than one from the list above (those are service IPs, not node IPs): 10.130.0.78.

There’s one more thing that caught my eye: in the envoy log I can see this warning:

[2019-01-28 13:15:17.816][000019][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:70] gRPC config for type.googleapis.com/envoy.api.v2.Cluster rejected: cluster: cluster type 'original_dst' may only be used with LB type 'original_dst_lb'

Running neither sidecar with --proxyLogLevel debug neither the pilot with --log_output_level default:debug gave me any further details, though.

The config follows below:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: app-gateway
  namespace: istio-scale
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
---
kind: VirtualService
metadata:
annotations:
  name: versionbased
  namespace: istio-scale
spec:
  gateways:
  - app-gateway
  hosts:
  - '*'
  http:
  - match:
      uri:
        prefix: app-1/
    rewrite:
      uri: /
    route:
    - destination:
        host: app
        port:
          number: 8080
        subset: app-1
# repeats couple more times for app-2, app-3....
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  annotations:
    name: app-subsets
    namespace: istio-scale
spec:
  host: '*'
  subsets:
  - labels:
      deploymentconfig: app-1
    name: app-1
# repeats for app-2, app-3...
  trafficPolicy:
    loadBalancer:
      simple: RANDOM
2

I found what was my issue - I had wrong selector on service and therefore no endpoints were created (I have messed this when reconfiguring the cluster). After fixing the labels the service got bound to the nodes and the proxy got its IP:port in listeners list, and could come up.

Right now I am trying to figure out why the ingressgateway is not routing the request properly but returning 404. I’ve fixed the wildcard hosts in the config above and set FQDN to route.destination.host; without success though.

3

Finally, after several days of investigation I got it to work. The main problem was in my virtualservice definition: I have used

- match:
    uri: ...

instead of

- match:
  - uri: ...

Took me a while to figure out what exactly is wrong on my definition even after istioctl validate was failing (since it complains that this is not a JSON etc. it’s hard to figure out if it’s working at all).

Another problem was that Gateway must be in the istio-system namespace (in the same namespace as istio-ingress-gateway) - I’ve found this information in sources but not in the docs (might have missed that, though). Virtualservice definition can be either in istio-system or reference the gateway by FQDN: app-gateway.istio-system.svc.cluster.local.

Finding out that I haven’t given anyuid and privileged SCCs to istio-galley-service-account and therefore it can’t open port 443 wasn’t such a problem; I haven’t noticed that it’s not opening that port until I got the virtualservice definition right and validating admission webhook started complaining.