I’ve been playing around with my Istio cluster configuration and I’ve ended up in a state I can’t debug my way out of.
I have the SDS+Gateway with a public IP configured. I have deployed the Istio HelloWorld app on port 5000. I can:
- exec into
curl localhost:5000/hello- this works fine
- check with
istioctl proxy-config cluster(and such) from /docs/ops/troubleshooting/network-issues/ and /docs/ops/troubleshooting/proxy-cmd/ — all report OK to everything, SYNC:ed and such
- I can do
kubectl exec istio-ingressgateway-[rnd] /bin/bashand then
curl helloworld.mynamespace:5000/hellosuccessfully (it returns
Hello version: v2 ...
But I can’t make the ingressgateway actually return anything but 503 when querying its publicly bound IP. If I query without the
/hello path, it returns
404 instead, so it’s obviously trying to route to the
helloworld service/deployment and failing.
So I’m in the state where I can actually contact my
helloworld service from the Istio Ingress Gateway, when asking the gateway itself
curl localhost/hello -i , or from ourside the network
curl -i http://35.x.y.z/hello I always get
503 Service Unavailable Back
I don’t have any DestinationRule nor Policy applying to
helloworld , and I have Istio in strict mTLS.
I could previously today access (other) services via the ingress gateway, but then I started cleaning things up (to the point when I only have the helloworld service VirtualService+Gateway and no others), and now it doesn’t work. It should be possible to debug.
What is wrong?
Not related (that I can tell):
- I have CNI + GKE Network Policy enabled (but turning it off didn’t help) and a Calico-allow-all rule didn’t help, so it should not be this; also, I can curl from ingressgateway, so there’s connectivity
- https://github.com/istio/istio/tree/master/samples/helloworld — config