503 between pod to pod communication (1.5.1)

15

@deepak_deore, I think, we got it working on our side. But still will need to properly test everything.
So far we found a few “acceptable” ways to make the direct pod-to-pod calls through istio-proxy:

  1. Via a headless service (as was already pointed out).
  • Adding a headless Service in k8s results in an ORIGINAL_DST cluster in Envoy (LB will be: CLUSTER_PROVIDED). Making calls through that cluster should allow you to connect directly to pods.
  • You might need to provide a Host header, ie something like curl -vvv 10.178.180.3:8000 -H "HOST: httpbin-headless.default.svc.cluster.local:8000", where 10.178.180.3 is a Pod IP to which you are trying to connect. Host header will allow Envoy to pick the correct VirtualService/Route
  • You might also need to verify that your requests are flowing through the VirtualOutbound. In our scenario, we had to use the traffic.sidecar.istio.io/includeOutboundPorts annotation.
  1. Via a “subset” defined in a DestinationRule.
  • This approach is similar to #1, but works when you have a regular service defined (ie. not Headless) and you also want to enable direct calls for the pods backing that service.
  • Change the DestinationRule rule and define a subset that uses the PASSTHROUGH LB:
    apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: httpbin
namespace: default
spec:
host: httpbin.default.svc.cluster.local
subsets:
    - name: direct
    trafficPolicy:
        loadBalancer:
        simple: PASSTHROUGH
...
  • Modfy your VirtualService to pick the direct subset on some criteria:
    apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin
namespace: default
spec:
hosts:
- httpbin.default.svc.cluster.local
http:
- match:
    - headers:
        use-direct:
        exact: "true"
    route:
    - destination:
        host: httpbin.default.svc.cluster.local
        port:
        number: 8000
        subset: direct
...
  • When that’s done, a new “direct” ORIGINAL_DST cluster will appear in envoy’s /config_dump. And you should be able to make direct calls with something like:
    curl -vvv 10.178.180.3:8000 -H "HOST: httpbin.default.svc.cluster.local:8000" -H "use-direct: true"

Our setup of Istio is a bit “non-standard” due to different reasons. But I would be curios to see if you can get this working in your environment.
A few things that may help you debugging your configs:

  • collect outputs from envoy’s /clusters and /stats admin endpoints and save them to “before” files
  • try to make curl/wget calls
  • collect the stats again and compare with the “before” versions (diff before after)

In the stats you will see where your calls are flowing to.
If they are ending up in the PassthroughCluster on the sending side, then probably a correct virtualservice/route wasn’t matched.
If your calls are ending up in the InboundPassthroughCluster on the receiving side, then something might be wrong with the listeners config.

1 Like