How ISTIO_KUBE_APP_PROBERS env variable works?

Config
1

Hi Team,

I’m running Istio version 1.8. There is a slight confusion here with Health-check implementation (readiness probes) of pods with istio-proxy injected.

My Deployment spec under readiness-probe section:

          httpGet:
            path: /app-health-check
            port: 8000
            scheme: HTTP

While doing istioctl kube-inject on the spec, I got my manifest file generated in yaml as follows:

        - name: ISTIO_KUBE_APP_PROBERS
          value: '{"/app-health/<APP_NAME>/readyz":{"httpGet":{"path":"/health","port":8000,"scheme":"HTTP"},"timeoutSeconds":1}}'

And I can see rewriteAppHTTPProbe is set to true:

$ kubectl get cm istio-sidecar-injector -n istio-system -o yaml |grep rewriteAppHTTPProbe
...
        "rewriteAppHTTPProbe": true
...

With this configurations everything is working fine.
But If I change ISTIO_KUBE_APP_PROBERS as follows (to my actual application’s healthcheck api):

        - name: ISTIO_KUBE_APP_PROBERS
          value: '{"/app-health/<APP_NAME>/readyz":{"httpGet":{"path":"/app-health-check","port":8000,"scheme":"HTTP"},"timeoutSeconds":1}}'

Then my pod goes to non-Ready state. And istio-proxy log shows:

[2021-12-01T07:57:39.860Z] "GET /app-health-check HTTP/1.1" 503 UF "-" 0 91 0 - "-" "kube-probe/1.19+" "d11022fc-625c-4afb-aabc-14a3e450cbde" "<PODIP>:8000" "<PODIP>:8000" InboundPassthroughClusterIpv4 - <PODIP>:8000 <HOSTIP>:46264 - default

Can someone help me understand why actual health check api (/app-health-check) in ISTIO_KUBE_APP_PROBERS doesn’t work, when /health in ISTIO_KUBE_APP_PROBERS works?

Also application running inside is bound to 127.0.0.1:8000 in the container.