Istioctl proxy-status failing with "exit code 126"

1

I have a nginx deployment. After istio-ingressgateway all calls go to this nginx pod (with envoy).

“istioctl proxy-status” command is failing for this particular pod. It is working fine with other microservices with envoy.

$ ./istioctl proxy-status ingress-1-6cf5f6b98d-4l9w7.r19-1-ingress-qa
Error: command terminated with exit code 126

$ ./istioctl proxy-status istio-ingressgateway-694576c7bb-sfphd.istio-system
Stderr when execute [/usr/local/bin/pilot-discovery request GET /debug/config_dump?proxyID=istio-ingressgateway-694576c7bb-sfphd.istio-system ]: gc 1 @0.016s 7%: 0.035+1.1+1.2 ms clock, 0.21+0.024/1.0/1.2+7.3 ms cpu, 4->4->1 MB, 5 MB goal, 6 P
gc 2 @0.027s 8%: 0.024+1.0+0.99 ms clock, 0.14+0.20/1.1/1.9+5.9 ms cpu, 4->4->2 MB, 5 MB goal, 6 P

Clusters Match
Listeners Match
Routes Match

2

Try adding --log_output_level default:debug to get more information.

3

Where I have to add this ? In deployment yaml’s “args” or somehwere in helm charts ?

4

Add to the istioctl command:

./istioctl proxy-status ingress-1-6cf5f6b98d-4l9w7.r19-1-ingress-qa --log_output_level default:debug

5

i wan the command with log output level for two different pods(one where it is giving the error and other where everything is fine):

Here are the Curl requests made by both the pods:

Pod with error:

2019-03-08T20:52:58.969056Z info curl -k -v -XGET -H “Accept: application/json, /” https://X.X.X.X/api/v1/namespaces/r19-1-ingress-qa/pods/ingress-1-6cf5f6b98d-pwdms
2019-03-08T20:52:59.203317Z info curl -k -v -XPOST -H “X-Stream-Protocol-Version: v4.channel.k8s.io” -H “X-Stream-Protocol-Version: v3.channel.k8s.io” -H “X-Stream-Protocol-Version: v2.channel.k8s.io” -H “X-Stream-Protocol-Version: channel.k8s.io” https://X.X.X.X/api/v1/namespaces/r19-1-ingress-qa/pods/ingress-1-6cf5f6b98d-pwdms/exec?command=%2Fusr%2Flocal%2Fbin%2Fpilot-agent&command=request&command=GET&command=config_dump&command=&container=ingress&container=ingress&stderr=true&stdout=true
Error: command terminated with exit code 126

pod with no error:

2019-03-08T20:53:48.778683Z info curl -k -v -XGET -H “Accept: application/json, /” https://X.X.X.X/api/v1/namespaces/r19-1-auth-qa/pods/auth-29-858987b888-c6zpb
2019-03-08T20:53:49.057399Z info curl -k -v -XPOST -H “X-Stream-Protocol-Version: v4.channel.k8s.io” -H “X-Stream-Protocol-Version: v3.channel.k8s.io” -H “X-Stream-Protocol-Version: v2.channel.k8s.io” -H “X-Stream-Protocol-Version: channel.k8s.io” https://X.X.X.X/api/v1/namespaces/r19-1-auth-qa/pods/auth-29-858987b888-c6zpb/exec?command=%2Fusr%2Flocal%2Fbin%2Fpilot-agent&command=request&command=GET&command=config_dump&command=&container=istio-proxy&container=istio-proxy&stderr=true&stdout=true
2019-03-08T20:53:50.187131Z info curl -k -v -XGET -H “Accept: application/json, /” https://X.X.X.X/api/v1/namespaces/istio-system/pods?labelSelector=istio%3Dpilot
2019-03-08T20:53:50.252224Z info curl -k -v -XPOST -H “X-Stream-Protocol-Version: v4.channel.k8s.io” -H “X-Stream-Protocol-Version: v3.channel.k8s.io” -H “X-Stream-Protocol-Version: v2.channel.k8s.io” -H “X-Stream-Protocol-Version: channel.k8s.io” https://X.X.X.X/api/v1/namespaces/istio-system/pods/istio-pilot-79f5f46dd5-chsl9/exec?command=%2Fusr%2Flocal%2Fbin%2Fpilot-discovery&command=request&command=GET&command=%2Fdebug%2Fconfig_dump%3FproxyID%3Dauth-29-858987b888-c6zpb.r19-1-auth-qa&command=&container=discovery&container=discovery&stderr=true&stdout=true

6

From there I would run the config_dump command directly on the pod.
Exec into the container:
kubectl exec -n r19-1-ingress-qa ingress-1-6cf5f6b98d-pwdms -it -- bash
then:
/usr/local/bin/pilot-agent request GET config_dump
And see if that errors.

7

I am getting all the info while running " /usr/local/bin/pilot-agent request GET config_dump" inside the envoy container pf the pod.

8

@dwradcliffe Can you help me figure out what could be the issue here. also now I am seeing a lot of “STALE” in “istioctl proxy-status” and 404s for all my calls.

25%20PM