Impossible to create a Gateway resource

Networking
1

Hello guys,
I am experiencing an issue while creating a Gateway resource.
I installed Istio 1.1.7 on a Kubernetes cluster, version 1.13.5. The cluster is on AWS, but it’s not EKS.
The installation is fine, but when I try to apply the bookinfo-gateway yaml file (to test eventually istio ingress) I get the following error:
Error from server (InternalError): error when creating "bookinfo-gateway.yaml": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: Service Unavailable

I know this is not a completely new error - it was actually a bug on 1.0.x versions (can’t remember exactly which one), therefore I was using the --set global.configValidation=false before. I would like to remove the flag (with the flag set to false it works) so I can actually validate the configuration.

To be entirely honest I ‘think’ this is not an Istio issue - I suspect there is something wrong with the aws CNI plugin we use (we use 1.3.0), but I couldn’t really relate any error to the one I am getting.

Can you please help me out here?
Thanks a lot,
Simone

2

Have a few more from kubectl (I used a verbose mode, level 8):

I0530 18:24:09.279494   15898 round_trippers.go:383] GET https://internal-weary-fal-masterel-z4pxtw6a3ply-1754695728.us-west-2.elb.amazonaws.com:6443/apis/networking.istio.io/v1alpha3/namespaces/default/gateways/bookinfo-gateway
I0530 18:24:09.279514   15898 round_trippers.go:390] Request Headers:
I0530 18:24:09.279517   15898 round_trippers.go:393]     Accept: application/json
I0530 18:24:09.279521   15898 round_trippers.go:393]     User-Agent: kubectl/v1.12.0 (darwin/amd64) kubernetes/0ed3388
I0530 18:24:09.468262   15898 round_trippers.go:408] Response Status: 404 Not Found in 188 milliseconds
I0530 18:24:09.468291   15898 round_trippers.go:411] Response Headers:
I0530 18:24:09.468299   15898 round_trippers.go:414]     Content-Type: application/json
I0530 18:24:09.468305   15898 round_trippers.go:414]     Content-Length: 258
I0530 18:24:09.468312   15898 round_trippers.go:414]     Date: Thu, 30 May 2019 16:24:09 GMT
I0530 18:24:09.468347   15898 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"gateways.networking.istio.io \"bookinfo-gateway\" not found","reason":"NotFound","details":{"name":"bookinfo-gateway","group":"networking.istio.io","kind":"gateways"},"code":404}
I0530 18:24:09.468691   15898 request.go:942] Request Body: {"apiVersion":"networking.istio.io/v1alpha3","kind":"Gateway","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"networking.istio.io/v1alpha3\",\"kind\":\"Gateway\",\"metadata\":{\"annotations\":{},\"name\":\"bookinfo-gateway\",\"namespace\":\"default\"},\"spec\":{\"selector\":{\"istio\":\"ingressgateway\"},\"servers\":[{\"hosts\":[\"*\"],\"port\":{\"name\":\"http\",\"number\":80,\"protocol\":\"HTTP\"}}]}}\n"},"name":"bookinfo-gateway","namespace":"default"},"spec":{"selector":{"istio":"ingressgateway"},"servers":[{"hosts":["*"],"port":{"name":"http","number":80,"protocol":"HTTP"}}]}}
I0530 18:24:09.468749   15898 round_trippers.go:383] POST https://internal-weary-fal-masterel-z4pxtw6a3ply-1754695728.us-west-2.elb.amazonaws.com:6443/apis/networking.istio.io/v1alpha3/namespaces/default/gateways
I0530 18:24:09.468759   15898 round_trippers.go:390] Request Headers:
I0530 18:24:09.468766   15898 round_trippers.go:393]     Accept: application/json
I0530 18:24:09.468773   15898 round_trippers.go:393]     Content-Type: application/json
I0530 18:24:09.468779   15898 round_trippers.go:393]     User-Agent: kubectl/v1.12.0 (darwin/amd64) kubernetes/0ed3388
I0530 18:24:09.663314   15898 round_trippers.go:408] Response Status: 500 Internal Server Error in 194 milliseconds
I0530 18:24:09.663346   15898 round_trippers.go:411] Response Headers:
I0530 18:24:09.663354   15898 round_trippers.go:414]     Content-Type: application/json
I0530 18:24:09.663362   15898 round_trippers.go:414]     Content-Length: 469
I0530 18:24:09.663368   15898 round_trippers.go:414]     Date: Thu, 30 May 2019 16:24:09 GMT
I0530 18:24:09.664525   15898 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"pilot.validation.istio.io\": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: Service Unavailable","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"pilot.validation.istio.io\": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: Service Unavailable"}]},"code":500}
I0530 18:24:09.666195   15898 loader.go:359] Config loaded from file /Users/I314665/kraken/deploykube
I0530 18:24:09.666660   15898 round_trippers.go:383] GET https://internal-weary-fal-masterel-z4pxtw6a3ply-1754695728.us-west-2.elb.amazonaws.com:6443/apis/networking.istio.io/v1alpha3/namespaces/default/virtualservices/bookinfo
I0530 18:24:09.666679   15898 round_trippers.go:390] Request Headers:
I0530 18:24:09.666687   15898 round_trippers.go:393]     Accept: application/json
I0530 18:24:09.666694   15898 round_trippers.go:393]     User-Agent: kubectl/v1.12.0 (darwin/amd64) kubernetes/0ed3388
I0530 18:24:09.856708   15898 round_trippers.go:408] Response Status: 404 Not Found in 190 milliseconds
I0530 18:24:09.856738   15898 round_trippers.go:411] Response Headers:
I0530 18:24:09.856746   15898 round_trippers.go:414]     Content-Type: application/json
I0530 18:24:09.856753   15898 round_trippers.go:414]     Content-Length: 256
I0530 18:24:09.856759   15898 round_trippers.go:414]     Date: Thu, 30 May 2019 16:24:09 GMT
I0530 18:24:09.856790   15898 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"virtualservices.networking.istio.io \"bookinfo\" not found","reason":"NotFound","details":{"name":"bookinfo","group":"networking.istio.io","kind":"virtualservices"},"code":404}
I0530 18:24:09.857181   15898 request.go:942] Request Body: {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"networking.istio.io/v1alpha3\",\"kind\":\"VirtualService\",\"metadata\":{\"annotations\":{},\"name\":\"bookinfo\",\"namespace\":\"default\"},\"spec\":{\"gateways\":[\"bookinfo-gateway\"],\"hosts\":[\"*\"],\"http\":[{\"match\":[{\"uri\":{\"exact\":\"/productpage\"}},{\"uri\":{\"exact\":\"/login\"}},{\"uri\":{\"exact\":\"/logout\"}},{\"uri\":{\"prefix\":\"/api/v1/products\"}}],\"route\":[{\"destination\":{\"host\":\"productpage\",\"port\":{\"number\":9080}}}]}]}}\n"},"name":"bookinfo","namespace":"default"},"spec":{"gateways":["bookinfo-gateway"],"hosts":["*"],"http":[{"match":[{"uri":{"exact":"/productpage"}},{"uri":{"exact":"/login"}},{"uri":{"exact":"/logout"}},{"uri":{"prefix":"/api/v1/products"}}],"route":[{"destination":{"host":"productpage","port":{"number":9080}}}]}]}}
I0530 18:24:09.857243   15898 round_trippers.go:383] POST https://internal-weary-fal-masterel-z4pxtw6a3ply-1754695728.us-west-2.elb.amazonaws.com:6443/apis/networking.istio.io/v1alpha3/namespaces/default/virtualservices
I0530 18:24:09.857254   15898 round_trippers.go:390] Request Headers:
I0530 18:24:09.857261   15898 round_trippers.go:393]     Accept: application/json
I0530 18:24:09.857267   15898 round_trippers.go:393]     Content-Type: application/json
I0530 18:24:09.857274   15898 round_trippers.go:393]     User-Agent: kubectl/v1.12.0 (darwin/amd64) kubernetes/0ed3388
I0530 18:24:10.049009   15898 round_trippers.go:408] Response Status: 500 Internal Server Error in 191 milliseconds
I0530 18:24:10.049043   15898 round_trippers.go:411] Response Headers:
I0530 18:24:10.049051   15898 round_trippers.go:414]     Content-Type: application/json
I0530 18:24:10.049060   15898 round_trippers.go:414]     Content-Length: 469
I0530 18:24:10.049066   15898 round_trippers.go:414]     Date: Thu, 30 May 2019 16:24:09 GMT
I0530 18:24:10.049103   15898 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"pilot.validation.istio.io\": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: Service Unavailable","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"pilot.validation.istio.io\": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: Service Unavailable"}]},"code":500}
I0530 18:24:10.049816   15898 helpers.go:201] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "error when creating \"/Users/I314665/istio-1.1.7/samples/bookinfo/networking/bookinfo-gateway.yaml\": Internal error occurred: failed calling webhook \"pilot.validation.istio.io\": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: Service Unavailable",
  "reason": "InternalError",
  "details": {
    "causes": [
      {
        "message": "failed calling webhook \"pilot.validation.istio.io\": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: Service Unavailable"
      }
    ]
  },
  "code": 500
}]
I0530 18:24:10.049894   15898 helpers.go:201] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "error when creating \"/Users/I314665/istio-1.1.7/samples/bookinfo/networking/bookinfo-gateway.yaml\": Internal error occurred: failed calling webhook \"pilot.validation.istio.io\": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: Service Unavailable",
  "reason": "InternalError",
  "details": {
    "causes": [
      {
        "message": "failed calling webhook \"pilot.validation.istio.io\": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: Service Unavailable"
      }
    ]
  },
  "code": 500
}]
F0530 18:24:10.049920   15898 helpers.go:119] Error from server (InternalError): error when creating "/Users/I314665/istio-1.1.7/samples/bookinfo/networking/bookinfo-gateway.yaml": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: Service Unavailable
Error from server (InternalError): error when creating "/Users/I314665/istio-1.1.7/samples/bookinfo/networking/bookinfo-gateway.yaml": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: Service Unavailable
➜  ~ kc apply -f ~/istio-1.1.7/samples/bookinfo/networking/bookinfo-gateway.yaml -n istio-system
Error from server (InternalError): error when creating "/Users/I314665/istio-1.1.7/samples/bookinfo/networking/bookinfo-gateway.yaml": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: Service Unavailable
Error from server (InternalError): error when creating "/Users/I314665/istio-1.1.7/samples/bookinfo/networking/bookinfo-gateway.yaml": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: Service Unavailable
➜```
3

The galley service is called by kube-apiserver, so first check why kube-apiserver calls it failed.

4

Thanks @hzxuzhonghu. It looks like (fingers crossed) I found out the culprit and it’s the proxy.
When Galley gets called, the domain is .istio-system.svc. I had to include it in the no_proxy directive in the api-server env variables.
I will run few more tests to confirm this.
Thanks a lot for your help!

5

Yes I confirm, the problem was the proxy.
Hope it helps.
Cheers,
Simone

6

Hi Simone,

I am also going through similar problem, could you let me know how did you fix it. what entry you added api-server manifest file.

Thanks,
Kasim Shaik.

7

Hi Kasimvali,
I think at the end I added the .istio-system.svc to the no_proxy env variable on each k8s node.
Hope it helps,
Simone