Accessing service using istio ingress gives 503 error when mTLS is enabled


#1

I have a mutual TLS enabled Istio mesh. My setup is as follows

  1. A service running inside a pod (Service container + envoy)
  2. An envoy gateway which stays in front of the above service. An Istio Gateway and Virtual Service attached to this. It routes /info/ route to the above service.
  3. Another Istio Gateway configured for ingress using the default istio ingress pod. This also has Gateway+Virtual Service combination. The virtual service directs /info/ path to the service described in 2

I’m attempting to access the service from the ingress gateway using a curl command such as:

$ curl -X GET http://istio-ingressgateway.istio-system:80/info/ -H "Authorization: Bearer $token" -v

But I’m getting a 503 not found error as below:

wso2carbon@gateway-6c5558f8b9-x9tvt:~$ curl -X GET http://istio-ingressgateway.istio-system:80/info/ -H "Authorization: Bearer $token" -v
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 10.105.138.94...
* Connected to istio-ingressgateway.istio-system (10.105.138.94) port 80 (#0)
> GET /info/ HTTP/1.1
> Host: istio-ingressgateway.istio-system
> User-Agent: curl/7.47.0
> Accept: */*
> Authorization: Bearer ...
>
< HTTP/1.1 503 Service Unavailable
< content-length: 57
< content-type: text/plain
< date: Sat, 12 Jan 2019 13:30:13 GMT
< server: envoy
<
* Connection #0 to host istio-ingressgateway.istio-system left intact

I managed to get a successful response for the same curl command when the mesh was created in mTLS disabled mode. There are no conflicts shown in mTLS setup.

Here are the config details for my service mesh in case you need additional info.

Pods

pasan@ubuntu:~/product-vick/system/scripts/all-in-one-installer$ kubectl get pods --all-namespaces
NAMESPACE       NAME                                                       READY     STATUS      RESTARTS   AGE
default         hr--gateway-deployment-688986c87c-z9nkh                    1/1       Running     0          37m
default         hr--hr-deployment-596946948d-c89bn                         2/2       Running     0          37m
default         hr--sts-deployment-694d7cff97-gjwdk                        1/1       Running     0          37m
ingress-nginx   default-http-backend-6586bc58b6-8qss6                      1/1       Running     0          42m
ingress-nginx   nginx-ingress-controller-6bd7c597cb-t4rwq                  1/1       Running     0          42m
istio-system    grafana-85dbf49c94-lfpbr                                   1/1       Running     0          42m
istio-system    istio-citadel-545f49c58b-dq5lq                             1/1       Running     0          42m
istio-system    istio-cleanup-secrets-bh5ws                                0/1       Completed   0          42m
istio-system    istio-egressgateway-7d59954f4-qcnxm                        1/1       Running     0          42m
istio-system    istio-galley-5b6449c48f-72vkb                              1/1       Running     0          42m
istio-system    istio-grafana-post-install-lwmsf                           0/1       Completed   0          42m
istio-system    istio-ingressgateway-8455c8c6f7-5khtk                      1/1       Running     0          42m
istio-system    istio-pilot-58ff4d6647-bct4b                               2/2       Running     0          42m
istio-system    istio-policy-59685fd869-h7v94                              2/2       Running     0          42m
istio-system    istio-security-post-install-cqj6k                          0/1       Completed   0          42m
istio-system    istio-sidecar-injector-75b9866679-qg88s                    1/1       Running     0          42m
istio-system    istio-statsd-prom-bridge-549d687fd9-bspj2                  1/1       Running     0          42m
istio-system    istio-telemetry-6ccf9ddb96-hxnwv                           2/2       Running     0          42m
istio-system    istio-tracing-7596597bd7-m5pk8                             1/1       Running     0          42m
istio-system    prometheus-6ffc56584f-4cm5v                                1/1       Running     0          42m
istio-system    servicegraph-5d64b457b4-jttl9                              1/1       Running     0          42m
kube-system     coredns-78fcdf6894-rxw57                                   1/1       Running     0          50m
kube-system     coredns-78fcdf6894-s4bg2                                   1/1       Running     0          50m
kube-system     etcd-ubuntu                                                1/1       Running     0          49m
kube-system     kube-apiserver-ubuntu                                      1/1       Running     0          49m
kube-system     kube-controller-manager-ubuntu                             1/1       Running     0          49m
kube-system     kube-flannel-ds-9nvf9                                      1/1       Running     0          49m
kube-system     kube-proxy-r868m                                           1/1       Running     0          50m
kube-system     kube-scheduler-ubuntu                                      1/1       Running     0          49m

Services

pasan@ubuntu:~/product-vick/system/scripts/all-in-one-installer$ kubectl get svc --all-namespaces
NAMESPACE       NAME                                    TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                                                                                                                   AGE

default         hr--gateway-service                     ClusterIP      10.100.238.144   <none>         80/TCP,443/TCP                                                                                                            39m
default         hr--hr-service                          ClusterIP      10.96.193.43     <none>         80/TCP                                                                                                                    39m
default         hr--sts-service                         ClusterIP      10.99.54.137     <none>         8080/TCP,8081/TCP,8090/TCP                                                                                                39m
default         kubernetes                              ClusterIP      10.96.0.1        <none>         443/TCP                                                                                                                   52m
ingress-nginx   default-http-backend                    ClusterIP      10.109.166.229   <none>         80/TCP                                                                                                                    44m
ingress-nginx   ingress-nginx                           NodePort       10.108.9.180     192.168.60.3   80:31001/TCP,443:32315/TCP                                                                                                44m
istio-system    grafana                                 ClusterIP      10.102.141.231   <none>         3000/TCP                                                                                                                  44m
istio-system    istio-citadel                           ClusterIP      10.101.128.187   <none>         8060/TCP,9093/TCP                                                                                                         44m
istio-system    istio-egressgateway                     ClusterIP      10.102.157.204   <none>         80/TCP,443/TCP                                                                                                            44m
istio-system    istio-galley                            ClusterIP      10.96.31.251     <none>         443/TCP,9093/TCP                                                                                                          44m
istio-system    istio-ingressgateway                    LoadBalancer   10.105.138.94    <pending>      80:31380/TCP,443:31390/TCP,31400:31400/TCP,15011:31219/TCP,8060:31482/TCP,853:30034/TCP,15030:31544/TCP,15031:32652/TCP   44m
istio-system    istio-pilot                             ClusterIP      10.100.170.73    <none>         15010/TCP,15011/TCP,8080/TCP,9093/TCP                                                                                     44m
istio-system    istio-policy                            ClusterIP      10.104.77.184    <none>         9091/TCP,15004/TCP,9093/TCP                                                                                               44m
istio-system    istio-sidecar-injector                  ClusterIP      10.100.180.152   <none>         443/TCP                                                                                                                   44m
istio-system    istio-statsd-prom-bridge                ClusterIP      10.107.39.50     <none>         9102/TCP,9125/UDP                                                                                                         44m
istio-system    istio-telemetry                         ClusterIP      10.110.55.232    <none>         9091/TCP,15004/TCP,9093/TCP,42422/TCP                                                                                     44m
istio-system    jaeger-agent                            ClusterIP      None             <none>         5775/UDP,6831/UDP,6832/UDP                                                                                                44m
istio-system    jaeger-collector                        ClusterIP      10.102.43.21     <none>         14267/TCP,14268/TCP                                                                                                       44m
istio-system    jaeger-query                            ClusterIP      10.104.182.189   <none>         16686/TCP                                                                                                                 44m
istio-system    prometheus                              ClusterIP      10.100.0.70      <none>         9090/TCP                                                                                                                  44m
istio-system    servicegraph                            ClusterIP      10.97.65.37      <none>         8088/TCP                                                                                                                  44m
istio-system    tracing                                 ClusterIP      10.109.87.118    <none>         80/TCP                                                                                                                    44m
kube-system     kube-dns                                ClusterIP      10.96.0.10       <none>         53/UDP,53/TCP                                                                                                             52m

Gateway and virtual service described in point 2

pasan@ubuntu:~/product-vick/system/scripts/all-in-one-installer$ kubectl describe gateways.networking.istio.io hr--gateway
Name:         hr--gateway
Namespace:    default
Labels:       app=hr--gateway
              vick.wso2.com/cell=hr
              vick.wso2.com/gateway=hr--gateway
Annotations:  <none>
API Version:  networking.istio.io/v1alpha3
Kind:         Gateway
Metadata:
  ...
Spec:
  Selector:
    App:                           hr--gateway
    Vick . Wso 2 . Com / Cell:     hr
    Vick . Wso 2 . Com / Gateway:  hr--gateway
  Servers:
    Hosts:
      *
    Port:
      Name:      http2
      Number:    80
      Protocol:  HTTP2
    Hosts:
      *
    Port:
      Name:      https
      Number:    443
      Protocol:  HTTPS
    Tls:
      Mode:  PASSTHROUGH

pasan@ubuntu:~/product-vick/system/scripts/all-in-one-installer$ kubectl describe virtualservices.networking.istio.io hr--gateway
Name:         hr--gateway
Namespace:    default
Labels:       app=hr--gateway
              vick.wso2.com/cell=hr
              vick.wso2.com/gateway=hr--gateway
Annotations:  <none>
API Version:  networking.istio.io/v1alpha3
Kind:         VirtualService
Metadata:
  ...
Spec:
  Gateways:
    hr--gateway
  Hosts:
    *
  Http:
    Match:
      Uri:
        Prefix:  /info/
    Rewrite:
      Uri:  /
    Route:
      Destination:
        Host:  hr--hr-service

Gateway and virtual service described in point 3

pasan@ubuntu:~/product-vick/system/scripts/all-in-one-installer$ kubectl describe gateways.networking.istio.io ingress-gateway
Name:         ingress-gateway
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"networking.istio.io/v1alpha3","kind":"Gateway","metadata":{"annotations":{},"name":"ingress-gateway","namespace":"default"},"spec":{"sel...
API Version:  networking.istio.io/v1alpha3
Kind:         Gateway
Metadata:
  ...
Spec:
  Selector:
    Istio:  ingressgateway
  Servers:
    Hosts:
      *
    Port:
      Name:      http2
      Number:    80
      Protocol:  HTTP2

pasan@ubuntu:~/product-vick/system/scripts/all-in-one-installer$ kubectl describe virtualservices.networking.istio.io hr--gateway-ingress-vs
Name:         hr--gateway-ingress-vs
Namespace:    default
Labels:       app=hr--gateway
              vick.wso2.com/cell=hr
              vick.wso2.com/gateway=hr--gateway
Annotations:  <none>
API Version:  networking.istio.io/v1alpha3
Kind:         VirtualService
Metadata:
Spec:
  Gateways:
    ingress-gateway
  Hosts:
    *
  Http:
    Match:
      Uri:
        Prefix:  /info/
    Route:
      Destination:
        Host:  hr--gateway-service
Events:        <none>