VirtualService stops working when hosts is set to fqdn


#1

Here’s my custom Gateway and VirtualService config:

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway

metadata:
  namespace: discourse
  name: discourse-gw

spec:

  selector:
    # use istio default controller
    istio: ingressgateway 
            
  servers:

  # The Port on which the proxy should listen for incoming connections.
  # In this case `ingressgateway` is listening on port 80 internally and
  # on the NodePort 31380 externally.
  - port:
      number: 80
      protocol: HTTP # one of HTTP|HTTPS|GRPC|HTTP2|MONGO|TCP|TLS
      name: http-discourse
    # A list of hosts exposed by this gateway. At least one host is required.
    # Typically applicable to HTTP services, but it can also be used for TCP
    # services using TLS with SNI. May contain a wildcard prefix:
    # *.foo.com --> bar.foo.com AND *.com --> bar.foo.com, example.com, etc.
    hosts:
    - discuss.example.com
---


---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService

metadata:
  namespace: discourse
  name: discourse-virt-svc

spec:

  gateways:
  - discourse-gw

  hosts:
   - "*"

  http:

  - route:
    - destination:
        host: discourse.discourse.svc.cluster.local
        port:
          number: 3000
---

This configuration works as I can get traffic from the pods at discuss.example.com:

$ curl -so /dev/null -LD- http://discuss.example.com:31380
HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
status: 200 OK
x-discourse-cached: true
cache-control: no-cache, no-store
referrer-policy: strict-origin-when-cross-origin
x-permitted-cross-domain-policies: none
x-xss-protection: 1; mode=block
x-request-id: 501cb0db-e9ad-4393-9f9f-717a12294151
x-discourse-route: list/latest
x-discourse-trackview: 1
x-download-options: noopen
x-runtime: 0.001691
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
date: Sat, 16 Feb 2019 21:24:10 GMT
x-powered-by: Phusion Passenger 6.0.1
server: envoy
x-envoy-upstream-service-time: 3
transfer-encoding: chunked

However if I change VirtualService config according to the docs recommendation (Note: hosts has been changed from "*" to discourse.discourse.svc.cluster.local):

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService

metadata:
  namespace: discourse
  name: discourse-virt-svc

spec:

  gateways:
  - discourse-gw

  hosts:
  - discourse.discourse.svc.cluster.local

  http:

  - route:
    - destination:
        host: discourse.discourse.svc.cluster.local
        port:
          number: 3000
---

It doesn’t work anymore:

$ curl -sv http://discuss.example.com:31380
*   Trying xxx.yyy.122.33...
* TCP_NODELAY set
* connect to xxx.yyy.122.33 port 31380 failed: Connection refused
* Failed to connect to discuss.example.com port 31380: Connection refused
* Closing connection 0

Any idea why?

BTW, I’m using NodePort in the istio-ingressgateway:

---
apiVersion: v1
kind: Service
metadata:
  name: istio-ingressgateway
  namespace: istio-system
  annotations:
  labels:
    chart: gateways-1.0.5
    release: istio
    heritage: Tiller
    app: istio-ingressgateway
    istio: ingressgateway
spec:
  type: NodePort
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  ports:
    -
      name: http2
      nodePort: 31380
      port: 80
      targetPort: 80
    -
      name: https
      nodePort: 31390
      port: 443
    -
      name: tcp
      nodePort: 31400
      port: 31400
    -
      name: tcp-pilot-grpc-tls
      port: 15011
      targetPort: 15011
    -
      name: tcp-citadel-grpc-tls
      port: 8060
      targetPort: 8060
    -
      name: tcp-dns-tls
      port: 853
      targetPort: 853
    -
      name: http2-prometheus
      port: 15030
      targetPort: 15030
    -
      name: http2-grafana
      port: 15031
      targetPort: 15031
---

I’ve made a script to install the istio, so I should have logs of the following components of the istio if you need them.

#! /bin/bash --

helm template install/kubernetes/helm/istio\
 --name istio\
 --namespace istio-system\
 --set kiali.enabled=true\
 --set grafana.enabled=true\
 --set tracing.enabled=true\
 --set servicegraph.enabled=true |\
 sed 's/type: LoadBalancer/type: NodePort/'

#2

My understanding from here is that for the VirtualService linked to the Gateway, you’d also need to specify the DNS name that external clients will be calling you at:

  hosts:
  - discuss.example.com # external
  - discourse.discourse.svc.cluster.local # internal

I think what you have might still work, but only if the Host header on your HTTP request had discourse.discourse.svc.cluster.local:

$ curl -sv -H 'discourse.discourse.svc.cluster.local' http://discuss.example.com:31380 

#3

Thanks, I tried what you suggest, but envoy now returns 404:

$ curl -sv -H 'discourse.discourse.svc.cluster.local' http://discuss.example.com:31380
*   Trying xxx.yyy.122.33...
* TCP_NODELAY set
* Connected to discuss.example.com (xxx.yyy.122.33) port 31380 (#0)
> GET / HTTP/1.1
> Host: discuss.example.com:31380
> User-Agent: curl/7.63.0
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< date: Sun, 17 Feb 2019 22:58:03 GMT
< server: envoy
< content-length: 0
< 
* Connection #0 to host discuss.example.com left intact

Also I don’t understand why are you suggesting to pass this header?
-H 'discourse.discourse.svc.cluster.local'

I own the domain on which I’m testing so it should work without a problem. Furthermore my website users will be accessing it directly, thus without passing any fancy headers, so it’s pretty much a requirement that it works from the regular browser.

My understanding is that DNS is caught by the Gateway. Then if there’s a VirtualService bound to it, further traffic paths are described in the VirtualService. Hosts in the VirtualService holds the addresses where the traffic might be delivered, thus in my case DNS names of the Kubernetes Services.


#4

tldr: I think you need to include the hostnames your external clients are calling you with in the host section of the VS. Try adding discuss.example.com.

The VirtualService is expecting traffic to come through with the hostnames that you’ve listed, which is only discourse.discourse.svc.cluster.local, so it’s going to 404 as there’s no route for that hostname.

Here’s a full example that I just tested locally with Bookinfo that’s agnostic of internal service and shows the same kind of problem you’re seeing.

The following VS will only know how to deal with foo.com even though the Gateway is letting everything through:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: bookinfo-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: bookinfo
spec:
  hosts:
  - "foo.com"
  gateways:
  - bookinfo-gateway
  http:
  - match:
    - uri:
        exact: /productpage
    - uri:
        exact: /login
    - uri:
        exact: /logout
    - uri:
        prefix: /api/v1/products
    route:
    - destination:
        host: productpage
        port:
          number: 9080

Curling this for foo.com works fine (I’m using --resolve as I don’t have DNS for my test cluster):

$ curl -I --resolve foo.com:80:$MY_IP http://foo.com/productpage
HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
content-length: 5719
server: envoy
date: Mon, 18 Feb 2019 00:08:37 GMT
x-envoy-upstream-service-time: 169

However, if I try something like bar.com that’s not going to work, as the host isn’t in the VS:

$ curl -i --resolve bar.com:80:$MY_IP http://bar.com/productpage
HTTP/1.1 404 Not Found
date: Mon, 18 Feb 2019 00:09:37 GMT
server: envoy
content-length: 0

If I update the VS to include bar.com to match on, everything works fine:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: bookinfo
spec:
  hosts:
  - "foo.com"
  - "bar.com"
... snip ...
$ curl -I --resolve bar.com:$MY_IP http://bar.com/productpage
HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
content-length: 5719
server: envoy
date: Mon, 18 Feb 2019 00:10:35 GMT
x-envoy-upstream-service-time: 176

Also I don’t understand why are you suggesting to pass this header?
-H 'discourse.discourse.svc.cluster.local'

Apologies, that probably warranted more explanation: I’m suggesting that only as a way of debugging, as you can force what the hostname is. I’m not suggesting your clients do that.

Maybe I’m misunderstanding what the problem is. Hope this helps though!


#5

OK, here’s more complete example, using standard manifests from istio docs page.

The first thing I do is update my hosts file: echo 'xxx.yyy.122.33 bookinfo.example.com' > /etc/hosts

I then create a custom namespace where automatic istio sidecar injection is enabled:

---
apiVersion: v1
kind: Namespace
metadata:
  name: bookinfo
  labels:
    istio-injection: enabled
---

Then deploy bookinfo components:

$ kubectl create -f https://raw.githubusercontent.com/istio/istio/release-1.0/samples/bookinfo/platform/kube/bookinfo.yaml
service "details" created
deployment.extensions "details-v1" created
service "ratings" created
deployment.extensions "ratings-v1" created
service "reviews" created
deployment.extensions "reviews-v1" created
deployment.extensions "reviews-v2" created
deployment.extensions "reviews-v3" created
service "productpage" created
deployment.extensions "productpage-v1" created

$ kubectl get po
NAME                            READY     STATUS    RESTARTS   AGE
details-v1-876bf485f-2d4sr      2/2       Running   0          16s
productpage-v1-8d69b45c-p7kmn   2/2       Running   0          15s
ratings-v1-7c9949d479-k55cb     2/2       Running   0          15s
reviews-v1-85b7d84c56-xgl8m     2/2       Running   0          15s
reviews-v2-cbd94c99b-79lx2      2/2       Running   0          15s
reviews-v3-748456d47b-kplrb     2/2       Running   0          15s

$ kubectl get svc
NAME          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
details       ClusterIP   10.30.224.45   <none>        9080/TCP   20s
productpage   ClusterIP   10.30.111.5    <none>        9080/TCP   19s
ratings       ClusterIP   10.30.50.27    <none>        9080/TCP   20s
reviews       ClusterIP   10.30.93.28    <none>        9080/TCP   19s

Finally I deploy Gateway and VirtualService:

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway

metadata:
  namespace: bookinfo
  name: bookinfo-gw

spec:

  selector:
    istio: ingressgateway

  servers:

  - port:
      number: 80
      protocol: HTTP
      name: http
    hosts:
    - bookinfo.example.com
---


---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService

metadata:
  namespace: bookinfo
  name: bookinfo-virt-svc

spec:

  gateways:
  - bookinfo-gw

  hosts:
  - "*"

  http:

  - route:
    - destination:
        host: productpage.bookinfo.svc.cluster.local
        port:
          number: 9080
---


$ kubectl create -f istio-test.yaml 
gateway.networking.istio.io "bookinfo-gw" created
virtualservice.networking.istio.io "bookinfo-virt-svc" created

So currently VirtualService.spec.hosts == "*". I test the connectivity:

$ curl -LsD- -o /dev/null bookinfo.example.com:31380
HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
content-length: 1836
server: envoy
date: Mon, 18 Feb 2019 17:19:32 GMT
x-envoy-upstream-service-time: 25

I have received http 200, so it seems that life is good.

Now I will change VirtualService.spec.hosts to bookinfo.example.com and deploy it:

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService

metadata:
  namespace: bookinfo
  name: bookinfo-virt-svc

spec:

  gateways:
  - bookinfo-gw

  hosts:
  - bookinfo.example.com

  http:

  - route:
    - destination:
        host: productpage.bookinfo.svc.cluster.local
        port:
          number: 9080
---

$ kubectl replace -f istio-test.yaml 
virtualservice.networking.istio.io "bookinfo-virt-svc" replaced

Test the connectivity again:

$ curl -LsD- -o /dev/null bookinfo.example.com:31380
HTTP/1.1 404 Not Found
date: Mon, 18 Feb 2019 17:23:23 GMT
server: envoy
content-length: 0

You can see that it doesn’t work anymore… :confused:

Am I doing something wrong? Can someone reproduce it?

Below is the routes config which I get when I set VirtualService.spec.hosts = bookinfo.example.com:

$ istioctl proxy-config routes istio-ingressgateway-78c6d8b8d7-pdtb8 -n istio-system -o json
[
    {
        "name": "http.80",
        "virtualHosts": [
            {
                "name": "bookinfo.example.com:80",
                "domains": [
                    "bookinfo.example.com",
                    "bookinfo.example.com:80"
                ],
                "routes": [
                    {
                        "match": {
                            "prefix": "/"
                        },
                        "route": {
                            "cluster": "outbound|9080||productpage.bookinfo.svc.cluster.local",
                            "timeout": "0.000s",
                            "maxGrpcTimeout": "0.000s"
                        },
                        "decorator": {
                            "operation": "productpage.bookinfo.svc.cluster.local:9080/*"
                        },
                        "perFilterConfig": {
                            "mixer": {
                                "forward_attributes": {
                                    "attributes": {
                                        "destination.service": {
                                            "string_value": "productpage.bookinfo.svc.cluster.local"
                                        },
                                        "destination.service.host": {
                                            "string_value": "productpage.bookinfo.svc.cluster.local"
                                        },
                                        "destination.service.name": {
                                            "string_value": "productpage"
                                        },
                                        "destination.service.namespace": {
                                            "string_value": "bookinfo"
                                        },
                                        "destination.service.uid": {
                                            "string_value": "istio://bookinfo/services/productpage"
                                        }
                                    }
                                },
                                "mixer_attributes": {
                                    "attributes": {
                                        "destination.service": {
                                            "string_value": "productpage.bookinfo.svc.cluster.local"
                                        },
                                        "destination.service.host": {
                                            "string_value": "productpage.bookinfo.svc.cluster.local"
                                        },
                                        "destination.service.name": {
                                            "string_value": "productpage"
                                        },
                                        "destination.service.namespace": {
                                            "string_value": "bookinfo"
                                        },
                                        "destination.service.uid": {
                                            "string_value": "istio://bookinfo/services/productpage"
                                        }
                                    }
                                }
                            }
                        }
                    }
                ]
            }
        ],
        "validateClusters": false
    },
    {
        "virtualHosts": [
            {
                "name": "backend",
                "domains": [
                    "*"
                ],
                "routes": [
                    {
                        "match": {
                            "prefix": "/stats/prometheus"
                        },
                        "route": {
                            "cluster": "prometheus_stats"
                        }
                    }
                ]
            }
        ]
    }
]

I’m also getting nowhere, if I’m testing from inside the cluster:

# curl -v -H 'Host: bookinfo.example.com' istio-ingressgateway.istio-system.svc.cluster.local
* Rebuilt URL to: istio-ingressgateway.istio-system.svc.cluster.local/
* Hostname was NOT found in DNS cache
*   Trying 10.30.47.21...
* Connected to istio-ingressgateway.istio-system.svc.cluster.local (10.30.47.21) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Accept: */*
> Host: bookinfo.example.com
> 
< HTTP/1.1 404 Not Found
< date: Mon, 18 Feb 2019 18:37:07 GMT
* Server envoy is not blacklisted
< server: envoy
< content-length: 0
< 
* Connection #0 to host istio-ingressgateway.istio-system.svc.cluster.local left intact

#6

See https://github.com/istio/istio/issues/11828


#7

@ed.snible I know, I myself opened that ticket.