404 thought protocol HTTP but not HTTP2

Networking
1

Hello!

I am getting a weird issue using HTTP and HTTP2, the thing I want to use HTTP, but i get a 404 on a istio 1.4.2:

kind: Gateway
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: xxxxxxx
  namespace: xxxxxxx
  selfLink: >-
    /apis/networking.istio.io/v1alpha3/namespaces/xxxxxxx/gateways/xxxxxxx
  uid: 69f6fdb7-1c1a-11ea-9e57-4201ac100205
  resourceVersion: '3530831'
  generation: 12
  creationTimestamp: '2019-12-11T13:30:36Z'
  labels:
    app: xxxxxxx
    chart: app
    environment: production
    heritage: Tiller
    release: rails-dashboard
spec:
  servers:
    - hosts:
        - xxxxxxx.api.xxxxxxx.co
      port:
        name: http
        number: 80
        protocol: HTTP
    - hosts:
        - xxxxxxx.api.xxxxxxx.co
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        credentialName: xxxxxxx-cert
        mode: SIMPLE
        privateKey: sds
        serverCertificate: sds
  selector:
    istio: ingressgateway

But if i change the protocol to use HTTP2, I get correctly the status 200.

Thats its not happening in another environment with istio 1.4.1.

Maybe a bug?

Thanks!

2

Just recently upgraded to 1.4.2 the other enviroment, and no problem here. Dont know why do not work, still trying to fix it.

3

Probably You have some misconfiguration with VirtualService attached to this Gateway

4

Both have same configuration files, only changing some env vars.

I just get telemetry to see the requests:

This one is HTTP2 protocol:

{"level":"info","time":"2019-12-16T20:32:46.914331Z","instance":"newlog.instance.istio-system","destinationIp":"10.28.6.125","latency":"9.039995ms","method":"GET","protocol":"http","requestSize":0,"responseCode":200,"responseSize":0,"sourceIp":"10.28.7.30","sourceUser":"cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account","url":"/healthz?bla"}
{"level":"info","time":"2019-12-16T20:32:46.913771Z","instance":"newlog.instance.istio-system","destinationIp":"10.28.6.125","latency":"10.071464ms","method":"GET","protocol":"http","requestSize":0,"responseCode":200,"responseSize":0,"sourceIp":"10.28.7.30","sourceUser":"","url":"/healthz?bla"}

This one is HTTP protocol:

{"level":"info","time":"2019-12-16T20:33:10.965161Z","instance":"newlog.instance.istio-system","destinationIp":"0.0.0.0","latency":"120.961µs","method":"GET","protocol":"http","requestSize":0,"responseCode":404,"responseSize":0,"sourceIp":"10.28.7.30","sourceUser":"","url":"/healthz?bla"}

The difference are, 2 requests on the HTTP2 and one only on HTTP, also the destination_ip on the HTTP one its 0.0.0.0

5

Did you change the port name from http to http2? Could you provide a config dump when using http?

6

Look at istio-ingress configmap. By default if I remember right it forwards 80 port to http2

  • port: 80
    targetPort: 80
    name: http2
7

This is the Gateway:

kind: Gateway
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: rails-dashboard-api-xxxxxx
  namespace: rails-dashboard
  selfLink: >-
    /apis/networking.istio.io/v1alpha3/namespaces/rails-dashboard/gateways/rails-dashboard-api-xxxxxx
  uid: 31f04573-2046-11ea-8a2b-4201ac100207
  resourceVersion: '3636775'
  generation: 7
  creationTimestamp: '2019-12-16T20:54:04Z'
  labels:
    app: rails-dashboard-api
    chart: api
    environment: production
    heritage: Tiller
    release: rails-dashboard
spec:
  servers:
    - hosts:
        - xxxxxx.api.xxxxxx.co
      port:
        name: http
        number: 80
        protocol: HTTP
      tls:
        httpsRedirect: false
    - hosts:
        - xxxxxx.api.xxxxxx.co
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        credentialName: xxxxxx-cert
        mode: SIMPLE
        privateKey: sds
        serverCertificate: sds
  selector:
    istio: ingressgateway

The Virtual Service:

kind: VirtualService
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: rails-dashboard-api
  namespace: rails-dashboard
  selfLink: >-
    /apis/networking.istio.io/v1alpha3/namespaces/rails-dashboard/virtualservices/rails-dashboard-api
  uid: 225bd847-2034-11ea-8a2b-4201ac100207
  resourceVersion: '3579102'
  generation: 1
  creationTimestamp: '2019-12-16T18:44:47Z'
  labels:
    app: rails-dashboard-api
    chart: api
    environment: production
    heritage: Tiller
    release: rails-dashboard
spec:
  hosts:
    - api.company.co
    - sub1.api.company.co
  gateways:
    - rails-dashboard-api-cloudflare
    - rails-dashboard-api-sub1
  http:
    - match:
        - uri:
            prefix: /api/public/v2/some_endpoint
      retries:
        attempts: 3
        perTryTimeout: 2s
      route:
        - destination:
            host: rails-dashboard-api.rails-dashboard.svc.cluster.local
            port:
              number: 8080
            subnet: production
          headers:
            response:
              add:
                x-app-dedicated: api
    - match:
        - uri:
            prefix: /api/public/v2/other1
      retries:
        attempts: 3
        perTryTimeout: 2s
      route:
        - destination:
            host: rails-dashboard-other1.rails-dashboard.svc.cluster.local
            port:
              number: 8080
            subnet: production
          headers:
            response:
              add:
                x-app-dedicated: other1
    - match:
        - uri:
            exact: /api/public/v2/orders/some_endpoint4
        - uri:
            exact: /api/v1/users/some_endpoint5
        - uri:
            exact: /api/v1/orders/some_endpoint6
      retries:
        attempts: 3
        perTryTimeout: 2s
      route:
        - destination:
            host: some-service-app.some-service.svc.cluster.local
            port:
              number: 8080
            subnet: production
          headers:
            response:
              add:
                x-app-dedicated: some-service
    - match:
        - uri:
            prefix: /api/public/v2
      retries:
        attempts: 3
        perTryTimeout: 2s
      route:
        - destination:
            host: rails-api-service-app.rails-api-service.svc.cluster.local
            port:
              number: 8080
            subnet: production
          headers:
            response:
              add:
                x-app-dedicated: api-service
    - match:
        - uri:
            prefix: /api/v2/one
        - uri:
            prefix: /api/v1/two/some_endpoint2
        - uri:
            prefix: /api/v1/three/some_endpoint3
      retries:
        attempts: 3
        perTryTimeout: 2s
      route:
        - destination:
            host: elixir-api-service-app.elixir-api-service.svc.cluster.local
            port:
              number: 8080
            subnet: production
          headers:
            response:
              add:
                x-app-dedicated: elixir-api-service
    - match:
        - uri:
            prefix: /api/public/v1/other2
        - uri:
            prefix: /admin/orders/build_label
      retries:
        attempts: 3
        perTryTimeout: 2s
      route:
        - destination:
            host: rails-dashboard-other2.rails-dashboard.svc.cluster.local
            port:
              number: 8080
            subnet: production
          headers:
            response:
              add:
                x-app-dedicated: other2
    - match:
        - uri:
            prefix: /api
      retries:
        attempts: 3
        perTryTimeout: 2s
      route:
        - destination:
            host: rails-dashboard-api.rails-dashboard.svc.cluster.local
            port:
              number: 8080
            subnet: production
          headers:
            response:
              add:
                x-app-dedicated: api
    - match:
        - uri:
            prefix: /
      retries:
        attempts: 3
        perTryTimeout: 2s
      route:
        - destination:
            host: rails-dashboard-app.rails-dashboard.svc.cluster.local
            port:
              number: 8080
            subnet: production
          headers:
            response:
              add:
                x-app-dedicated: app
8

The most weird thing its in the other environments works in the other way, http its 200 and http2 its 404, im getting mad :sob:

9

Check port definitions in svc and pod

10

the service is:

ports:
  - name: http
    nodePort: 31152
    port: 8080
    protocol: TCP
    targetPort: 80

And the pod:

ports:
    - containerPort: 80
      name: tcp
      protocol: TCP

Should be http? :thinking:

11

In pod better name it http, can be with suffix http-something.
But for what svc is this snippet? Is it svc bounded to this pod?

12

yes are bounced to the pod, HTTPS and HTTP2 both works properly, the weird things its in the other environment works in the other way, HTTPS and HTTP.

Why only changing to HTTP2 works, not seems necessary to do extra config between both protocols.

13

I just worried about nodeport used in svc. It is uncommon to use both ingress and nodeport.
BTW, what I see from information You provided, definitely two environments are have different configuration for ingress-gateway itself.
Compare istio-ingressgateway deployment definitions for ports section.

14

This http and http2 mismatch happens on ingress side

15

Thanks ill take a look on that

16

I did the change to ClusterIP, no difference. That ingress i shared are on the same environment, both have the same issue.

I hope i solve that soon :slight_smile:

17

Seriously, get describe for istio-ingressgateway svc

18

That?

Name:                     istio-ingressgateway
Namespace:                istio-system
Labels:                   app=istio-ingressgateway
                          chart=gateways
                          heritage=Tiller
                          istio=ingressgateway
                          release=istio
Annotations:              <none>
Selector:                 app=istio-ingressgateway,istio=ingressgateway,release=istio
Type:                     LoadBalancer
IP:                       10.0.43.189
IP:                       xx.xxx.xxx.xxx
LoadBalancer Ingress:     xx.xxx.xxx.xxx
Port:                     status-port  15020/TCP
TargetPort:               15020/TCP
NodePort:                 status-port  30920/TCP
Endpoints:                10.44.3.148:15020
Port:                     http2  80/TCP
TargetPort:               80/TCP
NodePort:                 http2  31380/TCP
Endpoints:                10.44.3.148:80
Port:                     https  443/TCP
TargetPort:               443/TCP
NodePort:                 https  31390/TCP
Endpoints:                10.44.3.148:443
Port:                     tcp  31400/TCP
TargetPort:               31400/TCP
NodePort:                 tcp  31400/TCP
Endpoints:                10.44.3.148:31400
Port:                     https-kiali  15029/TCP
TargetPort:               15029/TCP
NodePort:                 https-kiali  31092/TCP
Endpoints:                10.44.3.148:15029
Port:                     https-prometheus  15030/TCP
TargetPort:               15030/TCP
NodePort:                 https-prometheus  31829/TCP
Endpoints:                10.44.3.148:15030
Port:                     https-grafana  15031/TCP
TargetPort:               15031/TCP
NodePort:                 https-grafana  30142/TCP
Endpoints:                10.44.3.148:15031
Port:                     https-tracing  15032/TCP
TargetPort:               15032/TCP
NodePort:                 https-tracing  31245/TCP
Endpoints:                10.44.3.148:15032
Port:                     tls  15443/TCP
TargetPort:               15443/TCP
NodePort:                 tls  30866/TCP
Endpoints:                10.44.3.148:15443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

I see that http2 to 80.

19

Exactly, istio-proxy terminates http2 connection here, and proxy it to destination pod as http
It is because You see 2 telemetry entry in first case incoming&outgoing

20

Also there are Istio recommendations to place Gateway resource into the same ns where ingress-gateway located (istio-system), and VirtualService in destination ns