Send-proxy-v2-ssl with engressgateway dont work

Networking
1

Hi,

I have tried to configure istio in a cluster that sent the requests from external services to several internal clusters with istio.

But by activating the proxy protocol configuration, I only manage to make the proxy work on the internal clusters in http but for https it doesn’t work.

The visual diagram is as follows.

image
image847×828 40.2 KB

The environment configuration is:

Edge cluster:

Kubernetes Version: v1.22.9
Nodes OS: Ubuntu 21.10 (GNU/Linux 5.13.0-52-generic x86_64)
Rancher: 2.6.12
Istio-Proxy: 1.15.3
Istiod: 1.15.3
Kiali: 1.59.0

GATEWAY - INGRESS

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: gateway-ingress
  namespace: namespace
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - service.domain.com
      port:
        name: http
        number: 80
        protocol: HTTP
      tls:
        httpsRedirect: false
    - hosts:
        - service.domain.com
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        mode: PASSTHROUGH

GATEWAY - ENGRESS

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: gateway-engress
  namespace: namespace
spec:
  selector:
    istio: egressgateway
  servers:
    - hosts:
        - service.domain.com
      port:
        name: tls
        number: 443
        protocol: TLS
      tls:
        mode: PASSTHROUGH
    - hosts:
        - service.domain.com
      port:
        name: http
        number: 80
        protocol: HTTP

VIRTUALSERVICE

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: service-vs
  namespace: namespace
spec:
  gateways:
    - mesh
    - gateway-ingress
    - gateway-engress
  hosts:
    - service.domain.com
  http:
    - match:
        - gateways:
            - mesh
            - gateway-ingress
          port: 80
      route:
        - destination:
            host: istio-egressgateway.istio-system.svc.cluster.local
            port:
              number: 80
          weight: 100
    - match:
        - gateways:
            - gateway-engress
          port: 80
      route:
        - destination:
            host: service.domain.com
            port:
              number: 80
          weight: 100
  tls:
    - match:
        - gateways:
            - mesh
            - gateway-ingress
          port: 443
          sniHosts:
            - service.domain.com
      route:
        - destination:
            host: istio-egressgateway.istio-system.svc.cluster.local
            port:
              number: 443
          weight: 100
    - match:
        - gateways:
            - gateway-engress
          port: 443
          sniHosts:
            - service.domain.com
      route:
        - destination:
            host: service.domain.com
            port:
              number: 443
          weight: 100

DESTINATIONRULE

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: destinationrule
  namespace: namespace
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  subsets:
    - labels:
        servicio: passthrough
      name: service

ENVOYFILTERS

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: xff-proxy
  namespace: istio-system
spec:
  configPatches:
    - applyTo: NETWORK_FILTER
      match:
        context: ANY
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
      patch:
        operation: MERGE
        value:
          typed_config:
            '@type': >-
              type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
            skip_xff_append: false
            use_remote_address: true
            xff_num_trusted_hops: 0
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: externalingress-proxy-protocol
  namespace: istio-system
spec:
  configPatches:
  - applyTo: LISTENER
    patch:
      operation: MERGE
      value:
        listener_filters:
        - name: envoy.filters.listener.proxy_protocol
        - name: envoy.filters.listener.tls_inspector
  workloadSelector:
    labels:
      istio: ingressgateway

Internal cluster:

Kubernetes Version: v1.22.10
Nodes OS: Ubuntu 22.04 LTS (GNU/Linux 5.15.0-43-generic x86_64)
Rancher: 2.6.12
Istio-Proxy: 1.15.3
Istiod: 1.15.3
Kiali: 1.59.0

GATEWAY-HTTP

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: service-gateway-letsencrypt-http
  namespace: namespace
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - service.domain.com
      port:
        name: http
        number: 80
        protocol: HTTP

GATEWAY-HTTPS

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: service-gateway-letsencrypt
  namespace: namespace
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - service.domain.com
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        credentialName: service-letsencrypt-certificate
        mode: SIMPLE

VIRTUALSERVICE - HTTP

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: service-virtualservice-http
  namespace: namespace
spec:
  gateways:
    - service-gateway-letsencrypt-http
  hosts:
    - service.domain.com
  http:
    - match:
        - uri:
            prefix: /
      route:
        - destination:
            host: echo-service
            port:
              number: 80

VIRTUALSERVICE-HTTPS

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: service-virtualservice-letsencrypt
  namespace: namespace
spec:
  gateways:
    - service-gateway-letsencrypt
  hosts:
    - service.domain.com
  http:
    - match:
        - uri:
            prefix: /
      route:
        - destination:
            host: echo-service
            port:
              number: 80

SERVICE

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/instance: echo
    app.kubernetes.io/name: echo
  name: echo-service
  namespace: namespace
spec:
  internalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 8080
  selector:
    app.kubernetes.io/instance: echo
    app.kubernetes.io/name: echo
  sessionAffinity: None
  type: ClusterIP

ENVOYFILTERS

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: proxy-protocol
  namespace: istio-system
spec:
  configPatches:
    - applyTo: LISTENER
      patch:
        operation: MERGE
        value:
          listener_filters:
            - name: proxy_protocol
              typed_config:
                '@type': >-
                  type.googleapis.com/envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol
                allow_requests_without_proxy_protocol: true
            - name: tls_inspector
              typed_config:
                '@type': >-
                  type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector
  workloadSelector:
    labels:
      istio: ingressgateway
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: xff-proxy
  namespace: istio-system
spec:
  configPatches:
    - applyTo: NETWORK_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
      patch:
        operation: MERGE
        value:
          name: envoy.filters.network.http_connection_manager
          typed_config:
            '@type': >-
              type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
            skip_xff_append: false
            use_remote_address: true
            xff_num_trusted_hops: 2
  workloadSelector:
    labels:
      istio: ingressgateway

If I configure a haproxy with send-proxy-v2-ssl substituting for the edge proxy, it works fine. So I understand that the problem is in some missing configuration in the istio edge proxy.

I have tried using the upstream configuration on the edge proxy to add the protocol headers for https, but it doesn’t quite work.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: proxy-protocol-v2
  namespace: istio-system
spec:
  configPatches:
  - applyTo: CLUSTER
    patch:
      operation: MERGE
      value:
        transport_socket:
          name: envoy.transport_sockets.upstream_proxy_protocol
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.transport_sockets.proxy_protocol.v3.ProxyProtocolUpstreamTransport
            config:
              version: V2
            transport_socket:
              name: "envoy.transport_sockets.raw_buffer"
  workloadSelector:
    labels:
      istio: ingressgateway

Any suggestions?

2

Hi team,

any update?

let me know if any other info is required.

3

Hi team,

any update?

4

Hi,

someone? any hint?

5

Hi,

someone? any hint?

6

Your gateway-ingress is configured with tls.mode: PASSTHROUGH. This means that Istio is not looking inside the encrypted traffic, it just forwards it. If it can’t look inside the encrypted traffic, it cannot process the PROXY protocol.

7

Hi,

In part that is not correct, the proxy protocol is enabled at the transport layer and it needs to be enabled in both proxies, both the one that sends and the one that listens (upstream, downstream), which is why EnvoyFilter is used to enable the protocol.

It is independent that the connection is encrypted, to see it more clearly, it is like the https sni, that although the connection is encrypted the proxies can see where they have to send the connection.

Exploring the PROXY Protocol - Benjamin Boudreau.