Hi All,
I have a unique problem. I have a downstream system who would like to connect to my k8s cluster of course using Istio as the mesh. But this connection (TCP) has to be proxied untouched to a specific pod (which is dynamic per request) in the cluster. The reason being, this specific pod is connected via websocket to an upstream system. In essence, the downstream system should just proxy the connection via mesh, this pod to the upstream system.
From what I have researched, it seems under Network filters, TcpProxy along with the original destination cluster maybe the one to use.
Below are the configs I have tried with no luck.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: orig-dest-filter
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: NETWORK_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: envoy.filters.listener.original_dst
patch:
operation: INSERT_BEFORE
value:
name: "envoy.filters.network.tcp_proxy"
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
cluster: original_destination_cluster
- applyTo: HTTP_FILTER
match:
context: ANY
listener:
filterChain:
filter:
name: envoy.filters.network.http_connection_manager
subFilter:
name: envoy.filters.http.router
patch:
operation: MERGE
value:
name: "envoy.filters.network.http_connection_manager"
typed_config:
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
stat_prefix: http
route_config:
name: route_a
virtual_hosts:
- name: route_a
domains:
- "*"
routes:
- match:
prefix: /
route:
cluster: original_destination_cluster
- applyTo: CLUSTER
patch:
operation: ADD
value: # cluster specification
name: original_destination_cluster
type: ORIGINAL_DST
original_dst_lb_config:
use_http_header: true
connect_timeout: 6s
lb_policy: CLUSTER_PROVIDED
dns_lookup_family: V4_ONLY
Proxy config details, after creating Gateway object
hydraulics@hydraulics-virtual-machine:~$ istioctl proxy-config all istio-ingressgateway-77968dbd74-l2bft.istio-system
SERVICE FQDN PORT SUBSET DIRECTION TYPE DESTINATION RULE
BlackHoleCluster - - - STATIC
agent - - - STATIC
hello-world.test.svc.cluster.local 8080 - outbound EDS
istio-egressgateway.istio-system.svc.cluster.local 80 - outbound EDS
istio-egressgateway.istio-system.svc.cluster.local 443 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 80 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 443 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 15021 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 15443 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 31400 - outbound EDS
istiod.istio-system.svc.cluster.local 443 - outbound EDS
istiod.istio-system.svc.cluster.local 15010 - outbound EDS
istiod.istio-system.svc.cluster.local 15012 - outbound EDS
istiod.istio-system.svc.cluster.local 15014 - outbound EDS
kube-dns.kube-system.svc.cluster.local 53 - outbound EDS
kube-dns.kube-system.svc.cluster.local 9153 - outbound EDS
kubernetes.default.svc.cluster.local 443 - outbound EDS
metrics-server.kube-system.svc.cluster.local 443 - outbound EDS
original_destination_cluster - - - ORIGINAL_DST
prometheus_stats - - - STATIC
sds-grpc - - - STATIC
se-host 80 - outbound EDS
xds-grpc - - - STATIC
zipkin - - - STRICT_DNS
ADDRESS PORT MATCH DESTINATION
0.0.0.0 8080 ALL Route: http.8080
0.0.0.0 15021 ALL Inline Route: /healthz/ready*
0.0.0.0 15090 ALL Inline Route: /stats/prometheus*
NAME DOMAINS MATCH VIRTUAL SERVICE
http.8080 * /* 404
* /stats/prometheus*
* /healthz/ready*
Using a custom listener instead of using Gateway context. The Cluster and route seems to be created but the request just waits forever.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: orig-dest-filter
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: LISTENER
match:
context: ANY
patch:
operation: ADD
value:
name: proxy
# traffic_direction: INBOUND
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 8080
filter_chains:
- filters:
- name: "envoy.filters.network.http_connection_manager"
typed_config:
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
stat_prefix: ingress_proxy
route_config:
name: route_a
virtual_hosts:
- name: route_a
domains:
- "*"
routes:
- match:
prefix: "/abc"
route:
cluster: original_destination_cluster
- applyTo: CLUSTER
patch:
operation: ADD
value: # cluster specification
name: original_destination_cluster
type: ORIGINAL_DST
original_dst_lb_config:
use_http_header: true
connect_timeout: 6s
lb_policy: CLUSTER_PROVIDED
dns_lookup_family: V4_ONLY
proxy config details
$ istioctl proxy-config all istio-ingressgateway-77968dbd74-l2bft.istio-system
SERVICE FQDN PORT SUBSET DIRECTION TYPE DESTINATION RULE
BlackHoleCluster - - - STATIC
agent - - - STATIC
hello-world.test.svc.cluster.local 8080 - outbound EDS
istio-egressgateway.istio-system.svc.cluster.local 80 - outbound EDS
istio-egressgateway.istio-system.svc.cluster.local 443 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 80 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 443 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 15021 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 15443 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 31400 - outbound EDS
istiod.istio-system.svc.cluster.local 443 - outbound EDS
istiod.istio-system.svc.cluster.local 15010 - outbound EDS
istiod.istio-system.svc.cluster.local 15012 - outbound EDS
istiod.istio-system.svc.cluster.local 15014 - outbound EDS
kube-dns.kube-system.svc.cluster.local 53 - outbound EDS
kube-dns.kube-system.svc.cluster.local 9153 - outbound EDS
kubernetes.default.svc.cluster.local 443 - outbound EDS
metrics-server.kube-system.svc.cluster.local 443 - outbound EDS
original_destination_cluster - - - STATIC
prometheus_stats - - - STATIC
sds-grpc - - - STATIC
se-host 80 - outbound EDS
xds-grpc - - - STATIC
zipkin - - - STRICT_DNS
ADDRESS PORT MATCH DESTINATION
0.0.0.0 8080 ALL Inline Route: /abc*
0.0.0.0 15021 ALL Inline Route: /healthz/ready*
0.0.0.0 15090 ALL Inline Route: /stats/prometheus*
NAME DOMAINS MATCH VIRTUAL SERVICE
* /stats/prometheus*
* /healthz/ready*
route_a * /*
route_a * /abc*
Any direction is greatly appreciated. If my use case is something that can’t be solved using the above approach, please let me know.