Objective
Support routing HTTP request to a specific backend endpoint through ingress gateway. In our use case, there are some existing tools that monitor k8s Pods. After moving k8s Pods into mesh and Istio mutual TLS being enabled, these tools can’t access Pod anymore and they can’t be run in mesh. The problem can be resolved by making these tools send requests to ingress gateway and routing the request to the specified endpoint.
Replacing current DestinationRule Subset implementation with Envoy subset load balancing is not the goal of this feature request.
Background
Currently Istio generates one Envoy cluster for each subset defined in DestinationRule, backend endpoints are grouped into subset by labels. For using subset, labels are defined in Deployment/Pod spec, subsets are defined in DestinationRule, and route rules are defined in VirtualService, basically everything is defined statically.
It’s not practical to use the DestinationRule subset to route requests to any single specified endpoint without updating DestinationRule/VirtualService spec. If a service has a lot of endpoints(e.g. >1000), the size of RDS/CDS/EDS config will increase dramatically.
Requirements
In our use case, client needs to send HTTP requests to a specific endpoint of a service through ingress gateway, the destination endpoint can be specified by a HTTP header. This feature can be enabled for specific virtual hosts or route, e.g. only allowed for host foo.example.com for URI path `/foo’. When a new Pod is created or deleted, there is no need to update DestinationRule/VirtualService for adding new subset/route rule or clean up old subset/route rule.
Design Ideas
Envoy’s Subset Load Balancing comes in handy for this use case, endpoint can be generated with metadata in namespace envoy.lb, then Envoy Header-To-metadata filter can be used to transform header into metadata in namespace envoy.lb. In this way, the destination endpoint can be selected through a header, no need to update DestinationRule/VirtualService.
In our use case, an endpoint’s IP address can be used as metadata to select the endpoint. For adding its IP address as a metadata, a replacement pattern %ENDPOINT_IP% can be used, the DestinationRule spec could be:
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
spec:
subsetLB: # Envoy subset LB config
metadata:
endpoint-ip: %ENDPOINT_IP% # in metadata namespace `envoy.lb`
fallbackPolicy: NO_FALLBACK
subsets:
- name: v1
subsetLB: # this can also be applied to DestinationRule Subset
metadata:
endpoint-ip: %ENDPOINT_IP%
fallbackPolicy: NO_FALLBACK
The generated endpoint config will be:
{
"endpoint": {
"address": {
"socket_address": {
"address": "192.168.0.100",
"port_value": 8080
}
}
},
"metadata": {
"filter_metadata": {
...
"envoy.lb": {
"endpoint-ip": "192.168.0.100"
}
}
},
...
}
The corresponding subset LB config in Envoy Cluster should be:
{
"cluster": {
...
"lb_subset_config": {
"subset_selectors": [
{
"keys": [
"endpoint-ip"
],
"single_host_per_subset": true
}
]
}
}
}
A sample VirtualService for adding metadata to HTTP requests match /foo could be:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
...
spec:
hosts:
- foo.example.com
http:
- match:
- uri:
prefix: "/foo"
headers:
request:
...
subsetLb: # transform header into metadata namespace `envoy.lb`
- header: destination-endpoint
on_header_present:# same as header_to_metadata KeyValuePair,
# but no metadata_namespac field, so user
# can’t add metadata to other namespace
key: endpoint-ip
type: STRING
remove: false
...
The generated Header-To-Metadata filter config will be :
route_config:
virtual_hosts:
- domains:
- "foo.example.com"
routes:
- match:
prefix: "/foo"
typed_per_filter_config:
envoy.filters.http.header_to_metadata:
"@type": "type.googleapis.com/envoy.extensions.filters.http.header_to_metadata.v3.Config"
request_rules:
- header: instance
on_header_present:
metadata_namespace: envoy.lb
key: instance
type: STRING
remove: false