Our application requires the egress use case below. We have not been able to achieve a valid Istio configuration to implement it, we would like to get ideas / help from the community.
Our application needs to communicate with entities outside of the K8s cluster. Some of these entities are known at deployment time, and are expected to remain unchanged for a long time. For this kind of entities, it is acceptable to configure ServiceEntry definitions with the connection details.
But most of the external communication happens in a dynamic fashion, that is, the endpoints will be determined by runtime information provided to the application. We use HTTP / HTTPS traffic. In this case, using ServiceEntries introduces the following problems:
- requires the application to dynamically configure a ServiceEntry entry for every endpoint before it attempts to communicate with it.
- needs to wait until the ServiceEntry definition gets propagated and applied to all sidecars in the mesh before sending traffic, or it will be blocked
- it may end up creating a potentially very large set of ServiceEntries, with the scaling implications it may have.
- connected to the previous item, it requires the application to remove no longer used ServiceEntries for housekeeping reasons.
Bypassing the sidecar by setting global.proxy.includeIPRanges is not an option for us, as we would like to leverage Istio retries, timeouts, circuit breaker and (m)TLS features for the external traffic also, and for that you need the traffic to go through the sidecar.
Ideally, what we would need is a default ServiceEntry configuration for outgoing connections not matched by existing, more restrictive ServiceEntries, that would apply to all external traffic, and would allow us to set retries, timeouts, and (m)TLS settings as mentioned. It is OK to have the same retries, timeout, and (m)TLS config for all of the endpoints.
We already know that wildcards (*) cannot be used in hosts definitions. We have tried to use subdomains (e.g. *.org) and it seems to work. However this is suboptimal, as you would still need to define ServiceEntries for at least all the top-level domains.
Also, some endpoints are not registered in DNS, so they use IP addresses instead of FQDNs. You would need to define IP addreses one by one, as CIDRs are not supported.
And finally, for most of these endpoints, communication will not happen over standard HTTP and HTTPs ports (80, 443) but using custom ports. As ports attributes are mandatory in ServiceEntries, this makes very challenging to create ServiceEntry definitions to match all the different port numbers.
So, how could we best achieve the above with the current egress traffic management of Istio?
We believe this use case may be applicable to other applications also.
Thanks in advance for your help. Cheers!