Hallo
I want to route traffic to services outside of Kubernetes over the Egress Gateway.
I don’t want to trust our internal Root Certificates on all pods so I’m asking if its possible to add/trust them on the Egress Gateway so that I call them without trusting them on each Pod.
I am not sure I completely understand the question, but my hunch is that you are looking to set up TLS origination at the Egress Gateway. See https://istio.io/docs/tasks/traffic-management/egress/egress-gateway-tls-origination/ for instructions on how to do this.
In addition you can set the DestinationRule.TLSMode to SIMPLE and provide the CA certificate to be used by the Gateway to validate the external server’s certificate using DestinationRule.TLSSettings.caCertificate.
DISCLAIMER: I haven’t tried this personally but I suppose it should work.
The TLSSettings.caCertificates documentation states,
caCertificates string
OPTIONAL: The path to the file containing certificate authority certificates to use in verifying a presented server certificate. If omitted, the proxy will not verify the server’s certificate. Should be empty if mode is ISTIO_MUTUAL.
My understanding is that, if you don’t specify the caCertificates value the Gateway will skip validation of Server certificates, while if you do it will validate it based on the ca-certs you have uploaded.
Is my understanding accurate @Tao_Li ?
@nitishm you are right. The only issue is that you have to deploy the egress gateway in a separate namespace. See https://github.com/istio/istio/issues/15517.
Thank you, it works. 
I’ve a bit strange behavior, the connection was working but now I get a 503.
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: myservice-ext
namespace: mynamespace
spec:
hosts:
- myservice.domain.local
ports:
- number: 80
name: http
protocol: HTTP
- number: 443
name: https
protocol: HTTPS
resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-egressgateway-myservice
namespace: mynamespace
spec:
selector:
istio: egressgateway
servers:
- port:
number: 80
name: https
protocol: HTTPS
hosts:
- myservice.domain.local
tls:
mode: MUTUAL
serverCertificate: /etc/certs/cert-chain.pem
privateKey: /etc/certs/key.pem
caCertificates: /etc/certs/root-cert.pem
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: egressgateway-for-myservice
namespace: mynamespace
spec:
host: istio-egressgateway.istio-system.svc.cluster.local
subsets:
- name: myservice
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
portLevelSettings:
- port:
number: 80
tls:
mode: ISTIO_MUTUAL
sni: myservice.domain.local
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: direct-myservice-through-egress-gateway
namespace: mynamespace
spec:
hosts:
- myservice.domain.local
gateways:
- istio-egressgateway-myservice
- mesh
http:
- match:
- gateways:
- mesh
port: 80
route:
- destination:
host: istio-egressgateway.istio-system.svc.cluster.local
subset: myservice
port:
number: 80
weight: 100
- match:
- gateways:
- istio-egressgateway-myservice
port: 80
route:
- destination:
host: myservice.domain.local
port:
number: 443
weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: originate-tls-for-myservice
namespace: mynamespace
spec:
host: myservice.domain.local
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
portLevelSettings:
- port:
number: 443
tls:
mode: SIMPLE
caCertificates: /etc/certs/myrootcert.pem
In the proxy logs I see:
gRPC config for type.googleapis.com/envoy.api.v2.Cluster rejected: Error adding/updating cluster(s) outbound|443||myservice.domain.local: Invalid path: /etc/certs/myrootcert.pem
The certificate is mounted on the egress gateway pods.
Any hints whats wrong in my config?
So you had this working with the same caCertificate mounted and it randomly stopped working ? Could you make it work again with a clean setup ?
Yes it was working with the same config.
I think it has nothing to do with the ca certificate, its still working with another service.
I’ve opened a new Topic -> https://discuss.istio.io/t/upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-terminatio/
@nicolasmi I forgot to mention, sorry - you have to use exportTo: '.' in your destination rule, so the destination rule will be applied in the egress gateway’s namespace only and the certificate will be loaded in that namespace only (for the egress gateway only).
Still getting a “503 Service Unavailable” and “upstream connect error or disconnect/reset before headers. reset reason: connection Termination”
I can access my service from the istio-proxy container in the same pod.
Access from my app container in the same pod doesn’t work.
Really don’t understand that, also that its working for other services.
How can I debug the 503?
Is it possbile that there’s an issue with the certificate used on the service?
I have two services and they run on the same webserver and also share the same certificate.
The CN matches for the fqdn of service that is working, the fqdn of the other service is only a subject alternative name.
The only difference between these services is that they have different fqdn and this certificate thing…
Can you provide the cert detail? Not sure what do you mean “fqdn of other service is only a subject alt name” and why do we have the difference of the two services?
The certificate has:
CN = my-service.domain.local
and
SAN =
DNS Name=my-service.domain.local
DNS Name=my-service2.domain.local
Found out that on my-service2.domain.local I get HTTP_1_1_REQUIRED when I try to access it with curl.
On my-service.domain.local I can use HTTP/2.
Maybe thats the issue. Can I force the Egress Gateway to use HTTP/1.1?
@vadimeisenbergibm, any ideas?
@Tao_Li @nicolasmai I will have to check the issue, will take me some time.
I have some more details.
The services are running on a IIS Webserver (Windows Server 2016).
When I set the SSL option “Client Certificate” to accept or required I get:
“503 Service Unavailable” and “upstream connect error or disconnect/reset before headers. reset reason: connection Termination”
When I set the SSL option “Client Certificate” to ignore it works.
In the docs from Microsoft they state:
In a few cases, HTTP/2 can’t be used in combination with other features. In these situations, Windows will fall back to HTTP/1.1 and continue the transaction. This may involve negotiating HTTP/1.1 during the handshake, or sending an error code to the client instructing it to retry over an HTTP/1.1 connection.
Looks like the “negotiating HTTP/1.1 during the handshake, or sending an error code to the client instructing it to retry over an HTTP/1.1 connection” does not work with Istio.
I have opened an issue https://github.com/istio/istio/issues/15785
Thank you @vadimeisenbergibm