mTLS-secured HTTP: X-Forwarded-Proto header is `http`

Security
1

I’m currently investigating Istio as a service mesh solution for our data processing stack. A part of what I’m trying to solve is that our datalake service, Trino, only allows authentication when connected to via HTTPS. Istio’s mTLS would enable us to get other services within the cluster to automatically trust Trino without having to roll out our own CA certificate manually to every client.

The only thing that would be required for Trino to accept that it’s being served via an encrypted connection is that the downstream passes on an appropriate Forwarded header, such as X-Forwarded-Proto. Istio’s Envoy sidecar does that already, however in my testing with a simple echo server and cURL from another container with a sidecar, I only ever received the http value for the X-Forwarded-Proto header.

I was able to confirm that mTLS is working at all since the server also gets the X-Forwarded-Client-Cert header in the request. Am I simply wrong in my understanding of how the proxy is supposed to work? Since it’s terminating the TLS connection and understands that it’s HTTP traffic, I assumed it would also pass on https as the forwarded protocol.

The config for my test server looks like this, deployed on my local minikube instance running Istio v1.11.4 installed with the “minimal” profile:

apiVersion: v1
kind: Namespace
metadata:
  name: istio-test
---
apiVersion: v1
kind: Service
metadata:
  name: echo-server
  namespace: istio-test
  labels:
    app: echo-server
spec:
  type: ClusterIP
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
  selector:
    app: echo-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-server
  namespace: istio-test
spec:
  selector:
    matchLabels:
      app: echo-server
  replicas: 1
  template:
    metadata:
      labels:
        app: echo-server
        sidecar.istio.io/inject: "true"
    spec:
      containers:
      - name: echo-server
        image: gcr.io/google-containers/echoserver:1.10
        ports:
        - containerPort: 8080

No matter which proxied pod I send a request (e.g. curl http://echo-server.istio-test.svc.cluster.local:8080/) from, I only get ever see x-forwarded-proto=http in the eched request.

Am I missing some obvious configuration option or are my assumptions fundamentally wrong? Any help would be appreciated!

2

Assuming that your Trino service is front-ended by the Istio-proxy (envoy), since your curl is an HTTP, why would you expect HTTPS to be passed as the X-Forwarded-Proto? Istio’s HTTPS encryption is not something we want applications to be aware of AFAIK.

Also, given that Istio already handles authentication (at the TLS level) between services, what additional authentication logic do you expect the Trino service to handle?

3

Thanks for your response!

I was sort of assuming Envoy would act as a “regular” TLS termination proxy here in that it indicates to the server that the connection was indeed encrypted, even if that happened transparently. If that is not a goal of Istio though, that does make sense. I honestly have a hard time finding where the X-Forwarded-Proto comes from at all. If it’s already added on the outgoing request from the client-side proxy, I can see why it’s http all the way through.

We not only need service-level authentication, but different users (going through the same service) will also need to be distinguished and authenticated on Trino’s end to enable more granular authorization, so we need access to Trino’s own authentication system.

So if it’s not intended by Istio that the application knows that encryption is happening, but I do need it for Trino to accept credentials and know all incoming connections are secured by mTLS, would you say it’s safe to just force-set the X-Forwarded-Proto header ot HTTPS via a custom EnvoyFilter?

4

Yes. That is one option. Changing the x-forwarded-proto header using a custom Envoy filter to “HTTPS”.

However, is it possible to change/enhance Trino with a configuration that says it is being used under “Mesh” and hence skip x-forwarded-proto check to Http but still get xfcc header and validate client?

We (Salesforce) also have a similar need and we are moving forward with Envoy filter for now. But I am wondering if Trino is used more and more in Mesh style envs (Behind Istio proxy) and clients use Mesh it would be worthwhile to enable that option in Trino so that custom envoy filters are not needed?