Mutual TLS not working from istio-ingress-gateway to workloads in another namespace when enabling strict mode

1

Versions
AWS EKS version: 1.23

istioctl version
client version: 1.10.3
control plane version: 1.16.1
data plane version: 1.14.6

Issue
Mutual TLS connection between istio-ingress-gateway and workloads in other namespaces is not working

Description
We have workloads running in and EKS cluster in a namespace that is called transactions. On the other hand we have istio-ingress-gateway running in istio-system namespace. Traffic entering the cluster comes from an AWS ALB. The traffic path is follows:

client -> Cloudfront -> ALB -> istio-ingress-gateway (istio-system namespace) -> workload (transactions namespace)

In transactions namespace, we have a peerAuthentication policy to enable mTLS with permissive mode:

apiVersion: security.istio.io/v1beta1
  kind: PeerAuthentication
  metadata:
    name: default
  spec:
    mtls: {}

We also have a destinationrule resource to configure outbound connection to use mTLS

kind: DestinationRule
metadata:
  name: istio-mutual
  namespace: transactions
spec:
  host: '*.transactions.svc.cluster.local'
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

In istio-system namespace, istio-ingress-gateway is configured with TLS termination for inbound traffic coming from the ALB. This is configured with the following Gateway resource:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: https-port
      number: 443
      protocol: HTTPS
    tls:
      credentialName: credential
      minProtocolVersion: TLSV1_2
      mode: SIMPLE

Outbound traffic from istio-ingress-gateway is configured with a destinationrule to use mTLS

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: istio-mutual
  namespace: istio-system
spec:
  host: '*.svc.cluster.local'
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

When we want to enable strict mTLS in transactions namespace by updating the peerAuthentication policy. However, this breaks connections from the istio-ingress-gateway to the workloads in transactions namespace. This the behavior we observe:

  • We update the peerAuthentication to strict mode
  • Traffic still works fine between workloads in the transactions namespace
  • Traffic from istio-ingress-gateway to the workloads in transactions namespace no longer works. Curl commands return the error: curl: (56) Recv failure: Connection reset by peer

Our expectation here is that istio-ingress-gateway connects to workloads in the other namespace with mTLS, however this isn’t happening. Any idea what is the issue here?

1 Like
2

+1

I am facing with the same issue right now

@Hamza_Boulaares were you able to resolve this issue?

1 Like
3

@Jan_Michalik Unfortunately not, I am still facing the same issue and had to revert the peerAuthentication policy back to permissive mTLS.

4

@Hamza_Boulaares I bumped the istio version from 1.18.1 to 1.19.1 and the issue seems to be gone.
Istio ingress gateway is now able to communicate with istio sidecars using STRICT mTLS mode.

I bet curl command is not good example here, as this command should fail and it is expected with STRICT mTLS mode as plaintext is not allowed.

istioctl version:
client version: 1.19.1
control plane version: 1.19.1
data plane version: 1.19.1