How to debug service mesh mTLS settings with PeerAuthentication and DestinationRule applied

1

Hello. My team has three different environments which all have their own service mesh. We have one service which is bound to port 443 which we do not want to run Istio mTLS on (long story but it performs mTLS at the application level). Let’s say this service is named target.

We have defined the following PeerAuthentication rule and DestinationRule.

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  labels:
    ...
  name: ...
  namespace: ...
spec:
  portLevelMtls:
    "443":
      mode: DISABLE
  selector:
    matchLabels:
      ...

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  labels:
    ...
  name: ...
  namespace: ...
spec:
  host: target.<namespace>.svc.cluster.local
  trafficPolicy:
    portLevelSettings:
    - port:
        number: 443
      tls:
        mode: DISABLE

This has been working on all of our environments until recently. This no longer seems to work on our development environment and I am trying to understand why. I have used openssl to debug the issue and I have confirmed that the PeerAuthentication rule is not in effect since the client does not try to perform mTLS. When I try to reach the target service from the client, I will see the Istio signed certificate being presented by the target service.

In previous versions of Istio there was a useful istioctl command to check the effective TLS mode between client and server but that appears to be gone. I have looked into the proxy config as well but the output is enormous and difficult to grok.

What would be the best way to debug why this PeerAuthentication rule doesn’t seem to be working?

2

Hi,
Have you tried Kiali for security validation ?

Kiali offers validation at Graph level and also at detail level inside the Istio configuration, also looking if the all objects defined are synced.

https://kiali.io/documentation/latest/validations/

In any case, it’s an area of active development and we are eager to learn and understand more usecases that Kiali can improve.

cc @xeviknal

3

Hi Lucas. We haved used Kialia in the past (1.2.2 days) but don’t have it installed yet on our new kubernetes clusters. Thanks for sharing the link to the validations! That looks like something that would be really useful to us. I’ll check it out.

To respond to my own question… After several days of debugging we discovered that two PeerAuthentication rules (with different mTLS ports) were targeting the same workload. Apparently only one PeerAuthentication can target a workload so only one of the PeerAuthentication rules was effective.

It was quite difficult to figure out what was going on. We had to use the istioctl to look into the proxy configs and try and understand what would indicate a port enforcing Istio mTLS or not. This is the command that helped us understand what ports weren’t enforcing Istio mTLS.

istioctl proxy-config listener <pod name> -o json | jq '.[]|select(.trafficDirection != "OUTBOUND")|.filterChains[]|select(.transportSocket == null and .name != null)|.name'

4

Thanks @Sovietaced !
Kiali is currently working to improve the visualization of Sidecar proxy info.

If you navigate to the “Workload Details” page, you’ll see a “Show Envoy Config” action where the listeners are also listed both in raw config and also in a table format for easy consuption.

This feedback is quite useful to understand the scenario you had, we’ll work to see what is the best way to add it into that feature and the global validation scenario what Kiali offers.

Thanks again for taking the time to report it here !