Routing seems broken when enabling mtls on namespace basis

Security
1

I have 2 namespaces, bob and bobby, each have their own admin and managed servers. Bobby namespace has a policy and destination rule

apiVersion: [authentication.istio.io/v1alpha1](http://authentication.istio.io/v1alpha1)
kind: Policy
metadata:
name: default
namespace: bobby
spec:
peers:
- mtls: {}apiVersion: [networking.istio.io/v1alpha3](http://networking.istio.io/v1alpha3)

kind: DestinationRule
metadata:
name: bobby-destination-rule
namespace: bobby
spec:
host: '*.bobby.svc.cluster.local'
trafficPolicy:
tls:
mode: ISTIO_MUTUAL

bob namespace has a policy and destination rule

apiVersion: [authentication.istio.io/v1alpha1](http://authentication.istio.io/v1alpha1)
kind: Policy
metadata:
name: default
namespace: bob
spec:
peers:
- mtls: {}apiVersion: [networking.istio.io/v1alpha3](http://networking.istio.io/v1alpha3)

kind: DestinationRule
metadata:
name: bob-destination-rule
namespace: bob
spec:
host: '*.bob.svc.cluster.local'
trafficPolicy:
tls:
mode: ISTIO_MUTUAL

The Admin and Managed server in bobby can communicate fine.
However the Managed Server in bob fails because it can’t contact its local Admin server. Looking at the istio proxy log for the Managed Server one see this which prolly explains why it can’t talk to the Admin server in bob but what the heck?

[2020-05-26T16:51:07.240Z] "- - -" 0 UF,URX "-" "-" 0 0 3 - "-" "-" "-" "-" "10.244.0.183:7001" outbound|7001||bobbys-front-end-admin-server.bobby.svc.cluster.local - 10.244.0.183:7001 10.244.0.187:42956 - -
[2020-05-26T16:51:07.252Z] "- - -" 0 UF,URX "-" "-" 0 0 3 - "-" "-" "-" "-" "10.244.0.183:7001" outbound|7001||bobbys-front-end-admin-server.bobby.svc.cluster.local - 10.244.0.183:7001 10.244.0.187:42962 - -

The IP is for the adminserver in bob namespace but the svc name, bobbys-front-end-admin-server.bobby.svc.cluster.local, is bobby namespace. Sometimes the failed managed server will be in the bobby namespace. When that happens, the servers in bob namespace will boot successfully.

[2020-05-26T20:26:26.352Z] "- - -" 0 UF,URX "-" "-" 0 0 4 - "-" "-" "-" "-" "10.244.0.84:7001" outbound|7001||bobs-bookstore-admin-server.bob.svc.cluster.local - 10.244.0.84:7001 10.244.0.86:51450 - -
[2020-05-26T20:26:26.369Z] "- - -" 0 UF,URX "-" "-" 0 0 4 - "-" "-" "-" "-" "10.244.0.84:7001" outbound|7001||bobs-bookstore-admin-server.bob.svc.cluster.local - 10.244.0.84:7001 10.244.0.86:51460 - -

The IP is the endpoint of the Admin Server in bobby namespace but the svc name, bobs-bookstore-admin-server.bob.svc.cluster.local, is in bob namespace.
This is Istio 1.4.6
Any ideas how to investigate?
Tnx

One other possibly important fact, calls that are failing are on a protocol unsupported by istio. All this works fine w/o policy and destination rule but I may not be clear on the expected behavior of an unsupported protocol in the security case

2

cc @incfly
Can you share detailed steps for reproducing the issue?

3

Hi,

I have a client/server pair of microservices that speak both http and tcp.
I set is deployed to namespace bobby and the other to namespace bob.
These work fine until I add namespace scoped policy and destination rules
An example of these config files per namespace are above.

When the policy and destination rule files above are applied to the booby and bob namespace, one of the clients fails to connect to its corrsponding server on tcp.

The following comes from the istio-proxy log of the failing client instance in the bob namespace

[2020-05-26T16:51:02.633Z] "POST /admin-server-deployer/DeploymentService HTTP/1.1" 200 - "-" "-" 1507 1 5 4 "-" "Java1.8.0_211" "4db4d71c-1a63-9a06-ab1d-f33d96a46f20" "bobs-bookstore-admin-server:7001" "10.244.0.183:7001" outbound|7001||bobs-bookstore-admin-server.bob.svc.cluster.local - 10.244.0.183:7001 10.244.0.187:42636 - default
[2020-05-26T16:51:02.886Z] "POST /admin-server-deployer/DeploymentService HTTP/1.1" 200 - "-" "-" 1666 1 5 4 "-" "Java1.8.0_211" "5c937442-67ab-9741-a433-abd817854bb3" "bobs-bookstore-admin-server:7001" "10.244.0.183:7001" outbound|7001||bobs-bookstore-admin-server.bob.svc.cluster.local - 10.244.0.183:7001 10.244.0.187:42654 - default
[2020-05-26T16:51:02.959Z] "- - -" 0 UF,URX "-" "-" 0 0 3 - "-" "-" "-" "-" "10.244.0.183:7001" outbound|7001||bobbys-front-end-admin-server.bobby.svc.cluster.local - 10.244.0.183:7001 10.244.0.187:42668 - -
[2020-05-26T16:51:02.964Z] "- - -" 0 UF,URX "-" "-" 0 0 2 - "-" "-" "-" "-" "10.244.0.183:7001" outbound|7001||bobbys-front-end-admin-server.bobby.svc.cluster.local - 10.244.0.183:7001 10.244.0.187:42674 - -

It looks to me as http succeeds. The lines below that I assume are the tcp invocations from the client that are failing. They appear to be failing because the service name
bobbys-front-end-admin-server.bobby.svc.cluster.local
is from the admin server in the bobby namespace. This failing client is running in the bob namespace.

The IP address is the correct one for the admin server running in the bob namespace so something seems odd about that.

Here is the service definition for the admin-server in bobby namespace

apiVersion: v1
kind: Service
metadata:
  name: bobbys-front-end-admin-server
  namespace: bobby
spec:
  clusterIP: None
  ports:
  - name: istio-probe
    port: 8888
    protocol: TCP
    targetPort: 8888
  - name: istio-q3
    port: 7001
    protocol: TCP
    targetPort: 7001
  - name: istio-q3channel
    port: 30101
    protocol: TCP
    targetPort: 30101
  publishNotReadyAddresses: true
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Let me know if you need anything else, happy to try any recommendations.