Policy In Permissive Mode, mode: MUTUAL in DR, https, mTLS doesnt work

Networking
1

Hi,

I have a Service A(Frontend) taking to Service B(Backend). Service B has Policy and Destinationrule. Policy mode is PERMISSIVE and Destinationrule mode is MUTUAL. When service A sends the request through its sidecar, getting following error,
“upstream connect error or disconnect/reset before headers. reset reason: connection failure”

Looking at the inbound config of istio-proxy of Service B, two filter chains are getting added, one with match application_protocol: istio and tls configs to terminate, second is for plain text. I suspect Service A sidecar doesnt add “application_protocol: istio” , second filter chain matched which doesn’t have TLS context to terminate and failing.

Following Scenarios works,
Service A(policy mode, DR mode) ==> Service B(Policy mode, DR mode)
no policy, DR ==> STRICT, Mutual
PERMISSIVE,MUTUAL . ==> STRICT, Mutual

We use Custom CA, not Citadel.

Please let me if i am missing something…

2

Some further investigation:
In this PR (https://github.com/istio/istio/pull/6612) it seems like for PERMISSIVE authentication mode, the inbound listener has an additional filter chain match added which expects application_protocol to be set to “istio” if TLS termination is required.

However the corresponding code for clusters - https://github.com/istio/istio/blob/9b6d31b74d1c0cc9358cc82d395b53f71393326b/pilot/pkg/networking/core/v1alpha3/cluster.go#L980-L983 sets the alpn value to “istio” only if the mode is ISTIO_MUTUAL.

As mentioned above we use the mode MUTUAL since we have our custom CA, and hence the outbound cluster context does not have the alpn value set to “istio”.
As a result calls from the client sidecar istio-proxy to the server proxy are failing with “upstream connect error or disconnect/reset before headers. reset reason: connection failure”

3

Can you list the policy and DR? DR is for client side behavior and policy is for server side, if serviceA is a client, it’s policy mode does not matter.

4

Thanks a lot for your response. Below are the details,

Client(Service A) DR,

Server(Service B) Policy:

6

I think the issue might be that when you use PERMISSIVE as TLS mode on the server side, only unencrypted (no TLS) or ISTIO_MUTUAL modes are supported. The setting application_protocol: istio is only added when the DestinationRule uses tls.mode ISTIO_MUTUAL

7

Thanks @nrjpoddar for your response. Same configuration works fine in 1.0.6. Seems like 1.1.x has new change https://github.com/istio/istio/blob/9b6d31b74d1c0cc9358cc82d395b53f71393326b/pilot/pkg/networking/core/v1alpha3/cluster.go#L980-L983 which introduces this limitation.

8

The code snippet you pasted seems correct to me as it’s sets the application_protocol: istio only when ISTIO_MUTUAL is used as tls.mode.

Can you paste the Envoy config for both versions: .1.0.6 and 1.1.x for the relevant filter? You can get it by port-forwarding to Pod for Service A at port 15000 and go to localhost:15000/config_dump. Thanks!

9

@diemtvu Adding our expert on authentication policy.

10

The host in the destination rule should match the service correctly. In this particular example, it should be Service_A.test.svc.cluster.local (assuming your service name is ServiceA in namespace test). Also, as @ nrjpoddar pointed out, the TLS mode should be ISTIO_MUTUAL, and you don’t need to set any key/certs path. The whole DR should be like this:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  annotations:
  generation: 1
  labels:
  name: Service_A
  namespace: test
spec:
  host: Service_A.test.svc.cluster.local
  trafficPolicy:
    connectionPool:
      http:
        http1MaxPendingRequests: 100000
        http2MaxRequests: 100000
        maxRequestsPerConnection: 100000
        maxRetries: 3
    tls:
      mode: ISTIO_MUTUAL
11

1.1.x configs:
Corresponding configuration on Service A and B for this context

Service_A outbound cluster config - https://gist.github.com/karthikt-yahoo/66347b4aca58017ed42a2273a2974c73
Service_B listener config - https://gist.github.com/karthikt-yahoo/0f980e730a3bcf73638ad3dbd8d674f8

12

@diemtvu We do have the host: field containing ..svc.cluster.local

13

I’ve just realize that service_B is the server. Then your destination rule’s host should be service_b instead. Can you confirm? Thanks.

14

https://gist.github.com/karthikt-yahoo/16f1c343dd0070190c88d6029ece6ca1. Thanks

15

Thanks Karthik for the config snippet. I looked at the code and the config and this looks like a bug (or a feature request) to me. As you and @mdhume pointed out that if you have an Auth Policy in PERMISSIVE mode the server inbound listener only gets 2 filter chains (one with Istio ALPN and other with no TLS requirements) but if the client uses DestinationRule with MUTUAL mode set with its own certificates none of the filters on the server will match. We need to update server to have an additional filter if we need to support this use case. @diemtvu can you confirm? If my analysis is correct @Karthikeyan_Thangara please file a bug in Istio repo. I also don’t think that the Istio currently supports providing custom server side certificates via AuthPolicies which is your use case if I underststood correctly.

16

all we need is to add 3rd filter chain match to allow legacy client or client outside mesh to terminate TLS at the sidecar.

    filters:
    - name: ...
      [...]
  - filter_chain_match:
      transport_protocol: "tls"
    tls_context:
      common_tls_context:
        tls_certificates:
        - certificate_chain: { filename: "/etc/certs/cert-chain.pem" }
          private_key: { filename: "/etc/certs/key.pem" }
17

Opened an istio issue https://github.com/istio/istio/issues/14130

1 Like