mTLS is not working between services - istio-1.9.0

I have added PeerAuthentication mode: STRICT in namespace istio and there are 2 deployment running app1,app2

I have also injected the istio-injection=enabled on istio namespace
image

app1= nginx service
app2= httpd service

but when i curl -Iv http://app1 or http://app2 i don’t see any mTLS connection is being made. It’s only http connection.

Am i missing something? how do i achieve mTLS?

can you explain a little more about what you are doing? are you running curl within one of the containers? When you say you dont “see any mTLS”, what do you mean exactly?

Yes, I’m running the curl from app1 pod

When you say you dont “see any mTLS”, what do you mean exactly?

curl -Iv app2

   * Expire in 0 ms for 1 (transfer 0x5607bc98ebf0)
* Expire in 0 ms for 1 (transfer 0x5607bc98ebf0)
*   Trying 10.108.197.77...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x5607bc98ebf0)
* Connected to app2 (10.108.197.77) port 80 (#0)
> HEAD / HTTP/1.1
> Host: app2
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< date: Thu, 11 Feb 2021 13:28:57 GMT
date: Thu, 11 Feb 2021 13:28:57 GMT
< server: envoy
server: envoy
< last-modified: Mon, 11 Jun 2007 18:53:14 GMT
last-modified: Mon, 11 Jun 2007 18:53:14 GMT
< etag: "2d-432a5e4a73a80"
etag: "2d-432a5e4a73a80"
< accept-ranges: bytes
accept-ranges: bytes
< content-length: 45
content-length: 45
< content-type: text/html
content-type: text/html
< x-envoy-upstream-service-time: 7
x-envoy-upstream-service-time: 7

<
* Connection #0 to host app2 left intact

I don’t see any https/TLS connection being made

I also added DestinationRule as below

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: all-destnation-rule
spec:
  host: "*"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

but still same output as above

When i tried mode: SIMPLE and do curl -v app2 i see below error

the mTLS is made by the sidecar not your application: app1 <---> sidecar1 <=======> sidecar2 <---> app2, the mtls is between sidecar1 and sidecar2, the traffic between app and sidecar might be just plaintext.

1 Like

Yes, I understand that the mTLS will be between sidecar <==> sidecar

but i don’t see the mTLS badge between connection for example i have deployed the Bookinfo sample application from below link. It’s also not showing the mTLS badge between services in Kiali dashboard.

https://istio.io/latest/docs/setup/getting-started/

Below screenshot for above URL:

Though i’m still not able to get those mTLS badge on services but the traffic between istio sidecar proxy is on TLS

Here i have created a git repo to see how i checked TLS GitHub - koolwithk/istio-mTLS: istio-mTLS example

@ YangminZhu : Thank you, your answer gave me more confidence than what i was thinking earlier.

I believe its not configured by default in Kiali to display the mtls lock - check the “Display” settings.

1 Like

Enable the security legend but still don’t see the mTLS badge between app1 sidecar-proxy <==> app2 sidecar-proxy :frowning:

when i curl http://app2 from app1 container i see above green arrow app1 > app2 service > app2

root@app1-b6c8f6c99-x85r8:/# while  true; do curl -k http://app2:80 ; sleep 1; done
<html><body><h1>It works!</h1></body></html>
<html><body><h1>It works!</h1></body></html>
<html><body><h1>It works!</h1></body></html>
<html><body><h1>It works!</h1></body></html>

but when i do curl -k https://app2:80 from app1 container i see the mTLS badge but its giving below error.

root@app1-b6c8f6c99-x85r8:/# while  true; do curl -k https://app2:80 ; sleep 1; done
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Found out if we apply PeerAuthentication in istio 1.9.0 the mTLS badges appears on top right corner and if PeerAuthentication is not applied it displayed on pod and services.

With PeerAuthentication:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: "istio-system"
spec:
  mtls:
    mode: STRICT

see if this is any help: Kiali: Service mesh observability and configuration

1 Like

Hi John,

I also ran into the exact same issue as Alok. I encourage the Kiali team to take another review of the UX global security icon (on and off). I don’t think it is obvious and I had to go to my grafana dashboard to confirm mtls is enabled as istioctl authn command is also broken.

Also,in earlier versions of Kiali, if I have global strict authn enabled in istio, I would still see the security icon among microservices in Kiali. I think this is a lot more clear than what Kiali has now.

Thanks!

Lin

@linsun can you create a kiali issue and describe the problems? cc @xeviknal - I believe he’s the best one to handle looking at Kiali’s implementation of this.

Hi there,

we considered to hide all the locks when the global security is ON however edges without mTLS show a broken lock icon. The rationale behind was to reduce the number of locks in the graph and overview page because they are quite redundant with the global policy.
Kiali relies that Istio will enable mTLS unless there is a config that prevents it. When that config is in place, then kiali will show a broken lock.

Perhaps showing the mTLS status in the graph side panel would help to reduce the confusion.

I’ve created an issue to start discussions with the team: mTLS locks confusion when mTLS is strict at mesh-level · Issue #3707 · kiali/kiali · GitHub

1 Like

That’s true, you need to enable the Display->Security option. Perhaps this should be renamed from “Security” to “mTLS”.

While we revisit the UX around mTLS, lock badging, etc, I would also mention that you can use the “Graph Find” feature and enter “mtls” to see edges with mTLS active, or “!mtls” to see edges where it is not active. It will activate the display option automatically, if necessary.

You can confirm mtls by exec into Istio-proxy of the pod and try issuing the curl command. It should fail.

Hi Xavier,

Thank you for tracking this issue before I have the chance to, much appreciated! I will put my comment in the issue - my main concern is the mesh wide mTLS icon isn’t easily found and not showing the security badge when user asks for it seems very unintuitive.

Best,
Lin

Perhaps if you select the “Show Security” edge icons (Display>Security), if global mTLS is enabled, we somehow “flash” the lock icon in the masthead to bring the user’s attention to it.

Hi Xavier,

Thank you for tracking this issue before I have the chance to, much appreciated! I will put my comment in the issue - my main concern is the mesh wide mTLS icon isn’t easily found and not showing the security badge when user asks for it seems very unintuitive.

Best,
Lin