Hi there,
What is the easiest and fastest way to verify that mTLS is actually happening between the proxies of two services?
I can curl one service from another, but the only access logs I can see are within the receiving service, and at that point, its proxy had already changed it back into a plain HTTP request.
The proxy logs do not show me anything. Is there something I can tweak for their logs to show me the TLS activity happening within them?
Thanks,
jaid
Hi!
You can try envoy’s access log and then check some TLS property, such as %DOWNSTREAM_TLS_VERSION% on the server-side sidecar.
You can try something like:
istioctl x describe pod echoserver-v1-5f5b64d78-zzzz -n [namespace]
Service: echoserver
Port: http 8080/HTTP targets pod port 8080
DestinationRule: echoserver for “echoserver”
Traffic Policy TLS Mode: ISTIO_MUTUAL
Pod is STRICT and clients are ISTIO_MUTUAL
Changing settings to be Permissive I got (snippet of last 2 lines)
Traffic Policy TLS Mode: DISABLE
Pod is PERMISSIVE and clients are DISABLE
Kind Regards
Gerry
You can check mtls is enable or not for a service
istioctl authn tls-check -n $CLIENT_POD
reference: https://istio.io/docs/tasks/security/authentication/mutual-tls/#verify-mutual-tls-configuration
Thank you, all. I ended up updating the log format, per @kuat’s suggestion . I added the following:
localSubject=%DOWNSTREAM_LOCAL_SUBJECT% peerSubject=%DOWNSTREAM_PEER_SUBJECT% peerIssuer=%DOWNSTREAM_PEER_ISSUER%i localUriSan=%DOWNSTREAM_LOCAL_URI_SAN% peerUriSan=%DOWNSTREAM_PEER_URI_SAN%
This is what the receiving istio-proxy’s log looks like. I don’t know why the subjects are missing, but I do see other fields.
[03/03/2020T18:28:16+0000 1583260096] method=GET authority=hiworld.hw:5000 txId=- request=/hello alpn=HTTP/1.1 response_flag=- ssl_protocol=TLSv1.2 ssl_cipher=ECDHE-RSA-AES128-GCM-SHA256 status=200 status_details=via_upstream request_size=0 response_size=58 envoy_request_id=3e84e6d9-fb92-495a-b0ef-99c33caec71d user-agent=curl/7.38.0 xFor=192.168.125.199 x-envoy-upstream-service-time=176 total_time_ms=176 upstream=127.0.0.1:5000 localSubject=- peerSubject=- peerIssuer=emailAddress=<masked>,CN=<masked> localUriSan=spiffe://cluster.local/ns/hw/sa/default peerUriSan=spiffe://cluster.local/ns/hw/sa/default
This is what the sending istio-proxy’s log looks like. None of the SSL/TLS fields are populated. Is that to be expected for the sending proxy?
[03/03/2020T18:31:26+0000 1583260286] method=GET authority=hiworld.hw:5000 txId=- request=/hello alpn=HTTP/1.1 response_flag=- ssl_protocol=- ssl_cipher=- status=200 status_details=via_upstream request_size=0 response_size=58 envoy_request_id=67f40161-514e-482c-a055-7a27247cd197 user-agent=curl/7.38.0 xFor=192.168.125.199 x-envoy-upstream-service-time=181 total_time_ms=183 upstream=192.168.223.72:5000 localSubject=- peerSubject=- peerIssuer=-i localUriSan=- peerUriSan=-
Thanks again,
jaid
On the client server, you should use upstream TLS properties. Downstream on client means the connection from the app to the sidecar.
Doh! True. However, I didn’t find UPSTREAM equivalents in the doc. I can try them.
Also, if that’s the case, how would I specify two separate formats? A service can be a client or a server at any given time. And pardon me if the answer is obvious.
Yeah, there are some limitations to the envoy access log (such as lack of conditionals I believe and some upstream TLS properties have not been implemented). One way around that is to customize the access log format using EnvoyFilter (see the wiki).
There are also some subtle differences between upstream and downstream. You cannot check that the server validates certificates on the client side, even if the server requests a client certificate, for example. And similarly, you cannot tell that the client validates the server certificate on the server side.
Did anyone get to the bottom of why the Subjects are missing in the logs? With Istio 1.6.8 I can see the Issuer in the access logs but not the Subjects. Just checked 1.6.14 and I see the same behaviour.