Istio 1.3.5 : Problems with kafka when mtls enabled in the mesh

Hey Guys,

Looking for some help in debugging and understanding what is causing this behaviour in our cluster where we have enabled sidecar injection and mtls is enabled.
And we are trying to install kafka and it doesn’t work as intended.

Kafka Producer (client) errors with

02-04-20 09:22:58       INFO         Logging$class:68                        Fetching metadata from broker id:0,host:kafka,port:9092 with correlation id 13195 for 1 topic(s) Set(logs-metadata)
02-04-20 09:22:58       INFO         Logging$class:68                        Connected to kafka:9092 for producing
02-04-20 09:22:58 4ms   INFO         Logging$class:68                        Disconnecting from kafka:9092
02-04-20 09:22:58       WARNING      Logging$class:89                        Fetching topic metadata with correlation id 13195 for topics [Set(logs-metadata)] from broker [id:0,host:kafka,port:9092] failed
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223

Istio-proxy in Producer

[2020-02-04T16:57:24.938Z] "- - -" 0 UF,URX "-" "-" 0 0 2 - "-" "-" "-" "-" "[KAFKA-IP]:9092" outbound|9092||kafka.svc.cluster.local - [KAFKA-IP]:9092 [SOURCE-PRODUCER-IP]:43360 - -
[2020-02-04T16:57:26.044Z] "- - -" 0 UF,URX "-" "-" 0 0 3 - "-" "-" "-" "-" "[KAFKA-IP]:9092" outbound|9092||kafka.svc.cluster.local - [KAFKA-IP]:9092 [SOURCE-PRODUCER-IP]:43392 - -
[2020-02-04T16:57:26.148Z] "- - -" 0 UF,URX "-" "-" 0 0 3 - "-" "-" "-" "-" "[KAFKA-IP]:9092" outbound|9092||kafka.svc.cluster.local - [KAFKA-IP]:9092 [SOURCE-PRODUCER-IP]:43398 - -

And in the kafka instance - istio-proxy we do see inbound request. But doesn’t an outbound at all

[2020-02-04T16:57:24.938Z] "- - -" 0 UF,URX "-" "-" 0 0 1 - "-" "-" "-" "-" "127.0.0.1:9092" inbound|9092|broker|kafka.svc.cluster.local - [KAFKA-IP]:9092 [SOURCE-PRODUCER-IP]:43364 - -

Kafka Service exposes a headless service

kafka                      ClusterIP   None              <none>        9092/TCP,8000/TCP            3h30m

Istioctl points

istioctl authn tls-check [producer].default | grep kafka

kafka.svc.cluster.local:8000                                OK           mTLS       mTLS       default/         default/istio-system
kafka.svc.cluster.local:9092                                OK           mTLS       mTLS       default/         default/istio-system

Issue we are facing to summarize is, we are getting Connection resets in producer. But we do see inbound requests in the kafka.
Any pointers how to debug/fix our issue?

I’m having the same issue. :frowning: