Istio 1.3.5 : Problems with kafka when mtls enabled in the mesh

Networking
1

Hey Guys,

Looking for some help in debugging and understanding what is causing this behaviour in our cluster where we have enabled sidecar injection and mtls is enabled.
And we are trying to install kafka and it doesn’t work as intended.

Kafka Producer (client) errors with

02-04-20 09:22:58       INFO         Logging$class:68                        Fetching metadata from broker id:0,host:kafka,port:9092 with correlation id 13195 for 1 topic(s) Set(logs-metadata)
02-04-20 09:22:58       INFO         Logging$class:68                        Connected to kafka:9092 for producing
02-04-20 09:22:58 4ms   INFO         Logging$class:68                        Disconnecting from kafka:9092
02-04-20 09:22:58       WARNING      Logging$class:89                        Fetching topic metadata with correlation id 13195 for topics [Set(logs-metadata)] from broker [id:0,host:kafka,port:9092] failed
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223

Istio-proxy in Producer

[2020-02-04T16:57:24.938Z] "- - -" 0 UF,URX "-" "-" 0 0 2 - "-" "-" "-" "-" "[KAFKA-IP]:9092" outbound|9092||kafka.svc.cluster.local - [KAFKA-IP]:9092 [SOURCE-PRODUCER-IP]:43360 - -
[2020-02-04T16:57:26.044Z] "- - -" 0 UF,URX "-" "-" 0 0 3 - "-" "-" "-" "-" "[KAFKA-IP]:9092" outbound|9092||kafka.svc.cluster.local - [KAFKA-IP]:9092 [SOURCE-PRODUCER-IP]:43392 - -
[2020-02-04T16:57:26.148Z] "- - -" 0 UF,URX "-" "-" 0 0 3 - "-" "-" "-" "-" "[KAFKA-IP]:9092" outbound|9092||kafka.svc.cluster.local - [KAFKA-IP]:9092 [SOURCE-PRODUCER-IP]:43398 - -

And in the kafka instance - istio-proxy we do see inbound request. But doesn’t an outbound at all

[2020-02-04T16:57:24.938Z] "- - -" 0 UF,URX "-" "-" 0 0 1 - "-" "-" "-" "-" "127.0.0.1:9092" inbound|9092|broker|kafka.svc.cluster.local - [KAFKA-IP]:9092 [SOURCE-PRODUCER-IP]:43364 - -

Kafka Service exposes a headless service

kafka                      ClusterIP   None              <none>        9092/TCP,8000/TCP            3h30m

Istioctl points

istioctl authn tls-check [producer].default | grep kafka

kafka.svc.cluster.local:8000                                OK           mTLS       mTLS       default/         default/istio-system
kafka.svc.cluster.local:9092                                OK           mTLS       mTLS       default/         default/istio-system

Issue we are facing to summarize is, we are getting Connection resets in producer. But we do see inbound requests in the kafka.
Any pointers how to debug/fix our issue?

2

I’m having the same issue. :frowning:

3

Were you able to resolve this issue? If yes, can you please share

4

Sorry. Just catching up here.
Yes we did manage to figure out the root cause.

The application port was not bound to the localhost/127.0.0.1, thereby causing the envoy to fail to communicate. Once that was fixed, it worked

5

Can you share the statefulset file definition, facing the same problem