Has anyone been able to inject the istio sidecar on an existing Kafka cluster running in kubernetes?
I’ve managed to inject the sidecar to our apps with mTLS disabled and can have communication between the brokers and the apps work successfully.
However, once I try to inject the sidecar onto the kafka broker, it looks like communication stops working even with mTLS still disabled. Through some research, I’ve seen that Istio does not support headless services, which is what we’ve got set up given it’s a StatefulSet set up… so I attempted to create a ServiceEntry to explicitly create these hosts in the mesh. (I also created similar Service Entries for zookeeper)
apiVersion: networking.istio.io/v1alpha3 kind: ServiceEntry metadata: name: broker namespace: kafka spec: location: MESH_INTERNAL hosts: - kafka-0.broker.kafka.svc.cluster.local - kafka-1.broker.kafka.svc.cluster.local - kafka-2.broker.kafka.svc.cluster.local - kafka-3.broker.kafka.svc.cluster.local - kafka-4.broker.kafka.svc.cluster.local ports: - number: 9093 name: tcp-plaintext protocol: TCP - number: 9092 name: tcp-ssl protocol: TCP
At first it looked like the broker was able to be elected as a leader and things were looking good other than missing the JMX metrics (another issue I’d tackle later…) The apps are connecting over port 9092 currently, and as soon as the sidecar was injected onto one broker, they started to claim that broker as “dead” which indicates that network communication between the two was not successful.
I’d like to get to the point where I could get the sidecar injected on all pods in our mesh, but it looks like I may need to keep the kafka brokers outside of the mesh? Has anyone been able to get the sidecar injected even without mTLS and had communication still work successfully? Did I miss something obvious?
Thanks for help in advance!