Injecting Sidecar to Kafka Brokers Successfully

1

Has anyone been able to inject the istio sidecar on an existing Kafka cluster running in kubernetes?

I’ve managed to inject the sidecar to our apps with mTLS disabled and can have communication between the brokers and the apps work successfully.

However, once I try to inject the sidecar onto the kafka broker, it looks like communication stops working even with mTLS still disabled. Through some research, I’ve seen that Istio does not support headless services, which is what we’ve got set up given it’s a StatefulSet set up… so I attempted to create a ServiceEntry to explicitly create these hosts in the mesh. (I also created similar Service Entries for zookeeper)

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: broker
  namespace: kafka
spec:
  location: MESH_INTERNAL
  hosts:
  - kafka-0.broker.kafka.svc.cluster.local
  - kafka-1.broker.kafka.svc.cluster.local
  - kafka-2.broker.kafka.svc.cluster.local
  - kafka-3.broker.kafka.svc.cluster.local
  - kafka-4.broker.kafka.svc.cluster.local
  ports:
  - number: 9093
    name: tcp-plaintext
    protocol: TCP
  - number: 9092
    name: tcp-ssl
    protocol: TCP

At first it looked like the broker was able to be elected as a leader and things were looking good other than missing the JMX metrics (another issue I’d tackle later…) The apps are connecting over port 9092 currently, and as soon as the sidecar was injected onto one broker, they started to claim that broker as “dead” which indicates that network communication between the two was not successful.

I’d like to get to the point where I could get the sidecar injected on all pods in our mesh, but it looks like I may need to keep the kafka brokers outside of the mesh? Has anyone been able to get the sidecar injected even without mTLS and had communication still work successfully? Did I miss something obvious?

Thanks for help in advance!

1 Like
2

I had the same issues with getting the brokers to communicate. I also tried to service entry route without much luck. I assume you are using the helm cart by confluent?

3

We’re actually not, but it’s just a Kustomize overlay of the kubernetes-kafka GitHub repo. Not sure what the differences would be there.

4

probably nothing much. Were you able to get zk (zookeeper) on the mesh? I tried the same techniques, with the same outcome. They were never able to reach the headless service.

In case you wanted to check. They look almost identical.

5

Here is an example of Kafka I got working with sidecars. https://github.com/istio/tools/pull/127

Note I don’t have Zookeeper with sidecars yet, but in theory you should be able to. I would suggest turning off sidecars on everything first, making sure it works, then turning them on for Kafka, then zookeeper, so you can debug one step at a time

6

Awesome. Thanks for the help and advice. I will try to increment it from kafka to zookeeper. I went the other way around. Trying to inject into zookeeper first then kafka.

7

@mike @howardjohn have you managed to enable mTLS between Kafka Client and Kafka Brokers?

8

@mike @howardjohn @akhoshaba Did you manage to run Kafka with Istio mTLS enabled?
Any help would be appreciated.
Cheers,
~Ajit

9

Hello,
Try using mtls mode STRICT in policy and set tls mode to ISTIO_MUTUAL in destinationrule as follow,

apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  annotations:
  name: kafka-mtls-authn
  namespace: ns-kafka
spec:
  peers:
  - mtls:
      mode: STRICT
  targets:
  - name: service-kafka
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  annotations:
  name: kafka-mtls-dr
  namespace: ns-kafka
spec:
  host: service-kafka.ns-kafka.svc.cluster.local
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL