Sucessful deployment of CockroachDB in istio-injected namespace

For anyone who might be struggling with this, I was able to deploy CockroachDB in secure mode with istio injection. I have elected to use CRDBs own internal TLS workflow, and have not enabled istio mTLS for the CRDB service, as this seems redundant.

I used a GKE hosted cluster, installed istio 1.2.2 from helm templates. Created the namespace: crdb and labeled it for istio injection.

Since cockroachDB is a statefulset, it is required to create a service entry resource for the MESH_INTERNAL hosts associated with its headless service. Mine looks like this:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: crdb-stateful-service-entry
  namespace: crdb
spec:
  hosts:
  - "*.cockroachdb.crdb.svc.cluster.local"
  - "*.cockroachdb"
  location: MESH_INTERNAL
  ports:
  - number: 26257
    name: crdbheadless1
    protocol: TCP
  - number: 8080
    name: crdbheadless2
    protocol: HTTP
  resolution: NONE

You can now follow the “orchestrated installation” instructions provided by cockroachDB documentation, however, the deployment manifests require alteration in port naming, and also should be labeled for creation inside the “crdb” namespace. Here are my altered versions of their provided deployment manifests:

Hope this is helpful for anyone looking into deploying this DB cluster with istio injection. Enjoy!

2 Likes

The protocol and port name must be changed to “TCP” on both this service entry, and also in the port name in the headless service and StatefulSet manifests.

For whatever reason, when it’s labeled GRPC it cannot make the TLS handshake, but does work when labeled TCP. The other port (8080) works fine as HTTP.

1 Like

Hi,
I followed the steps, and cluster is now up.

but i cannot connect it using public service, are you facing the same issue

Did you have to disable the mTLS for the service explicitly using a Policy and DestinationRule ?

Hi,

I`m new to istio and trying to expose cocroachdb deployed in a rke2 cluster and can’t get it to work. I’m unable to connect to the database from outside. The loadbalancer part and control plane ports are fine.
The operator is configured as Daemonset and uses host ports. This works for many things but cockroachdb.
Does anyone see any misconfiguration in the following manifests? Any help would be appreciated.

Edit: The namespace with database has no sidecar injection activated

Here the operator part. (Istio 1.10.2)

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: foris-istio-controlplane
spec:
  components:
    cni:
      enabled: true
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          service:
            type: ClusterIP
            ports:
              ...
            - port: 26257
              targetPort: 26257
              name: cocroachdb
          overlays:
          - kind: Deployment
            name: istio-ingressgateway
            patches:
              - path: kind
                value: DaemonSet
                ...
              - path: spec.template.spec.containers.[name:istio-proxy].ports.[containerPort:26257].hostPort
                value: 26257
...

As suggested by mike, changed the port names to others than grpc on the cockroachdb services.

apiVersion: v1
kind: Service
metadata:
  namespace: crdb
  name: cockroachdb-public
  labels:
    app: cockroachdb
spec:
  ports:
  - port: 26257
    targetPort: 26257
    name: tcp-public
  - port: 8080
    targetPort: 8080
    name: http-public
  selector:
    app: cockroachdb
---
apiVersion: v1
kind: Service
metadata:
  namespace: crdb
  name: cockroachdb
  labels:
    app: cockroachdb
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
  ports:
  - port: 26257
    targetPort: 26257
    protocol: TCP
    name: tcp-headless
  - port: 8080
    targetPort: 8080
    name: http-headless
  publishNotReadyAddresses: true
  clusterIP: None
  selector:
    app: cockroachdb

Deployed a service entry

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  namespace: crdb
  name: cockroachdb-service-entry
spec:
  hosts:
  - "*.cockroachdb.crdb.svc.cluster.local"
  - "*.cockroachdb"
  location: MESH_INTERNAL
  ports:
  - number: 26257
    name: tcp-headless
    protocol: TCP
  - number: 8080
    name: http-headless
    protocol: HTTP
  resolution: NONE

A destination rule (is this needed?)

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  namespace: crdb
  name: cockroachdb-simple
spec:
  host: "*.cockroachdb.crdb.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: SIMPLE

a virtual service and a gateway

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  namespace: crdb
  name: cocroachdb-service
spec:
  hosts:
  - "*"
  gateways:
  - crdb/cockroachdb-gateway
  tls:
  - match:
    - port: 26257
      sniHosts:
        - crdb.mydomain.com
    route:
    - destination:
        host: cockroachdb-service-entry
        port:
          number: 26257
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  namespace: crdb
  name: cockroachdb-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 26257
      name: https
      protocol: HTTPS
    tls:
      mode: PASSTHROUGH
    hosts:
    - crdb.mydomain.com

Kind regards,
Michael