Hello, I’ve got issue with cross cluster trafficing in multi-cluster, multi-network Istio setup, I’ve set it up mostly according to docs (Istio / Install Multi-Primary on different networks), main difference is that I’ve been using cert-manager for issuing workloads certificates.
Setup consist of two clusters with domains qwe.local and asd.local
After installing sample application, request are properly routed to service within cluster, but request to other cluster ends up with following error in eastwest gateway in cluster asd
[2022-12-20T20:51:53.272Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 192.12.118.77:15443 192.12.99.0:58158 outbound_.8080_._.hostname.istio-example.svc.qwe.local -
[2022-12-20T20:51:54.433Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 192.12.118.77:15443 192.12.99.0:45808 outbound_.8080_._.hostname.istio-example.svc.qwe.local -
[2022-12-21T08:02:37.173Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 1 - "-" "-" "-" "-" "-" - - 192.12.118.77:15443 192.12.99.0:50448 hostname.istio-example.global -
[2022-12-21T08:02:38.036Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 192.12.118.77:15443 192.12.99.0:6134 hostname.istio-example.global -
Following manifests are used:
IstioOperator for both clusters are similar, difference is only with qwe/asd cluster names
---
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: istiod-istio-east-west-istiooperator
spec:
values:
global:
meshID: mesh-1
multiCluster:
clusterName: cluster-qwe
network: network-qwe
caAddress: cert-manager-istio-csr.cert-manager.svc:443
proxy:
clusterDomain: qwe.local
cni:
psp_cluster_role: istio-cni
excludeNamespaces:
- istio-system
- kube-system
pilot:
autoscaleEnabled: false
replicaCount: 1
gateways:
istio-ingressgateway:
injectionTemplate: gateway
telemetry:
v2:
prometheus:
enabled: true
components:
ingressGateways:
- name: istio-ingressgateway
enabled: false
- name: istio-eastwestgateway
enabled: true
label:
istio: eastwestgateway
app: istio-eastwestgateway
topology.istio.io/network: network-qwe
k8s:
env:
- name: CA_ADDR
value: cert-manager-istio-csr.cert-manager.svc:443
- name: ISTIO_META_ROUTER_MODE
value: "sni-dnat"
- name: ISTIO_META_REQUESTED_NETWORK_VIEW
value: "network-qwe"
overlays:
- apiVersion: apps/v1
kind: Deployment
name: istio-eastwestgateway
patches:
- path: spec.template.spec.containers.[name:istio-proxy].volumeMounts[-1]
value:
name: ca-root-cert
mountPath: "/etc/cert-manager/ca/ca-certificates.crt"
readOnly: true
- path: spec.template.spec.volumes[-1]
value:
name: ca-root-cert
hostPath:
path: /etc/ssl/certs/ca-certificates.crt
service:
ports:
- name: status-port
port: 15021
targetPort: 15021
- name: tls
port: 15443
targetPort: 15443
- name: tls-istiod
port: 15012
targetPort: 15012
- name: tls-webhook
port: 15017
targetPort: 15017
- name: http-monitoring
port: 15020
targetPort: 15020
pilot:
k8s:
env:
# Disable istiod CA Sever functionality
- name: ENABLE_CA_SERVER
value: "false"
overlays:
- apiVersion: apps/v1
kind: Deployment
name: istiod
patches:
# Mount istiod serving and webhook certificate from Secret mount
- path: spec.template.spec.containers.[name:discovery].args[-1]
value: "--tlsCertFile=/etc/cert-manager/tls/tls.crt"
- path: spec.template.spec.containers.[name:discovery].args[-1]
value: "--tlsKeyFile=/etc/cert-manager/tls/tls.key"
- path: spec.template.spec.containers.[name:discovery].args[-1]
value: "--caCertFile=/etc/cert-manager/tls/ca.crt"
- path: spec.template.spec.containers.[name:discovery].volumeMounts[-1]
value:
name: cert-manager
mountPath: "/etc/cert-manager/tls"
readOnly: true
- path: spec.template.spec.containers.[name:discovery].volumeMounts[-1]
value:
name: ca-root-cert
mountPath: "/etc/cert-manager/ca/ca-certificates.crt"
readOnly: true
- path: spec.template.spec.volumes[-1]
value:
name: cert-manager
secret:
secretName: istiod-tls
- path: spec.template.spec.volumes[-1]
value:
name: ca-root-cert
hostPath:
path: /etc/ssl/certs/ca-certificates.crt
cni:
enabled: true
meshConfig:
enableTracing: true
trustDomain: qwe.local
defaultConfig:
proxyMetadata:
ECC_SIGNATURE_ALGORITHM: ECDSA
ISTIO_META_DNS_CAPTURE: "true"
ISTIO_META_DNS_AUTO_ALLOCATE: "true"
TRUST_DOMAIN: "qwe.local"
accessLogFile: /dev/stdout
serviceSettings:
- settings:
clusterLocal: true
hosts:
#
- "*.projectx.svc.qwe.local"
- "*.projectx.svc.cluster.local"
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-east-west-mesh-cross-network-gateway
spec:
selector:
istio: eastwestgateway
servers:
- port:
number: 15443
name: tls
protocol: TLS
tls:
mode: AUTO_PASSTHROUGH
hosts:
- "*.local"
hostname service is basic, and points to one pod
---
apiVersion: v1
kind: Service
metadata:
name: hostname
spec:
type: ClusterIP
ports:
- port: 8080
targetPort: http
protocol: TCP
name: http
selector:
app: hostname
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hostname
labels:
app: hostname
spec:
replicas: 1
selector:
matchLabels:
app: hostname
template:
metadata:
labels:
app: hostname
spec:
containers:
- name: hostname
image: "test/hostname:latest"
imagePullPolicy: IfNotPresent
env:
- name: CLUSTER
value: "qwe" # asd in second
ports:
- name: http
containerPort: 8080
protocol: TCP
DR/SE in qwe
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: hostname.istio-example.global
spec:
host: hostname.istio-example.global
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
sni: hostname.istio-example.global
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: hostname.istio-example.global
spec:
endpoints:
- address: 192.205.114.2
locality: cluster-asd
ports:
http: 15443
- address: hostname.istio-example.svc.qwe.local
locality: cluster-qwe
ports:
http: 8080
hosts: [hostname.istio-example.global]
location: MESH_INTERNAL
ports:
- name: http
number: 8080
protocol: http
resolution: DNS
DR/SE in asd
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: hostname.istio-example.global
spec:
host: hostname.istio-example.global
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
sni: hostname.istio-example.global
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: hostname.istio-example.global
spec:
endpoints:
- address: hostname.istio-example.svc.asd.local
locality: cluster-asd
ports:
http: 8080
- address: 192.210.114.22
locality: cluster-qwe
ports:
http: 15443
hosts: [hostname.istio-example.global]
location: MESH_INTERNAL
ports:
- name: http
number: 8080
protocol: http
resolution: DNS
Also there is sleep service from helloworld example from docs.
$ istioctl --context qwe-istio-example pc endpoint service/sleep | grep hostname
192.128.86.24:8080 HEALTHY OK outbound|8080||hostname.istio-example.svc.qwe.local
192.130.217.40:8080 HEALTHY OK outbound|8080||hostname.istio-example.global
192.205.114.1:15443 HEALTHY OK outbound|8080||hostname.istio-example.svc.qwe.local
192.205.114.2:15443 HEALTHY OK outbound|8080||hostname.istio-example.global
$ istioctl --context asd-istio-example pc endpoint service/sleep | grep hostname
192.132.89.157:8080 HEALTHY OK outbound|8080||hostname.istio-example.svc.asd.local
192.135.146.182:8080 HEALTHY OK outbound|8080||hostname.istio-example.global
192.210.114.22:15443 HEALTHY OK outbound|8080||hostname.istio-example.global
192.210.114.22:15443 HEALTHY OK outbound|8080||hostname.istio-example.svc.asd.local
As for the default service discovery I can understood that it might be problem with different cluster domains, but I don’t really know why it doesn’t work for hostname.istio-example.global.
And is there a way to make hostname.istio-example.svc.qwe.local work?