Hi,
I have configured Multi-Cluster with Multi-Network, version 12.2, as specified in official guide. As long as my application containers are listening on HTTP, everything is working. For legacy reasons, I have a lot of containers that are HTTPS-only enabled and I need to be able to connect to them using HTTPS. Everything is working except for one cross-cluster scenario:
I can connect from within cluster1 to cluster1 using curl
I can connect from within cluster1 to cluster2 using curl
I can connect using browser via cluster1 ingress-gateway to cluster1
I cannot connect using browser via cluster1 ingress-gateway to cluster2
To avoid issue with double TLS within cluster, I have two ports for abc-vt service:
443:443 - standard to be used for intra-cluster traffic
8443:443 - to be used by ingress-gateway with TLS origination enabled in DestinationRule
Now from the logs I can see that:
- if I connect from within cluster1, my HTTPS request is wrapped in TCP proxy and arrives correctly to destination in cluster1 or cluster2
- if I use ingress-gateway, behaviour is different and TCP proxy is not used
I am struggling to understand what I am doing wrong - especially since traffic originating in cluster1 can reach cluster2 using HTTPS without issues.
Logs:
CLUSTER1, CURL FROM WITHIN CLUSTER1 TO https://abc-vt-blue-svc.abc-test3.svc.cluster.local:443/vt/health, RETURNED CLUSTER1, OK
2022-01-25T20:26:24.746910Z debug envoy filter [C14024[] new tcp proxy session
2022-01-25T20:26:24.746936Z debug envoy filter [C14024[] Creating connection to cluster outbound|443||abc-vt-blue-svc.abc-test3.svc.cluster.local
2022-01-25T20:26:24.747159Z debug envoy conn_handler [C14024[] new connection from 172.27.4.81:43830
2022-01-25T20:26:24.752876Z debug envoy filter [C14024[] TCP:onUpstreamEvent(), requestedServerName:
2022-01-25T20:26:24.791942Z debug envoy connection [C14024[] closing data_to_write=40 type=0
2022-01-25T20:26:24.792043Z debug envoy connection [C14024[] write flush complete
2022-01-25T20:26:24.792049Z debug envoy connection [C14024[] closing socket: 1
2022-01-25T20:26:24.792202Z debug envoy conn_handler [C14024[] adding to cleanup list
CLUSTER1, CURL FROM WITHIN CLUSTER1 TO https://abc-vt-blue-svc.abc-test3.svc.cluster.local:443/vt/health, RETURNED CLUSTER2, OK
2022-01-25T20:33:48.252857Z debug envoy filter [C14292[] new tcp proxy session
2022-01-25T20:33:48.252903Z debug envoy filter [C14292[] Creating connection to cluster outbound|443||abc-vt-blue-svc.abc-test3.svc.cluster.local
2022-01-25T20:33:48.252936Z debug envoy misc Allocating TCP conn pool
2022-01-25T20:33:48.252968Z debug envoy pool trying to create new connection
2022-01-25T20:33:48.252974Z debug envoy pool creating a new connection
2022-01-25T20:33:48.253046Z debug envoy connection [C14293[] connecting to 10.77.6.57:15443
2022-01-25T20:33:48.253155Z debug envoy connection [C14293[] connection in progress
2022-01-25T20:33:48.253186Z debug envoy conn_handler [C14292[] new connection from 172.27.4.81:36396
2022-01-25T20:33:48.254086Z debug envoy connection [C14293[] connected
2022-01-25T20:33:48.257237Z debug envoy pool [C14293[] attaching to next stream
2022-01-25T20:33:48.257259Z debug envoy pool [C14293[] creating stream
2022-01-25T20:33:48.257282Z debug envoy filter [C14292[] TCP:onUpstreamEvent(), requestedServerName:
2022-01-25T20:33:48.273785Z debug envoy connection [C14293[] SSL shutdown: rc=0
2022-01-25T20:33:48.278489Z debug envoy connection [C14293[] remote close
2022-01-25T20:33:48.278516Z debug envoy connection [C14293[] closing socket: 0
2022-01-25T20:33:48.278585Z debug envoy pool [C14293[] client disconnected, failure reason:
2022-01-25T20:33:48.278603Z debug envoy pool invoking idle callbacks - is_draining_for_deletion_=false
2022-01-25T20:33:48.278618Z debug envoy pool [C14293[] destroying stream: 0 remaining
2022-01-25T20:33:48.278622Z debug envoy pool invoking idle callbacks - is_draining_for_deletion_=false
2022-01-25T20:33:48.278628Z debug envoy connection [C14292[] closing data_to_write=40 type=0
2022-01-25T20:33:48.278703Z debug envoy connection [C14292[] write flush complete
2022-01-25T20:33:48.278713Z debug envoy connection [C14292[] closing socket: 1
2022-01-25T20:33:48.278898Z debug envoy conn_handler [C14292[] adding to cleanup list
CLUSTER1, BROWSER CALL TO CLUSTER1 TO https://vipa01.dc.uat.k8s.abc-emea.xyz.com/vt/health, RETURNED CLUSTER1, OK
2022-01-25T20:47:22.071738Z debug envoy connection [C3631[] connecting to 172.27.9.126:8943
2022-01-25T20:47:22.071873Z debug envoy connection [C3631[] connection in progress
2022-01-25T20:47:22.072179Z debug envoy connection [C3631[] connected
2022-01-25T20:47:22.082363Z debug envoy client [C3631[] connected
2022-01-25T20:47:22.082396Z debug envoy pool [C3631[] attaching to next stream
2022-01-25T20:47:22.082407Z debug envoy pool [C3631[] creating stream
2022-01-25T20:47:22.082422Z debug envoy router [C1702[][S972152758931264896[] pool ready
2022-01-25T20:47:22.090307Z debug envoy router [C1702[][S972152758931264896[] upstream headers complete: end_stream=false
2022-01-25T20:47:22.090457Z debug envoy http [C1702[][S972152758931264896[] encoding headers via codec (end_stream=false):
CLUSTER1, BROWSER CALL TO CLUSTER1 TO https://vipa01.dc.uat.k8s.abc-emea.xyz.com/vt/health, FAILS TO CONNECT TO CLUSTER1
INGRESSGATEWAY:
istio-ingressgateway-69bb8ccdcf-bpm9x 2022-01-25T20:56:16.131955Z debug envoy pool trying to create new connection
istio-ingressgateway-69bb8ccdcf-bpm9x 2022-01-25T20:56:16.131962Z debug envoy pool creating a new connection
istio-ingressgateway-69bb8ccdcf-bpm9x 2022-01-25T20:56:16.132149Z debug envoy client [C3902[] connecting
istio-ingressgateway-69bb8ccdcf-bpm9x 2022-01-25T20:56:16.132163Z debug envoy connection [C3902[] connecting to 10.77.6.57:15443
istio-ingressgateway-69bb8ccdcf-bpm9x 2022-01-25T20:56:16.132581Z debug envoy connection [C3902[] connection in progress
istio-ingressgateway-69bb8ccdcf-bpm9x 2022-01-25T20:56:16.133422Z debug envoy connection [C3902[] connected
istio-ingressgateway-69bb8ccdcf-bpm9x 2022-01-25T20:56:16.134357Z debug envoy connection [C3902[] TLS error: 33554536:system library:OPENSSL_internal:Connection reset by peer
istio-ingressgateway-69bb8ccdcf-bpm9x 2022-01-25T20:56:16.134378Z debug envoy connection [C3902[] closing socket: 0
istio-ingressgateway-69bb8ccdcf-bpm9x 2022-01-25T20:56:16.134393Z debug envoy connection [C3902[] TLS error: 33554536:system library:OPENSSL_internal:Connection reset by peer
istio-ingressgateway-69bb8ccdcf-bpm9x 2022-01-25T20:56:16.134505Z debug envoy client [C3902[] disconnect. resetting 0 pending requests
istio-ingressgateway-69bb8ccdcf-bpm9x 2022-01-25T20:56:16.134526Z debug envoy pool [C3902[] client disconnected, failure reason: TLS error: 33554536:system library:OPENSSL_internal:Connection reset by peer
istio-ingressgateway-69bb8ccdcf-bpm9x 2022-01-25T20:56:16.134543Z debug envoy router [C1702[][S2107624254085186837[] upstream reset: reset reason: connection failure, transport failure reason: TLS error: 33554536:system library:OPENSSL_internal:Connection reset by peer
istio-ingressgateway-69bb8ccdcf-bpm9x 2022-01-25T20:56:16.134569Z debug envoy pool invoking idle callbacks - is_draining_for_deletion_=false
istio-ingressgateway-69bb8ccdcf-bpm9x 2022-01-25T20:56:16.158730Z debug envoy router [C1702[][S2107624254085186837[] performing retry
istio-ingressgateway-69bb8ccdcf-bpm9x 2022-01-25T20:56:16.158894Z debug envoy pool queueing stream due to no available connections
EASTWESTGATEWAY:
2022-01-25T20:56:16.133941Z debug envoy conn_handler closing connection from 172.27.9.0:62021: no matching filter chain found
2022-01-25T20:56:16.160633Z debug envoy filter tls inspector: new connection accepted
2022-01-25T20:56:16.160730Z debug envoy conn_handler closing connection from 172.27.9.0:38282: no matching filter chain found
2022-01-25T20:56:16.174963Z debug envoy filter tls inspector: new connection accepted
2022-01-25T20:56:16.175054Z debug envoy conn_handler closing connection from 172.27.9.0:38846: no matching filter chain found
2022-01-25T20:56:17.427651Z debug envoy conn_handler [C7763[] new connection from 10.77.7.90:53957
CONFIG:
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: abc-vt
spec:
host: abc-vt-blue-svc.abc-test3.svc.cluster.local
trafficPolicy:
portLevelSettings:
- port:
number: 8443
tls:
mode: SIMPLE
outlierDetection:
consecutiveGatewayErrors: 5
interval: 1s
baseEjectionTime: 1m
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: abc-vt
spec:
gateways:
- main-gateway
hosts:
- '*'
http:
- route:
- destination:
host: abc-vt-blue-svc.abc-test3.svc.cluster.local
port:
number: 8443
match:
- uri:
prefix: "/vt"
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: abc-vt
spec:
hosts:
- abc-vt-blue-svc.abc-test3.svc.cluster.local
ports:
- number: 8443
name: ingress-https
protocol: HTTPS
resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: main-gateway
spec:
servers:
- hosts:
- 'vipa01.dc.uat.k8s.xyz-emea.xyz.com'
port:
name: https
number: 443
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: istio-cert
selector:
istio: ingressgateway
---
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
annotations:
meta.helm.sh/release-name: istio
meta.helm.sh/release-namespace: istio-system
creationTimestamp: "2022-01-21T15:27:26Z"
finalizers:
- istio-finalizer.install.istio.io
generation: 5
labels:
app.kubernetes.io/managed-by: Helm
name: istio-controlplane
namespace: istio-system
resourceVersion: "223628659"
uid: cb327aca-becd-4c5b-805c-ce79dcf6af77
spec:
components:
ingressGateways:
- k8s:
env:
- name: ISTIO_META_ROUTER_MODE
value: sni-dnat
service:
externalIPs:
- 10.77.6.56
sessionAffinityConfig:
clientIP:
timeoutSeconds: 150
type: ClusterIP
label:
version: 1.12.2
name: istio-ingressgateway
- enabled: true
k8s:
env:
- name: ISTIO_META_REQUESTED_NETWORK_VIEW
value: network2
- name: ISTIO_META_ROUTER_MODE
value: sni-dnat
service:
externalIPs:
- 10.77.6.57
ports:
- name: status-port
port: 15021
targetPort: 15021
- name: tls
port: 15443
targetPort: 15443
- name: tls-istiod
port: 15012
targetPort: 15012
- name: tls-webhook
port: 15017
targetPort: 15017
sessionAffinityConfig:
clientIP:
timeoutSeconds: 150
type: ClusterIP
label:
app: istio-eastwestgateway
istio: eastwestgateway
topology.istio.io/network: network2
name: istio-eastwestgateway
hub: harbor-emea.xyz.com/istio
meshConfig:
defaultConfig:
tracing:
zipkin:
address: jaeger-es-collector.observability:9411
enableTracing: true
profile: default
tag: 1.12.2
values:
gateways:
istio-ingressgateway:
injectionTemplate: gateway
global:
meshID: mesh1
multiCluster:
clusterName: cluster2
network: network2
status:
componentStatus:
Base:
status: HEALTHY
IngressGateways:
status: HEALTHY
Pilot:
status: HEALTHY
status: HEALTHY