Upstream connect error or disconnect/reset before headers. reset reason: connection termination for STRICT Peer Authentication

Security
1

The issue i have is that, everything seems to be working well with peerAuthentication set to PERMISSIVE at namespace level. Now i change it to STRICT i get

upstream connect error or disconnect/reset before headers. reset reason: connection termination .

Here is a look at my configurations:


apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: prod
  namespace: prod
spec:
  mtls:
    mode: STRICT

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: mygateway
  namespace: prod
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - api-gateway.com
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: secret-tls
      mode: SIMPLE

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: api-gateway
  namespace: prod
spec:
  gateways:
  - mygateway
  hosts:
  - api-gateway.com
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: api-gateway
        port:
          number: 8765

apiVersion: v1
kind: Service
metadata:
  labels:
    app: api-gateway
  name: api-gateway
  namespace: prod
  spec:
  ports:
  - name: http-web
    nodePort: 31991
    port: 8765
    protocol: TCP
    targetPort: 8765
  selector:
    app: api-gateway
    version: v1
  sessionAffinity: None
  type: LoadBalancer

istioctl proxy-config route istio-ingressgateway-66d5c58595-th9r6 -n istio-system -o json:

[
    {
        "name": "https.443.https.mygateway.prod",
        "virtualHosts": [
            {
                "name": "api-gateway.com:443",
                "domains": [
                    "api-gateway.com"
                ],
                "routes": [
                    {
                        "match": {
                            "prefix": "/",
                            "caseSensitive": true
                        },
                        "route": {
                            "cluster": "outbound|8765||api-gateway.prod.svc.cluster.local",
                            "timeout": "0s",
                            "retryPolicy": {
                                "retryOn": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes",
                                "numRetries": 2,
                                "retryHostPredicate": [
                                    {
                                        "name": "envoy.retry_host_predicates.previous_hosts",
                                        "typedConfig": {
                                            "@type": "type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate"
                                        }
                                    }
                                ],
                                "hostSelectionRetryMaxAttempts": "5",
                                "retriableStatusCodes": [
                                    503
                                ]
                            },
                            "maxStreamDuration": {
                                "maxStreamDuration": "0s",
                                "grpcTimeoutHeaderMax": "0s"
                            }
                        },
                        "metadata": {
                            "filterMetadata": {
                                "istio": {
                                    "config": "/apis/networking.istio.io/v1alpha3/namespaces/prod/virtual-service/api-gateway"
                                }
                            }
                        },
                        "decorator": {
                            "operation": "api-gateway.prod.svc.cluster.local:8765/*"
                        }
                    }
                ],
                "requireTls": "ALL",
                "includeRequestAttemptCount": true
            }
        ],
        "validateClusters": false,
        "ignorePortInHostMatching": true
    },
    {
        "name": "http.8080",
        "virtualHosts": [
            {
                "name": "blackhole:80",
                "domains": [
                    "*"
                ]
            }
        ],
        "validateClusters": false,
        "ignorePortInHostMatching": true
    },
    {
        "virtualHosts": [
            {
                "name": "backend",
                "domains": [
                    "*"
                ],
                "routes": [
                    {
                        "match": {
                            "prefix": "/stats/prometheus"
                        },
                        "route": {
                            "cluster": "prometheus_stats"
                        }
                    }
                ]
            }
        ]
    },
    {
        "virtualHosts": [
            {
                "name": "backend",
                "domains": [
                    "*"
                ],
                "routes": [
                    {
                        "match": {
                            "prefix": "/healthz/ready"
                        },
                        "route": {
                            "cluster": "agent"
                        }
                    }
                ]
            }
        ]
    }
]

Please what am i doing wrong? Any help is highly welcome

2

Please i need all the help i can get

3

@Tomas_Kohout
@vadimeisenbergibm
@YangminZhu
Please any help will be appreciated

4

I solved the issue by unistalling istio using istioctl and then re installed it. Set timeout in the virtual services and then expose the services on port 80

5

So did you solve your problem or did you find a workaround?
If you still need help, could you first describe your use case from high level (what do you want to expose, how do you want to secure it, etc.)?

6

I actually re installed istio then I got 502 bad gateway error… But I changed my port to 80, then everything went fine… Is it OK using 80 for STRICT mode in production ?