ISTIO External Authorization : getting error '503 upstream connect error or disconnect/reset before headers. reset reason: connection terminationroot' when access over HTTPS

ISTIO version: 1.9.4
EKS Cluster version: 1.14

We have deployed ISTIO APP mesh in our project. We have deployed External Authorization using istio’s documentation i.e. Istio / External Authorization.

External authorizer used (as mentioned in above documentation) : https://raw.githubusercontent.com/istio/istio/release-1.9/samples/extauthz/ext-authz.yaml

When we access any API from going into pod of another API (i.e. over http), using curl command, all works fine. External auth service gets call and all the headers are passed into external authorizer’s v3 check method. Below information is passed
source, principal, destination, headers: authority, method, path, accept, content-length, user-agent, x-b3-sampled, x-b3-spanid, x-b3-traceid, x-envoy-attempt-count, x-ext-authz, x-forwarded-client-certx-forwarded-proto, x-request-id.

But when we try to access the same service over https using postman, browser or from going into pod of another API and using curl with https endpoint, we get denied response from external authorizer’s v3 check method. Also when we check the logs of external authorizer’s v3 check method no headers are passed to it in this case.

Will really appreciate any help to fix this

Below is setup

Name spaces with ISTIO ejection enable : foo

Attached file has all the setup details. Please zoom in and check

But when we try to access the same service over https …
By https, do you mean you call your service in curl like "curl https://your-service/and you get the503 upstream connect error …`?

This seems not related to ext-authz but probably some misconfiguration in the mTLS. In your yaml file, it seems you enabled Istio mTLS in the PeerAuthN, it won’t work with the curl https request because the mTLS is in the Envoy level, not in your application level.

Did you plan to do HTTPS-in-mTLS by wrapping the HTTPS within Istio mTLS?

Hello @YangminZhu , Thanks a lot for your reply.

We have configure istio gateway with proxy on ssl and that works fine without external auth for all services and it only causing issue with external auth.

And yes we want to do HTTPS-in-mTLS

Please suggest.

I am still facing the issue

Below are the logs from ingress gateway

2021-07-08T02:13:56.325854Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T02:13:56.465960Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T02:44:53.108353Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T02:44:53.300615Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T03:16:08.573839Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T03:16:08.598611Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T03:48:16.290671Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T03:48:16.650628Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T04:21:18.936938Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T04:21:19.388311Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T04:48:36.554156Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T04:48:36.672130Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T05:19:43.851559Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T05:19:43.969333Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T05:51:11.040081Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T05:51:11.232936Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T06:04:13.420507Z info ads XDS: Incremental Pushing:0 ConnectedEndpoints:2 Version: 2021-07-08T06:04:13.871080Z info cache generated new workload certificate latency=450.256107ms ttl=23h59m59.128934125s 2021-07-08T06:04:13.871201Z info sds SDS: PUSH resource=default 2021-07-08T06:20:12.252942Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T06:20:12.591504Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T06:47:33.612995Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T06:47:33.701139Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T07:20:21.000443Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T07:20:21.312464Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T07:49:54.955433Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T07:49:55.278159Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T08:19:37.092565Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T08:19:37.260526Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T08:47:36.603014Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T08:47:36.807715Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T09:15:02.338493Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T09:15:02.399156Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T09:46:02.333524Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T09:46:02.708266Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T10:13:15.227794Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T10:13:15.686407Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T10:40:05.284785Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T10:40:06.482284Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp 172.20.0.51:15012: connect: connection refused" 2021-07-08T10:40:07.954178Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp 172.20.0.51:15012: connect: connection refused" 2021-07-08T10:40:09.212389Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T11:11:12.027052Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T11:11:12.107397Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T11:13:32.133303Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T11:13:33.554104Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp 172.20.0.51:15012: connect: connection refused" 2021-07-08T11:13:35.420052Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T11:43:24.012961Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T11:43:24.485189Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012 2021-07-08T12:11:32.608376Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0, 2021-07-08T12:11:32.904944Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012

I have also checked that the port is 15012 is Open