Upstream connect error or disconnect/reset before headers. reset reason: local reset

I have installed istio 1.7.0 in gke environment through IstioOperator and deployed bookinfo application when i am trying to access the bookinfo application using istio-ingressgateway external-ip - I am seeing the following error

GET /productpage HTTP/1.1" 503 LR “-” “-” 0 84 10001 - “IP” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36” “d99c7ea4-f9e6-4697-bf48-2678df6616cb” “EXTERNAL_IP” “IP:9080” outbound|9080||productpage.do-bookinfo-phani.svc.cluster.local - IP:8080 IP:57846 - -

I don’t see any logs on productpage istio-proxy - however i can access the productpage service from ratings, details ,etc… but not vice versa - not able to access rating, detatils, etc… services from productpage as shown below

#kubectl exec -it productpage-v1-6f9df695b7-bspbb -c istio-proxy -n do-bookinfo-phani – bash
istio-proxy@productpage-v1-6f9df695b7-bspbb:/$ curl -v -I ratings:9080

  • Rebuilt URL to: ratings:9080/
  • Trying ip…
  • TCP_NODELAY set
  • Connected to ratings (ip) port 9080 (#0)

HEAD / HTTP/1.1
Host: ratings:9080
User-Agent: curl/7.58.0
Accept: /

< HTTP/1.1 503 Service Unavailable
HTTP/1.1 503 Service Unavailable
< content-length: 95
content-length: 95
< content-type: text/plain
content-type: text/plain
< date: Mon, 08 Mar 2021 10:43:10 GMT
date: Mon, 08 Mar 2021 10:43:10 GMT
< server: istio-envoy
server: istio-envoy
< x-envoy-decorator-operation: ratings.do-bookinfo-phani.svc.cluster.local:9080/*
x-envoy-decorator-operation: ratings.do-bookinfo-phani.svc.cluster.local:9080/*

<

  • Connection #0 to host ratings left intact

Below error is showing in rating istio-proxy.
[2021-03-08T10:34:34.267Z] “HEAD / HTTP/1.1” 503 UC “-” “-” 0 0 120097 - “-” “curl/7.58.0” “dc67f769-30e2-4ac7-b0ae-c747989c2211” “ratings:9080” “127.0.0.1:9080” inbound|9080|http|ratings.do-bookinfo-phani.svc.cluster.local 127.0.0.1:53328 240.0.13.31:9080 240.0.6.46:34288 - default


NAME CDS LDS EDS RDS ISTIOD VERSION
details-v1-5f449bdbb9-j47l9.do-bookinfo-phani SYNCED SYNCED SYNCED SYNCED istiod-7bf95f9-75sbz 1.7.0
istio-egressgateway-6bb8b89cb4-fp8n2.istio-system SYNCED SYNCED SYNCED NOT SENT istiod-7bf95f9-75sbz 1.7.0
istio-ingressgateway-6fbf7884dc-zp5rg.istio-system SYNCED SYNCED SYNCED SYNCED istiod-7bf95f9-75sbz 1.7.0
productpage-v1-6f9df695b7-bspbb.do-bookinfo-phani SYNCED SYNCED SYNCED SYNCED istiod-7bf95f9-75sbz 1.7.0
ratings-v1-857bb87c57-f26pm.do-bookinfo-phani SYNCED SYNCED SYNCED SYNCED istiod-7bf95f9-75sbz 1.7.0
reviews-v1-68f9c47f69-qc8pj.do-bookinfo-phani SYNCED SYNCED SYNCED SYNCED istiod-7bf95f9-75sbz 1.7.0
reviews-v2-5d56c488f5-tf69q.do-bookinfo-phani SYNCED SYNCED SYNCED SYNCED istiod-7bf95f9-75sbz 1.7.0
reviews-v3-869ff44845-vwh6c.do-bookinfo-phani SYNCED SYNCED SYNCED SYNCED istiod-7bf95f9-75sbz 1.7.0


127.0.0.1:9080 HEALTHY OK inbound|9080|http|productpage.do-bookinfo-phani.svc.cluster.local
ip:9080 HEALTHY OK outbound|9080||ratings.do-bookinfo-phani.svc.cluster.local
ip:9080 HEALTHY OK outbound|9080||details.do-bookinfo-phani.svc.cluster.local
ip:9080 HEALTHY OK outbound|9080||reviews.do-bookinfo-phani.svc.cluster.local
ip:9080 HEALTHY OK outbound|9080||reviews.do-bookinfo-phani.svc.cluster.local
ip:9080 HEALTHY OK outbound|9080||productpage.do-bookinfo-phani.svc.cluster.local
ip:9080 HEALTHY OK outbound|9080||reviews.do-bookinfo-phani.svc.cluster.local


#istioctl proxy-config cluster -n istio-system istio-ingressgateway-6fbf7884dc-zp5rg | grep bookinfo
details.do-bookinfo-phani.svc.cluster.local 9080 - outbound EDS
outbound_.9080_..details.do-bookinfo-phani.svc.cluster.local - - - EDS
outbound
.9080_..productpage.do-bookinfo-phani.svc.cluster.local - - - EDS
outbound
.9080_..ratings.do-bookinfo-phani.svc.cluster.local - - - EDS
outbound
.9080_._.reviews.do-bookinfo-phani.svc.cluster.local - - - EDS
productpage.do-bookinfo-phani.svc.cluster.local 9080 - outbound EDS
ratings.do-bookinfo-phani.svc.cluster.local 9080 - outbound EDS
reviews.do-bookinfo-phani.svc.cluster.local 9080 - outbound EDS


#istioctl x describe pod productpage-v1-6f9df695b7-bspbb -n do-bookinfo-phani
Pod: productpage-v1-6f9df695b7-bspbb
Pod Ports: 9080 (productpage), 15090 (istio-proxy)

Service: productpage
Port: http 9080/HTTP targets pod port 9080

Exposed on Ingress Gateway http://EXTERNAL_IP
VirtualService: bookinfo
/productpage, /static*, /login, /logout, /api/v1/products*


Can you please let me know what am i missing?

please let me know if more log info needed

PS: have not configured mTLS.