Intermittent connection to PostgreDB RDS

Networking
1

Hi,

We deployed Istio 1.13.5 on EKS with K8s 1.20.

There is no issue with egress to forward proxy and AWS Redis (Elasticache).
However, we have been having issue with intermittent connection with PostgreDB RDS.
In some deployments, the application could connect to the RDS, while in other deployments the application could not connect to the RDS. Hence, we can rule out network/security group issues.

We noticed the following when the app could not connect to the DB:

  • From the log, we noticed there was an error UF,URX:
    [2022-06-27T05:51:28.149Z] "- - -" 0 UF,URX - - "-" 0 0 10009 - "-" "-" "-" "-" "10.XXX.YYY.238:ABCD" outbound|ABCD||dbs-XXX.rds.amazonaws.com - 10.X.Y.238:ABCD 10.A.B.40:58848 - -

Following is the ServiceEntry:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  labels:
    app.kubernetes.io/instance: ne71-sa
spec:
  hosts:
  - dbs-XXX.rds.amazonaws.com
  location: MESH_EXTERNAL
  ports:
  - name: psql
    number: ABCD
    protocol: tcp
  resolution: DNS

Has anybody seen a similar behavior and the solutions?