Intermittent connection reset error on external SSL connections

Networking
1

We are seeing intermittent (max 3 per hour) “Connection reset” error from Java code with following stack trace.
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[?:1.8.0_252]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_252]
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[?:1.8.0_252]
at sun.security.ssl.InputRecord.read(InputRecord.java:503) ~[?:1.8.0_252]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:990) ~[?:1.8.0_252]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1388) ~[?:1.8.0_252]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1416) ~[?:1.8.0_252]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1400) ~[?:1.8.0_252]

Curl command running in same container is also getting same error.
* Connected to host_masked (ip_masked) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* NSS error -5961 (PR_CONNECT_RESET_ERROR)
* TCP connection reset by peer

This host is open to public on internet.
Istio version is 1.5.8.
There is no egress gateway.
Istio treats these URLs as PassthroughCluster as seen from logs.

We don’t get any error when we run the same script on same k8s worker node, or on a standalone EC2 node.

As per AWS support there is no issue with NAT gateway or the ELB.

Has anybody seen a similar behavior?

Thanks!

2

We were getting this error because default connection timeout for passthroughcluster is 1s.
We increased it to 5s via EnvoyFilter.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata: 
  name: external-connect-timeout-filter
  namespace: {{ .Release.Namespace }}
spec: 
  configPatches: 
  - applyTo: CLUSTER
    match: 
      #PassthroughCluster
      #connectTimeout
      context: SIDECAR_OUTBOUND
      cluster: 
        name: "PassthroughCluster"
    patch: 
      operation: MERGE
      value: 
        connectTimeout: 5s
1 Like
3

@Sameer_Naik we are also having same issue getting below error. I tried to add Envoy Filter but still it is not working. Any suggestions?

org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://myendpoint": Connection reset; nested exception is javax.net.ssl.SSLException: Connection reset at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:748) ~[spring-web-5.2.8.RELEASE.jar!/:5.2.8.RELEASE] at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:641) ~[spring-web-5.2.8.RELEASE.jar!/:5.2.8.RELEASE] at com.charter.scp.scl.commons.client.diplomata.auth.AuthenticatedRequestorBase.lambda$request$1(AuthenticatedRequestorBase.java:136) ~[scl-client-commons-1.12.0.jar!/:1.12.0] at net.jodah.failsafe.Functions.lambda$resultSupplierOf$11(Functions.java:283) ~[failsafe-2.0.1.jar!/:2.0.1] at net.jodah.failsafe.internal.executor.RetryPolicyExecutor.lambda$supply$0(RetryPolicyExecutor.java:67) ~[failsafe-2.0.1.jar!/:2.0.1] at net.jodah.failsafe.Execution.executeSync(Execution.java:117) ~[failsafe-2.0.1.jar!/:2.0.1] at net.jodah.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:319) ~[failsafe-2.0.1.jar!/:2.0.1] at net.jodah.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:71) ~[failsafe-2.0.1.jar!/:2.0.1]

Here is the envoy filter which i applied in my application namespace

   ```
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata: 
  name: external-connect-timeout-filter
  namespace: {my-app-namespace}
spec: 
  configPatches: 
  - applyTo: CLUSTER
    match: 
      #PassthroughCluster
      #connectTimeout
      context: SIDECAR_OUTBOUND
      cluster: 
        name: "InboundPassthroughClusterIpv4"
    patch: 
      operation: MERGE
      value: 
        connectTimeout: 10s
4

Hi Guys , Am facing similar issue. One of the microservices unable to connect to another service. This is Intermittent issue as well and started after migrating Istio mesh (Could be co-incident) Also works perfectly fine in dev environment and Tst faces this issue.

org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://servicename-tst101.com/preferences/5266": Connection reset; nested exception is javax.net.ssl.SSLException: Connection reset
com.lm.pt.commonelements.errorHandling.AppException: Unable to Connect to service
at com.servicename.commonelements.errorHandling.AppExceptionHandler.handleException(AppExceptionHandler.java:52) ~[Java.pt.java.common-0.0.163.jar!/:na]

Any solutions found ? Please share it. Many thanks

5

Hi @Sameer_Naik. I have a question. Regarding the EnvoyFilter, did it already exist before you apply the timeout or it was added later because you needed to apply the timeout?

6

connectTimeout is already set as 10s in some version (i think from 1.7 istio)