Standard service to Stateful service communication error 503

Networking
1

Hi,

I have a technical difficulty, with Istio service to service communication.
First I think that the problem comes from mTLS but this is not the case (I disabled it)…

I search the internet and found that the old version of Istio had problems with Stateful, but I won’t be able to find a solution!

I use Istio 1.13.2 I configure HTTP Redirector, HTTPS, JWK and route some paths to several services, it works fine.

mTLS is disabled and I am working on my own namespace.

Standard services can communicate with other services and with my StatefulSet Mongodb using TCP clients.

My problem:
I create an HTTP server as a StatefulSet service listed on the pod as follows:
my-app-0 2/2 Running 2 (48m ago) 54m

If I add an entry on the virtual service, I can access it on port 80.

If I do port forwarding, I can access it on port 2002→8080.

Everything seems to work fine, but when I try to access it from a standard service (not stateful) using curl like this:
kubectl -n my-namespace exec my-service-fdd8cb667-mgd5h -c app – curl -v “my-app-0.my-app.my-namespace.svc.cluster.local:80/version”

The client sends me this error: 0upstream connect error or disconnect/reset before headers. Reset

And the proxy gives this server-side error:
[2022-05-01T14:32:18.703Z] “GET /version HTTP/1.1” 503 UF upstream_reset_before_response_started{connection_failure,delayed_connect_error:111} - “-” 0 145 0 - “-” “curl/7.64.0” “b7f96317-d31d-42c7-8090-3dda0e773d89” “my-app-0.my-app.my-namespace.svc.cluster.local” “172.17.0.14:80” InboundPassthroughClusterIpv4 - 172.17.0.14:80 172.17.0.15:51438 outbound.80_._.my-app.my-namespace.svc.cluster.local default
So the connection is established but something is broken on proxy?!

apiVersion: v1
kind: ServiceAccount
metadata:
name: my-app
namespace: my-namespace

apiVersion: v1
kind: Service
metadata:
name: my-app
namespace: my-namespace
labels:
app: my-app
spec:
clusterIP: None # Headless.
ports:

  • name: http-web
    port: 80
    targetPort: 8080
    selector:
    app: my-app

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: my-app
namespace: my-namespace
labels:
app: my-app
spec:
selector:
matchLabels:
app: my-app
serviceName: my-app
replicas: 1
template:
metadata:
labels:
app: my-app
version: 0.0.1-beta2
annotations:
sidecar.istio.io/rewriteAppHTTPProbers: “false”
spec:
serviceAccountName: my-app
containers:
- env:
- name: KUBERNETES_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: my-app:latest
imagePullPolicy: Never
name: my-app
ports:
- containerPort: 8080
securityContext:
privileged: false

readinessProbe:

httpGet:

path: /ready

port: 8080

periodSeconds: 5

failureThreshold: 10

livenessProbe:

httpGet:

path: /live

port: 8080

initialDelaySeconds: 10

periodSeconds: 30

failureThreshold: 2

—

Thanks for your help,

2

Hi, any help after several weeks…sad…

Is there somewhere on ISTIO examples a sample to test (direct internal cluster) communication between a “deployment” and a “Headless service” using an HTTP request (with or without mTLS enabled)?

Thanks for your help,

3

Hi,

More information: If I call using Curl/HTTP the “StatefulSet” service located in “my-namespace” from a “deployment” service located in the “default” namespace using is IP it works…
Note: “deployment” to “deployment” on “my-namespace” work too.

I don’t have any “DestinationRules” in “default” and “my-namespace”, and mTLS is not enabled…

According to the documentation there is no restriction of access by default… I misunderstood something?

Regards

Note: I have “AuthorizationPolicy” & “RequestAuthentication” on “istio-system” / matchLabels: istio: ingressgateway and I have a “VirtualService” on “my-namespace”…

4

I create a ticket here: "Deployment" to "StatefulSet" HTTP communication don’t work across same namespace... · Issue #11357 · istio/istio.io · GitHub
and add a “circumvention”…
Regards

5

You could try to follow the official 503 error doc to fix.