We are using istio ingress gateway in front of a Docker registry (Docker/Distribution) that serves large blobs of data in long-running connections.
When using istio-ingress-gateway we have hit 2 issues.
- We appear to often get connection hangs which sometimes resume after an amount of time.
- When this happens we have noticed that there appears to be a 15s idle connection timeout to the gateway. We can recreate the 15s timeout with a simple
nc -v <IP> 443
We have not configured anything on the gateway in terms of timeout so would not expect the default to be 15s as the envoy and istio documentation show that it should default to much higher than this. We also did a config_dump for istio gateway and found that
stream_idle_timeout was set to 0s.
We see neither of these issues when we switch back to an Nginx ingress but leave the rest of the microservices in the mesh, this coupled with the client-side symptom being an EOF makes me think this points to a gateway configuration issue.
We have a reliable recreate of the hangs and EOF (believe this is caused by a hang of more than 15s) which is to do a
skopeo copy from one location to another in the same Registry.
We have both the Istio and Nginx Ingress set up at the same time and have tried the following combinations:
Combinations that fail:
- Both the pull and push going via the Istio ingress gateway
- Pull via istio and push via Nginx
Combinations that work reliably:
- Pull via Nginx and Push via Istio
- Pull from Istio Push to local disk (Possible key difference is skopeo does this one layer at a time in stated 5 layers in parallel)
Config I think will be relevant.
apiVersion: networking.istio.io/v1beta1 kind: Gateway metadata: creationTimestamp: "2020-05-29T14:59:57Z" generation: 2 name: front-door namespace: istio-system spec: selector: istio: ingressgateway servers: - hosts: <redacted> port: name: https number: 8443 protocol: HTTPS tls: mode: SIMPLE privateKey: /etc/istio/ingressgateway-certs/tls.key serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
apiVersion: networking.istio.io/v1beta1 kind: VirtualService metadata: annotations: meta.helm.sh/release-namespace: default generation: 3 labels: app.kubernetes.io/managed-by: Helm name: registry-front-door namespace: istio-system spec: gateways: - front-door hosts: - '*' http: - match: - uri: prefix: /v2 route: - destination: host: registry-v2.default.svc.cluster.local port: number: 8080 headers: request: set: ingress-type: front-door x-envoy-force-trace: "true" response: remove: - x-envoy-upstream-service-time - x-envoy-force-trace - x-server-node - server set: cache-control: no-cache, no-store docker-distribution-api-version: registry/2.0 expires: "0" pragma: no-cache strict-transport-security: max-age=31536000; includeSubDomains x-registry-supports-signatures: "1" x-xss-protection: 1; mode=block
apiVersion: networking.istio.io/v1beta1 kind: DestinationRule metadata: labels: app.kubernetes.io/managed-by: Helm name: default namespace: istio-system spec: host: '*.local' trafficPolicy: tls: mode: ISTIO_MUTUAL