We are using istio ingress gateway in front of a Docker registry (Docker/Distribution) that serves large blobs of data in long-running connections.
When using istio-ingress-gateway we have hit 2 issues.
- We appear to often get connection hangs which sometimes resume after an amount of time.
- When this happens we have noticed that there appears to be a 15s idle connection timeout to the gateway. We can recreate the 15s timeout with a simple
nc -v <IP> 443
We have not configured anything on the gateway in terms of timeout so would not expect the default to be 15s as the envoy and istio documentation show that it should default to much higher than this. We also did a config_dump for istio gateway and found that stream_idle_timeout was set to 0s.
We see neither of these issues when we switch back to an Nginx ingress but leave the rest of the microservices in the mesh, this coupled with the client-side symptom being an EOF makes me think this points to a gateway configuration issue.
We have a reliable recreate of the hangs and EOF (believe this is caused by a hang of more than 15s) which is to do a skopeo copy from one location to another in the same Registry.
We have both the Istio and Nginx Ingress set up at the same time and have tried the following combinations:
Combinations that fail:
- Both the pull and push going via the Istio ingress gateway
- Pull via istio and push via Nginx
Combinations that work reliably:
- Pull via Nginx and Push via Istio
- Pull from Istio Push to local disk (Possible key difference is skopeo does this one layer at a time in stated 5 layers in parallel)
Config I think will be relevant.
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
creationTimestamp: "2020-05-29T14:59:57Z"
generation: 2
name: front-door
namespace: istio-system
spec:
selector:
istio: ingressgateway
servers:
- hosts:
<redacted>
port:
name: https
number: 8443
protocol: HTTPS
tls:
mode: SIMPLE
privateKey: /etc/istio/ingressgateway-certs/tls.key
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
VirtualService
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
annotations:
meta.helm.sh/release-namespace: default
generation: 3
labels:
app.kubernetes.io/managed-by: Helm
name: registry-front-door
namespace: istio-system
spec:
gateways:
- front-door
hosts:
- '*'
http:
- match:
- uri:
prefix: /v2
route:
- destination:
host: registry-v2.default.svc.cluster.local
port:
number: 8080
headers:
request:
set:
ingress-type: front-door
x-envoy-force-trace: "true"
response:
remove:
- x-envoy-upstream-service-time
- x-envoy-force-trace
- x-server-node
- server
set:
cache-control: no-cache, no-store
docker-distribution-api-version: registry/2.0
expires: "0"
pragma: no-cache
strict-transport-security: max-age=31536000; includeSubDomains
x-registry-supports-signatures: "1"
x-xss-protection: 1; mode=block
Destination Rule
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
labels:
app.kubernetes.io/managed-by: Helm
name: default
namespace: istio-system
spec:
host: '*.local'
trafficPolicy:
tls:
mode: ISTIO_MUTUAL