I’ve been trying to set up a circuit breaker for an external service using a
DestinationRule and haven’t been able to get it working.
First, I set up a
ServiceEntry like this:
apiVersion: networking.istio.io/v1alpha3 kind: ServiceEntry metadata: name: vertex spec: hosts: - restconnect.vertexsmb.com ports: - number: 443 name: https protocol: HTTPS resolution: DNS location: MESH_EXTERNAL
Then, I added a
DestinationRule like this (basically copied from the tutorial for testing):
apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: vertex spec: host: vertex trafficPolicy: connectionPool: tcp: maxConnections: 1 http: http1MaxPendingRequests: 1 maxRequestsPerConnection: 1 outlierDetection: consecutiveErrors: 1 interval: 1s baseEjectionTime: 3m maxEjectionPercent: 100
Then I ran
fortio (with the istio-proxy sidecar) to see if I get some 503s with concurrent connections:
kubectl exec -it $FORTIO_POD -c fortio /usr/bin/fortio -- load -c 3 -qps 0 -n 30 -loglevel Warning https://restconnect.vertexsmb.com/test
And it only returned 404s (expected since the endpoint I’m hitting doesn’t actually exist). I expected at least some % of 503s due to there being many concurrent connections.
I noticed the tutorial was using a plain Kubernetes
Service to do circuit breaker but the documentation for a
DestinationRule mentions that
ServiceEntry hosts can be used as well.
Furthermore, I did try using a
Service entry (specifically an
ExternalName) but that did not work either. Here’s the
Service for reference:
kind: Service apiVersion: v1 metadata: name: vertex spec: type: ExternalName externalName: restconnect.vertexsmb.com
This is further complicated by the fact that I can’t access the
ExternalName via https since it gives a certificate mismatch error. The host is interpreted as
vertex but the certificate is only valid for
*.vertexsmb.com so trying to hit that endpoint gives back SSL errors. Requesting the http endpoint returns 400s though so I was able to test the circuit breaker with that. Again, I expected 503s but only got 400s.
I feel like I’m missing something obvious. I was able to replicate 503s following the tutorial with httpbin so I know Istio is configured fine.
For context, I’m using Istio 1.2 on Kubernetes 1.12 (EKS).