Circuit breaker for external service

1

Hi all,

I’ve been trying to set up a circuit breaker for an external service using a DestinationRule and haven’t been able to get it working.

First, I set up a ServiceEntry like this:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: vertex
spec:
  hosts:
  - restconnect.vertexsmb.com
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
  location: MESH_EXTERNAL

Then, I added a DestinationRule like this (basically copied from the tutorial for testing):

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: vertex
spec:
  host: vertex
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 1
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
    outlierDetection:
      consecutiveErrors: 1
      interval: 1s
      baseEjectionTime: 3m
      maxEjectionPercent: 100

Then I ran fortio (with the istio-proxy sidecar) to see if I get some 503s with concurrent connections:

kubectl exec -it $FORTIO_POD  -c fortio /usr/bin/fortio -- load -c 3 -qps 0 -n 30 -loglevel Warning https://restconnect.vertexsmb.com/test

And it only returned 404s (expected since the endpoint I’m hitting doesn’t actually exist). I expected at least some % of 503s due to there being many concurrent connections.

I noticed the tutorial was using a plain Kubernetes Service to do circuit breaker but the documentation for a DestinationRule mentions that ServiceEntry hosts can be used as well.

Furthermore, I did try using a Service entry (specifically an ExternalName) but that did not work either. Here’s the Service for reference:

kind: Service
apiVersion: v1
metadata:
  name: vertex
spec:
  type: ExternalName
  externalName: restconnect.vertexsmb.com

This is further complicated by the fact that I can’t access the ExternalName via https since it gives a certificate mismatch error. The host is interpreted as vertex but the certificate is only valid for *.vertexsmb.com so trying to hit that endpoint gives back SSL errors. Requesting the http endpoint returns 400s though so I was able to test the circuit breaker with that. Again, I expected 503s but only got 400s.

I feel like I’m missing something obvious. I was able to replicate 503s following the tutorial with httpbin so I know Istio is configured fine.

For context, I’m using Istio 1.2 on Kubernetes 1.12 (EKS).

Thanks!

1 Like
2

Just wanted to give an update here on my own research.

Turns out Istio 1.2 outlier detection simply doesn’t work. Upgrading to Istio 1.3 seems to fix this issue based on my own tests using Minikube. connectionPool config does work though!

Furthermore, it looks like Istio only counts 502-504 errors towards circuit breaking, which is inconsistent in documentation. There are issues open to track this.

I ended up testing this using the following resources on httpbin:

---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: httpbin
spec:
  hosts:
  - httpbin.org
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: httpbin.org
  trafficPolicy:
    outlierDetection:
      consecutiveErrors: 3
      interval: 10s
      baseEjectionTime: 1m
      maxEjectionPercent: 100
      minHealthPercent: 0

This is pretty great for testing since you can have httpbin return whatever status codes you want to try to trigger the circuit breaker.

Relevant links: