Outlier Detection: Pods are getting ejected one by one

Networking
1

Hi folks!

We are using outlier detection in the traffic policy of Istio in production. We have three replicas of hello-world deployment managed by Knative serving, running in a namespace. We have the below outlier detection configured for the host - hello-world-kl49t-00008.

Pods:

NAME                                                              READY   STATUS    RESTARTS   AGE
hello-world-kl49t-00008-deployment-58ccfkxb   3/3     Running   0          16m
hello-world-kl49t-00008-deployment-58cfgn27   3/3     Running   0          16m
hello-world-kl49t-00008-deployment-58clqsnl   3/3     Running   0          16m

Our configuration:

  host: hello-world-kl49t-00008
  trafficPolicy:
    outlierDetection:
      baseEjectionTime: 60s
      consecutive5xxErrors: 5
      interval: 20s
      maxEjectionPercent: 100

Observations:
We can see the pods are getting ejected one by one. That is, we see the ejection only after around 15 5xx calls. Is this the expected behavior?

Version:

$ istioctl version
client version: 1.17.1
control plane version: 1.16-dev
data plane version: 1.16-dev (44 proxies)

$ kubectl version --short
Client Version: v1.25.0
Kustomize Version: v4.5.7
Server Version: v1.24.1