Inconsistency in Rate Limiting

Policies and Telemetry
1

I am using istio of version 1.1.3, I am trying to implement rate limiting at service level. For testing I configured 10 request/minute against source ip. I am expecting istio will allow first 10 requests and blocks following requests for a minute. But istio blocks some requests in the first 10 requests.

I tried the above scenario in memquota and redisquota(fixed window, rolling window).

The following is my configuration

apiVersion: config.istio.io/v1alpha2
kind: handler
metadata:
  name: redishandler
  namespace: istio-system
spec:
  compiledAdapter: redisquota
  params:
    redisServerUrl: redis-master.cache.svc.cluster.local:6379
    connectionPoolSize: 10
    quotas:
    - name: requestcountquota.instance.istio-system
      maxAmount: 100
      validDuration: 60s
      bucketDuration: 500ms
      rateLimitAlgorithm: ROLLING_WINDOW
      overrides:
      - dimensions:
          destination: my-service
        maxAmount: 10
---
apiVersion: config.istio.io/v1alpha2
kind: instance
metadata:
  name: requestcountquota
  namespace: istio-system
spec:
  compiledTemplate: quota
  params:
    dimensions:
      source: request.headers["x-forwarded-for"] | "unknown"
      destination: destination.labels["app"] | destination.workload.name | "unknown"
      destinationVersion: destination.labels["version"] | "unknown"
---
apiVersion: config.istio.io/v1alpha2
kind: QuotaSpec
metadata:
  name: request-count
  namespace: istio-system
spec:
  rules:
  - quotas:
    - charge: 1
      quota: requestcountquota
---
apiVersion: config.istio.io/v1alpha2
kind: QuotaSpec
metadata:
  name: request-count
  namespace: istio-system
spec:
  rules:
  - quotas:
    - charge: 1
      quota: requestcountquota
---
apiVersion: config.istio.io/v1alpha2
kind: QuotaSpecBinding
metadata:
  name: request-count
  namespace: istio-system
spec:
  quotaSpecs:
  - name: request-count
    namespace: istio-system
  services:
  - name: my-service
    namespace: my-namespace
    ###  - service: '*'  # Uncomment this to bind *all* services to request-count
 ---
apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
  name: quota
  namespace: istio-system
spec:
  ### quota only applies if you are not logged in.
  ### match: match(request.headers["cookie"], "session=*") == false
  actions:
  - handler: redishandler
    instances:
    - requestcountquota
---

output:
{errorCode:6009,errorMessage:Symbol not found}
{errorCode:6009,errorMessage:Symbol not found}
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
{errorCode:6009,errorMessage:Symbol not found}
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
{errorCode:6009,errorMessage:Symbol not found}
{errorCode:6009,errorMessage:Symbol not found}
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
{errorCode:6009,errorMessage:Symbol not found}
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
{errorCode:6009,errorMessage:Symbol not found}
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
{errorCode:6009,errorMessage:Symbol not found}
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
{errorCode:6009,errorMessage:Symbol not found}
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota
RESOURCE_EXHAUSTED:Quota is exhausted for: requestcountquota

In the above output istio rate limiting allows 9 request in a minute, But the 9 requests are random, not first 9 requests. Is there a way to achieve the output I am expecting that allowing requests at beginning(not randomly allowing) and blocking the rest.

FYI: I am just using configuration taken from istio.io without modification.

2

What is the rate at which you are sending the requests?
What happens if you increase bucketDuration or change rateLimitAlgorithm to FIXED_WINDOW?

3

My request rate is ~2.3 per second, Yes I have tried both FIXED_WINDOW, ROLLING_WINDOW in redisquota. And I also tried with memquota, Got the same output.

4

This output seems to be about the expected behavior for a ROLLING_WINDOW with bucket size of 500ms.

Can you re-confirm that this does not change when changed to FIXED_WINDOW? Make sure the memquota and redisquotas are not both deployed at the same time.

5

Hi,
I tried with FIXED_WINDOW, But still the same. No change in output. And I also running redisquota alone.
Can you please suggest any starting point to debug this problem.

Thanks,
Siva.

6

@DataSweeper: You could be hitting bug https://github.com/istio/istio/issues/3028