Rate Limiting is not working in Istio 1.9

I have been trying to setup rate limiting using EnvoyFilters following the steps mentioned in this doc https://istio.io/latest/docs/tasks/policy-enforcement/rate-limit/.

But it’s not working as expected. Global rate limiting is not working in particular. It shows 500 response code for all the requests. But local rate limiting seems to be working fine.

This is the redis config that I am using

apiVersion: v1
kind: Service
metadata:
  name: redis
  labels:
    app: redis
spec:
  ports:
  - name: grpc
    port: 6379
    targetPort: 6379
    protocol: TCP   
  selector:
    app: redis

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - image: redis:alpine
        imagePullPolicy: Always
        name: redis
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "528Mi"
            cpu: "500m"
        ports:
        - name: redis
          containerPort: 6379
          protocol: TCP   
      restartPolicy: Always

And this is the config for the rate limit app. It’s based on this app GitHub - envoyproxy/ratelimit: Go/gRPC service designed to enable generic rate limit scenarios from different types of applications.

apiVersion: v1
kind: Service
metadata:
  name: ratelimit
  labels:
    app: ratelimit
spec:
  type: ClusterIP
  ports:
    - port: 6070
      targetPort: http-debug
      protocol: TCP
      name: http-debug
    - port: 8081
      targetPort: grpc-server
      protocol: TCP
      name: grpc-server
    - port: 8080
      targetPort: http-server
      protocol: TCP
      name: http-server
  selector:
    app: ratelimit
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ratelimit
  labels:
    app: ratelimit
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ratelimit
  template:
    metadata:
      labels:
        app: ratelimit
    spec:
      containers:
        - env:
          - name: LOG_LEVEL
            value: debug
          - name: LOG_FORMAT
            value: JSON
          - name: REDIS_SOCKET_TYPE
            value: tcp
          - name: REDIS_URL
            value: redis.default.svc.cluster.local:6379
          - name: USE_STATSD
            value: "false"
          - name: RUNTIME_ROOT
            value: /data
          - name: RUNTIME_SUBDIRECTORY
            value: ratelimit
          name: ratelimit
          image: envoyproxy/ratelimit:v1.4.0
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: 500m
              memory: 528Mi
            requests:
              cpu: 250m
              memory: 64Mi
          command: ["/bin/ratelimit"]
          ports:
          - name: http-debug 
            containerPort: 6070
          - name: grpc-server
            containerPort: 8081
          - name: http-server
            containerPort: 8080
          volumeMounts:
          - name: commonconfig-volume
            mountPath: /data/ratelimit/config/config.yaml
            subPath: config.yaml
      volumes:
        - name: commonconfig-volume
          configMap:
            name: ratelimit-config

The EnvoyFilters and the ConfigMap are used from the Isito doc mentioned earlier. Any help would be much appreciated.

can you provide more information about your setup? where are you seeing the 500s? what is the response? can you post the proxy routes and listeners?

My limit doesn’t work either. Here’s the complete configuration:(istio 1.9)

**service.yam**
        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: ratelimit-config
        data:
          config.yaml: |
            domain: echo-ratelimit
            descriptors:
              - key: PATH
                value: "/"
                rate_limit:
                  unit: minute
                  requests_per_unit: 1
              - key: PATH
                rate_limit:
                  unit: minute
                  requests_per_unit: 1
        ---
        apiVersion: v1
        kind: Service
        metadata:
          name: redis
          labels:
            app: redis
        spec:
          ports:
          - name: grpc
            port: 6379
            targetPort: 6379
            protocol: TCP
          selector:
            app: redis

        ---

        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: redis
        spec:
          replicas: 1
          selector:
            matchLabels:
              app: redis
          template:
            metadata:
              labels:
                app: redis
            spec:
              containers:
              - image: redis:alpine
                imagePullPolicy: Always
                name: redis
                resources:
                  requests:
                    memory: "64Mi"
                    cpu: "250m"
                  limits:
                    memory: "528Mi"
                    cpu: "500m"
                ports:
                - name: redis
                  containerPort: 6379
                  protocol: TCP
              restartPolicy: Always
    ---
        apiVersion: v1
        kind: Service
        metadata:
          name: ratelimit
          labels:
            app: ratelimit
        spec:
          ports:
          - name: "8080"
            port: 8080
            targetPort: 8080
            protocol: TCP
          - name: "8081"
            port: 8081
            targetPort: 8081
            protocol: TCP
          - name: "6070"
            port: 6070
            targetPort: 6070
            protocol: TCP
          selector:
            app: ratelimit
    ---
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: ratelimit
        spec:
          replicas: 1
          selector:
            matchLabels:
              app: ratelimit
          strategy:
            type: Recreate
          template:
            metadata:
              labels:
                app: ratelimit
            spec:
              containers:
              - image: envoyproxy/ratelimit:v1.4.0
                imagePullPolicy: Always
                name: ratelimit
                command: ["/bin/ratelimit"]
                env:
                - name: LOG_LEVEL
                  value: debug
                - name: REDIS_SOCKET_TYPE
                  value: tcp
                - name: REDIS_URL
                  value: redis.maixiaolan-test.svc.cluster.local:6379
                - name: USE_STATSD
                  value: "false"
                - name: RUNTIME_ROOT
                  value: /data
                - name: RUNTIME_SUBDIRECTORY
                  value: ratelimit
                ports:
                - containerPort: 8080
                - containerPort: 8081
                - containerPort: 6070
                volumeMounts:
                - name: config-volume
                  mountPath: /data/ratelimit/config/config.yaml
                  subPath: config.yaml
              volumes:
              - name: config-volume
                configMap:
                  name: ratelimit-config

----------------------------


**filter.yaml**

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: filter-ratelimit
  namespace: istio-system
spec:
  workloadSelector:
    # select by label in the same namespace
    labels:
      istio: ingressgateway
  configPatches:
    # The Envoy config you want to modify
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: "envoy.http_connection_manager"
              subFilter:
                name: "envoy.router"
      patch:
        operation: INSERT_BEFORE
        # Adds the Envoy Rate Limit Filter in HTTP filter chain.
        value:
          name: envoy.filters.http.ratelimit
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
            # domain can be anything! Match it to the ratelimter service config
            domain: echo-ratelimit
            failure_mode_deny: true
            rate_limit_service:
              grpc_service:
                envoy_grpc:
                  cluster_name: rate_limit_cluster
                timeout: 10s
              transport_api_version: V3
    - applyTo: CLUSTER
      match:
        cluster:
          service: ratelimit.maixiaolan-test.svc.cluster.local
      patch:
        operation: ADD
        # Adds the rate limit service cluster for rate limit service defined in step 1.
        value:
          name: rate_limit_cluster
          type: STRICT_DNS
          connect_timeout: 10s
          lb_policy: ROUND_ROBIN
          http2_protocol_options: {}
          load_assignment:
            cluster_name: rate_limit_cluster
            endpoints:
            - lb_endpoints:
              - endpoint:
                  address:
                     socket_address:
                      address: ratelimit.maixiaolan-test.svc.cluster.local
                      port_value: 8081
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: filter-ratelimit-svc
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: VIRTUAL_HOST
      match:
        context: GATEWAY
        routeConfiguration:
          vhost:
            name: "*:58080"
            route:
              action: ANY
      patch:
        operation: MERGE
        # Applies the rate limit rules.
        value:
          rate_limits:
            - actions: # any actions in here
              - request_headers:
                  header_name: ":path"
                  descriptor_key: "PATH"

@nick_tetrate I am using the book info sample application

kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

I am using the EnvoyFilters exactly as provided in the istio doc Istio / Enabling Rate Limits using Envoy

apiVersion: v1
kind: ConfigMap
metadata:
  name: ratelimit-config
data:
  config.yaml: |
    domain: productpage-ratelimit
    descriptors:
      - key: PATH
        value: "/productpage"
        rate_limit:
          unit: minute
          requests_per_unit: 1
      - key: PATH
        rate_limit:
          unit: minute
          requests_per_unit: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: filter-ratelimit
  namespace: istio-system
spec:
  workloadSelector:
    # select by label in the same namespace
    labels:
      istio: ingressgateway
  configPatches:
    # The Envoy config you want to modify
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: "envoy.http_connection_manager"
              subFilter:
                name: "envoy.router"
      patch:
        operation: INSERT_BEFORE
        # Adds the Envoy Rate Limit Filter in HTTP filter chain.
        value:
          name: envoy.filters.http.ratelimit
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
            # domain can be anything! Match it to the ratelimter service config
            domain: productpage-ratelimit
            failure_mode_deny: true
            rate_limit_service:
              grpc_service:
                envoy_grpc:
                  cluster_name: rate_limit_cluster
                timeout: 10s
              transport_api_version: V3
    - applyTo: CLUSTER
      match:
        cluster:
          service: ratelimit.default.svc.cluster.local
      patch:
        operation: ADD
        # Adds the rate limit service cluster for rate limit service defined in step 1.
        value:
          name: rate_limit_cluster
          type: STRICT_DNS
          connect_timeout: 10s
          lb_policy: ROUND_ROBIN
          http2_protocol_options: {}
          load_assignment:
            cluster_name: rate_limit_cluster
            endpoints:
            - lb_endpoints:
              - endpoint:
                  address:
                     socket_address:
                      address: ratelimit.default.svc.cluster.local
                      port_value: 8081
---

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: filter-ratelimit-svc
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: VIRTUAL_HOST
      match:
        context: GATEWAY
        routeConfiguration:
          vhost:
            name: "*:80"
            route:
              action: ANY
      patch:
        operation: MERGE
        # Applies the rate limit rules.
        value:
          rate_limits:
            - actions: # any actions in here
              - request_headers:
                  header_name: ":path"
                  descriptor_key: "PATH"

NOTE

When I send curl request to /productpage, I get 500 response code from all the request. When I make the failure_mode_deny: false, it gives a 200 response for all request.

maybe look here? RateLimit v3 with 1.9.0 Kills the Ingressgatway · Issue #30900 · istio/istio · GitHub

1 Like

@nick_tetrate Thanks for the link. Followed the instructions in the comments and now it’s working.

@catman002 and anyone coming here with the same or similar issue, this is how my issue was resolved

Use a docker image from the docker hub Docker Hub with one of the latest build. I used envoyproxy/ratelimit:40393342 docker image and it’s working.

It seems the docker image that I was using and most of the docs and tutorials mention and even the envoy rate limit docker-compose uses envoyproxy/ratelimit:v1.4.0 is really outdated. And it does not support V3 filters. Even after adding transport_api_version: V3 to the filter does not seems to work. Adding transport_api_version: V3 seems suppress the error messages in the logs, but it starts returning 500 error for all requests.

2 Likes

@fai555
kubectl describe pod ratelimit-76cdf9659-zkrz9
error: Back-off restarting failed container

Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  55s                 default-scheduler  Successfully assigned maixiaolan-test/ratelimit-76cdf9659-zkrz9 to node2
  Normal   Pulling    2m14s               kubelet            Pulling image "docker.io/istio/proxyv2:1.9.0"
  Normal   Pulled     2m4s                kubelet            Successfully pulled image "docker.io/istio/proxyv2:1.9.0" in 10.408854639s
  Normal   Started    2m3s                kubelet            Started container istio-init
  Normal   Created    2m3s                kubelet            Created container istio-init
  Normal   Pulled     116s                kubelet            Successfully pulled image "envoyproxy/ratelimit:40393342" in 6.349474151s
  Normal   Pulling    115s                kubelet            Pulling image "docker.io/istio/proxyv2:1.9.0"
  Normal   Pulled     108s                kubelet            Successfully pulled image "docker.io/istio/proxyv2:1.9.0" in 7.512116491s
  Normal   Started    107s                kubelet            Started container istio-proxy
  Normal   Created    107s                kubelet            Created container istio-proxy
  Warning  Unhealthy  105s                kubelet            Readiness probe failed: Get "http://10.244.104.5:15021/healthz/ready": dial tcp 10.244.104.5:15021: connect: connection refused
  Normal   Pulled     100s                kubelet            Successfully pulled image "envoyproxy/ratelimit:master" in 5.981558377s
  Normal   Pulling    86s (x3 over 2m2s)  kubelet            Pulling image "envoyproxy/ratelimit:master"
  Normal   Pulled     70s                 kubelet            Successfully pulled image "envoyproxy/ratelimit:master" in 16.545751943s
  Normal   Created    69s (x3 over 115s)  kubelet            Created container ratelimit
  Normal   Started    69s (x3 over 115s)  kubelet            Started container ratelimit
  Warning  BackOff    54s (x4 over 99s)   kubelet            Back-off restarting failed container

my ratelimit.yml:

    spec:
      containers:
      - image: envoyproxy/ratelimit:40393342
        imagePullPolicy: Always
        name: ratelimit
        command: ["/bin/ratelimit"]
        env:
        - name: LOG_LEVEL
          value: debug
        - name: REDIS_SOCKET_TYPE
          value: tcp
        - name: REDIS_URL
          value: redis:6379
        - name: USE_STATSD
          value: "false"
        - name: RUNTIME_ROOT
          value: /data
        - name: RUNTIME_SUBDIRECTORY
          value: ratelimit
        ports:
        - containerPort: 8080
        - containerPort: 8081
        - containerPort: 6070

[root@node0 bak]# kubectl logs ratelimit-57bd7cb68b-ml6nw


time="2021-03-02T00:54:26Z" level=warning msg="statsd is not in use"
{"@message":"runtime changed. loading new snapshot at /data/ratelimit","@timestamp":"2021-03-02T00:54:26.759277459Z","level":"debug"}
{"@message":"runtime: processing /data/ratelimit","@timestamp":"2021-03-02T00:54:26.759371361Z","level":"debug"}
{"@message":"runtime: processing /data/ratelimit/config","@timestamp":"2021-03-02T00:54:26.759418029Z","level":"debug"}
{"@message":"runtime: processing /data/ratelimit/config/config.yaml","@timestamp":"2021-03-02T00:54:26.759437261Z","level":"debug"}
{"@message":"runtime: adding key=config.config.yaml value=domain: echo-ratelimit\ndescriptors:\n  - key: PATH\n    value: \"/\"\n    rate_limit:\n      unit: minute\n      requests_per_unit: 1\n  - key: PATH\n    rate_limit:\n      unit: minute\n      requests_per_unit: 1\n uint=false","@timestamp":"2021-03-02T00:54:26.759484386Z","level":"debug"}
{"@message":"connecting to redis on redis.maixiaolan-test.svc.cluster.local:6379 with pool size 10","@timestamp":"2021-03-02T00:54:26.759731609Z","level":"warning"}
{"@message":"Implicit pipelining enabled: false","@timestamp":"2021-03-02T00:54:26.759785423Z","level":"debug"}
panic: EOF

goroutine 1 [running]:
github.com/envoyproxy/ratelimit/src/redis.checkError(...)
	/ratelimit/src/redis/driver_impl.go:51
github.com/envoyproxy/ratelimit/src/redis.NewClientImpl(0xd67c80, 0xc0000b6ba0, 0x0, 0x0, 0x0, 0xac81c8, 0x6, 0xc00003c00a, 0x2c, 0xa, ...)
	/ratelimit/src/redis/driver_impl.go:121 +0xbbd
github.com/envoyproxy/ratelimit/src/redis.NewRateLimiterCacheImplFromSettings(0x0, 0x0, 0x1f90, 0x1f91, 0x17b6, 0xc00003801a, 0x5, 0xc00003802b, 0x4, 0x0, ...)
	/ratelimit/src/redis/cache_impl.go:20 +0x127
github.com/envoyproxy/ratelimit/src/service_cmd/runner.createLimiter(0xd64e60, 0xc0002de000, 0x0, 0x0, 0x1f90, 0x1f91, 0x17b6, 0xc00003801a, 0x5, 0xc00003802b, ...)
	/ratelimit/src/service_cmd/runner/runner.go:50 +0x25b
github.com/envoyproxy/ratelimit/src/service_cmd/runner.(*Runner).Run(0xc00000a1e0)
	/ratelimit/src/service_cmd/runner/runner.go:101 +0x348
main.main()
	/ratelimit/src/service_cmd/main.go:10 +0x203

@fai555: Hi, it looks like I have an issue similar to yours but I am unable to make ratelimit deployment running: Istio 1.9.0 - unable to configure rate limit example. My configuration seems to be similar, so could you provide more information about the solution that helped you? I’ve tried envoyproxy/ratelimit:40393342 but it keeps crashing.

When I tried running the demo in this repo,

transport_api_version: V3

was missing. Adding that resolved the crashing.

Check if your EnvoyFilter has the

transport_api_version: V3

set.

rate limit service is used in current version v1.10,need to restart ratelimit pod for the ConfigMap update to take effect.
is there any configuration that will make the ConfigMap update take effect immediately?
ref: the rate limit service update ConfigMap does not take effect · Issue #34890 · istio/istio · GitHub

another question
i made a request from the sleep pod,the ratelimit service counted two requests

i don’t know what caused this, can anyone tell me?

curl -vv http://details.sun-system:9080/details/0
time="2021-08-27T03:37:55Z" level=debug msg="got descriptor: (PATH=/details/0)"
time="2021-08-27T03:37:55Z" level=debug msg="starting get limit lookup"
time="2021-08-27T03:37:55Z" level=debug msg="looking up key: PATH_/details/0"
time="2021-08-27T03:37:55Z" level=debug msg="looking up key: PATH"
time="2021-08-27T03:37:55Z" level=debug msg="found rate limit: PATH"
time="2021-08-27T03:37:55Z" level=debug msg="applying limit: 10 requests per MINUTE"
time="2021-08-27T03:37:55Z" level=debug msg="starting cache lookup"
time="2021-08-27T03:37:55Z" level=debug msg="looking up cache key: productpage-ratelimit_PATH_/details/0_1630035420"
time="2021-08-27T03:37:55Z" level=debug msg="cache key: productpage-ratelimit_PATH_/details/0_1630035420 current: 1"
time="2021-08-27T03:37:55Z" level=debug msg="returning normal response"
time="2021-08-27T03:37:55Z" level=debug msg="got descriptor: (PATH=/details/0)"
time="2021-08-27T03:37:55Z" level=debug msg="starting get limit lookup"
time="2021-08-27T03:37:55Z" level=debug msg="looking up key: PATH_/details/0"
time="2021-08-27T03:37:55Z" level=debug msg="looking up key: PATH"
time="2021-08-27T03:37:55Z" level=debug msg="found rate limit: PATH"
time="2021-08-27T03:37:55Z" level=debug msg="applying limit: 10 requests per MINUTE"
time="2021-08-27T03:37:55Z" level=debug msg="starting cache lookup"
time="2021-08-27T03:37:55Z" level=debug msg="looking up cache key: productpage-ratelimit_PATH_/details/0_1630035420"
time="2021-08-27T03:37:55Z" level=debug msg="cache key: productpage-ratelimit_PATH_/details/0_1630035420 current: 2"
time="2021-08-27T03:37:55Z" level=debug msg="returning normal response"
time="2021-08-27T03:37:55Z" level=debug msg="[gostats] flushing counter ratelimit.go.mallocs: 968"
time="2021-08-27T03:37:55Z" level=debug msg="[gostats] flushing counter ratelimit.go.frees: 69"
time="2021-08-27T03:37:55Z" level=debug msg="[gostats] flushing counter ratelimit.service.rate_limit.productpage-ratelimit.PATH.total_hits: 2"

This is not a ratelimit or istio specific thing. It’s how K8s ConfigMap and Deployment works. To the best of my knowledge, the Deployment has to be restarted if you want the configmap change to take effect.

One of the ways that kustomize solves this problem is to automatically add a name suffix depending on the hash of the configmap content. And since the name of the ConfigMap updates and hence Deployment spec also changes, your deployment is restarted automatically upon

kustomize build /yaml-config/folder/path | kubectl apply -f -

We use this all the time to solve this problem. You can have a look into kustomization documentation.

kustomize can help us restart automatically. is there a better way? don’t restart

@Dui_Liu Let me know if you find a way without restarting the pods. :joy: I have searched for months. Then settled down with the Kustomize based approach.

look here: Enable ratelimit config reload automatically when the configMap has been updated by devincd · Pull Request #34948

this PR might help us Enable ratelimit config reload automatically when the configMap has been updated by devincd · Pull Request #34948

Yes, see my comment here Global ratelimit not working · Issue #32381 · istio/istio · GitHub

can rate limit be used on service? always return HTTP code 200

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: filter-local-ratelimit-svc-httpbin
  namespace: boook
spec:
  workloadSelector:
    labels:
      app: httpbin
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.local_ratelimit
          typed_config:
            "@type": type.googleapis.com/udpa.type.v1.TypedStruct
            type_url: type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
            value:
              stat_prefix: http_local_rate_limiter
              token_bucket:
                max_tokens: 10
                tokens_per_fill: 10
                fill_interval: 60s
              filter_enabled:
                runtime_key: local_rate_limit_enabled
                default_value:
                  numerator: 100
                  denominator: HUNDRED
              filter_enforced:
                runtime_key: local_rate_limit_enforced
                default_value:
                  numerator: 100
                  denominator: HUNDRED
              response_headers_to_add:
                - append: false
                  header:
                    key: x-local-rate-limit
                    value: 'true'
              descriptors:
                - entries:
                  - key: path
                    value: /ip
                  token_bucket:
                    max_tokens: 1
                    tokens_per_fill: 1
                    fill_interval: 60s

check envoyfilter config

[root@10-20-11-81 bookinfo]# istioctl pc listener httpbin-66cdbdb6c5-zjfwq.boook -oyaml | grep -5 /ip
            typeUrl: type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
            value:
              descriptors:
              - entries:
                - key: path
                  value: /ip
                token_bucket:
                  fill_interval: 60s
                  max_tokens: 1
                  tokens_per_fill: 1
              filter_enabled: