EnvoyProxy Rate Limit Not Working in Istio 1.7

I’m not able to see rate limit applied in istio 1.7 by applying the following scripts.

---
apiVersion: v1
kind: Namespace
metadata:
  name: sock-shop
  labels:
    istio-injection: enabled
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: filter-ratelimit
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: "envoy.http_connection_manager"
              subFilter:
                name: "envoy.router"
      patch:
        operation: INSERT_BEFORE
        value:
         name: envoy.rate_limit
         config:
           domain: sock-shop-ratelimit
           rate_limit_service:
             grpc_service:
               envoy_grpc:
                 cluster_name: rate_limit_service
               timeout: 0.25s
    - applyTo: CLUSTER
      match:
        cluster:
          service: ratelimit.rate-limit.svc.cluster.local
      patch:
        operation: ADD
        value:
          name: rate_limit_service
          type: STRICT_DNS
          connect_timeout: 0.25s
          lb_policy: ROUND_ROBIN
          http2_protocol_options: {}
          hosts:
            - socket_address:
                address: ratelimit.rate-limit.svc.cluster.local
                port_value: 8081
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: filter-ratelimit-svc
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: VIRTUAL_HOST
      match:
        context: GATEWAY
        routeConfiguration:
          vhost:
            name: "*:80"
            route:
              action: ANY
      patch:
        operation: MERGE
        value:
          # rate limit service descriptors config relays on the order of the request headers (desriptor_key)
          rate_limits:
            - actions:
                - request_headers:
                    header_name: "x-plan"
                    descriptor_key: "plan"               
                - request_headers:
                    header_name: "x-account"
                    descriptor_key: "account" 
apiVersion: v1
kind: Namespace
metadata:
  name: rate-limit
  labels:
    istio-injection: enabled
---
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: rate-limit
  labels:
    app: redis
spec:
  ports:
  - name: redis
    port: 6379
  selector:
    app: redis
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  namespace: rate-limit
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - image: redis:alpine
        imagePullPolicy: Always
        name: redis
        ports:
        - name: redis
          containerPort: 6379
      restartPolicy: Always
      serviceAccountName: ""
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ratelimit-config
  namespace: rate-limit
data:
  # check this example: https://github.com/envoyproxy/ratelimit#example-4
  config.yaml: |
    domain: sock-shop-ratelimit
    descriptors:
      - key: plan
        value: BASIC
        descriptors:
          - key: account
            rate_limit:
              unit: minute
              requests_per_unit: 1
      - key: plan
        value: PLUS
        descriptors:
          - key: account
            rate_limit:
              unit: minute
              requests_per_unit: 2
---
apiVersion: v1
kind: Service
metadata:
  name: ratelimit
  namespace: rate-limit
  labels:
    app: ratelimit
spec:
  ports:
  - name: "8080"
    port: 8080
    targetPort: 8080
    protocol: TCP
  - name: "8081"
    port: 8081
    targetPort: 8081
    protocol: TCP
  - name: "6070"
    port: 6070
    targetPort: 6070
    protocol: TCP
  selector:
    app: ratelimit
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ratelimit
  namespace: rate-limit
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ratelimit
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: ratelimit
    spec:
      containers:
      - image: envoyproxy/ratelimit:v1.4.0
        imagePullPolicy: Always
        name: ratelimit
        command: ["/bin/ratelimit"]
        env:
        - name: LOG_LEVEL
          value: debug
        - name: REDIS_SOCKET_TYPE
          value: tcp
        - name: REDIS_URL
          value: redis:6379
        - name: USE_STATSD
          value: "false"
        - name: RUNTIME_ROOT
          value: /data
        - name: RUNTIME_SUBDIRECTORY
          value: ratelimit
        ports:
        - containerPort: 8080
        - containerPort: 8081
        - containerPort: 6070
        volumeMounts:
        - name: config-volume
          mountPath: /data/ratelimit/config/config.yaml
          subPath: config.yaml
      volumes:
        - name: config-volume
          configMap:
            name: ratelimit-config
---

Other than that I have deployed the istio bookinfo sample application

kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml

Configured istio ingress gateway route to this app

kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

When I access the app through the ingress gateway by applying the associated rate limit headers

    curl -I "http://"$GATEWAY_URL/productpage" --header 'x-plan: BASIC' --header 'x-account: user'

It keeps giving me 200 OK response doesn’t matter how many times I apply it although the applied rimit is 2 requests per minute.

Moreover when I try to see data in redis, I see no data there.

export REDIS_POD=$(kubectl get pod -n rate-limit | grep redis | awk '{ print $1 }')

kubectl -n rate-limit exec -it $REDIS_POD -c redis /bin/sh

redis-cli

keys *

Anyone help would be appreciated.

1 Like

@talha: you have sidecar injection enabled in rate-limit namespace. Is it needed?

I tried tests in https://github.com/istio/istio/pull/23513/files on master and 1.6 branch and they are working… but they don’t have sidecar injection enabled in rate-limit namespace…

@talha: Looks like my master was not synced properly to latest and thus couldn’t see the errors.

If you look at logs in ingressgateway and istiod, you would see logs that would indicate problems with config.
For me, to make it work in master I had to do following changes:

  1. Change EnvoyFilter from config to typed_config. This is a breaking change in EnvoyFilter API and applies to 1.7 too… https://istio.io/latest/news/releases/1.7.x/announcing-1.7/upgrade-notes/#envoyfilter-syntax-change
  2. Change hosts to endpoints in cluster config above as v3 api of clusters have deprecated hosts support.

Ref: https://github.com/istio/istio.io/pull/8119 -> working test.

2 Likes

Thanks @gargnupur for the help :slight_smile:

1 Like

Hi @gargnupur,
can you please elaborate a bit on the 2nd point -
2. Change hosts to endpoints in cluster config above as v3 api of clusters have deprecated hosts support.

I am also getting the below error -
warning envoy config gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) 0.0.0.0_8080: Proto constraint validation failed (RateLimitValidationError.Domain: [“value length must be at least " ‘\x01’ " bytes”]):

0.0.0.0_8443: Proto constraint validation failed (RateLimitValidationError.Domain: [“value length must be at least " ‘\x01’ " bytes”]):

@talha
Can you please help to get the config right.

@nayan-kalita: Can you share your config?
You can take a look at https://github.com/istio/istio/blob/450b8f06912743ded8169445c9a8ec42a08de499/tests/integration/telemetry/policy/testdata/enable_envoy_ratelimit.yaml config to see the difference. The error probably means you are missing some required field.

@gargnupur,

Thanks a lot. I am able to fix the issue by referring to the config mentioned by you.
Thank you.

1 Like

can rate limit be used on service? always return HTTP code 200

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: filter-local-ratelimit-svc-httpbin
  namespace: boook
spec:
  workloadSelector:
    labels:
      app: httpbin
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.local_ratelimit
          typed_config:
            "@type": type.googleapis.com/udpa.type.v1.TypedStruct
            type_url: type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
            value:
              stat_prefix: http_local_rate_limiter
              token_bucket:
                max_tokens: 10
                tokens_per_fill: 10
                fill_interval: 60s
              filter_enabled:
                runtime_key: local_rate_limit_enabled
                default_value:
                  numerator: 100
                  denominator: HUNDRED
              filter_enforced:
                runtime_key: local_rate_limit_enforced
                default_value:
                  numerator: 100
                  denominator: HUNDRED
              response_headers_to_add:
                - append: false
                  header:
                    key: x-local-rate-limit
                    value: 'true'
              descriptors:
                - entries:
                  - key: path
                    value: /ip
                  token_bucket:
                    max_tokens: 1
                    tokens_per_fill: 1
                    fill_interval: 60s

check envoyfilter config

[root@10-20-11-81 bookinfo]# istioctl pc listener httpbin-66cdbdb6c5-zjfwq.boook -oyaml | grep -5 /ip
            typeUrl: type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
            value:
              descriptors:
              - entries:
                - key: path
                  value: /ip
                token_bucket:
                  fill_interval: 60s
                  max_tokens: 1
                  tokens_per_fill: 1
              filter_enabled:

Also having the same problem, the example local rate limit from Istio / Enabling Rate Limits using Envoy doesn’t work. Any updates?