The global Rate limit is not taking the actual remote_address but taking any IP from internal GCP VPC range. For example, If I hit request from workstataion
which has IP 202.121.112.113, the rate limit deployment is limiting request correctly but it’s treating the remote IP as 10.128.0.51 & 10.128.0.50 which is internal.
Rate Limit Deployment Logs from workstation 1:
time=“2023-07-24T06:32:26Z” level=debug msg=“looking up key: remote_address_10.128.0.51”
time=“2023-07-24T06:32:26Z” level=debug msg=“looking up key: remote_address”
time=“2023-07-24T06:32:26Z” level=debug msg=“iterating to next level”
time=“2023-07-24T06:32:26Z” level=debug msg=“looking up key: PATH_/”
time=“2023-07-24T06:32:26Z” level=debug msg=“looking up key: PATH”
time=“2023-07-24T06:32:26Z” level=debug msg=“found rate limit: PATH”
time=“2023-07-24T06:32:26Z” level=debug msg=“applying limit: 2 requests per MINUTE, shadow_mode: false”
time=“2023-07-24T06:32:26Z” level=debug msg=“starting cache lookup”
time=“2023-07-24T06:32:26Z” level=debug msg=“looking up cache key: edge_proxy_per_ip_remote_address_10.128.0.51_PATH_/1690180320"
time=“2023-07-24T06:32:26Z” level=debug msg="cache key: edge_proxy_per_ip_remote_address_10.128.0.51_PATH/_1690180320 current: 3”
Rate Limit Deployment Logs from workstation 2:
time=“2023-07-24T06:33:32Z” level=debug msg=“looking up key: PATH_/”
time=“2023-07-24T06:33:32Z” level=debug msg=“looking up key: PATH”
time=“2023-07-24T06:33:32Z” level=debug msg=“found rate limit: PATH”
time=“2023-07-24T06:33:32Z” level=debug msg=“applying limit: 2 requests per MINUTE, shadow_mode: false”
time=“2023-07-24T06:33:32Z” level=debug msg=“starting cache lookup”
time=“2023-07-24T06:33:32Z” level=debug msg=“looking up cache key: edge_proxy_per_ip_remote_address_10.128.0.50_PATH_/1690180380"
time=“2023-07-24T06:33:32Z” level=debug msg="cache key: edge_proxy_per_ip_remote_address_10.128.0.50_PATH/_1690180380 current: 3”
time=“2023-07-24T06:33:32Z” level=debug msg=“returning normal response”
time=“2023-07-24T06:33:32Z” level=debug msg=“[gostats] flushing time ratelimit_server.ShouldRateLimit.response_time: 0.000000”
time=“2023-07-24T06:33:34Z” level=debug msg=“[gostats] flushing counter ratelimit.service.rate_limit.edge_proxy_per_ip.remote_address.PATH.total_hits: 5”
time=“2023-07-24T06:33:34Z” level=debug msg=“[gostats] flushing counter ratelimit.go.totalAlloc: 103896”
time=“2023-07-24T06:33:34Z” level=debug msg=“[gostats] flushing counter ratelimit.service.rate_limit.edge_proxy_per_ip.remote_address.PATH.within_limit: 4”
time=“2023-07-24T06:33:34Z” level=debug msg=“[gostats] flushing counter ratelimit.go.mallocs: 2387”
time=“2023-07-24T06:33:34Z” level=debug msg=“[gostats] flushing counter ratelimit.service.rate_limit.edge_proxy_per_ip.remote_address.PATH.near_limit: 2”
time=“2023-07-24T06:33:34Z” level=debug msg=“[gostats] flushing counter ratelimit.go.frees: 175”
time=“2023-07-24T06:33:34Z” level=debug msg=“[gostats] flushing counter ratelimit_server.ShouldRateLimit.total_requests: 5”
time=“2023-07-24T06:33:34Z” level=debug msg=“[gostats] flushing counter ratelimit.service.rate_limit.edge_proxy_per_ip.remote_address.PATH.over_limit: 1”
time=“2023-07-24T06:33:34Z” level=debug msg=“[gostats] flushing gauge runtime.num_values: 1”
time=“2023-07-24T06:33:34Z” level=debug msg=“[gostats] flushing gauge ratelimit.go.alloc: 2244880”
Rate Limit Deployment Logs from workstation 3:
ime=“2023-07-24T06:34:46Z” level=debug msg=“looking up cache key: edge_proxy_per_ip_remote_address_10.128.0.51_PATH_/productpage_1690180440”
time=“2023-07-24T06:34:46Z” level=debug msg=“cache key: edge_proxy_per_ip_remote_address_10.128.0.51_PATH_/productpage_1690180440 current: 6”
time=“2023-07-24T06:34:46Z” level=debug msg=“returning normal response”
time=“2023-07-24T06:34:46Z” level=debug msg=“[gostats] flushing time ratelimit_server.ShouldRateLimit.response_time: 0.000000”
time=“2023-07-24T06:34:48Z” level=debug msg=“got descriptor: (remote_address=10.128.0.51),(PATH=/productpage)”
time=“2023-07-24T06:34:48Z” level=debug msg=“starting get limit lookup”
time=“2023-07-24T06:34:48Z” level=debug msg=“looking up key: remote_address_10.128.0.51”
time=“2023-07-24T06:34:48Z” level=debug msg=“looking up key: remote_address”
time=“2023-07-24T06:34:48Z” level=debug msg=“iterating to next level”
time=“2023-07-24T06:34:48Z” level=debug msg=“looking up key: PATH_/productpage”
time=“2023-07-24T06:34:48Z” level=debug msg=“looking up key: PATH”
time=“2023-07-24T06:34:48Z” level=debug msg=“found rate limit: PATH”
time=“2023-07-24T06:34:48Z” level=debug msg=“applying limit: 2 requests per MINUTE, shadow_mode: false”
time=“2023-07-24T06:34:48Z” level=debug msg=“starting cache lookup”
time=“2023-07-24T06:34:48Z” level=debug msg=“looking up cache key: edge_proxy_per_ip_remote_address_10.128.0.51_PATH_/productpage_1690180440”
time=“2023-07-24T06:34:48Z” level=debug msg=“cache key: edge_proxy_per_ip_remote_address_10.128.0.51_PATH_/productpage_1690180440 current: 7”
time=“2023-07-24T06:34:48Z” level=debug msg=“returning normal response”
time=“2023-07-24T06:34:48Z” level=debug msg=“[gostats] flushing time ratelimit_server.ShouldRateLimit.response_time: 1.000000”
My Yaml files are like this:
apiVersion: v1
kind: ConfigMap
metadata:
name: ratelimit-config
data:
config.yaml: |
domain: edge_proxy_per_ip
descriptors:
- key: remote_address
descriptors:
- key: PATH
rate_limit:
unit: minute
requests_per_unit: 2
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: filter-ratelimit
namespace: istio-system
spec:
workloadSelector:
# select by label in the same namespace
labels:
istio: ingressgateway
configPatches:
# The Envoy config you want to modify
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.filters.network.http_connection_manager"
subFilter:
name: "envoy.filters.http.router"
patch:
operation: INSERT_BEFORE
# Adds the Envoy Rate Limit Filter in HTTP filter chain.
value:
name: envoy.filters.http.ratelimit
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
# domain can be anything! Match it to the ratelimter service config
domain: edge_proxy_per_ip
failure_mode_deny: true
timeout: 10s
rate_limit_service:
grpc_service:
envoy_grpc:
cluster_name: outbound|8081||ratelimit.default.svc.cluster.local
authority: ratelimit.default.svc.cluster.local
transport_api_version: V3
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: filter-ratelimit-svc
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: VIRTUAL_HOST
match:
context: GATEWAY
routeConfiguration:
vhost:
name: ""
route:
action: ANY
patch:
operation: MERGE
# Applies the rate limit rules.
value:
rate_limits:
- actions: # any actions in heres
- remote_address: {}
- request_headers:
header_name: ":path"
descriptor_key: "PATH"
My environment is GKE private zonal cluster running in GCP cloud.