Hi !
Here is my use case:
I have a Microsoft AKS cluster with Istio installed.
I would like to do IP based filtering at the ingress level, using resource AuthorizationPolicy
When a request arrives at Istio:
x-forwarded-for header is <IP source client>,<IP Frontdoor proxy>x-azure-clientip header is <IP source client> (header set by the Azure Frontdoor proxy)sourceIP: is <IP Frontdoor proxy>:<port>remoteIP is <IP Frontdoor proxy>:<port>directRemoteIP is <IP Frontdoor proxy>:<port>
First, it seems that using remoteIpBlocks does not work because x-forwaded-for is a list of IPs and it is not checking all of them ? (taking 2nd one for the remote.ip and not the first one)
Then I used a rule instead to check IP against request.headers[x-azure-clientip]: - when: - key: request.headers[x-azure-clientip] notValues: - "<IP to whitelist 1>" - "<IP to whitelist 2>"
It worked. However with this I cannot allow a range of IP (with CIDR e.g xx.xx.xx.xx/22)
So now I am thinking of using EnvoyFilter to:
take IP from x-azure-clientip if exists
Rewrite source IP with this IPipBlocks in AuthorizationPolicy which can process IP ranges
Is it the best strategy ? Does someone encountered the same kind of challenge ?
Thank you !
1 Like
Hello! I have the exact same issue as you. Did you find a solution to this?
It seem strange to me that Istio doesn’t have configuration option to control which header to look at.
Hi @Dogsbody
Actually I dig further on existing issues and I found:
opened 07:54PM - 18 Jan 21 UTC
closed 05:02AM - 25 Jun 21 UTC
area/security
lifecycle/stale
lifecycle/automatically-closed
**Bug description**
When `AuthorizationPolicy` is applied to injected istio proxy, `remoteIpBlocks` does not work as expected when istio gateway is behind another reverse proxy (Azure Front Door). RemoteIP seems to set to the IP of the reverse-proxy deployed in front of istio gateway. (see diagram above: requests in green, configuration in blue)
```yaml
kind: AuthorizationPolicy
metadata:
name: test-auth-policy
spec:
action: ALLOW
selector:
matchLabels:
app: test
rules:
- from:
- source:
remoteIpBlocks:
- <clientIp>
```
Request from `<clientIp>` gets denied with the following entry in the logs:
```bash
2021-01-18T17:40:15.872025Z debug envoy rbac checking request: requestedServerName: outbound_.80_._.test-service.test-ns.svc.cluster.local, sourceIP: <IstioIngressGatewayIp>:37834, directRemoteIP: <IstioIngressGatewayIp>:37834, remoteIP: <AzureFrontDoorIP>:0,localAddress: 10.240.2.152:8080, ssl: uriSanPeerCertificate: spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-public-service-account, dnsSanPeerCertificate: , subjectPeerCertificate: , headers: ':authority', 'test.example.com'
...
'x-forwarded-for', '<clientIp>,<AzureFrontDoorIP>'
...
'x-envoy-external-address', '<clientIp>'
...
'x-forwarded-client-cert', 'By=spiffe://cluster.local/ns/test-ns/sa/default;Hash=2a3d7357260f29961fd26f74bf7e9d637eaa9c23c4d735488f0541027b4c48ed;Subject="";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-public-service-account'
, dynamicMetadata: filter_metadata {
key: "istio_authn"
value {
fields {
key: "request.auth.principal"
value {
string_value: "cluster.local/ns/istio-system/sa/istio-ingressgateway-public-service-account"
}
}
fields {
key: "source.namespace"
value {
string_value: "istio-system"
}
}
fields {
key: "source.principal"
value {
string_value: "cluster.local/ns/istio-system/sa/istio-ingressgateway-public-service-account"
}
}
fields {
key: "source.user"
value {
string_value: "cluster.local/ns/istio-system/sa/istio-ingressgateway-public-service-account"
}
}
}
}
2021-01-18T17:40:15.872065Z debug envoy rbac enforced denied, matched policy none
```
**Affected product area (please put an X in all that apply)**
[ ] Docs
[ ] Installation
[X] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[X] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
[ ] Upgrade
**Affected features**
[ ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane
**Expected behavior**
`AuthorizationPolicy` allows requests from `<clientIp>`.
**Steps to reproduce the bug**
1. Install istio using `istioctl` and configure with:
```yaml
ingressGateways:
- enabled: false
name: istio-ingressgateway
k8s:
replicaCount: 2
podAnnotations:
proxy.istio.io/config: '{"gatewayTopology" : { "numTrustedProxies": 1 } }'
service:
externalTrafficPolicy: Local
```
2. Create `AuthorizationPolicy`
```yaml
kind: AuthorizationPolicy
metadata:
name: test-auth-policy
spec:
action: ALLOW
selector:
matchLabels:
app: test
rules:
- from:
- source:
remoteIpBlocks:
- <clientIp>
```
3. Deploy a dummy service with the labels matching istio `AuthorizationPolicy`
4. Make a request from `<clientIp>` through a reverse proxy (or simulate with XFF header)
5. Observe the request beeing blocked, but should be allowed
**Version**
* `istioctl version --remote`
```bash
client version: 1.8.2
control plane version: 1.8.2
data plane version: 1.8.2 (7 proxies)
```
* `kubectl version --short`
```bash
Client Version: v1.18.15
Server Version: v1.18.14
```
**How was Istio installed?**
`istioctl install -f install.yaml`
<details>
<summary>install.yaml</summary>
```yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: default
tag: 1.8.2-distroless
meshConfig:
outboundTrafficPolicy:
mode: REGISTRY_ONLY
components:
cni:
enabled: true
ingressGateways:
- enabled: false
name: istio-ingressgateway
- enabled: true
name: istio-ingressgateway-public
k8s:
replicaCount: 2
podAnnotations:
proxy.istio.io/config: '{"gatewayTopology" : { "numTrustedProxies": 1 } }'
serviceAnnotations:
service.beta.kubernetes.io/azure-allowed-service-tags: AzureFrontDoor.Backend
service:
externalTrafficPolicy: Local
loadBalancerIP: 51.145.104.39
ports:
- name: http2
port: 80
targetPort: 8080
- name: https
port: 443
targetPort: 8443
hpaSpec:
maxReplicas: 5
minReplicas: 2
resources:
limits:
cpu: 650m
memory: 400Mi
requests:
cpu: 100m
memory: 400Mi
overlays:
- kind: HorizontalPodAutoscaler
name: istio-ingressgateway-public
patches:
- path: metadata.labels.app
value: istio-ingressgateway-public
- path: metadata.labels.istio
value: public-ingressgateway
- path: spec.scaleTargetRef.name
value: istio-ingressgateway-public
- kind: Deployment
name: istio-ingressgateway-public
patches:
- path: metadata.labels.app
value: istio-ingressgateway-public # Change the label to istio-ingressgateway-public
- path: metadata.labels.istio
value: public-ingressgateway
- path: spec.selector.matchLabels.app
value: istio-ingressgateway-public
- path: spec.selector.matchLabels.istio
value: public-ingressgateway
- path: spec.template.metadata.labels.app
value: istio-ingressgateway-public
- path: spec.template.metadata.labels.istio
value: public-ingressgateway
- path: spec.template.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution[0].labelSelector.matchExpressions[0].values[0]
value: istio-ingressgateway-public
- kind: Service
name: istio-ingressgateway-public
patches:
- path: metadata.labels.app
value: istio-ingressgateway-public
- path: metadata.labels.istio
value: public-ingressgateway
- path: spec.selector.app
value: istio-ingressgateway-public
- path: spec.selector.istio
value: public-ingressgateway
- kind: ServiceAccount
name: istio-ingressgateway-public-service-account
patches:
- path: metadata.labels.app
value: istio-ingressgateway-public
- path: metadata.labels.istio
value: public-ingressgateway
- kind: PodDisruptionBudget
name: istio-ingressgateway-public
patches:
- path: metadata.labels.app
value: istio-ingressgateway-public
- path: metadata.labels.istio
value: public-ingressgateway
- path: spec.selector.matchLabels.app
value: istio-ingressgateway-public
- path: spec.selector.matchLabels.istio
value: public-ingressgateway
pilot:
k8s:
replicaCount: 2
values:
global:
defaultNodeSelector:
beta.kubernetes.io/os: linux
proxy:
resources:
limits:
cpu: 500m
memory: 400Mi
requests:
cpu: 10m
memory: 80Mi
proxy_init:
resources:
limits:
cpu: 500m
memory: 400Mi
requests:
cpu: 10m
memory: 10Mi
gateways:
istio-ingressgateway:
podAntiAffinityLabelSelector:
- key: app
operator: In
values: istio-ingressgateway
topologyKey: kubernetes.io/hostname
cni:
excludeNamespaces:
- istio-system
- kube-system
- kube-public
- kube-node-lease
pilot:
autoscaleMin: 2
```
</details>
**Environment where the bug was observed (cloud vendor, OS, etc)**
* Cloud: Azure AKS
* Networking: Azure (advanced)
* OS: `Ubuntu 18.04.5 LTS`
* Kernel: `5.4.0-1035-azure`
* Container Runtime: `docker://19.3.14`
opened 06:25AM - 27 Jul 22 UTC
closed 05:02AM - 15 Nov 22 UTC
lifecycle/stale
lifecycle/automatically-closed
I am using istio authorization policy for IP whitelisting. I have an issue with … the existing environment where the x-forwarded-for header has a complete hop of IPs
example:
x-forwarded-for: client ip, front door IP ,service ip
I am unable to complete my requirement with ipBlock and remoteIpBlock. so I am using request. header rule. I have created a manifest shown below
```yaml
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: ingress-policy
namespace: istio-system
spec:
selector:
matchLabels:
app: istio-ingressgateway
action: ALLOW
rules:
- when:
- key: request.headers[x-azure-socket]
values:
- x.x.x.x
- x.x.x.x
```
It's works for external traffic. But i have issue with internal traffic which is cluster-to-cluster communication. so have to use a multiply istio authorization policy with a different header. if any internal traffic passes the istio ingress gateway the x-forwarded-for has a single IP. I don't see x-azure-socket for internal traffic.
```
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: ingress-policy
namespace: istio-system
spec:
selector:
matchLabels:
app: istio-ingressgateway
action: ALLOW
rules:
- when:
- key: request.headers[x-forwarded-for]
values:
- x.x.x.x
- x.x.x.x
```
I have tried creating a custom header with envoyfilter. I am not sure anything wrong I did on envoyfilter which is shown below
```yaml
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: ingressgateway-user-ip
namespace: ingress
spec:
workloadLabels:
app: istio-ingressgateway
filters:
- listenerMatch:
portNumber: 443
listenerType: ANY
filterName: envoy.lua
filterType: HTTP
filterConfig:
inlineCode: |
function envoy_on_request(request_handle)
local xff_header = request_handle:headers():get("X-Forwarded-For")
local client_ip = string.gmatch(xff_header, "(%d+.%d+.%d+.%d+)")();
request_handle:headers():add("X-Custom-User-IP", first_ip);
end
```
It's creating a custom header as expected after adding custom access logs. but when i try to create istio authorization policy with my custom header. It's not working and in logs shows - (null)
So I have decided to use a multiple istio authorization policy for internal and external traffic. But now I see the request. header rule doesn't support CIDR and as well as TCP protocol.
So it seems it is a common use case
I haven’t tried yet but I am thinking of using envoyFilter at ingress level to rewrite source IP, and do a AuthorizationPolicy on ipBlocks at VirtualService level.
1 Like
ams0
July 17, 2023, 6:34am
4
Have you tried ipBlocks in your AuthorizationPolocy? According to this artcile when externalTrafficPolicy: Local is set that’s what you should use.
Hello @ams0
I tried ipBlocks without success because the client IP is set with IP of first proxy (Azure FrontDoor in my use case), and not the real remote client IP. I guess it works fine if the cluster is not behind a proxy (so reached out directly by client IP). And as I wrote in the issue description, remoteIpBlocks does not seem to work on x-forwarded-for that contain a list of IPs ( <IP source client>,<IP Frontdoor proxy>).