Hello, I am attempting to make an AuthorizationPolicy for one of my services (SRV1 in namespace ‘org-delcowater’) I would like external traffic from the gateway to require a JWT token to access the service.
However I would like the other service in the namespace (SRV2) to also access SRV1 but not require a token. My config is below. When I apply this config it works for the external traffic but when I SSH into SRV2 and do a CURL on http://SRV1/ I get RBAC: access denied.
Any help would be appreciated, thank you.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: default-gateway
namespace: istio-system
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- 'example.com'
port:
name: https
number: 443
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: ingress-secert
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: virtual-svc
namespace: org-delcowater
spec:
hosts:
- "*"
gateways:
- istio-system/default-gateway
http:
- name: "SRV1"
match:
- uri:
prefix: "/config-api"
rewrite:
uri: "/config-api"
route:
- destination:
host: SRV1
---
apiVersion: "security.istio.io/v1beta1"
kind: RequestAuthentication
metadata:
name: backend-jwt-auth
namespace: istio-system
spec:
selector:
matchLabels:
app: backend-secure # Label applied to SRV1
jwtRules:
- issuer: "https://example.auth0.com/"
jwksUri: "https://example.auth0.com/.well-known/jwks.json"
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: backend-secure
namespace: istio-system
spec:
selector:
matchLabels:
app: backend-secure # Label applied to SRV1
action: ALLOW
rules:
- from:
- source:
requestPrincipals: ["*"]
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: backend
namespace: istio-system
spec:
selector:
matchLabels:
app: backend-secure # Label applied to SRV1
action: ALLOW
rules:
- from:
- source:
namespaces: ["org-delcowater"]
---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: org-delcowater
spec:
mtls:
mode: STRICT
---
edited namespace*
could you change the NS to the lower case and try again? I think k8s requires the namespace to be in lower case. If still not working, could you send enable the Envoy debug logging, send some requests from the SRV2 and get the Envoy debug log in the app: backend-secure pod?
See Istio / Security Problems for more details.
Thanks a lot Yangmin, yeah the NS is an example namespace. In practice I have a all lowercase namespace listed.
I will enable debug logging and get back to you.
I have got the logs of app: backend-sure, SRV1’s actual name is ‘config-api’ NS actual name is ‘org-delcowater’
2021-05-06T23:00:48.844621Z info FLAG: --concurrency="2"
2021-05-06T23:00:48.844649Z info FLAG: --domain="org-delcowater.svc.cluster.local"
2021-05-06T23:00:48.844655Z info FLAG: --help="false"
2021-05-06T23:00:48.844658Z info FLAG: --log_as_json="false"
2021-05-06T23:00:48.844662Z info FLAG: --log_caller=""
2021-05-06T23:00:48.844668Z info FLAG: --log_output_level="default:info"
2021-05-06T23:00:48.844672Z info FLAG: --log_rotate=""
2021-05-06T23:00:48.844787Z info FLAG: --log_rotate_max_age="30"
2021-05-06T23:00:48.844794Z info FLAG: --log_rotate_max_backups="1000"
2021-05-06T23:00:48.844831Z info FLAG: --log_rotate_max_size="104857600"
2021-05-06T23:00:48.844835Z info FLAG: --log_stacktrace_level="default:none"
2021-05-06T23:00:48.844877Z info FLAG: --log_target="[stdout]"
2021-05-06T23:00:48.844881Z info FLAG: --meshConfig="./etc/istio/config/mesh"
2021-05-06T23:00:48.844911Z info FLAG: --outlierLogPath=""
2021-05-06T23:00:48.844922Z info FLAG: --proxyComponentLogLevel="misc:error"
2021-05-06T23:00:48.844926Z info FLAG: --proxyLogLevel="warning"
2021-05-06T23:00:48.844962Z info FLAG: --serviceCluster="backend-secure.org-delcowater"
2021-05-06T23:00:48.844987Z info FLAG: --stsPort="0"
2021-05-06T23:00:48.844991Z info FLAG: --templateFile=""
2021-05-06T23:00:48.845026Z info FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2021-05-06T23:00:48.845040Z info Version 1.9.4-13fb8ac89420d8cc5b3f895adc614233e805a61c-Clean
2021-05-06T23:00:48.845225Z info Apply proxy config from env {}
2021-05-06T23:00:48.846120Z info Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: backend-secure.org-delcowater
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
zipkin:
address: zipkin.istio-system:9411
2021-05-06T23:00:48.846143Z info Proxy role ips=[10.244.3.21] type=sidecar id=config-api-ddd857f64-n62cw.org-delcowater domain=org-delcowater.svc.cluster.local
2021-05-06T23:00:48.846881Z info JWT policy is third-party-jwt
2021-05-06T23:00:48.846902Z info Pilot SAN: [istiod.istio-system.svc]
2021-05-06T23:00:48.846912Z info CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2021-05-06T23:00:48.846946Z info Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2021-05-06T23:00:48.847033Z info citadelclient Citadel client using custom root cert: istiod.istio-system.svc:15012
2021-05-06T23:00:48.874918Z info ads All caches have been synced up in 32.145321ms, marking server ready
2021-05-06T23:00:48.875151Z info sds SDS server for workload certificates started, listening on "./etc/istio/proxy/SDS"
2021-05-06T23:00:48.877310Z info xdsproxy Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2021-05-06T23:00:48.877520Z info Starting proxy agent
2021-05-06T23:00:48.878562Z info sds Start SDS grpc server
2021-05-06T23:00:48.878619Z info Opening status port 15020
2021-05-06T23:00:48.878711Z info Received new config, creating new Envoy epoch 0
2021-05-06T23:00:48.878737Z info Epoch 0 starting
2021-05-06T23:00:49.118581Z info cache Root cert has changed, start rotating root cert
2021-05-06T23:00:49.118621Z info ads XDS: Incremental Pushing:0 ConnectedEndpoints:0 Version:
2021-05-06T23:00:49.118633Z info cache generated new workload certificate latency=243.275971ms ttl=23h59m59.88137364s
2021-05-06T23:00:49.897311Z warn HTTP request unsuccessful with status: 400 Bad Request
2021-05-06T23:00:49.902065Z info Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster backend-secure.org-delcowater --service-node sidecar~10.244.3.21~config-api-ddd857f64-n62cw.org-delcowater~org-delcowater.svc.cluster.local --local-address-ip-version v4 --bootstrap-version 3 --log-format %Y-%m-%dT%T.%fZ %l envoy %n %v -l warning --component-log-level misc:error --concurrency 2]
2021-05-06T23:00:49.940258Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-05-06T23:00:49.940303Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-05-06T23:00:49.940803Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-05-06T23:00:49.940843Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-05-06T23:00:49.982774Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-05-06T23:00:50.005009Z info ads ADS: new connection for node:sidecar~10.244.3.21~config-api-ddd857f64-n62cw.org-delcowater~org-delcowater.svc.cluster.local-1
2021-05-06T23:00:50.005071Z info cache returned workload certificate from cache ttl=23h59m58.994937852s
2021-05-06T23:00:50.005267Z info sds SDS: PUSH resource=default
2021-05-06T23:00:50.006033Z info ads ADS: new connection for node:sidecar~10.244.3.21~config-api-ddd857f64-n62cw.org-delcowater~org-delcowater.svc.cluster.local-2
2021-05-06T23:00:50.006158Z info sds SDS: PUSH resource=ROOTCA
2021-05-06T23:00:51.200178Z info Initialization took 2.361948722s
2021-05-06T23:00:51.200196Z info Envoy proxy is ready
[2021-05-06T23:02:07.693Z] "GET /config-api/sites/3fa85f64-5717-4562-b3fc-2c963f66afa6 HTTP/1.1" 200 - via_upstream - "-" 0 4413 199 199 "-" "Wget" "ca72a28e-0d1d-4bf7-b5c7-a699e341e42c" "config-api" "127.0.0.1:80" inbound|80|| 127.0.0.1:54362 10.244.3.21:80 10.244.0.21:40132 outbound_.80_._.config-api.org-delcowater.svc.cluster.local default
[2021-05-06T23:02:07.756Z] "- - -" 0 - - - "-" 939 6063 138 - "-" "-" "-" "-" "13.71.170.1:443" PassthroughCluster 10.244.3.21:57634 13.71.170.1:443 10.244.3.21:57632 - -
[2021-05-06T23:02:07.811Z] "- - -" 0 - - - "-" 3834 11035 85 - "-" "-" "-" "-" "13.71.170.1:443" PassthroughCluster 10.244.3.21:57646 13.71.170.1:443 10.244.3.21:57644 - -
[2021-05-06T23:02:07.758Z] "- - -" 0 - - - "-" 939 6063 152 - "-" "-" "-" "-" "13.71.170.1:443" PassthroughCluster 10.244.3.21:57640 13.71.170.1:443 10.244.3.21:57636 - -
[2021-05-06T23:02:07.868Z] "- - -" 0 - - - "-" 1113 11069 43 - "-" "-" "-" "-" "20.38.146.2:14130" PassthroughCluster 10.244.3.21:33326 20.38.146.2:14130 10.244.3.21:33324 - -
[2021-05-06T23:02:07.758Z] "- - -" 0 - - - "-" 939 6063 153 - "-" "-" "-" "-" "13.71.170.1:443" PassthroughCluster 10.244.3.21:57642 13.71.170.1:443 10.244.3.21:57638 - -
This is the log I get on SRV2
[2021-05-06T23:11:18.757Z] "GET /config-api/sites/3fa85f64-5717-4562-b3fc-2c963f66afa6 HTTP/1.1" 200 - via_upstream - "-" 0 4413 318 317 "-" "Wget" "b9b6e31a-3b62-4671-a49f-54c552fd1dea" "config-api" "10.244.3.21:80" outbound|80||config-api.org-delcowater.svc.cluster.local 10.244.0.21:45590 10.0.231.21:80 10.244.0.21:60872 - default
Looks like there is something weird going on with that pod, when I use busy box wget on the URL I get permission denied. But if I SSH into another Pod in the same namespace and I run a CURL on that URL I al able to get the response as I expect.