Authorization Policy issue with 503

murbano · November 27, 2019, 3:38pm

Hello,

I want to disable the access from external to certain endpoints on one of my projects. So I started to use the AuthorizationPolicy without success.

I use Istio 1.4.0 and I have enabled mTls on my namespace

HOST:PORT                                                      STATUS     SERVER     CLIENT           AUTHN POLICY                DESTINATION RULE
xxxx-app.xxxxx.svc.cluster.local:8080     OK         STRICT     ISTIO_MUTUAL     xxxxx/default     xxxxx/xxxx-app

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
    metadata:
    name: "default"
    labels: 
        app: {{ template "application.name" . }}
        chart: {{ template "application.chart" . }}
        heritage: {{ .Release.Service }}
        release: {{ template "system.name" . }}
        environment: {{ .Values.global.environment }}
 spec:
     peers:
         - mtls: {}

And my Authorization Policty:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: {{ template "application.name" . }}
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  selector:
    matchLabels:
      app: {{ template "application.name" . }}
  rules:
  - to:
    - operation:
        methods: ["GET", "POST", "OPTIONS", "HEAD", "PUT"]

But when I go to any endpoint I just get a 503 error with “upstream connect error or disconnect/reset before headers. reset reason: connection termination”

With rules empty everythings comes back to normal:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: {{ template "application.name" . }}
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  selector:
    matchLabels:
      app: {{ template "application.name" . }}
  rules:
  - {}

Any idea on what could happen? I miss something?

Thanks!

philliple · November 27, 2019, 5:21pm

Do you have any destination rule to configure clients to use mtls when talking to your mtls-enabled services?

murbano · November 27, 2019, 5:40pm

I have these two DestinationRule:

apiVersion: networking.istio.io/v1alpha3
kind: "DestinationRule"
metadata:
  name: "default"
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  host: "*.my-system.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

and

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: {{ template "application.name" . }}
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  host: my-service
  subsets:
  - name: {{ .Values.global.environment }}
    labels:
      main: "true"

The second DestinationRule gets override and removes my subsets, dont know why.

All these are on the same namespace (its added by helm)

murbano · November 27, 2019, 6:10pm

Maybe that could be part of the problem:

kubectl exec $(kubectl get pod -l app=my-app -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl http://my-app:8080/healthz -o /dev/null -s -w '%{http_code}\n' -k
kubectl exec $(kubectl get pod -l app=my-app -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl https://my-app:8080/healthz -o /dev/null -s -w '%{http_code}\n' -k
kubectl exec $(kubectl get pod -l app=my-app -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl https://my-app:8080/healthz -o /dev/null -s -w '%{http_code}\n' --key /etc/certs/key.pem --cert /etc/certs/cert-chain.pem --cacert /etc/certs/root-cert.pem -k
000
command terminated with exit code 56
000
command terminated with exit code 16
000
command terminated with exit code 16

I’m getting that exit code 16.

murbano · November 28, 2019, 4:35pm

Policy:

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "default"
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  peers:
    - mtls: {}

Destination Rule

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: {{ template "application.name" . }}
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  host: "{{ template "application.name" . }}.{{ template "system.name" . }}.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
  subsets:
  - name: {{ .Values.global.environment }}
    labels:
      main: "true"

Virtual Service:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: {{ template "application.name" . }}
  labels:
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  {{ include "application.hosts" . }}
  {{ include "application.gateways" . }}
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: "{{ template "application.name" . }}.{{ template "system.name" . }}.svc.cluster.local"
        subset: {{ .Values.global.environment }}
        port:
          number: {{ template "application.service.port" . }}
      headers:
        response:
          add:
            x-app-dedicated: "app"
    retries:
        attempts: 3
        perTryTimeout: 2s

Authorization Policy

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: {{ template "application.name" . }}
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  selector:
    matchLabels:
      app: {{ template "application.name" . }}
  rules:
  - to:
    - operation:
        methods: ["GET"]

After apply the AuthorizationPolicy, the istio-proxy starts to show the next:

│ [2019-11-28T16:01:29.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:40873 10.40.10.156:15090 10.40.11.143:46502 - -                                                        │
│ [2019-11-28T16:01:44.947Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:48599 10.40.10.156:15090 10.40.11.143:46702 - -                                                        │
│ [2019-11-28T16:01:59.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:36675 10.40.10.156:15090 10.40.11.143:46896 - -                                                        │
│ [2019-11-28T16:02:14.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:55289 10.40.10.156:15090 10.40.11.143:47090 - -                                                        │
│ [2019-11-28T16:02:29.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:45871 10.40.10.156:15090 10.40.11.143:47284 - -                                                        │
│ [2019-11-28T16:02:44.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:40379 10.40.10.156:15090 10.40.11.143:47494 - -                                                        │
│ [2019-11-28T16:02:59.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:33319 10.40.10.156:15090 10.40.11.143:47686 - -                                                        │
│ [2019-11-28T16:03:14.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:57011 10.40.10.156:15090 10.40.11.143:47886 - -                                                        │
│ [2019-11-28T16:03:29.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:59227 10.40.10.156:15090 10.40.11.143:48078 - -                                                        │
│ [2019-11-28T16:03:44.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:52511 10.40.10.156:15090 10.40.11.143:48280 - -

I still do not understand why im getting this kind of error:

curl -X GET \
  https://mysystem.com/healthz \
  -H 'Accept: */*' \
  -H 'Accept-Encoding: gzip, deflate' \
  -H 'Cache-Control: no-cache' \
  -H 'Connection: keep-alive' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -H 'Host: mysystem.com' \
  -H 'Postman-Token: f4455c4e-34bb-41d2-8a0d-d219e5def917,b35173cb-8334-4c82-9003-e0b044e0fdc9' \
  -H 'User-Agent: PostmanRuntime/7.20.1' \
  -H 'cache-control: no-cache'


503 - upstream connect error or disconnect/reset before headers. reset reason: connection termination

This AuthorizationPolicy should work without mTLS enabled?

Thanks

murbano · November 28, 2019, 5:40pm

On the environment where im testing this policies kiali only shows me istio-telemetry > telemetry.

But on other environment where I dont have mTls active:

It shows traffic to policy.

Maybe some wrong installation with istio on that environment?

philliple · December 2, 2019, 7:35pm

@YangminZhu @diemtvu

YangminZhu · December 2, 2019, 7:51pm

AuthorizationPolicy should never return 503 error, it should only cause 401. It seems something wrong in the networking or mTLS part.

Could you confirm the following:

Do you see 503 if you remove the authorization policy?
Do you see 503 if you remove the authentication policy?
What’s your service definition?

diemtvu · December 2, 2019, 9:14pm

Can you also check the DR for policy and telemetry (mixer). Failure to talk to policy also shows up as 503 in data plane.

murbano · December 16, 2019, 5:32pm

Sorry for late response was a stupid thing, just putting on the service http as protocol instead of tcp.

MediceTiago · May 3, 2020, 2:59pm

you mean you gateway protocol ?

Cory_Silva · June 3, 2020, 9:47pm

I had to specifically name the port “http” in the kubernetes service definition.

“Kubernetes also supports DNS SRV (Service) records for named ports. If the "my-service.my-ns" Service has a port named "http" with the protocol set to TCP , you can do a DNS SRV query for _http._tcp.my-service.my-ns to discover the port number for "http" , as well as the IP address.”

henry · June 15, 2020, 1:05pm

I got 503 error while replacing Policy with RequestAuthentication & AuthorizationPolicy for JWT validation, after upgrade istio from 1.4 to 1.5.

I had to change the port name from tcp to http in Service. The root cause is Istio apply envoy.tcp_proxy filter rather than envoy.http_connection_manager filter which includes “envoy.filters.http.jwt_authn”, “istio_authn” & “envoy.filters.http.rbac” HTTP specific features.

Reference: https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/

Topic		Replies	Views
Authorization polices issue Security	1	166	November 15, 2023
503 errors with AuthorizationPolicy and ALLOW with a Rule Security	3	953	January 11, 2021
Using AuthorizationPolicy for access control of legacy clients located outside of Istio	2	1458	October 25, 2021
AuthorizationPolicy Issue Security	3	734	October 23, 2020
Need help to add authorization policy to filter internal and external calls Security	5	1084	July 13, 2023

Authorization Policy issue with 503

Related Topics