Authorization Policy issue with 503

#1

Hello,

I want to disable the access from external to certain endpoints on one of my projects. So I started to use the AuthorizationPolicy without success.

I use Istio 1.4.0 and I have enabled mTls on my namespace

HOST:PORT                                                      STATUS     SERVER     CLIENT           AUTHN POLICY                DESTINATION RULE
xxxx-app.xxxxx.svc.cluster.local:8080     OK         STRICT     ISTIO_MUTUAL     xxxxx/default     xxxxx/xxxx-app

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
    metadata:
    name: "default"
    labels: 
        app: {{ template "application.name" . }}
        chart: {{ template "application.chart" . }}
        heritage: {{ .Release.Service }}
        release: {{ template "system.name" . }}
        environment: {{ .Values.global.environment }}
 spec:
     peers:
         - mtls: {}

And my Authorization Policty:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: {{ template "application.name" . }}
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  selector:
    matchLabels:
      app: {{ template "application.name" . }}
  rules:
  - to:
    - operation:
        methods: ["GET", "POST", "OPTIONS", "HEAD", "PUT"]

But when I go to any endpoint I just get a 503 error with “upstream connect error or disconnect/reset before headers. reset reason: connection termination”

With rules empty everythings comes back to normal:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: {{ template "application.name" . }}
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  selector:
    matchLabels:
      app: {{ template "application.name" . }}
  rules:
  - {}

Any idea on what could happen? I miss something?

Thanks!

#2

Do you have any destination rule to configure clients to use mtls when talking to your mtls-enabled services?

#3

I have these two DestinationRule:

apiVersion: networking.istio.io/v1alpha3
kind: "DestinationRule"
metadata:
  name: "default"
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  host: "*.my-system.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

and

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: {{ template "application.name" . }}
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  host: my-service
  subsets:
  - name: {{ .Values.global.environment }}
    labels:
      main: "true"

The second DestinationRule gets override and removes my subsets, dont know why.

All these are on the same namespace (its added by helm)

#4

Maybe that could be part of the problem:

kubectl exec $(kubectl get pod -l app=my-app -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl http://my-app:8080/healthz -o /dev/null -s -w '%{http_code}\n' -k
kubectl exec $(kubectl get pod -l app=my-app -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl https://my-app:8080/healthz -o /dev/null -s -w '%{http_code}\n' -k
kubectl exec $(kubectl get pod -l app=my-app -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl https://my-app:8080/healthz -o /dev/null -s -w '%{http_code}\n' --key /etc/certs/key.pem --cert /etc/certs/cert-chain.pem --cacert /etc/certs/root-cert.pem -k
000
command terminated with exit code 56
000
command terminated with exit code 16
000
command terminated with exit code 16

I’m getting that exit code 16.

#5

Policy:

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "default"
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  peers:
    - mtls: {}

Destination Rule

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: {{ template "application.name" . }}
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  host: "{{ template "application.name" . }}.{{ template "system.name" . }}.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
  subsets:
  - name: {{ .Values.global.environment }}
    labels:
      main: "true"

Virtual Service:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: {{ template "application.name" . }}
  labels:
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  {{ include "application.hosts" . }}
  {{ include "application.gateways" . }}
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: "{{ template "application.name" . }}.{{ template "system.name" . }}.svc.cluster.local"
        subset: {{ .Values.global.environment }}
        port:
          number: {{ template "application.service.port" . }}
      headers:
        response:
          add:
            x-app-dedicated: "app"
    retries:
        attempts: 3
        perTryTimeout: 2s

Authorization Policy

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: {{ template "application.name" . }}
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  selector:
    matchLabels:
      app: {{ template "application.name" . }}
  rules:
  - to:
    - operation:
        methods: ["GET"]

After apply the AuthorizationPolicy, the istio-proxy starts to show the next:

│ [2019-11-28T16:01:29.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:40873 10.40.10.156:15090 10.40.11.143:46502 - -                                                        │
│ [2019-11-28T16:01:44.947Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:48599 10.40.10.156:15090 10.40.11.143:46702 - -                                                        │
│ [2019-11-28T16:01:59.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:36675 10.40.10.156:15090 10.40.11.143:46896 - -                                                        │
│ [2019-11-28T16:02:14.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:55289 10.40.10.156:15090 10.40.11.143:47090 - -                                                        │
│ [2019-11-28T16:02:29.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:45871 10.40.10.156:15090 10.40.11.143:47284 - -                                                        │
│ [2019-11-28T16:02:44.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:40379 10.40.10.156:15090 10.40.11.143:47494 - -                                                        │
│ [2019-11-28T16:02:59.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:33319 10.40.10.156:15090 10.40.11.143:47686 - -                                                        │
│ [2019-11-28T16:03:14.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:57011 10.40.10.156:15090 10.40.11.143:47886 - -                                                        │
│ [2019-11-28T16:03:29.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:59227 10.40.10.156:15090 10.40.11.143:48078 - -                                                        │
│ [2019-11-28T16:03:44.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:52511 10.40.10.156:15090 10.40.11.143:48280 - -

I still do not understand why im getting this kind of error:

curl -X GET \
  https://mysystem.com/healthz \
  -H 'Accept: */*' \
  -H 'Accept-Encoding: gzip, deflate' \
  -H 'Cache-Control: no-cache' \
  -H 'Connection: keep-alive' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -H 'Host: mysystem.com' \
  -H 'Postman-Token: f4455c4e-34bb-41d2-8a0d-d219e5def917,b35173cb-8334-4c82-9003-e0b044e0fdc9' \
  -H 'User-Agent: PostmanRuntime/7.20.1' \
  -H 'cache-control: no-cache'


503 - upstream connect error or disconnect/reset before headers. reset reason: connection termination

This AuthorizationPolicy should work without mTLS enabled?

Thanks

#6

On the environment where im testing this policies kiali only shows me istio-telemetry > telemetry.

But on other environment where I dont have mTls active:

13

It shows traffic to policy.

Maybe some wrong installation with istio on that environment?

#7

@YangminZhu @diemtvu

#8

AuthorizationPolicy should never return 503 error, it should only cause 401. It seems something wrong in the networking or mTLS part.

Could you confirm the following:

  1. Do you see 503 if you remove the authorization policy?
  2. Do you see 503 if you remove the authentication policy?
  3. What’s your service definition?
#9

Can you also check the DR for policy and telemetry (mixer). Failure to talk to policy also shows up as 503 in data plane.