Authorization Policy issue with 503

Hello,

I want to disable the access from external to certain endpoints on one of my projects. So I started to use the AuthorizationPolicy without success.

I use Istio 1.4.0 and I have enabled mTls on my namespace

HOST:PORT                                                      STATUS     SERVER     CLIENT           AUTHN POLICY                DESTINATION RULE
xxxx-app.xxxxx.svc.cluster.local:8080     OK         STRICT     ISTIO_MUTUAL     xxxxx/default     xxxxx/xxxx-app

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
    metadata:
    name: "default"
    labels: 
        app: {{ template "application.name" . }}
        chart: {{ template "application.chart" . }}
        heritage: {{ .Release.Service }}
        release: {{ template "system.name" . }}
        environment: {{ .Values.global.environment }}
 spec:
     peers:
         - mtls: {} 

And my Authorization Policty:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: {{ template "application.name" . }}
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  selector:
    matchLabels:
      app: {{ template "application.name" . }}
  rules:
  - to:
    - operation:
        methods: ["GET", "POST", "OPTIONS", "HEAD", "PUT"]

But when I go to any endpoint I just get a 503 error with “upstream connect error or disconnect/reset before headers. reset reason: connection termination”

With rules empty everythings comes back to normal:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: {{ template "application.name" . }}
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  selector:
    matchLabels:
      app: {{ template "application.name" . }}
  rules:
  - {}

Any idea on what could happen? I miss something?

Thanks!

Do you have any destination rule to configure clients to use mtls when talking to your mtls-enabled services?

I have these two DestinationRule:

apiVersion: networking.istio.io/v1alpha3
kind: "DestinationRule"
metadata:
  name: "default"
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  host: "*.my-system.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

and

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: {{ template "application.name" . }}
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  host: my-service
  subsets:
  - name: {{ .Values.global.environment }}
    labels:
      main: "true"

The second DestinationRule gets override and removes my subsets, dont know why.

All these are on the same namespace (its added by helm)

Maybe that could be part of the problem:

kubectl exec $(kubectl get pod -l app=my-app -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl http://my-app:8080/healthz -o /dev/null -s -w '%{http_code}\n' -k
kubectl exec $(kubectl get pod -l app=my-app -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl https://my-app:8080/healthz -o /dev/null -s -w '%{http_code}\n' -k
kubectl exec $(kubectl get pod -l app=my-app -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl https://my-app:8080/healthz -o /dev/null -s -w '%{http_code}\n' --key /etc/certs/key.pem --cert /etc/certs/cert-chain.pem --cacert /etc/certs/root-cert.pem -k
000
command terminated with exit code 56
000
command terminated with exit code 16
000
command terminated with exit code 16

I’m getting that exit code 16.

Policy:

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "default"
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  peers:
    - mtls: {}

Destination Rule

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: {{ template "application.name" . }}
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  host: "{{ template "application.name" . }}.{{ template "system.name" . }}.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
  subsets:
  - name: {{ .Values.global.environment }}
    labels:
      main: "true"

Virtual Service:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: {{ template "application.name" . }}
  labels:
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  {{ include "application.hosts" . }}
  {{ include "application.gateways" . }}
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: "{{ template "application.name" . }}.{{ template "system.name" . }}.svc.cluster.local"
        subset: {{ .Values.global.environment }}
        port:
          number: {{ template "application.service.port" . }}
      headers:
        response:
          add:
            x-app-dedicated: "app"
    retries:
        attempts: 3
        perTryTimeout: 2s

Authorization Policy

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: {{ template "application.name" . }}
  labels: 
    app: {{ template "application.name" . }}
    chart: {{ template "application.chart" . }}
    heritage: {{ .Release.Service }}
    release: {{ template "system.name" . }}
    environment: {{ .Values.global.environment }}
spec:
  selector:
    matchLabels:
      app: {{ template "application.name" . }}
  rules:
  - to:
    - operation:
        methods: ["GET"]

After apply the AuthorizationPolicy, the istio-proxy starts to show the next:

│ [2019-11-28T16:01:29.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:40873 10.40.10.156:15090 10.40.11.143:46502 - -                                                        │
│ [2019-11-28T16:01:44.947Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:48599 10.40.10.156:15090 10.40.11.143:46702 - -                                                        │
│ [2019-11-28T16:01:59.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:36675 10.40.10.156:15090 10.40.11.143:46896 - -                                                        │
│ [2019-11-28T16:02:14.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:55289 10.40.10.156:15090 10.40.11.143:47090 - -                                                        │
│ [2019-11-28T16:02:29.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:45871 10.40.10.156:15090 10.40.11.143:47284 - -                                                        │
│ [2019-11-28T16:02:44.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:40379 10.40.10.156:15090 10.40.11.143:47494 - -                                                        │
│ [2019-11-28T16:02:59.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:33319 10.40.10.156:15090 10.40.11.143:47686 - -                                                        │
│ [2019-11-28T16:03:14.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:57011 10.40.10.156:15090 10.40.11.143:47886 - -                                                        │
│ [2019-11-28T16:03:29.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:59227 10.40.10.156:15090 10.40.11.143:48078 - -                                                        │
│ [2019-11-28T16:03:44.946Z] "- - -" 0 - "-" "-" 0 0 0 - "-" "-" "-" "-" "10.40.10.156:15090" InboundPassthroughClusterIpv4 127.0.0.6:52511 10.40.10.156:15090 10.40.11.143:48280 - -

I still do not understand why im getting this kind of error:

curl -X GET \
  https://mysystem.com/healthz \
  -H 'Accept: */*' \
  -H 'Accept-Encoding: gzip, deflate' \
  -H 'Cache-Control: no-cache' \
  -H 'Connection: keep-alive' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -H 'Host: mysystem.com' \
  -H 'Postman-Token: f4455c4e-34bb-41d2-8a0d-d219e5def917,b35173cb-8334-4c82-9003-e0b044e0fdc9' \
  -H 'User-Agent: PostmanRuntime/7.20.1' \
  -H 'cache-control: no-cache'


503 - upstream connect error or disconnect/reset before headers. reset reason: connection termination

This AuthorizationPolicy should work without mTLS enabled?

Thanks

On the environment where im testing this policies kiali only shows me istio-telemetry > telemetry.

But on other environment where I dont have mTls active:

13

It shows traffic to policy.

Maybe some wrong installation with istio on that environment?

@YangminZhu @diemtvu

AuthorizationPolicy should never return 503 error, it should only cause 401. It seems something wrong in the networking or mTLS part.

Could you confirm the following:

  1. Do you see 503 if you remove the authorization policy?
  2. Do you see 503 if you remove the authentication policy?
  3. What’s your service definition?

Can you also check the DR for policy and telemetry (mixer). Failure to talk to policy also shows up as 503 in data plane.