I am using Istio 1.8.0 with on-prem k8s v1.19…We have several microservices running where I am using STRICT mode for peerauthentication. And I can verify that if I use PERMISSIVE mode I did not receive any 503 errors.
I really get stuck to find any solution cause I do not want to use PERMISSIVE mode as recommended.
Here is the log for istio ingressgateway.
$kubectl logs -f istio-ingressgateway-75496c97df-44g6l -n istio-system
[2021-01-16T07:28:51.852Z] "GET /config HTTP/1.1" 503 URX "-" 0 95 57 57 "95.0.145.40,10.6.0.21" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36" "foo-bar.example.com" "10.6.25.34:3000" outbound|80||oneapihub-ui-dev.hub-dev.svc.cluster.local 10.6.5.216:46364 10.6.5.216:8080 10.6.0.21:14387 - -
Here is the log for my ui microservice… I did not catch any 503 errors for rest of services
$ kubectl get ns --show-labels
NAME STATUS AGE LABELS
hub-dev Active 2d19h istio-injection=enabled
istio-system Active 3d5h istio-injection=disabled
$ kubectl get pods -n hub-dev
NAME READY STATUS RESTARTS AGE
oneapihub-api-dev-79dff67cdb-hx754 3/3 Running 0 15h
oneapihub-auth-dev-76cfcb6cb4-74ljq 3/3 Running 0 15h
oneapihub-backend-dev-d76799bcd-bmwjn 2/2 Running 0 15h
oneapihub-cronjob-dev-6879dbf9b8-wvpnp 3/3 Running 0 15h
oneapihub-mp-dev-864794d446-cfqj7 3/3 Running 0 15h
oneapihub-ui-dev-67d7bb6779-8z4xt 2/2 Running 0 14h
redis-hub-master-0 2/2 Running 0 15h
$ kubectl logs -f oneapihub-ui-dev-67d7bb6779-8z4xt -n hub-dev -c istio-proxy
[2021-01-15T14:17:24.698Z] "GET /config HTTP/1.1" 503 URX "-" 0 95 65 64 "95.0.145.40,10.6.0.19" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36" "foo-bar.example.com" "10.6.25.241:3000" outbound|80||oneapihub-ui-dev.hub-dev.svc.cluster.local 10.6.5.216:37584 10.6.5.216:8080 10.6.0.19:31138 - -
[2021-01-15T14:17:24.817Z]
Here is peerauthentication
$ kubectl get peerauthentication -n istio-system -o yaml
apiVersion: v1
items:
- apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"security.istio.io/v1beta1","kind":"PeerAuthentication","metadata":{"annotations":{},"name":"default","namespace":"istio-system"},"spec":{"mtls":{"mode":"STRICT"}}}
name: default
namespace: istio-system
spec:
mtls:
mode: STRICT
Here is the service.yaml of my frontend app.
capel0068340585:~ semural$ kubectl get svc -n hub-dev
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
oneapihub-ui-dev ClusterIP 10.254.47.95 <none> 80/TCP 48m
# Source: oneapihub-ui/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: RELEASE-NAME-oneapihub-ui
labels:
app.kubernetes.io/name: oneapihub-ui
app.kubernetes.io/instance: RELEASE-NAME
app.kubernetes.io/managed-by: Helm
helm.sh/chart: oneapihub-ui-0.1.0
spec:
type: ClusterIP
ports:
- port: 80
targetPort: 3000
protocol: TCP
name: http
Here is the Gateway and VS that I created in same namepace where microservices are running.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: hub-gateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- foo-bar.example.com
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: hub
spec:
hosts:
- foo-bar.example.com
gateways:
- hub-gateway
http:
- route:
- destination:
host: oneapihub-ui-dev.hub-dev.svc.cluster.local
port:
number: 80
Here is the proxy config for ingressgateway
$ istioctl proxy-config route istio-ingressgateway-75496c97df-44g6l -n istio-system -o json
[
{
"name": "http.80",
"virtualHosts": [
{
"name": "foo-baexample.com:80",
"domains": [
"foo-bar.example.com",
"foo-bar.example.com:*"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"cluster": "outbound|80||oneapihub-ui-dev.hub-dev.svc.cluster.local",
"timeout": "0s",
"retryPolicy": {
"retryOn": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes",
"numRetries": 2,
"retryHostPredicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts"
}
],
Here are the outputs of istioctl describe pods.
$ istioctl x describe pod oneapihub-api-dev-78fbccf48c-5hb4c -n hub-dev
Pod: oneapihub-api-dev-78fbccf48c-5hb4c
Pod Ports: 50004 (oneapihub-api), 15090 (istio-proxy)
--------------------
Service: oneapihub-api-dev
Port: http 80/HTTP targets pod port 50004
$ istioctl x describe pod oneapihub-auth-dev-7f8998cd69-gzmnm -n hub-dev
Pod: oneapihub-auth-dev-7f8998cd69-gzmnm
Pod Ports: 50002 (oneapihub-auth), 15090 (istio-proxy)
--------------------
Service: oneapihub-auth-dev
Port: http 80/HTTP targets pod port 5000
$ istioctl x describe pod oneapihub-backend-dev-849b4bcd5d-fcm4l -n hub-dev
Pod: oneapihub-backend-dev-849b4bcd5d-fcm4l
Pod Ports: 50001 (oneapihub-backend), 15090 (istio-proxy)
--------------------
Service: oneapihub-backend-dev
Port: http 80/HTTP targets pod port 50001
$ istioctl x describe pod oneapihub-cronjob-dev-58b64d9c68-lv5bk -n hub-dev
Pod: oneapihub-cronjob-dev-58b64d9c68-lv5bk
Pod Ports: 50005 (oneapihub-cronjob), 15090 (istio-proxy)
--------------------
Service: oneapihub-cronjob-dev
Port: http 80/HTTP targets pod port 50005
$ istioctl x describe pod oneapihub-mp-dev-74fd6ffc9f-65gh5 -n hub-dev
Pod: oneapihub-mp-dev-74fd6ffc9f-65gh5
Pod Ports: 50003 (oneapihub-mp), 15090 (istio-proxy)
--------------------
Service: oneapihub-mp-dev
Port: http 80/HTTP targets pod port 50003
$ istioctl x describe pod oneapihub-ui-dev-7fd56f747c-nr5fk -n hub-dev
Pod: oneapihub-ui-dev-7fd56f747c-nr5fk
Pod Ports: 3000 (oneapihub-ui), 15090 (istio-proxy)
--------------------
Service: oneapihub-ui-dev
Port: http 80/HTTP targets pod port 3000
Exposed on Ingress Gateway http://53.6.48.168
VirtualService: hub
1 HTTP route(s)
$ istioctl x describe pod redis-hub-master-0 -n hub-dev
Pod: redis-hub-master-0
Pod Ports: 6379 (redis-hub), 15090 (istio-proxy)
Suggestion: add 'version' label to pod for Istio telemetry.
--------------------
Service: redis-hub-headless
Port: redis 6379/Redis targets pod port 6379
--------------------
Service: redis-hub-master
Port: redis 6379/Redis targets pod port 6379