Hi All #networking #security #policies-and-telemetry #config ,
I have a simple application with
- .Net core MVC (UI) service called GiftShopUI
- .Net core WebAPI (REST) service called GiftShopAPI
I have deployed them on AKS (K8s version 1.13.10), istio version 1.4. I have created a namespace called “microservices” in my cluster and enabled automatic sidecar injection on that namespace.
~ $kubectl get namespace -L istio-injection
NAME STATUS AGE ISTIO-INJECTION
cert-manager Active 23h
default Active 30d
istio-system Active 7d16h
kube-public Active 30d
kube-system Active 30d
microservices Active 4d22h enabled
Please find my service, virtual service and gateway definitions below.
GiftShopAPI Service and Deployment
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: giftshopapi-v2
spec:
replicas: 1
template:
metadata:
labels:
app: giftshopapi
tier: backend
version: v2
spec:
containers:
- name: giftshopapi
image: myacr.azurecr.io/giftshop/sreeni-giftshopapi:1.0
imagePullPolicy: Always
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 80
imagePullSecrets:
- name: myacrsecret
---
apiVersion: v1
kind: Service
metadata:
name: giftshopapi
spec:
selector:
app: giftshopapi
ports:
- port: 80
name: http-giftshopapi
GiftShopUI Service and Deployment
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: giftshopui-v6
spec:
replicas: 1
template:
metadata:
labels:
app: giftshopui
tier: frontend
version: v6
spec:
containers:
- name: giftshopui
image: myacr.azurecr.io/giftshop/sreeni-giftshopui:1.0
imagePullPolicy: Always
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 80
env:
- name: ASPNETCORE_ENVIRONMENT
value: "Development"
- name: ExternalDependencies_GiftShopAPI_BaseUrl
valueFrom:
configMapKeyRef:
name: giftshopui-config
key: GiftShopAPIBaseUrl
imagePullSecrets:
- name: myacrsecret
---
apiVersion: v1
kind: Service
metadata:
name: giftshopui
spec:
selector:
app: giftshopui
ports:
- port: 80
name: http-giftshopui
GiftShopUI ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: giftshopui-config
data:
GiftShopAPIBaseUrl: http://giftshopapi.microservices.svc.cluster.local/
Istio Gateway and Virtual Services for GiftShopAPI and GiftShopUI
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: demo-ingress-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- giftshopapi.<my-godaddy-domain-name>.com
- giftshopui.<my-godaddy-domain-name>.com
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: giftshopapi
spec:
hosts:
- giftshopapi.<my-godaddy-domain-name>.com
gateways:
- demo-ingress-gateway
http:
- match:
- uri:
prefix: /swagger
- uri:
prefix: /api
route:
- destination:
host: giftshopapi.microservices.svc.cluster.local
port:
number: 80
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: giftshopui
spec:
hosts:
- giftshopui.<my-godaddy-domain-name>.com
gateways:
- demo-ingress-gateway
http:
- route:
- destination:
host: giftshopui.microservices.svc.cluster.local
port:
number: 80
Until this step, I have not enabled mTLS and the services are able to communicate with each other and the app works as expected. The app is fully functional at this point in time.
Please find the settings returned by istioctl authn tls-check below.
~ $istioctl authn tls-check giftshopui-v6-66fbd6b784-qcb6s.microservices giftshopapi.microservices.svc.cluster.local
HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
giftshopapi.microservices.svc.cluster.local:80 OK PERMISSIVE - /default -
~ $istioctl authn tls-check giftshopapi-v2-6ddd8b6668-gst48.microservices giftshopui.microservices.svc.cluster.local
HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
giftshopui.microservices.svc.cluster.local:80 OK PERMISSIVE - /default -
I have enabled krew and WireShark as suggested here (https://blog.getambassador.io/verifying-service-mesh-tls-in-kubernetes-using-ksniff-and-wireshark-454b1e3f4dc9) to sniff the traffic between GiftShopUI and GiftShopAPI pods. It looks all good at this point and I am able to see the calls from GiftShopUI to GiftShopAPI.
Now I enable mTLS at namespace level using the below yamls.
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
name: "enable-mtls"
namespace: "microservices"
spec:
host: "*.microservices.svc.cluster.local"
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
name: "default"
namespace: "microservices"
spec:
peers:
- mtls: {}
Please find the settings returned by istioctl authn tls-check below.
~ $istioctl authn tls-check giftshopui-v6-66fbd6b784-qcb6s.microservices giftshopapi.microservices.svc.cluster.local
HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
giftshopapi.microservices.svc.cluster.local:80 OK STRICT ISTIO_MUTUAL microservices/default microservices/enable-mtls
~ $istioctl authn tls-check giftshopapi-v2-6ddd8b6668-gst48.microservices giftshopui.microservices.svc.cluster.local
HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
giftshopui.microservices.svc.cluster.local:80 OK STRICT ISTIO_MUTUAL microservices/default microservices/enable-mtls
Now i do not see any traffic flowing in WireShark and I start getting the below error.
upstream connect error or disconnect/reset before headers. reset reason: connection failure
I have verified the certs in my pods and they seem to be ok.
istio-1.3.3 $kubectl exec giftshopui-v6-66fbd6b784-qcb6s -c istio-proxy -- ls /etc/certs
cert-chain.pem
key.pem
root-cert.pem
istio-1.3.3 $kubectl get pods
NAME READY STATUS RESTARTS AGE
giftshopapi-v2-6ddd8b6668-gst48 2/2 Running 0 3h
giftshopui-v6-66fbd6b784-qcb6s 2/2 Running 0 3h
istio-1.3.3 $kubectl exec giftshopapi-v2-6ddd8b6668-gst48 -c istio-proxy -- ls /etc/certs
cert-chain.pem
key.pem
root-cert.pem
istio-1.3.3 $kubectl exec giftshopui-v6-66fbd6b784-qcb6s -c istio-proxy -- cat /etc/certs/cert-chain.pem | openssl x509 -text -noout | grep Validity -A 2
Validity
Not Before: Oct 13 15:29:15 2019 GMT
Not After : Jan 11 15:29:15 2020 GMT
istio-1.3.3 $kubectl exec giftshopapi-v2-6ddd8b6668-gst48 -c istio-proxy -- cat /etc/certs/cert-chain.pem | openssl x509 -text -noout | grep Validity -A 2
Validity
Not Before: Oct 13 15:29:15 2019 GMT
Not After : Jan 11 15:29:15 2020 GMT
Any help is much appreciated.