Hi,
I am trying to setup a mesh-external nginx and talk to it from a Pod inside the mesh via plain HTTP, but get the sidecar to upgrade the connection to mutual TLS.
I have setup a service entry and a destination rule to advertise the external service and upgrade the connection. Curl from the istio-proxy container, specifying all certificates manually, works fine. But curl from the application container, I cannot seem to get to work.
Would anybody be so kind as to take a peek at my configs so far, and point out what could be going wrong?
Thanks a lot already in advance!
Services & deployments:
cat <<EOF | kubectl apply -n $NAMESPACE -f -
apiVersion: v1
kind: Service
metadata:
name: my-nginx
labels:
run: my-nginx
spec:
ports:
- port: 443
protocol: TCP
selector:
run: my-nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 1
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 443
volumeMounts:
- name: nginx-config
mountPath: /etc/nginx
readOnly: true
- name: nginx-certs
mountPath: /etc/nginx/server_certs/nginx.example.com
readOnly: true
- name: ca-certs
mountPath: /etc/nginx/client_certs/example.com
readOnly: true
volumes:
- name: nginx-config
configMap:
name: nginx-configmap
- name: nginx-certs
secret:
secretName: nginx-credential
- name: ca-certs
secret:
secretName: nginx-credential-cacert
EOF
cat <<EOF | istioctl kube-inject -f - | kubectl apply -n $NAMESPACE -f -
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: sleep
name: sleep
spec:
replicas: 1
selector:
matchLabels:
app: sleep
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
annotations:
sidecar.istio.io/inject: "true"
sidecar.istio.io/userVolume: '{"client-secret":{"secret":{"secretName":"sleep-credential"}},"ca-secret":{"secret":{"secretName":"sleep-credential-cacert"}}}'
sidecar.istio.io/userVolumeMount: '{"client-secret":{"mountPath":"/etc/sleep/client_certs/client.example.com","readOnly":true},"ca-secret":{"mountPath":"/etc/sleep/ca_certs/example.com"}}'
labels:
app: sleep
spec:
containers:
- command:
- /bin/sleep
- 3650d
image: pstauffer/curl
imagePullPolicy: IfNotPresent
name: sleep
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
EOF
Service entry & destination rule:
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: nginx-example-com
spec:
hosts:
- nginx.example.com
location: MESH_EXTERNAL
ports:
- name: http-port
number: 443
protocol: HTTP
resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: nginx-example-com
spec:
host: nginx.example.com
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
portLevelSettings:
- port:
number: 443
tls:
mode: MUTUAL
clientCertificate: /etc/sleep/client_certs/client.example.com/tls.crt
privateKey: /etc/sleep/client_certs/client.example.com/tls.key
caCertificates: /etc/sleep/ca_certs/example.com/cacert
Curl from sleep / curl from istio-proxy:
user@node:~$ export NAMESPACE=egress-tls-test
user@node:~$ export NGINX_CLUSTER_IP=$(kubectl get svc -n $NAMESPACE my-nginx -o jsonpath={.spec.clusterIP})
user@node:~$ export SLEEP_POD=$(kubectl get pod -n $NAMESPACE -l app=sleep -o jsonpath={.items..metadata.name})
user@node:~$ k exec -n $NAMESPACE $SLEEP_POD -c sleep -- curl -v --resolve nginx.example.com:443:$NGINX_CLUSTER_IP -HHost:nginx.example.com http://nginx.example.com:443
* Added nginx.example.com:443:10.105.47.172 to DNS cache
* Rebuilt URL to: http://nginx.example.com:443/
* Hostname nginx.example.com was found in DNS cache
* Trying 10.105.47.172...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to nginx.example.com (10.105.47.172) port 443 (#0)
> GET / HTTP/1.1
> Host:nginx.example.com
> User-Agent: curl/7.60.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< date: Tue, 17 Mar 2020 16:06:16 GMT
< server: envoy
< content-length: 0
<
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
* Connection #0 to host nginx.example.com left intact
user@node:~$ k exec -n $NAMESPACE $SLEEP_POD -c istio-proxy -- curl --resolve nginx.example.com:443:$NGINX_CLUSTER_IP --cacert /etc/sleep/ca_certs/example.com/cacert --cert /etc/sleep/client_certs/client.example.com/tls.crt --key /etc/sleep/client_certs/client.example.com/tls.key https://nginx.example.com
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
100 612 100 612 0 0 149k 0 --:--:-- --:--:-- --:--:-- 149k