Failing to secure egress HTTP with TLS

Hi,

I am trying to setup a mesh-external nginx and talk to it from a Pod inside the mesh via plain HTTP, but get the sidecar to upgrade the connection to mutual TLS.
I have setup a service entry and a destination rule to advertise the external service and upgrade the connection. Curl from the istio-proxy container, specifying all certificates manually, works fine. But curl from the application container, I cannot seem to get to work.

Would anybody be so kind as to take a peek at my configs so far, and point out what could be going wrong?

Thanks a lot already in advance!

Services & deployments:

cat <<EOF | kubectl apply -n $NAMESPACE -f -
apiVersion: v1
kind: Service
metadata:
  name: my-nginx
  labels:
    run: my-nginx
spec:
  ports:
  - port: 443
    protocol: TCP
  selector:
    run: my-nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  selector:
    matchLabels:
      run: my-nginx
  replicas: 1
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx
        ports:
        - containerPort: 443
        volumeMounts:
        - name: nginx-config
          mountPath: /etc/nginx
          readOnly: true
        - name: nginx-certs
          mountPath: /etc/nginx/server_certs/nginx.example.com
          readOnly: true
        - name: ca-certs
          mountPath: /etc/nginx/client_certs/example.com
          readOnly: true
      volumes:
      - name: nginx-config
        configMap:
          name: nginx-configmap
      - name: nginx-certs
        secret:
          secretName: nginx-credential
      - name: ca-certs
        secret:
          secretName: nginx-credential-cacert
EOF
cat <<EOF | istioctl kube-inject -f - | kubectl apply -n $NAMESPACE -f -
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: sleep
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
        sidecar.istio.io/userVolume: '{"client-secret":{"secret":{"secretName":"sleep-credential"}},"ca-secret":{"secret":{"secretName":"sleep-credential-cacert"}}}'
        sidecar.istio.io/userVolumeMount: '{"client-secret":{"mountPath":"/etc/sleep/client_certs/client.example.com","readOnly":true},"ca-secret":{"mountPath":"/etc/sleep/ca_certs/example.com"}}'
      labels:
        app: sleep
    spec:
      containers:
      - command:
        - /bin/sleep
        - 3650d
        image: pstauffer/curl
        imagePullPolicy: IfNotPresent
        name: sleep
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
EOF

Service entry & destination rule:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: nginx-example-com
spec:
  hosts:
  - nginx.example.com
  location: MESH_EXTERNAL
  ports:
  - name: http-port
    number: 443
    protocol: HTTP
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: nginx-example-com
spec:
  host: nginx.example.com
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 443
      tls:
        mode: MUTUAL
        clientCertificate: /etc/sleep/client_certs/client.example.com/tls.crt
        privateKey: /etc/sleep/client_certs/client.example.com/tls.key
        caCertificates: /etc/sleep/ca_certs/example.com/cacert

Curl from sleep / curl from istio-proxy:

user@node:~$ export NAMESPACE=egress-tls-test
user@node:~$ export NGINX_CLUSTER_IP=$(kubectl get svc -n $NAMESPACE my-nginx -o jsonpath={.spec.clusterIP})
user@node:~$ export SLEEP_POD=$(kubectl get pod -n $NAMESPACE -l app=sleep -o jsonpath={.items..metadata.name})
user@node:~$ k exec -n $NAMESPACE $SLEEP_POD -c sleep -- curl -v --resolve nginx.example.com:443:$NGINX_CLUSTER_IP -HHost:nginx.example.com http://nginx.example.com:443
* Added nginx.example.com:443:10.105.47.172 to DNS cache
* Rebuilt URL to: http://nginx.example.com:443/
* Hostname nginx.example.com was found in DNS cache
*   Trying 10.105.47.172...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to nginx.example.com (10.105.47.172) port 443 (#0)
> GET / HTTP/1.1
> Host:nginx.example.com
> User-Agent: curl/7.60.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< date: Tue, 17 Mar 2020 16:06:16 GMT
< server: envoy
< content-length: 0
<
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host nginx.example.com left intact
user@node:~$ k exec -n $NAMESPACE $SLEEP_POD -c istio-proxy -- curl --resolve nginx.example.com:443:$NGINX_CLUSTER_IP --cacert /etc/sleep/ca_certs/example.com/cacert --cert /etc/sleep/client_certs/client.example.com/tls.crt --key /etc/sleep/client_certs/client.example.com/tls.key https://nginx.example.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
100   612  100   612    0     0   149k      0 --:--:-- --:--:-- --:--:--  149k
1 Like

@JimmyChen Jimmy, can you take a look?

I use the similar method and it can work in 1.7.3 and 1.6.7.
The new problem is after using this configuration, the ROOTCA is replaced by the ca cert assigned in annotation, do you meet the same problem. I will raise new issue soon. You can use command "istioctl pc secret . to find it.
The new issue is #28143 sidecar egress mtls configuration will override ROOTCA to make internal mtls fail