Failing to secure egress HTTP with TLS

Hi,

I am trying to setup a mesh-external nginx and talk to it from a Pod inside the mesh via plain HTTP, but get the sidecar to upgrade the connection to mutual TLS.
I have setup a service entry and a destination rule to advertise the external service and upgrade the connection. Curl from the istio-proxy container, specifying all certificates manually, works fine. But curl from the application container, I cannot seem to get to work.

Would anybody be so kind as to take a peek at my configs so far, and point out what could be going wrong?

Thanks a lot already in advance!

Services & deployments:

cat <<EOF | kubectl apply -n $NAMESPACE -f -
apiVersion: v1
kind: Service
metadata:
  name: my-nginx
  labels:
    run: my-nginx
spec:
  ports:
  - port: 443
    protocol: TCP
  selector:
    run: my-nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  selector:
    matchLabels:
      run: my-nginx
  replicas: 1
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx
        ports:
        - containerPort: 443
        volumeMounts:
        - name: nginx-config
          mountPath: /etc/nginx
          readOnly: true
        - name: nginx-certs
          mountPath: /etc/nginx/server_certs/nginx.example.com
          readOnly: true
        - name: ca-certs
          mountPath: /etc/nginx/client_certs/example.com
          readOnly: true
      volumes:
      - name: nginx-config
        configMap:
          name: nginx-configmap
      - name: nginx-certs
        secret:
          secretName: nginx-credential
      - name: ca-certs
        secret:
          secretName: nginx-credential-cacert
EOF
cat <<EOF | istioctl kube-inject -f - | kubectl apply -n $NAMESPACE -f -
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: sleep
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
        sidecar.istio.io/userVolume: '{"client-secret":{"secret":{"secretName":"sleep-credential"}},"ca-secret":{"secret":{"secretName":"sleep-credential-cacert"}}}'
        sidecar.istio.io/userVolumeMount: '{"client-secret":{"mountPath":"/etc/sleep/client_certs/client.example.com","readOnly":true},"ca-secret":{"mountPath":"/etc/sleep/ca_certs/example.com"}}'
      labels:
        app: sleep
    spec:
      containers:
      - command:
        - /bin/sleep
        - 3650d
        image: pstauffer/curl
        imagePullPolicy: IfNotPresent
        name: sleep
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
EOF

Service entry & destination rule:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: nginx-example-com
spec:
  hosts:
  - nginx.example.com
  location: MESH_EXTERNAL
  ports:
  - name: http-port
    number: 443
    protocol: HTTP
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: nginx-example-com
spec:
  host: nginx.example.com
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 443
      tls:
        mode: MUTUAL
        clientCertificate: /etc/sleep/client_certs/client.example.com/tls.crt
        privateKey: /etc/sleep/client_certs/client.example.com/tls.key
        caCertificates: /etc/sleep/ca_certs/example.com/cacert

Curl from sleep / curl from istio-proxy:

user@node:~$ export NAMESPACE=egress-tls-test
user@node:~$ export NGINX_CLUSTER_IP=$(kubectl get svc -n $NAMESPACE my-nginx -o jsonpath={.spec.clusterIP})
user@node:~$ export SLEEP_POD=$(kubectl get pod -n $NAMESPACE -l app=sleep -o jsonpath={.items..metadata.name})
user@node:~$ k exec -n $NAMESPACE $SLEEP_POD -c sleep -- curl -v --resolve nginx.example.com:443:$NGINX_CLUSTER_IP -HHost:nginx.example.com http://nginx.example.com:443
* Added nginx.example.com:443:10.105.47.172 to DNS cache
* Rebuilt URL to: http://nginx.example.com:443/
* Hostname nginx.example.com was found in DNS cache
*   Trying 10.105.47.172...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to nginx.example.com (10.105.47.172) port 443 (#0)
> GET / HTTP/1.1
> Host:nginx.example.com
> User-Agent: curl/7.60.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< date: Tue, 17 Mar 2020 16:06:16 GMT
< server: envoy
< content-length: 0
<
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host nginx.example.com left intact
user@node:~$ k exec -n $NAMESPACE $SLEEP_POD -c istio-proxy -- curl --resolve nginx.example.com:443:$NGINX_CLUSTER_IP --cacert /etc/sleep/ca_certs/example.com/cacert --cert /etc/sleep/client_certs/client.example.com/tls.crt --key /etc/sleep/client_certs/client.example.com/tls.key https://nginx.example.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
100   612  100   612    0     0   149k      0 --:--:-- --:--:-- --:--:--  149k

@JimmyChen Jimmy, can you take a look?