I just ran into this exact issue, and adding proxy_ssl_server_name fixed my broken attempts at using nginx as a proxy between services in two kubernetes clusters.
I did stumble upon one clue that hints at this solution in the envoy access logs on the ingress gateways. A particular field named REQESTED_SERVER_NAME was always empty in the logs for my failed requests, the 404s. When I curled the same virtual service directly REQUESTED_SERVER_NAME would list the target virtual service host.
This GH discussion in the envoy code ties REQUESTED_SERVER_NAME back to SNI, and as soon as enabled proxy_ssl_server_name in nginx config I saw REQUESTED_SERVER_NAME list the correct host in the access logs of my now successful proxied requests.