Bug Description
When enabling strict mTLS with sidecar.istio.io/rewriteAppHTTPProbers: true annotation (both at workload level) the startup health probes are failing for a http2 only service. Changing the mTLS to permissive mode works just fine (w/o the annotation)
Tested the same approach on the another service (not http2 only) and the probes work just fine. So, the problem is only when mTLS is set to strict and the the app HTTP probers are overwritten for http2 only service. So, suspecting that the rewriteAppHTTPProbers annotation is sending the probe as http 1.x and not http2?
Also, tried setting h2UpgradePolicy: UPGRADE on the destination rule for the service but no luck (may be since kubelet is not behind sidecar?)
Version
istioctl:
client version: 1.14.1
control plane version: 1.14.0
data plane version: 1.14.0
kubectl:
Client Version: v1.23.1
Server Version: v1.23.5
helm:
v3.7.2+g663a896
Additional Information
istio proxy log:
2022-07-27T14:01:56.064088Z error Request to probe app failed: Get "https://<pod-ip>:443/": dial tcp 127.0.0.6:0-><pod-ip>:443: connect: connection refused, original URL path = /app-health/service-pds/startupz
app URL path = /
2022-07-27T14:02:06.452825Z error Request to probe app failed: Get "https://<pod-ip>:443/": EOF, original URL path = /app-health/service-pds/startupz
app URL path = /
2022-07-27T14:02:16.070669Z error Request to probe app failed: Get "https://<pod-ip>:443/": EOF, original URL path = /app-health/service-pds/startupz
app URL path = /
service container (http2) log:
[14:02:26 ERR] ({SourceContext="Microsoft.AspNetCore.Server.Kestrel", ConnectionId="0HMJFR2HC85QE"}) HTTP/2 over TLS was not negotiated on an HTTP/2-only endpoint.
[14:02:36 ERR] ({SourceContext="Microsoft.AspNetCore.Server.Kestrel", ConnectionId="0HMJFR2HC85QF"}) HTTP/2 over TLS was not negotiated on an HTTP/2-only endpoint.
[14:02:46 ERR] ({SourceContext="Microsoft.AspNetCore.Server.Kestrel", ConnectionId="0HMJFR2HC85QG"}) HTTP/2 over TLS was not negotiated on an HTTP/2-only endpoint.