After upgrading to Istio 1.10.0 from 1.9.1, all gRPC requests across the internet (via public url) to publicly exposed services fail with this in the sidecar proxy:
2021-05-27T00:45:29.328138305Z [2021-05-27T00:45:25.603Z[] "POST /REDACTEDROUTE/REDACTEDSUBROUTE HTTP/2" 200 UR upstream_reset_before_response_started{remote_reset} - "-" 8 0 1 - "10.252.40.95" "grpc-dotnet/2.29.0.0" "5743dd73-c019-4522-93e4-d77549e12830" "[REDACTED_PUBLIC_URL]" "10.252.40.205:5010" inbound|5010|| 127.0.0.6:42273 10.252.40.205:5010 10.252.40.95:0 [REDACTED_PUBLIC_URL] default
This occurs many times before the request gives up.
Below is the configuration with names/urls/keys redacted:
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: api-gateway
namespace: default
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- '*'
port:
name: http-apis
number: 80
protocol: HTTP2
tls:
httpsRedirect: true
- hosts:
- '*'
port:
name: https-apis
number: 443
protocol: HTTPS
tls:
minProtocolVersion: TLSV1_2
mode: SIMPLE
privateKey: [key]
serverCertificate: [cert]
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
spec:
gateways:
- api-gateway
- mesh
hosts:
- [REDACTED PUBLIC URL]
- [servicename].default.svc.cluster.local
- [servicename]
http:
- corsPolicy:
allowHeaders:
- DNT
- X-CustomHeader
- Keep-Alive
- User-Agent
- X-Requested-With
- If-Modified-Since
- Cache-Control
- Content-Type
- Authorization
- ...
- ...
- ...
match:
- headers:
content-type:
exact: application/grpc
route:
- destination:
host: [servicename]
port:
number: 5010
- match:
- port: 80
- port: 443
route:
- destination:
host: [servicename]
port:
number: 80
headers:
response:
add:
Referrer-Policy: no-referrer
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-DNS-Prefetch-Control: "off"
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
remove:
- server
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
...
spec:
host: [servicename]
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
Any help would be greatly appreciated.
istio
client version: 1.10.0
control plane version: 1.10.0
data plane version: 1.10.0 (72 proxies)
k8s
Server Version: v1.20.5
How was Istio installed?
Istio-Operator
Environment where the bug was observed (cloud vendor, OS, etc)
Azure AKS