Missing selected ALPN property, TLS with AWS Certificate, AWS classic LB

Security
1

Hi there,

I am having an issue with the ALPN properties when using gRPC across TLS with AWS certificates.

The TLS negotiation appears to proceed but appears to fall back to HTTP 1.1 rather than gRPC HTTP 2.0. I see this with curl and also the same issue with the python gRPC client with TLS. When this happens the python gRPC client treats this as a fatal error.

I have read https://github.com/istio/istio/issues/5769 and related items with no clues found.

kops is being used to create the cluster and I am using the service.beta.kubernetes.io/aws-load-balancer- annotations to set the istio-ingressgateway in the istio-system namespace to load the AWS ARN into the AWS classic load balancer successfully.

When I used self signed certs and relied completely on the istio gateway to perform TLS security things appear to work.

Does anyone have suggestions on using HTTP/2 gRPC with the AWS load balancer option ?

Thanks,
Karl

__call__
return _end_unary_response_blocking(state, call, False, None)
File "/home/kmutch/project/src/github.com/leaf-ai/ascend-autoseg/autoseg_grpc/python/esp/.direnv/python-3.6.9/lib/python3.6/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
    status = StatusCode.UNAVAILABLE
    details = "Connect Failed"
    debug_error_string = "{"created":"@1570048658.193487347","description":"Failed to create subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2721,"referenced_errors":[{"created":"@1570048658.193428990","description":"Pick Cancelled","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":241,"referenced_errors":[{"created":"@1570048658.193203968","description":"Connect Failed","file":"src/core/ext/filters/client_channel/subchannel.cc","file_line":689,"grpc_status":14,"referenced_errors":[{"created":"@1570048658.193012936","description":"Cannot check peer: missing selected ALPN property.","file":"src/core/lib/security/security_connector/security_connector.cc","file_line":869}]}]}]}"

When using curl without the --http2-prior-knowledge option the fall back can be seen as follows

curl --http2 -v https://esp.cognizant-ai.net  
* Rebuilt URL to: https://esp.cognizant-ai.net/
*   Trying 52.43.76.166...
* TCP_NODELAY set
* Connected to esp.cognizant-ai.net (52.43.76.166) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=esp.cognizant-ai.net
*  start date: Oct  2 00:00:00 2019 GMT
*  expire date: Nov  2 12:00:00 2020 GMT
*  subjectAltName: host "esp.cognizant-ai.net" matched cert's "esp.cognizant-ai.net"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: esp.cognizant-ai.net
> User-Agent: curl/7.58.0
> Accept: */*

When using the --http2-prior-knowledge option we still see the following:

* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=esp.cognizant-ai.net
....
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x56540429f580)
> GET / HTTP/2
> Host: esp.cognizant-ai.net
> User-Agent: curl/7.58.0
> Accept: */*
> 
* http2 error: Remote peer returned unexpected data while we expected SETTINGS frame.  Perhaps, peer does not support HTTP/2 properly.
* Connection #0 to host esp.cognizant-ai.net left intact