Istio Egress breaking Apache httpd reverse proxy

Networking
1

I am trying to implement Istio for some workloads that are already working in Kubernetes. I have encountered one case where an existing workload fails with Istio. I want to make sure I have the correct understanding. Here’s a diagram.

istio-apache-rp
istio-apache-rp.png1552×900 50 KB

The top user request path is working. The request successfully goes through the Apache reverse proxy to the external workload and is returned as expected.

The lower path is what I understand is happening in Istio. I know that the traffic is making it to the Apache container, and I think when it makes the call to the external workload, that request goes through the Envoy proxy. My first question is whether this is correct?

If my understanding is correct, then I need to figure out why the request shown by the red arrow is failing. Here is a snippet from the Apache logs.

[Wed Aug 21 21:44:07.857888 2019] [proxy:error] [pid 8:tid 139724907849472] (20014)Internal error (specific information not available): [client 127.0.0.1:34160] AH01084: pass request body failed to 192.168.15.50:443 (dev10.hprt.com), referer: https://story-6421.api.cicd.k8s.com/
[Wed Aug 21 21:44:07.858016 2019] [proxy:error] [pid 8:tid 139724907849472] [client 127.0.0.1:34160] AH00898: Error during SSL Handshake with remote server returned by /favicon.ico, referer: https://story-6421.api.cicd.k8s.com/
[Wed Aug 21 21:44:07.858024 2019] [proxy_http:error] [pid 8:tid 139724907849472] [client 127.0.0.1:34160] AH01097: pass request body failed to 192.168.15.50:443 (dev10.hprt.com) from 127.0.0.1 (), referer: https://story-6421.api.cicd.k8s.com/

I know SSL is setup correctly on dev10.hprt.com, so I think Apache must be having a problem with Envoy. I found some threads about how to fix this error in Apache, which is usually caused by a bad certificate or SSL configuration (e.g. https://serverfault.com/questions/538086/proxyerror-ah00898-error-during-ssl-handshake-with-remote-server), but even when I add those configuration changes to ignore and not validate, I get the same error.

Is it possible to achieve what I’m trying to do? Can egress be configured to just pass through the request and not attempt to terminate SSL?

1 Like
2

@Daniel_Watrous The question is how did you configure egress traffic. There are multiple options described at https://istio.io/docs/tasks/traffic-management/egress/egress-control/.

3

Hi @vadimeisenbergibm. I just double checked, and I see this in my istio ConfigMap

mode: ALLOW_ANY

I use kubectl exec... to get a shell in the container that can’t connect to the outside service, and ran a curl command. This is what I got.

root@api-termination-deployment-feature-optms-6421-7955d664fc-dn2hh:/usr/local/apache2# curl -v https://dev10.hprt.com/favicon.ico
* Expire in 0 ms for 6 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 0 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 1 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
* Expire in 2 ms for 1 (transfer 0x55ee2bcc6720)
*   Trying 192.168.15.50...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55ee2bcc6720)
* Connected to dev10.hprt.com (192.168.15.50) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to dev10.hprt.com:443
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to dev10.hprt.com:443
4

Can you try some other hostname, e.g. www.google.com, edition.cnn.com? Also can you run this code from some other pod with istio sidecar, so we will have a more full picture?

5

I see this issue https://github.com/istio/istio/issues/16458, maybe it is related. Can you try to define a ServiceEntry for dev10.hrt.com? See https://istio.io/docs/tasks/traffic-management/egress/egress-control/#access-an-external-https-service.

6

I ran the same curl command for the two URL’s you suggested from the original Pod and go this.

root@api-termination-deployment-feature-optms-6421-7955d664fc-dn2hh:/usr/local/apache2# curl -v https://www.google.com
*   Trying 172.217.3.68...
* TCP_NODELAY set
* Expire in 149970 ms for 3 (transfer 0x5644da0dd720)
* Expire in 200 ms for 4 (transfer 0x5644da0dd720)
* Connected to www.google.com (172.217.3.68) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

The result was the same for the edition.cnn.com.

I then ran it from a different container, and the call to www.google.com succeeded, but the call to dev10.hprt.com failed.

root@api-termination-deployment-feature-optms-6421-7955d664fc-dn2hh:/usr/src/app# curl -v https://www.google.com
* Rebuilt URL to: https://www.google.com/
* Hostname was NOT found in DNS cache
*   Trying 172.217.3.68...
* Connected to www.google.com (172.217.3.68) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* Server certificate:
* 	 subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=www.google.com
* 	 start date: 2019-07-29 18:43:22 GMT
* 	 expire date: 2019-10-21 18:23:00 GMT
* 	 subjectAltName: www.google.com matched
* 	 issuer: C=US; O=Google Trust Services; CN=Google Internet Authority G3
* 	 SSL certificate verify ok.
> GET / HTTP/1.1
> User-Agent: curl/7.38.0
> Host: www.google.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 22 Aug 2019 14:52:29 GMT
< Expires: -1
< Cache-Control: private, max-age=0
< Content-Type: text/html; charset=ISO-8859-1
< P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
* Server gws is not blacklisted
< Server: gws
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
< Set-Cookie: 1P_JAR=2019-08-22-14; expires=Sat, 21-Sep-2019 14:52:29 GMT; path=/; domain=.google.com
< Set-Cookie: NID=188=Qq0QSIvb3x39yWjPmdbZr9znoXIu1OQ9VDJ7KfF2izPGLAF00mhvX5dRvR8BNRyAkeEIDvN1VxMTrfm17HTE5z1jSqUkMiypCA3rC2J_TeyAoKVs2hS9e0Lgn1Ddk6SIJLanlW8HkvEKAWn6VvSGl9mmfWs3ILkhrDn4Bdjvnd4; expires=Fri, 21-Feb-2020 14:52:29 GMT; path=/; domain=.google.com; HttpOnly
< Alt-Svc: quic=":443"; ma=2592000; v="46,43,39"
< Accept-Ranges: none
< Vary: Accept-Encoding
< Transfer-Encoding: chunked
<
<!doctype html><html...

and the call that failed

root@api-termination-deployment-feature-optms-6421-7955d664fc-dn2hh:/usr/src/app# curl -v https://dev10.hprt.com/favicon.ico
* Hostname was NOT found in DNS cache
*   Trying 192.168.15.50...
* Connected to dev10.hprt.com (192.168.15.50) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to dev10.hprt.com:443
* Closing connection 0
curl: (35) Unknown SSL protocol error in connection to dev10.hprt.com:443
root@api-termination-deployment-feature-optms-6421-7955d664fc-dn2hh:/usr/src/app#

I’m not sure what I learn from the fact that the call to www.google.com succeeded in a different container.

7

So there are two issues here:

  1. www.google.com is not accessible from the pod with httpd. There is something special about that pod.
  2. there is something special about dev10.hprt.com - it is not accessible from another pod while www.google.com is accessible.

for the full picture, if you use a pod without Istio proxy, is dev10.hprt.com accessible?

Regarding the httpd pod - how is it different from your other pod in which www.google.com is accessible? What is the httpd’s service definition? Which ports does it use?

8

Here are some results from your previous suggestion. I created this ServiceEntry in the same namespace where my Pod is deployed. I wasn’t sure if it should be in the istio-system namespace…

~ # cat dev10.hprt.com-serviceentry.yaml
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: dev10
spec:
  hosts:
  - dev10.hprt.com
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
  location: MESH_EXTERNAL

I can see the request in the istio-proxy

~ # kubectl logs api-termination-deployment-feature-optms-6421-7955d664fc-dn2hh -c istio-proxy
[2019-08-22T15:03:48.035Z] "- - -" 0 UF,URX "-" "-" 0 0 10000 - "-" "-" "-" "-" "192.168.15.50:443" outbound|443||dev10.hprt.com - 192.168.15.50:443 10.233.72.218:44018 -

But I cannot see anything in mixer

[centos@k8s-master-0 ~]$ kubectl -n istio-system logs -l istio-mixer-type=telemetry -c mixer | grep 'dev10.hprt.com'

For httpd I use https://hub.docker.com//httpd without any modifications. I provide the httpd.conf by way of a ConfigMap. The other container is derived from https://hub.docker.com//python (3-slim tag). All I do is copy in some python files and run a flask app.

I’m going to look more closely at how the SSL certificate was installed on dev10.hprt.com. Maybe there is something wrong with how the certificate was installed there or the version of SSL supported.

9

I created a container from the same httpd:2.4 image on my local computer to troubleshoot that a bit more. I could successfully make a request using the -k option.

root@43b8498882da:/usr/local/apache2# curl -k -v https://dev10.hprt.com/
*   Trying 192.168.15.50...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55a82e6ef310)
* Connected to dev10.hprt.com (192.168.15.50) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / DHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Texas; L=Austin; O=TriNet Group Inc; CN=*.hrpt.com
*  start date: Jan 14 13:09:01 2019 GMT
*  expire date: Apr 13 13:39:01 2021 GMT
*  issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2012 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1K
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET / HTTP/1.1
> Host: dev10.hprt.com
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 302 Found
< Date: Thu, 22 Aug 2019 16:21:10 GMT
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips OpenAM Web Agent/4.2.1.1
...
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html>...</html>
* Connection #0 to host dev10.hprt.com left intact

I do get an error without the -k, so I can look into the way the certificate was installed, but the request still succeeds. I also see that the request downgrades from TLSv1.3 to TLSv1.2. In the Istio environment, the TLSv1.3 Server hello never makes it back. That’s where it hangs every time.

I also tried the request from the container in the Istio environment with --tls-max 1.2, but it the Server hello still never makes it back.

10

One more thing, this has been working (top of original diagram) without Istio for a year and a half, and still works in the non-istio environments.

11

Hi,

I do not have any solutions for that, but only a question.

Shouldn’t it be possible to let ISTIO do the reverse proxy
thing, so that no one needs a webserver (httpd/nginx/
lighthttpd/…) to do the reverse proxy job?

And, if yes, how? .-)

Best regards,
Jan

12

@palic The job of the Istio control plane is to configure a fleet of reverse proxies. The purpose of the webserver is to serve content, not reverse proxy. The reverse proxy technology at the heart of Istio is Envoy, and Envoy can be use as a replacement for HAProxy, nginx, Apache, F5, or any other component that is being used as a reverse proxy.

I recommend starting here: https://www.envoyproxy.io/

13

Good day to you, Daniel.

Thank you for your answer.
Especially I tried to get a Wildfly/JBoss working with ISTIO Gateway and VirtualServices and
I didn’t get anything work. I tried the native approach, which I know from Apache/NGinX…

I am pretty sure, that I did miss something…

Best,
Jan