Hi *,
this is a repost of my original question asked on Stackoverflow due to lack of response over there. I am hoping for some help here.
Background
We run a GitLab CI runner inside of our cluster. It executes cypress.io end-to-end tests. To do so the runner launches a pod with some cypress docker container. Our frontend web app under testing is also hosted by the very same cluster. The pod for running the cypress test s shall connect to the frontend via our public user-facing domain, such as webapp.example.dev.
Simplified:
We have 1 Frontend Service available under https://webapp.example.dev/ by
- Load Balancer
- Istio Ingress Gateway (with Let’s Encrypt Certs Wildcard cert on
*.example.dev) - Istio Virtual Service for the host
webapp.example.dev - connected to a K8s Service + K8s Deployment
The GitLab Runner stuff can be simplified with the following pod:
apiVersion: v1
kind: Pod
metadata:
name: debugger-pod
labels:
app: debugger-pod
spec:
containers:
- name: debugger
image: curlimages/curl
command: [ "/bin/sh", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
From that pod I am facing an SSL connection error.
curl https://webapp.example.dev -v
* Trying 100.200.100.200:443...
* Connected to webapp.example.dev (100.200.100.200) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /cacert.pem
* CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection to webapp.example.dev:443
* Closing connection 0
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to webapp.example.dev:443
(IP Address is changed & made up)
Making the same request through an HTTP proxy works fine, as you can see from these logs. (You may find a proxy over here: https://free-proxy-list.net/)
/ $ curl -I https://webapp.example.dev -x 178.217.216.184:49086 -v
* Trying 178.217.216.184:49086...
* Connected to 178.217.216.184 (178.217.216.184) port 49086 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to webapp.example.dev:443
> CONNECT webapp.example.dev:443 HTTP/1.1
> Host: webapp.example.dev:443
> User-Agent: curl/7.75.0-DEV
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /cacert.pem
* CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.example.dev
* start date: Jan 4 09:31:45 2021 GMT
* expire date: Apr 4 09:31:45 2021 GMT
* subjectAltName: host "webapp.example.dev" matched cert's "*.example.dev"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55a551522940)
> HEAD /metainfo HTTP/2
> Host: webapp.example.dev
> user-agent: curl/7.75.0-DEV
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 200
HTTP/2 200
< server: istio-envoy
server: istio-envoy
< date: Tue, 23 Feb 2021 19:30:33 GMT
date: Tue, 23 Feb 2021 19:30:33 GMT
< content-type: text/html
content-type: text/html
< content-length: 4273
content-length: 4273
< etag: "5fd87257-10b1"
etag: "5fd87257-10b1"
< x-envoy-upstream-service-time: 2
x-envoy-upstream-service-time: 2
<
* Closing connection 0
Istio Details:
- Version: 1.8.3
- mTLS mode: STRICT (I tried to disable it without success)
Question:
It seems that the connection from my debugger-pod’s curl is made within the cluster. Even though curl's verbose outputs that the connection is made to the external IP address of our cluster / its load balancer. The fact that it is working by forcing the traffic to be routed outside the cluster using a proxy is giving me evidence.
Now, is there any way to directly connect to the frontend app without the SSL issues?
