Istio tls origination problem

Hi all. I have what looks like a tls origination problem, but the traffic is not going from my pod to an external service. It is between pods.

I am installing Tyk with Istio integration. Tyk creates 2 services - dashboard and gateway. I seem to be getting this error in my Tyk Gateway

2021-05-07T14:39:09.176883515Z

[2021-05-07T14:39:05.231Z] "GET /register/node HTTP/1.1" 503 UF,URX "TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER" 0 190 45 - "-" "Go-http-client/1.1" "47cb6d92-6a2a-4302-b2a7-51f38bceb769" "dashboard-svc-tyk-pro.coma-apigateway:3000" "10.88.2.8:3000" outbound|3000||dashboard-svc-tyk-pro.coma-apigateway.svc.cluster.local - 10.92.4.146:3000 10.88.4.2:52430 - default

The Tyk gateway is trying to register itself with the Tyk dashboard but it’s failing - it’s saying the Tyk dashboard is down (it isn’t). The problem just appears to be some type of SSL issue.

I am running my Kubernetes cluster on GCP. I have already tried setting values.pilot.enableProtocolSniffingForOutbound=false but to no avail.

Help?

Can provide a bit more information on how things are setup? I’ll be useful to know how the services expect to talk to each other, such as if the endpionts expect http or https. Also, if there’s any VirtualServices or Destination rules defined for them.

I have not defined any virtual services or destination rules.

This is a plain vanilla istio installation using

istioctl install --set meshConfig.accessLogFile=/dev/stdout --set values.pilot.enableProtocolSniffingForOutbound=false --set profile=default

The dashboard has the following ports defined:

ports:

        - containerPort: 3000

          protocol: TCP

        readinessProbe:

          failureThreshold: 3

          httpGet:

            path: /

            port: 3000

            scheme: HTTP

          initialDelaySeconds: 1

          periodSeconds: 10

          successThreshold: 1

          timeoutSeconds: 3

The gateway has this:

ports:

        - containerPort: 8080

          protocol: TCP

        readinessProbe:

          failureThreshold: 3

          httpGet:

            path: /hello

            port: 8080

            scheme: HTTP

          initialDelaySeconds: 1

          periodSeconds: 10

          successThreshold: 1

          timeoutSeconds: 3

The gateway is expecting http200 response from the dashboard when it is attempting to register itself. I hope that helps.

Please excuse me if I am mistaken in providing the information above. I am quite new to Istio and Kubernetes.

If I rebuild everything without Istio, I face no issues. Tyk functions correctly. So I must be doing something really silly :frowning:

Let me add one more thing.

If I open a bash shell into the Gateway pod and issue a curl command to connect to the Dashboard, I get the following results - with HTTP:

 curl http://10.92.4.146:3000/hello -v -k
* Expire in 0 ms for 6 (transfer 0x56004a8a2f90)
*   Trying 10.92.4.146...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x56004a8a2f90)
* Connected to 10.92.4.146 (10.92.4.146) port 3000 (#0)
> GET /hello HTTP/1.1
> Host: 10.92.4.146:3000
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< content-length: 190
< content-type: text/plain
< date: Fri, 07 May 2021 15:12:31 GMT
< server: envoy
<
* Connection #0 to host 10.92.4.146 left intact
upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER

And with HTTPS:

curl https://10.92.4.146:3000/hello -v -k
* Expire in 0 ms for 6 (transfer 0x560dd3df7f90)
*   Trying 10.92.4.146...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x560dd3df7f90)
* Connected to 10.92.4.146 (10.92.4.146) port 3000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Thanks for the additional info, but I can’t know exactly what might be going on. As a suggestion, maybe you can disable mTLS to the address you’re trying o access to see if that is where the problem resides, with a destination rule:

---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name:  mtls-disable-test
spec:
  exportTo:
  - .
  host: "dashboard-svc-tyk-pro.coma-apigateway.svc.cluster.local"
  trafficPolicy:
    portLevelSettings:
    - port:
        number: 3000
      tls:
        mode: NONE

If it starts working, something with the mTLS configuration for the services might be incorrect.

It works! Thank you! I wonder why it fails with a default installation though. Any tips on how I can check further?

Just a minor change your destinationrule. the tls.mode value of NONE isn’t supported. I had to change it to DISABLE.