Hi all. I have what looks like a tls origination problem, but the traffic is not going from my pod to an external service. It is between pods.
I am installing Tyk with Istio integration. Tyk creates 2 services - dashboard and gateway. I seem to be getting this error in my Tyk Gateway
2021-05-07T14:39:09.176883515Z
[2021-05-07T14:39:05.231Z] "GET /register/node HTTP/1.1" 503 UF,URX "TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER" 0 190 45 - "-" "Go-http-client/1.1" "47cb6d92-6a2a-4302-b2a7-51f38bceb769" "dashboard-svc-tyk-pro.coma-apigateway:3000" "10.88.2.8:3000" outbound|3000||dashboard-svc-tyk-pro.coma-apigateway.svc.cluster.local - 10.92.4.146:3000 10.88.4.2:52430 - default
The Tyk gateway is trying to register itself with the Tyk dashboard but it’s failing - it’s saying the Tyk dashboard is down (it isn’t). The problem just appears to be some type of SSL issue.
I am running my Kubernetes cluster on GCP. I have already tried setting values.pilot.enableProtocolSniffingForOutbound=false but to no avail.
Help?
Can provide a bit more information on how things are setup? I’ll be useful to know how the services expect to talk to each other, such as if the endpionts expect http or https. Also, if there’s any VirtualServices or Destination rules defined for them.
I have not defined any virtual services or destination rules.
This is a plain vanilla istio installation using
istioctl install --set meshConfig.accessLogFile=/dev/stdout --set values.pilot.enableProtocolSniffingForOutbound=false --set profile=default
The dashboard has the following ports defined:
ports:
- containerPort: 3000
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /
port: 3000
scheme: HTTP
initialDelaySeconds: 1
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 3
The gateway has this:
ports:
- containerPort: 8080
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /hello
port: 8080
scheme: HTTP
initialDelaySeconds: 1
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 3
The gateway is expecting http200 response from the dashboard when it is attempting to register itself. I hope that helps.
Please excuse me if I am mistaken in providing the information above. I am quite new to Istio and Kubernetes.
If I rebuild everything without Istio, I face no issues. Tyk functions correctly. So I must be doing something really silly 
Let me add one more thing.
If I open a bash shell into the Gateway pod and issue a curl command to connect to the Dashboard, I get the following results - with HTTP:
curl http://10.92.4.146:3000/hello -v -k
* Expire in 0 ms for 6 (transfer 0x56004a8a2f90)
* Trying 10.92.4.146...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x56004a8a2f90)
* Connected to 10.92.4.146 (10.92.4.146) port 3000 (#0)
> GET /hello HTTP/1.1
> Host: 10.92.4.146:3000
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< content-length: 190
< content-type: text/plain
< date: Fri, 07 May 2021 15:12:31 GMT
< server: envoy
<
* Connection #0 to host 10.92.4.146 left intact
upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
And with HTTPS:
curl https://10.92.4.146:3000/hello -v -k
* Expire in 0 ms for 6 (transfer 0x560dd3df7f90)
* Trying 10.92.4.146...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x560dd3df7f90)
* Connected to 10.92.4.146 (10.92.4.146) port 3000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
Thanks for the additional info, but I can’t know exactly what might be going on. As a suggestion, maybe you can disable mTLS to the address you’re trying o access to see if that is where the problem resides, with a destination rule:
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: mtls-disable-test
spec:
exportTo:
- .
host: "dashboard-svc-tyk-pro.coma-apigateway.svc.cluster.local"
trafficPolicy:
portLevelSettings:
- port:
number: 3000
tls:
mode: NONE
If it starts working, something with the mTLS configuration for the services might be incorrect.
It works! Thank you! I wonder why it fails with a default installation though. Any tips on how I can check further?
Just a minor change your destinationrule. the tls.mode value of NONE isn’t supported. I had to change it to DISABLE.