I have Istio installed successfully on a web application in a couple of environments. One environment does not seem to have any issues, but the other one is running into an issue where its trying to route to an internal service but for some reason istio thinks that its an external service. I can create a Service Entry for the internal service and traffic is routed successfully, but this is not good as i have to use an IP addess which may change since we are deploying inside Kubernetes. If the service gets deleted and recreated it will have a new IP. When i set outbound traffic policy to ALLOW_ANY i see it as a PassthoroughCluster and if i do REGISTRY_ONLY i get a BlackHoleCluster stopping the internal call. Any thoughts in how i can troubleshoot this?
2019-12-27T19:58:49.025881271Z [2019-12-27T19:58:35.074Z] “- - -” 0 - “-” “-” 10203 56600 8071 - “-” “-” “-” “-” “100.69.112.133:80” PassthroughCluster 100.119.229.52:34152 100.69.112.133:80 100.119.229.52:34150 - -
Above you will see the istio proxy log and the ip address 100.69.112.133:80 refers to an internal service inside K8S 100.119.229.52:34152 refers to the Pod that is sending the request to the service.
We have a public api that is attached to a gateway, a virtual service for the public-api that routes it to a service in example-ns, a service entry that allows other services to call the public-api, and all of our other services have virtual services that route to their correct svc/ports - examples below
service-A -> api.example.com/foo -> public-gw -> public-api-se -> public-api-vs -> service-B-vs -> service-B
I hope this paints a picture of our setup. All of our services 55+ seem to work except for this one in this one environment, whereas it works in another environment. This specific call and i cant seem to figure out what is happening. Please give me some guidance or let me know what you need and i will try to get you some more info.
---
kind: Gateway
apiVersion: networking.istio.io/v1alpha3
metadata:
name: public-gw
namespace: example-ns
spec:
servers:
- hosts:
- api.example.com
port:
name: http
number: 80
protocol: HTTP
selector:
istio: ingressgateway
---
kind: VirtualService
apiVersion: networking.istio.io/v1alpha3
metadata:
name: example-public-vs
namespace: example-ns
spec:
hosts:
- api.example.com
gateways:
- public-gw
http:
- route:
- destination:
host: service.namespace.svc.cluster.local
port:
number: 3000
---
kind: ServiceEntry
apiVersion: networking.istio.io/v1alpha3
metadata:
name: example-api-se
namespace: example-ns
spec:
hosts:
- api.example.com
ports:
- name: http-example
number: 80
protocol: HTTP
location: MESH_INTERNAL
resolution: NONE
---
kind: VirtualService
apiVersion: networking.istio.io/v1alpha3
metadata:
name: service-vs
namespace: example-ns
spec:
hosts:
- service.namespace.svc.cluster.local
http:
- route:
- destination:
host: service.namespace.svc.cluster.local
port:
number: 80
---