Testing istio with Filebeat -> Logstash -> Elasticsearch

1

So i am new to istio and wanted to give it a try.
i installed on my EKS K8S the istio from Helm and the only thing that i added is a new Internal Gateway for internal vpc networking.

Now i created the gateway and virtualservice and and set all my servers filebeat to the dns i set in the gateway
So it was running good for some time and in the middle of the night i notice all my filbeat logs say
Failed to publish events caused by: EOF
as i understand its TCP closed
now its not working at all
what am i doing wrong and how can i debug it?

kubectl get gateway -n elk -o yaml
apiVersion: v1
items:

  • apiVersion: networking.istio.io/v1alpha3
    kind: Gateway
    metadata:
    clusterName: “”
    creationTimestamp: 2019-01-13T14:39:30Z
    generation: 1
    name: devops-logstash-service-sc-gateway
    namespace: elk
    spec:
    selector:
    istio: ingressgateway-internal
    servers:
    • hosts:
      • xxx
      • xxx
        port:
        name: tcp-filebeat
        number: 5044
        protocol: TCP
        kind: List
        metadata:
        resourceVersion: “”
        selfLink: “”

kubectl get virtualservice -n elk -o yaml
apiVersion: v1
items:

  • apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
    clusterName: “”
    creationTimestamp: 2019-01-13T14:39:30Z
    generation: 1
    name: devops-logstash-service-sc
    namespace: elk
    spec:
    gateways:
    • devops-logstash-service-sc-gateway
      hosts:
    • xxx
    • xxx
      tcp:
    • match:
      • port: 5044
        route:
      • destination:
        host: devops-logstash-service-sc
        port:
        number: 5044
        kind: List
        metadata:
        resourceVersion: “”
        selfLink: “”