Using custom gateway like zuul in istio not generating metrics


#1

Any documentation on setting up a custom gateway like zuul in Istio, instead of using ingress gateway and generate metrics, logs when the gateway and services are in different namespace.

I am able to use zuul as a gateway in istio by configuring ribbon to route using kubeservice name and port to service which is deployed using istio config and have a sidecar. But the metrics are not getting reflected in grafana with reponse codes, time taken and all attributes. Are these attributes generation linked to gateway or a service or a namespace. please forward me a documentation by using which i can generate metrics which has reponse and request attributes.

Is there a way I can forward my java,spring specific logs which are being dispayed on my console using logger to be forwarded to istio logging filters like stdio or something, when services and zuul are deployed in different namespace otherthan istio-system?


#2

It sounds like you have a custom gateway that is then talking to a service with a sidecar. Is that correct?

Can you tell me a bit about the sidecar (how it was injected?, what image is it?, etc.)?

Can you also provide some details about how you installed Istio and what your configuration looks like?

There is a FAQ entry on Missing Metrics that might help explain things and troubleshoot.

You should see automatic metric generation for any istio-proxy sidecar properly configured to point to istio-telemetry. In your case, this should be in front of your services.

If you want the custom gateway to also report to istio-telemetry, you will need to implement a client for Mixer, based on the Mixer protocol.

Some things to check as well: how is the custom gateway contacting the services? Do you see traffic hitting the sidecar proxy?

FWIW, the namespace of the service should not matter. The configuration for the “default” set of metrics that would have been installed is applied in the istio-system namespace, but that namespace has special significance. It means that they are generated for traffic for all namespaces.

Istio, at the moment, does not offer a way to ingest fully-formed logs. There are proposals for supporting such a “Direct API”, but there is no existing implementation.


#3

Thanks for the reply.
I have tried the Istio setup in 2 ways by helm and by auto istio-demo.yml none worked for metrics for cutom zuul gateway.

I think the problem might be I am calling the custom gateway by public IP may be the requests are not flowing through istio-proxy(sidecar) and even service to service communications may not be going through proxy after that.

Even when I call through istio-gateway I am getting metrics till next downstream service call but the call to the dependent service metrics are not getting reflected this says I am not calling the istio-proxy(sidecar) corretly but I am getting responses from service may be the requests are not going through proxy.

Below are the answers to questions u asked

*Yes, I have custom gateway as zuul, I am using kube DNS service name like ‘product:{port}’ or name like ‘product.default.svc.cluster.local:{port}’ as base url. Does this mean I am communicating to sidecar?

*Injection of the sidecar in 2 ways one by ‘istioctl kube-inject -f samples/sleep/sleep.yaml | kubectl apply -f -’ while doing istio-demo.yml istio setup and the other by ‘istio-ingestion=enabled’ for default namespace while using helm.

*I have tried 2 types of setup too one is by crds and Istio-demo.yaml to install istio and the version is 1.0.5 refferred ‘https://istio.io/docs/setup/kubernetes/quick-start/’ and other by helm by enabling gatling, service graph, tracing using "–set for helm template command.

My config:

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: rc-sample
  namespace: default
  labels:
    app: rc-sample
  annotations:
    deployment.kubernetes.io/revision: '1'
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rc-sample
  template:
    metadata:
      name: rc-sample
      labels:
        app: rc-sample
        version: v1
    spec:
      containers:
      - name: rc-sample
        image: gcr.io/cognitive-genie-63754/mesh_rc_sample:latest
        env: 
        - name: eureka_instance_hostname
          value: 'rc-sample'
        envFrom:
        - configMapRef:
            name: mesh-config
        resources: {}
        imagePullPolicy: Always
        securityContext:
          privileged: false
        ports:
        - containerPort: 3333
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      imagePullSecrets:
      - name: acr-mesh
      schedulerName: default-scheduler
  strategy:
    type: Recreate
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: rc-sample
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: rc-sample
  minReplicas: 1
  maxReplicas: 1
  targetCPUUtilizationPercentage: 80
---
kind: Service
apiVersion: v1
metadata:
  name: rc-sample
  namespace: default
  labels:
    app: rc-sample
spec:
  ports:
  - name: tcp-6050-6050-rc-sample
    protocol: TCP
    port: 6050
    targetPort: 6050
  selector:
    app: rc-sample
  type: ClusterIP
  sessionAffinity: None
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: config-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: rcsampleroute
spec:
  hosts:
  - "*"
  gateways:
  - config-gateway
  http:
  - match:
    - uri:
        prefix: /sample-service
    route:
    - destination:
        host: rc-sample
        port:
          number: 6050
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: rc-sample
spec:
  hosts:
  - rc-sample
  http:
  - route:
    - destination:
        host: rc-sample
  • I tried “Missing Metrics” everything looks fine. I am able to generate new metrics but response and request attributes are not reflecting when service gets called through zuul gateway. Below is the error which i get when i add custom metric.

“[libprotobuf ERROR src/istio/mixerclient/report_batch.cc:83] Mixer Report failed with: UNKNOWN:2 errors occurred:%0A%0A* evaluation failed at [newlog.logentry.istio-system]‘Timestamp’: ‘lookup failed: ‘request.time’’%0A* failed to report all values: 1 error occurred:%0A%0A* could not find metric info from adapter config for doublerequestcount.metric.default”

*How can i validate this or implement this " You should see automatic metric generation for any istio-proxy sidecar properly configured to point to istio-telemetry . In your case, this should be in front of your services."

  • How can I validate traffic hitting sidecar since I am not able to see any log changes in istio-proxy even when metrics are reflected when calls are made through istio-ingess gateway?

There is one more problem which I observed in both the setup is servicegraph is not coming up. it just says starting at 8088 in logs that is it.


#4

Your rc-sample service is using a TCP port. Are you seeing TCP metrics instead? Is that possibly the issue at play here?

Try changing the port in the service definition to be named http-6050-rc-sample for instance and see what happens.


#5

Thanks it worked. changing the port name helped me to see metrics. even before that as u said I was getting TCP metrics.

Now as u aware of all my installation procedures can u please help me with below 2 issues I am facing.

  • SERVICE GRAPH is not coming up.

  • Is there any fallback logic which I can write when a circuit fails by using ISTIO Cirtcuitbraking logic on applying destination rule?


#6

The service graph add-on, as much as it held a special place in my heart, has been deprecated. It was not really maintained. It has been replaced in the ecosystem by Kiali – which has a dedicated team behind it and is under active development. I don’t know what would be causing it to fail, but I would suggest trying Kiali instead.


#7

ok great Kiali is working fine.

Can you please also let me know ur comments on below 3 isuues

So does above statement mean I wont be able to pass whatever log statements I make in java code using log4j to istio, right?

  • any update on below question

*Service tracing through jaeger is not coming up as expected.


#8

If you make in-app logging statements, there is currently no Istio-related feature for ingest.

Fallback logic for circuit breaking is not supported, as far as I understand. You should open a separate thread to ask about this feature and tag the Networking group.

I don’t know how to help with your comment about service tracing. The best I can do is point you to: https://preliminary.istio.io/help/faq/distributed-tracing/