I deploy istio 1.2.5. on some ks8 cluster. According to documentation https://istio.io/faq/mixer/ in rules section: kubectl get rules --all-namespaces You will get the list . In my cluster I got No resources found.
But if I use kubectl get rules.config.istio.io -n istio-system NAME AGE kubeattrgenrulerule 5h promhttp 5h promtcp 5h promtcpconnectionclosed 5h promtcpconnectionopen 5h stdio 5h stdiotcp 5h
I got the list. Someone know the diff? Also if I try kubectl -n istio-system logs -f istio-telemetry-7df96d454b-4kxs9 -c mixer
I didn’t got the log of request in the log ( I found it work in another cluster) Do you know why?
On a test cluster, running $ kubectl get rules --all-namespaces returns the following for me:
NAMESPACE NAME AGE
istio-system analyticsrule 6d
istio-system egresspromhttp 2d
istio-system kubeattrgenrulerule 15d
istio-system newlogtofluentd 8d
istio-system promhttp 15d
istio-system promtcp 15d
istio-system promtcpconnectionclosed 15d
istio-system promtcpconnectionopen 15d
istio-system quota 15d
istio-system stdio 5d
istio-system stdiotcp 5d
istio-system tcpkubeattrgenrulerule 15d
You should see something like the following if you query the crd itself ($ kubectl get crds rules.config.istio.io -o yaml)
names:
categories:
- istio-io
- policy-istio-io
kind: rule
listKind: ruleList
plural: rules
singular: rule
I think that this may be an issue with the cache for kubectl. Somewhere else I saw a suggetion to: “delete all files under .kube/http-cache/ and .kube/cache/” and try again.
Hi like i was written if I use ```
kubectl get rules.config.istio.io -n istio-system
I got the list .
So what the diff between rules.config.istio.io and rules?
rules.config.istio.io is just the full name for the CRD. rules is the short name that is meant for use in the CLI. You can read more about this in the kubernetes docs on CRDs.
Your issue appears to be a k8s-related issue. That’s why I was suggesting removing your local caches and trying again.
kubectl it’s only client tool so why if i connect to 2 cluster it’s show my the list in rules and only in one cluster no. And the big question is why if it installed i didn’t get the metrics in mixer log?
Have you tried the suggestion? Has that produced any results?
The other thing to check would be to see if you have any other CRDs installed in your problem cluster that also try to use “rule” as their singular name. Other than that, I don’t know what else to say on this issue.
I can’t tell if you are suggesting that the logging is working in one cluster and not in the other. If that is the case, I think that is another suggestion that something is broken in the one cluster related to configuration.
There is a troubleshooting guide on Missing Metrics that you should be able to tailor for your investigation.
Hi
I using helm to deploy it so on both clusters are the same helm so what configuration that related to it can be the diff?
I saw in missing metric doc that I have no grpc_io_server_completed_rpcs - in Verify Mixer is receiving Report calls section. But like it written there to check the side card injection I checked and it’s all good. so what can be the problem?
It is hard to say what has gone wrong, but it appears that something has failed during install (maybe a partial install, or something else happened). If you aren’t seeing Report calls to Mixer, I would check the proxy logs for the client and server proxies. You should check for errors.
you mean to check the logs of istio-proxy container in the service and in mixer?
check istio-proxy logs in the application that is sending traffic and in the application that is receiving traffic (both should be sending traffic to mixer).
hi I dont see ant error in both istio-proxy containers.
I mean by not geting the request in mixer log that When I run :kubectl logs -f istio-telemetry-6dc458ff7-468dv -n istio-system -c mixer I got in the log : something like this :gc 1537 @183375.442s 0%: 0.033+13+0.009 ms clock, 0.20+0/18/63+0.054 ms cpu, 39->39->34 MB, 69 MB goal, 6 P scvg1222: 0 MB released scvg1222: inuse: 46, idle: 12, sys: 58, released: 10, consumed: 48 (MB) GC forced .
I should get something like: {“level”:“info”,“time”:“2019-09-18T10:59:28.071530Z”,“instance”:“accesslog.instance.istio-system”,“apiClaims”:"",“apiKey”:"",“clientTraceId”:"",“connection_security_policy”:“none”,“destinationApp”:“telemetry”,“destinationIp”:“10.5.108.34”,…} this is in the good env
If you aren’t seeing errors in your proxies, and you are not seeing requests to Mixer, then it sounds like traffic is not actually going through your proxies.
You could try to turn on Envoy access logs on the proxies in question to see if that provides more insight (Access Log Task).
You could also try looking at traces, if you have distributed tracing enabled to see if traffic is flowing as expected.
I try do turn the access log on and didn’t get anything something new in the logs. what can be the problem? how to debug it?
If you aren’t seeing anything in the Envoy access log (this is distinct from any mixer-generated logging, per the task I linked above), then requests are not coming through your proxies.
You may want to look at the Networking troubleshooting guide for more information.
https://istio.io/docs/ops/troubleshooting/network-issues/?_ga=2.263893259.609529861.1568800435-1210262183.1565597267 - still try to log the istio proxy but still not seeing access log or something …
I’t’s seems like istio config env problem because if i deploy it in another envs it’s work…
Any other direction?
Problem resolved.Was a problem with some mix between cni and init-container in helm chat deployment.
I’m happy you were able to get this resolved. I’m sorry it took so long to troubleshoot. If you have the time and inclination, I’m sure a write-up of how things went wrong, how to discover that issue, and how to fix it would be very welcomed by the community.