Collecting Logs doesn't work

I just testd with task telemetry-log-collecting metrics, but I can’t see any logs in the istio-telemetry pods. I also debug it according to operations-telemetry-missing metrics, it seems normal yet.

root@controller-mk-0:~# kubectl logs -n istio-system -l istio-mixer-type=telemetry -c mixer | grep "newlog" | grep -v '"destination":"telemetry"' | grep -v '"destination":"pilot"' | grep -v '"destination":"policy"' | grep -v '"destination":"unknown"'
root@controller-mk-0:~# 

the sample bookinfo seems to work normally, I can access bookinfo through http://gatewayhost/productpage in browser, there is my k8s-cluster profile below.

istio version

root@controller-mk-0:~# istioctl version
version.BuildInfo{Version:"1.1.7", GitRevision:"eec7a74473deee98cad0a996f41a32a47dd453c2", User:"root", Host:"341b3bf0-76ac-11e9-b644-0a580a2c0404", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.6-6-geec7a74"

bookinfo pod status

root@controller-mk-0:~# kubectl get po
NAME                              READY   STATUS    RESTARTS   AGE
details-v1-65b966b497-56f7w       2/2     Running   0          11d
productpage-v1-79458795bc-vn9t5   2/2     Running   0          11d
ratings-v1-5b7cd6c58f-6bdnf       2/2     Running   0          11d
reviews-v1-54c7c79486-cmqqr       2/2     Running   0          11d
reviews-v2-7dc5785684-88vx9       2/2     Running   0          11d
reviews-v3-6c464d7bf4-wxrvn       2/2     Running   0          11d

routing related objects

root@controller-mk-0:~# kubectl get gw
NAME               AGE
bookinfo-gateway   106m
root@controller-mk-0:~# kubectl get vs
NAME       GATEWAYS             HOSTS   AGE
bookinfo   [bookinfo-gateway]   [*]     11d

metric related objects

root@controller-mk-0:~# kubectl get instance -n istio-system 
NAME     AGE
newlog   87m
root@controller-mk-0:~# kubectl get handler -n istio-system 
NAME            AGE
kubernetesenv   11d
newloghandler   94m
prometheus      11d
root@controller-mk-0:~# kubectl get rule -n istio-system 
NAME                      AGE
kubeattrgenrulerule       12d
newlogstdio               95m
promhttp                  12d
promtcp                   12d
promtcpconnectionclosed   12d
promtcpconnectionopen     12d
tcpkubeattrgenrulerule    12d

verify Mixer is receiving report calls

# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 4.015e-05
go_gc_duration_seconds{quantile="0.25"} 7.5219e-05
go_gc_duration_seconds{quantile="0.5"} 9.4484e-05
go_gc_duration_seconds{quantile="0.75"} 0.000120525
go_gc_duration_seconds{quantile="1"} 0.009851564
go_gc_duration_seconds_sum 3.532235828
go_gc_duration_seconds_count 9475
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 2113
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.10.4"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 9.948568e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 6.066894292e+10
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.942579e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 6.92677314e+08
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 2.5914291853277276e-05
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 1.41312e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 9.948568e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 9.322496e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 1.417216e+07
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 66733
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 2.301952e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 2.3494656e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.5622345820353892e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 973818
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 6.92744047e+08
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 10416
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 246696
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 344064
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 1.6931584e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.295557e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 7.962624e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 7.962624e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 3.6468984e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 15
# HELP grpc_io_server_completed_rpcs Count of RPCs by method and status.
# TYPE grpc_io_server_completed_rpcs counter
grpc_io_server_completed_rpcs{grpc_server_method="istio.mixer.v1.Mixer/Report",grpc_server_status="OK"} 73716
# HELP grpc_io_server_received_bytes_per_rpc Distribution of received bytes per RPC, by method.
# TYPE grpc_io_server_received_bytes_per_rpc histogram
grpc_io_server_received_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="0"} 0
grpc_io_server_received_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="1024"} 72459
grpc_io_server_received_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="2048"} 73474
grpc_io_server_received_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="4096"} 73696
grpc_io_server_received_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="16384"} 73716
grpc_io_server_received_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="65536"} 73716
grpc_io_server_received_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="262144"} 73716
grpc_io_server_received_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="1.048576e+06"} 73716
grpc_io_server_received_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="4.194304e+06"} 73716
grpc_io_server_received_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="1.6777216e+07"} 73716
grpc_io_server_received_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="6.7108864e+07"} 73716
grpc_io_server_received_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="2.68435456e+08"} 73716
grpc_io_server_received_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="1.073741824e+09"} 73716
grpc_io_server_received_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="4.294967296e+09"} 73716
grpc_io_server_received_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="+Inf"} 73716
grpc_io_server_received_bytes_per_rpc_sum{grpc_server_method="istio.mixer.v1.Mixer/Report"} 4.401408299999983e+07
grpc_io_server_received_bytes_per_rpc_count{grpc_server_method="istio.mixer.v1.Mixer/Report"} 73716
# HELP grpc_io_server_sent_bytes_per_rpc Distribution of total sent bytes per RPC, by method.
# TYPE grpc_io_server_sent_bytes_per_rpc histogram
grpc_io_server_sent_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="0"} 0
grpc_io_server_sent_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="1024"} 73716
grpc_io_server_sent_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="2048"} 73716
grpc_io_server_sent_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="4096"} 73716
grpc_io_server_sent_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="16384"} 73716
grpc_io_server_sent_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="65536"} 73716
grpc_io_server_sent_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="262144"} 73716
grpc_io_server_sent_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="1.048576e+06"} 73716
grpc_io_server_sent_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="4.194304e+06"} 73716
grpc_io_server_sent_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="1.6777216e+07"} 73716
grpc_io_server_sent_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="6.7108864e+07"} 73716
grpc_io_server_sent_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="2.68435456e+08"} 73716
grpc_io_server_sent_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="1.073741824e+09"} 73716
grpc_io_server_sent_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="4.294967296e+09"} 73716
grpc_io_server_sent_bytes_per_rpc_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="+Inf"} 73716
grpc_io_server_sent_bytes_per_rpc_sum{grpc_server_method="istio.mixer.v1.Mixer/Report"} 0
grpc_io_server_sent_bytes_per_rpc_count{grpc_server_method="istio.mixer.v1.Mixer/Report"} 73716
# HELP grpc_io_server_server_latency Distribution of server latency in milliseconds, by method.
# TYPE grpc_io_server_server_latency histogram
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="0"} 0
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="0.01"} 0
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="0.05"} 0
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="0.1"} 0
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="0.3"} 0
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="0.6"} 4370
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="0.8"} 63119
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="1"} 71397
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="2"} 72784
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="3"} 73269
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="4"} 73311
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="5"} 73333
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="6"} 73348
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="8"} 73386
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="10"} 73446
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="13"} 73644
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="16"} 73658
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="20"} 73663
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="25"} 73665
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="30"} 73668
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="40"} 73677
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="50"} 73681
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="65"} 73690
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="80"} 73701
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="100"} 73714
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="130"} 73716
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="160"} 73716
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="200"} 73716
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="250"} 73716
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="300"} 73716
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="400"} 73716
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="500"} 73716
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="650"} 73716
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="800"} 73716
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="1000"} 73716
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="2000"} 73716
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="5000"} 73716
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="10000"} 73716
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="20000"} 73716
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="50000"} 73716
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="100000"} 73716
grpc_io_server_server_latency_bucket{grpc_server_method="istio.mixer.v1.Mixer/Report",le="+Inf"} 73716
grpc_io_server_server_latency_sum{grpc_server_method="istio.mixer.v1.Mixer/Report"} 60689.22521899968
grpc_io_server_server_latency_count{grpc_server_method="istio.mixer.v1.Mixer/Report"} 73716
# HELP istio_build Istio component build info
# TYPE istio_build gauge
istio_build{component="mixer",tag="1.1.6-6-geec7a74"} 1
# HELP mixer_config_attributes_total The number of known attributes in the current config.
# TYPE mixer_config_attributes_total counter
mixer_config_attributes_total{configID="0"} 23
mixer_config_attributes_total{configID="1"} 112
# HELP mixer_config_handler_configs_total The number of known handlers in the current config.
# TYPE mixer_config_handler_configs_total counter
mixer_config_handler_configs_total{configID="1"} 2
# HELP mixer_config_instance_configs_total The number of known instances in the current config.
# TYPE mixer_config_instance_configs_total counter
mixer_config_instance_configs_total{configID="1"} 9
# HELP mixer_config_rule_configs_total The number of known rules in the current config.
# TYPE mixer_config_rule_configs_total counter
mixer_config_rule_configs_total{configID="1"} 6
# HELP mixer_dispatcher_destinations_per_request Number of handlers dispatched per request by Mixer
# TYPE mixer_dispatcher_destinations_per_request histogram
mixer_dispatcher_destinations_per_request_bucket{le="0"} 0
mixer_dispatcher_destinations_per_request_bucket{le="1"} 0
mixer_dispatcher_destinations_per_request_bucket{le="2"} 115292
mixer_dispatcher_destinations_per_request_bucket{le="3"} 151818
mixer_dispatcher_destinations_per_request_bucket{le="4"} 151818
mixer_dispatcher_destinations_per_request_bucket{le="5"} 151818
mixer_dispatcher_destinations_per_request_bucket{le="6"} 151818
mixer_dispatcher_destinations_per_request_bucket{le="7"} 151818
mixer_dispatcher_destinations_per_request_bucket{le="8"} 151818
mixer_dispatcher_destinations_per_request_bucket{le="10"} 151818
mixer_dispatcher_destinations_per_request_bucket{le="15"} 151818
mixer_dispatcher_destinations_per_request_bucket{le="20"} 151818
mixer_dispatcher_destinations_per_request_bucket{le="+Inf"} 151818
mixer_dispatcher_destinations_per_request_sum 188343.9999999983
mixer_dispatcher_destinations_per_request_count 151818
# HELP mixer_dispatcher_instances_per_request Number of instances created per request by Mixer
# TYPE mixer_dispatcher_instances_per_request histogram
mixer_dispatcher_instances_per_request_bucket{le="0"} 0
mixer_dispatcher_instances_per_request_bucket{le="1"} 0
mixer_dispatcher_instances_per_request_bucket{le="2"} 39547
mixer_dispatcher_instances_per_request_bucket{le="3"} 112107
mixer_dispatcher_instances_per_request_bucket{le="4"} 112271
mixer_dispatcher_instances_per_request_bucket{le="5"} 151818
mixer_dispatcher_instances_per_request_bucket{le="6"} 151818
mixer_dispatcher_instances_per_request_bucket{le="7"} 151818
mixer_dispatcher_instances_per_request_bucket{le="8"} 151818
mixer_dispatcher_instances_per_request_bucket{le="10"} 151818
mixer_dispatcher_instances_per_request_bucket{le="15"} 151818
mixer_dispatcher_instances_per_request_bucket{le="20"} 151818
mixer_dispatcher_instances_per_request_bucket{le="+Inf"} 151818
mixer_dispatcher_instances_per_request_sum 343346.9999999901
mixer_dispatcher_instances_per_request_count 151818
# HELP mixer_handler_daemons_total The current number of active daemon routines in a given adapter environment.
# TYPE mixer_handler_daemons_total gauge
mixer_handler_daemons_total{handler="kubernetesenv.istio-system",initConfigID="1"} 4
mixer_handler_daemons_total{handler="prometheus.istio-system",initConfigID="1"} 1
# HELP mixer_handler_new_handlers_total The number of handlers that were newly created during config transition.
# TYPE mixer_handler_new_handlers_total counter
mixer_handler_new_handlers_total{configID="1"} 2
# HELP mixer_mcp_sink_reconnections The number of times the sink has reconnected.
# TYPE mixer_mcp_sink_reconnections counter
mixer_mcp_sink_reconnections 1
# HELP mixer_mcp_sink_request_acks_total The number of request acks received by the source.
# TYPE mixer_mcp_sink_request_acks_total counter
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/adapters",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/apikeys",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/authorizations",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/bypasses",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/checknothings",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/circonuses",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/cloudwatches",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/deniers",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/dogstatsds",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/edges",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/fluentds",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/kubernetesenvs",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/kuberneteses",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/listcheckers",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/listentries",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/logentries",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/memquotas",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/metrics",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/noops",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/opas",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/prometheuses",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/quotas",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/rbacs",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/redisquotas",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/reportnothings",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/signalfxs",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/solarwindses",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/stackdrivers",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/statsds",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/stdios",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/tracespans",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/legacy/zipkins",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/config/v1alpha2/templates",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/policy/v1beta1/attributemanifests",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/policy/v1beta1/handlers",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/policy/v1beta1/instances",connectionID="0"} 1
mixer_mcp_sink_request_acks_total{collection="istio/policy/v1beta1/rules",connectionID="0"} 1
# HELP mixer_runtime_dispatch_duration_seconds Duration in seconds for adapter dispatches handled by Mixer.
# TYPE mixer_runtime_dispatch_duration_seconds histogram
mixer_runtime_dispatch_duration_seconds_bucket{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes",le="0.0001"} 42898
mixer_runtime_dispatch_duration_seconds_bucket{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes",le="0.00025"} 111560
mixer_runtime_dispatch_duration_seconds_bucket{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes",le="0.0005"} 111843
mixer_runtime_dispatch_duration_seconds_bucket{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes",le="0.001"} 112069
mixer_runtime_dispatch_duration_seconds_bucket{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes",le="0.0025"} 112140
mixer_runtime_dispatch_duration_seconds_bucket{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes",le="0.005"} 112149
mixer_runtime_dispatch_duration_seconds_bucket{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes",le="0.01"} 112208
mixer_runtime_dispatch_duration_seconds_bucket{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes",le="0.025"} 112256
mixer_runtime_dispatch_duration_seconds_bucket{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes",le="0.05"} 112259
mixer_runtime_dispatch_duration_seconds_bucket{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes",le="0.1"} 112271
mixer_runtime_dispatch_duration_seconds_bucket{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes",le="0.25"} 112271
mixer_runtime_dispatch_duration_seconds_bucket{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes",le="0.5"} 112271
mixer_runtime_dispatch_duration_seconds_bucket{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes",le="1"} 112271
mixer_runtime_dispatch_duration_seconds_bucket{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes",le="2.5"} 112271
mixer_runtime_dispatch_duration_seconds_bucket{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes",le="5"} 112271
mixer_runtime_dispatch_duration_seconds_bucket{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes",le="10"} 112271
mixer_runtime_dispatch_duration_seconds_bucket{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes",le="+Inf"} 112271
mixer_runtime_dispatch_duration_seconds_sum{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes"} 14.673212938999988
mixer_runtime_dispatch_duration_seconds_count{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes"} 112271
mixer_runtime_dispatch_duration_seconds_bucket{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric",le="0.0001"} 66285
mixer_runtime_dispatch_duration_seconds_bucket{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric",le="0.00025"} 73207
mixer_runtime_dispatch_duration_seconds_bucket{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric",le="0.0005"} 73586
mixer_runtime_dispatch_duration_seconds_bucket{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric",le="0.001"} 73670
mixer_runtime_dispatch_duration_seconds_bucket{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric",le="0.0025"} 73682
mixer_runtime_dispatch_duration_seconds_bucket{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric",le="0.005"} 73683
mixer_runtime_dispatch_duration_seconds_bucket{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric",le="0.01"} 73704
mixer_runtime_dispatch_duration_seconds_bucket{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric",le="0.025"} 73711
mixer_runtime_dispatch_duration_seconds_bucket{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric",le="0.05"} 73714
mixer_runtime_dispatch_duration_seconds_bucket{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric",le="0.1"} 73716
mixer_runtime_dispatch_duration_seconds_bucket{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric",le="0.25"} 73716
mixer_runtime_dispatch_duration_seconds_bucket{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric",le="0.5"} 73716
mixer_runtime_dispatch_duration_seconds_bucket{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric",le="1"} 73716
mixer_runtime_dispatch_duration_seconds_bucket{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric",le="2.5"} 73716
mixer_runtime_dispatch_duration_seconds_bucket{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric",le="5"} 73716
mixer_runtime_dispatch_duration_seconds_bucket{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric",le="10"} 73716
mixer_runtime_dispatch_duration_seconds_bucket{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric",le="+Inf"} 73716
mixer_runtime_dispatch_duration_seconds_sum{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric"} 5.922906316
mixer_runtime_dispatch_duration_seconds_count{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric"} 73716
# HELP mixer_runtime_dispatches_total Total number of adapter dispatches handled by Mixer.
# TYPE mixer_runtime_dispatches_total counter
mixer_runtime_dispatches_total{adapter="kubernetesenv",error="false",handler="kubernetesenv.istio-system",meshFunction="kubernetes"} 112271
mixer_runtime_dispatches_total{adapter="prometheus",error="false",handler="prometheus.istio-system",meshFunction="metric"} 73716
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 2575.91
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 17
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 6.2976e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.56119577163e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 8.353792e+07
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1

I have the same issue.Any luck mate?

Looking at your Mixer self-monitoring metrics, I don’t see any dispatches to the stdio logger (or any adapter other than prometheus and kubernetesenv).

And, according to Mixer, there is only config for two adapters: mixer_config_handler_configs_total{configID="1"} 2

This leads me to believe that Mixer is not receiving configuration for the newloghandler. The fact that you would have added this config after install, but still only have "configID=1" in these metrics is also concerning. I would have expected more configuration pushes as you added the logging config.

Just out of curiosity, have you tried bouncing your istio-telemetry pod(s) and/or galley?