Errorlog "One or more TimeSeries could not be written" by Istio 1.5 with Stackdriver on GKE

Policies and Telemetry
1

I’m trying Istio 1.5 on GKE with Stackdriver. But I can’t see any metrics from Istio and I found a lot of following error message in the log. Could anyone help me to solve this error?

CreateTimeSeries request failed (1 RPCs, 8 views, 24 timeseries): INVALID_ARGUMENT: One or more TimeSeries could not be written: Unrecognized metric labels: destination_canonical_revision, source_canonical_revision, destination_canonical_service_name, source_canonical_service_namespace, destination_canonical_service_namespace, source_canonical_service_name: timeSeries[1,4,6,21]; Unrecognized metric labels: destination_canonical_service_name, source_canonical_service_namespace, destination_canonical_service_namespace, source_canonical_service_name, destination_canonical_revision, source_canonical_revision: timeSeries[8,14,15]; Unrecognized metric labels: destination_canonical_service_namespace, source_canonical_service_name, destination_canonical_revision, source_canonical_revision, destination_canonical_service_name, source_canonical_service_namespace: timeSeries[9,12]; Unrecognized metric labels: source_canonical_revision, destination_canonical_service_name, source_canonical_service_namespace, destination_canonical_service_namespace, source_canonical_service_name, destination_canonical_revision: timeSeries[0,2,7,10,11,13,19]; Unrecognized metric labels: source_canonical_service_namespace, destination_canonical_service_namespace, source_canonical_service_name, destination_canonical_revision, source_canonical_revision, destination_canonical_service_name: timeSeries[3,5,16-18,20,22,23]

Procedure

1 Move to istio-1.5.0 installer directory

❯ cd istio-1.5.0

2 Prepare a config file

❯ cat << EOF > config.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: default

  addonComponents:
    prometheus:
      enabled: false

  values:
    global:
      proxy:
        tracer: "stackdriver"
    telemetry:
      v2:
        enabled: true
        prometheus:
          enabled: false
        stackdriver:
          enabled: true
          logging: true
          monitoring: true
          topology: true
          configOverride: {}
EOF

3 Deploy Istio with the config

❯ istioctl manifest apply -f config.yaml

4 Deploy bookinfo sample application

❯ kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
❯ kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

5 Access to the bookinfo application

Result

  • Stackdriver Trace shows information
  • There’s no metrics shown on Stackdriver Metrics
  • The error logs above are shown

Versions

❯ kubectl version --short
Client Version: v1.16.3
Server Version: v1.14.10-gke.17

❯ istioctl version
client version: 1.5.0
control plane version: 1.5.0
data plane version: 1.5.0 (7 proxies)
2

The schema changes to SD were rolling out to production as you were testing. Would you mind trying again and updating your status here?

3

Thank you! I understood. I will try it again and update the result here.

4

It works now! Thank you!