Istio 1.10 with stackdriver tracing [GKE]

I have installed istio 1.10 with the following config on GCP.
I want to use the google strackdriver tracing instead of jager or zipkin, but does not seems to work.
I can the metrics working fine and was able to create dashboards.
Wonder what is the issue ?
if anyone can help would really apperaicate it.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
spec:
  hub: docker.io/istio
  tag: 1.10.0
  profile: default
  # You may override parts of meshconfig by uncommenting the following lines.
  meshConfig:
    defaultConfig:
      proxyMetadata:
        SECRET_TTL: 2160h
        SECRET_GRACE_DURATION: 2160h
    enablePrometheusMerge: false
    accessLogFile: /dev/stdout
    accessLogEncoding: JSON
    # Opt-out of global http2 upgrades.
    # Destination rule is used to opt-in.
    # h2_upgrade_policy: DO_NOT_UPGRADE


  # Traffic management feature
  components:
    base:
      enabled: true
    pilot:
      enabled: true
      k8s:
        strategy:
          rollingUpdate:
            maxUnavailable: 0
            maxSurge: 1
    # Istio Gateway feature
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
      k8s:
        strategy:
          rollingUpdate:
            maxUnavailable: 0
            maxSurge: 1
    egressGateways:
    - name: istio-egressgateway
      enabled: false

    # Istio CNI feature
    cni:
      enabled: false

    # istiod remote configuration when istiod isn't installed on the cluster
    istiodRemote:
      enabled: false
  # Global values passed through to helm global.yaml.
  # Please keep this in sync with manifests/charts/global.yaml
  values:
    global:
      istioNamespace: istio-system
      istiod:
        enableAnalysis: true
      logging:
        level: "default:info"
      logAsJson: true
      pilotCertProvider: istiod
      proxy:
        image: proxyv2
        clusterDomain: "cluster.local"
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 2000m
            memory: 1024Mi
        lifecycle:
          preStop:
            exec:
              command: ["sh", "-c", "sleep 5"]
        logLevel: info
        componentLogLevel: "misc:error"
        privileged: false
        enableCoreDump: true
        statusPort: 15020
        readinessInitialDelaySeconds: 1
        readinessPeriodSeconds: 2
        readinessFailureThreshold: 30
        includeIPRanges: "*"
        excludeIPRanges: ""
        excludeOutboundPorts: ""
        excludeInboundPorts: ""
        autoInject: enabled
        tracer: "stackdriver"
      proxy_init:
        image: proxyv2
        resources:
          requests:
            cpu: 100m
            memory: 10Mi
      # Specify image pull policy if default behavior isn't desired.
      # Default behavior: latest images will be Always else IfNotPresent.
      imagePullPolicy: ""
      operatorManageWebhooks: false
      imagePullSecrets: []
      oneNamespace: false
      defaultNodeSelector: {}
      configValidation: true
      multiCluster:
        enabled: false
        clusterName: ""
      omitSidecarInjectorConfigMap: false
      network: ""
      defaultResources:
        requests:
          cpu: 10m
      defaultPodDisruptionBudget:
        enabled: true
      priorityClassName: ""
      useMCP: false
      sds:
        token:
          aud: istio-ca
      sts:
        servicePort: 0
      meshNetworks: {}
      mountMtlsCerts: false
    base:
      enableCRDTemplates: false
      validationURL: ""
    pilot:
      autoscaleEnabled: true
      autoscaleMin: 1
      autoscaleMax: 5
      replicaCount: 1
      image: pilot
      traceSampling: 1.0
      env: {}
      cpu:
        targetAverageUtilization: 80
      nodeSelector: {}
      keepaliveMaxServerConnectionAge: 30m
      enableProtocolSniffingForOutbound: true
      enableProtocolSniffingForInbound: true
      deploymentLabels:
      configMap: true

    telemetry:
      enabled: true
      v2:
        enabled: true
        metadataExchange:
          wasmEnabled: false
        prometheus:
          wasmEnabled: false
          enabled: true
        stackdriver:
          enabled: true
          # logging: true
          outboundAccessLogging: FULL
          inboundAccessLogging: FULL
          monitoring: true
          topology: true
          configOverride: {}

    gateways:
      istio-ingressgateway:
        autoscaleEnabled: true
        autoscaleMin: 1
        autoscaleMax: 5
        type: NodePort
        ports:
        ## You can add custom gateway ports in user values overrides, but it must include those ports since helm replaces.
        # Note that AWS ELB will by default perform health checks on the first port
        # on this list. Setting this to the health check port will ensure that health
        # checks always work. https://github.com/istio/istio/issues/12503
        - port: 15021
          targetPort: 15021
          name: status-port
          protocol: TCP
          nodePort: 31021
        - port: 80
          targetPort: 8080
          name: http2
          protocol: TCP
          nodePort: 31080
        - port: 443
          targetPort: 8443
          name: https
          protocol: TCP
          nodePort: 31443
        name: istio-ingressgateway

Service account for the GKE cluster has the following permissions

Cloud Trace Agent
Cloud Trace User
Logs Writer
Monitoring Metric Writer
Monitoring Viewer
Stackdriver Resource Metadata Write

To see the telemetry data in StackDriver you need to enable it in the Istio profile config.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istio-production-controlplane
spec:
  values:
  ...
  ...
    telemetry:
      enabled: true
      v2:
        stackdriver:
          configOverride: {}
          enabled: true
          logging: true
          monitoring: true
          topology: true

The following values are false by default. Set them to true.

        stackdriver:
          configOverride: {}
          enabled: true
          logging: true
          monitoring: true
          topology: true
1 Like

thanks let me give it a try and get back to you

does not seems to work.

        stackdriver:
          enabled: true
          # logging: true
          outboundAccessLogging: FULL
          inboundAccessLogging: FULL
          monitoring: true
          topology: true
          configOverride: {}

using jaeger tracing instead

@fai555 sorry for even late update.
but I got it to work with the following config, plus in GCP GKE needs to have Workload Identity enabled and the pod’s service account linked to GCP service account that has permission to create metrics and send logs to the stack driver. otherwise, the ISTIO sidecar will keep throwing errors that it does not have permission to send logs and metrics to stackdriver.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
spec:
  hub: docker.io/istio
  tag: 1.10.0
  profile: default
  # You may override parts of meshconfig by uncommenting the following lines.
  meshConfig:
    defaultConfig:
      proxyMetadata:
        SECRET_TTL: 2160h
        SECRET_GRACE_DURATION: 2160h
    enablePrometheusMerge: false
    accessLogFile: /dev/stdout
    accessLogEncoding: JSON
    # Opt-out of global http2 upgrades.
    # Destination rule is used to opt-in.
    # h2_upgrade_policy: DO_NOT_UPGRADE


  # Traffic management feature
  components:
    base:
      enabled: true
    pilot:
      enabled: true
      k8s:
        strategy:
          rollingUpdate:
            maxUnavailable: 0
            maxSurge: 1
    # Istio Gateway feature
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
      k8s:
        strategy:
          rollingUpdate:
            maxUnavailable: 0
            maxSurge: 1
    egressGateways:
    - name: istio-egressgateway
      enabled: false

    # Istio CNI feature
    cni:
      enabled: false

    # istiod remote configuration when istiod isn't installed on the cluster
    istiodRemote:
      enabled: false
  # Global values passed through to helm global.yaml.
  # Please keep this in sync with manifests/charts/global.yaml
  values:
    global:
      istioNamespace: istio-system
      istiod:
        enableAnalysis: true
      logging:
        level: "default:info"
      logAsJson: true
      pilotCertProvider: istiod
      proxy:
        image: proxyv2
        clusterDomain: "cluster.local"
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 2000m
            memory: 1024Mi
        lifecycle:
          preStop:
            exec:
              command: ["sh", "-c", "sleep 5"]
        logLevel: info
        componentLogLevel: "misc:error"
        privileged: false
        enableCoreDump: true
        statusPort: 15020
        readinessInitialDelaySeconds: 1
        readinessPeriodSeconds: 2
        readinessFailureThreshold: 30
        includeIPRanges: "*"
        excludeIPRanges: ""
        excludeOutboundPorts: ""
        excludeInboundPorts: ""
        autoInject: enabled
        tracer: "stackdriver"
      proxy_init:
        image: proxyv2
        resources:
          requests:
            cpu: 100m
            memory: 10Mi
      # Specify image pull policy if default behavior isn't desired.
      # Default behavior: latest images will be Always else IfNotPresent.
      imagePullPolicy: ""
      operatorManageWebhooks: false
      imagePullSecrets: []
      oneNamespace: false
      defaultNodeSelector: {}
      configValidation: true
      multiCluster:
        enabled: false
        clusterName: ""
      omitSidecarInjectorConfigMap: false
      network: ""
      defaultResources:
        requests:
          cpu: 10m
      defaultPodDisruptionBudget:
        enabled: true
      priorityClassName: ""
      useMCP: false
      sds:
        token:
          aud: istio-ca
      sts:
        servicePort: 0
      meshNetworks: {}
      mountMtlsCerts: false
    base:
      enableCRDTemplates: false
      validationURL: ""
    pilot:
      autoscaleEnabled: true
      autoscaleMin: 1
      autoscaleMax: 5
      replicaCount: 1
      image: pilot
      traceSampling: 100.0
      env: {}
      cpu:
        targetAverageUtilization: 80
      nodeSelector: {}
      keepaliveMaxServerConnectionAge: 30m
      enableProtocolSniffingForOutbound: true
      enableProtocolSniffingForInbound: true
      deploymentLabels:
      configMap: true
    telemetry:
      enabled: true
      v2:
        enabled: true
        metadataExchange:
          wasmEnabled: false
        prometheus:
          wasmEnabled: false
          enabled: true
        stackdriver:
          enabled: true
          # logging: true
          outboundAccessLogging: FULL
          inboundAccessLogging: FULL
          monitoring: true
          topology: true
          configOverride: {}
    gateways:
      istio-ingressgateway:
        autoscaleEnabled: true
        autoscaleMin: 1
        autoscaleMax: 5
        type: NodePort
        ports:
        ## You can add custom gateway ports in user values overrides, but it must include those ports since helm replaces.
        # Note that AWS ELB will by default perform health checks on the first port
        # on this list. Setting this to the health check port will ensure that health
        # checks always work. https://github.com/istio/istio/issues/12503
        - port: 15021
          targetPort: 15021
          name: status-port
          protocol: TCP
          nodePort: 31021
        - port: 80
          targetPort: 8080
          name: http2
          protocol: TCP
          nodePort: 31080
        - port: 443
          targetPort: 8443
          name: https
          protocol: TCP
          nodePort: 31443
        name: istio-ingressgateway