Hello all:
We’re upgrading our K8S clusters on GCP and we also have been upgrading the Istio version.
After our rollout on the development environment, where we moved our K8s from 1.16 and Istio 1.4.5 to K8s version 1.19 and Istio 1.10.1. So far, so good.
Apart from that, we use GCP Trace. Previously, we have to do customizations on Istio Mixer to have stack driver enabled on Istio and also spanning tags. Here is how the configuration looked like when we were configuring with Mixer:
apiVersion: "config.istio.io/v1alpha2"
kind: handler
metadata:
name: stackdriver-tracing-handler-example
namespace: istio-system
spec:
compiledAdapter: stackdriver
params:
trace:
sampleProbability: 1
---
apiVersion: "config.istio.io/v1alpha2"
kind: instance
metadata:
name: stackdriver-span-example
namespace: istio-system
spec:
compiledTemplate: tracespan
params:
traceId: request.headers["x-b3-traceid"]
spanId: request.headers["x-b3-spanid"] | ""
parentSpanId: request.headers["x-b3-parentspanid"] | ""
spanName: destination.service.name + request.path
startTime: request.time
endTime: response.time
clientSpan: (context.reporter.kind | "inbound") == "outbound"
spanTags:
http.method: request.method | ""
http.status_code: response.code | 200
http.url: request.path | ""
destination_service_name: destination.service.name | "unknown"
destination_service_namespace: destination.service.namespace | "unknown"
destination_port: destination.port | 0
request_operation: conditional((context.protocol | "unknown") == "grpc", request.path | "unknown", request.method | "unknown")
request_protocol: context.protocol | "unknown"
api_version: api.version | "unknown"
api_name: api.service | "unknown"
response_code: response.code | 0
service_authentication_policy: conditional((context.reporter.kind | "inbound") == "outbound", "unknown", conditional(connection.mtls | false, "mutual_tls", "none"))
source_workload_namespace: source.workload.namespace | "unknown"
source_workload_name: source.workload.name | "unknown"
source_owner: source.owner | "unknown"
destination_workload_namespace: destination.workload.namespace | "unknown"
destination_workload_name: destination.workload.name | "unknown"
destination_owner: destination.owner | "unknown"
http_url: request.path | ""
request_size: request.size | 0
response_size: response.size | 0
source_ip: source.ip | ip("0.0.0.0")
---
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
name: stackdriver-tracing-rule-example
namespace: istio-system
spec:
match: (context.protocol == "http" || context.protocol == "grpc") && conditional(match(request.path, "*.php"), false, true) && destination.service.namespace == "example" && request.path != "/heartbeat" && request.path != "/" && request.path != "/metrics"
actions:
- handler: stackdriver-tracing-handler-example
instances:
- stackdriver-span-example
---
Since Istio 1.8, Mixer support was dropped and some configurations that were done with CRDs such as rule must be done on top of envoy using envoyfilter CRD. I didn’t think I would need to port those resources on Istio 1.10.
However, looking at the GCP Trace Dashboard, I can notice some differences, wherein the old cluster I can see the tags are being correctly being spanned, but with the newer cluster, I don’t see in the same format as previously. Here is a screenshot showing how it looks like:
The effects, in short, are the following:
- Previously we have our tracing labels as “Sent.SOMETHING” and “Recv.SOMETHING”. It’s all now starting with FQDN such as “example.default.svc.cluster.local”.
- There are duplications on the tracing. You can see that the “example.default.svc.cluster.local” appears twice. I guess that’s the part where we should see “Sent.SOMETHING” and “Recv.SOMETHING”.
- We no longer have our spanned tags like we used to have.
I tried to find a good example on how to expand those tags back on this link or this link but nothing seemed to be very clear on how to accomplish that.
Can you please advise me on how should I create those tags back?
Thank you,
Willian