Hi,
Can anyone point me towards any docs regarding how to install the Istio cloudwatch adapter?
I see the handler parameters are documented here: https://istio.io/docs/reference/config/policy-and-telemetry/adapters/cloudwatch/
But I can’t find any examples of the actual YAML that shows the resources that need to be created.
Many thanks in advance!
If you look in the repo, there is example operator config supplied. I’m not sure how up-to-date that configuration has been kept, however.
Thanks, I did try that config earlier but from what i can see this is not compatible with the later istio builds - it uses logentry/metric/cloudwatch CRDs that are not present.
I eventually got it working via configuring handler/instance/rule resources based upon the prometheus adapter config included in the helm charts.
Appreciate your response though.
Thanks
Happy to hear you were able to get it working. Sorry it took a bit of digging.
If you have the time, I’d very much welcome a PR to update the example config.
sure - I’ll raise once I’ve cleaned up the config
Were you ever able to update docs? I am new to istio and trying to figure out how to send istio logs to cloudwatch, and am completely stumped. I read up on the mixer adapter model and learned a bit about handlers, templates, and instances, but am confused if I need to write my own for the cloudwatch adapter, and if so what files I actually need to create, and what they should look like.
I tried looking for the prometheus adapter config but didn’t have any luck
Hi,
I’m afraid I still haven’t got round to creating a sample config suitable for PR.
Here is an example of our configuration for simple request count / durations though:
apiVersion: "config.istio.io/v1alpha2"
kind: instance
metadata:
labels:
app: mixer
name: requests
spec:
compiledTemplate: metric
params:
dimensions:
service: destination.service.name | "unknown"
env: destination.service.namespace | "unknown"
response_code: response.code
monitored_resource_type: '"UNSPECIFIED"'
value: "1"@studees Did you ever get the cloudwatch template working with a logentry? I keep getting error adapters failed to get message for instance: accesslogentry.instance.istio-system. failed to evaluate template for log instance: &{accesslogentry.instance.istio-system map[destination_ip:�� destination_service:unknown latency:199.372µs method:GET protocol:https request_size:%!s(int64=0) response_code:%!s(int64=404) response_size:%!s(int64=0) source_ip:��
Sorry for the late reply, if you are still looking at this then this config seemed to successfully configure access logs within cloudwatch:
apiVersion: "config.istio.io/v1alpha2"
kind: instance
metadata:
name: accesslog
labels:
app: mixer
spec:
compiledTemplate: logentry
params:
severity: '"Info"'
timestamp: request.time
variables:
sourceApp: source.labels["app"] | ""
sourcePrincipal: source.principal | ""
sourcePod: source.name | ""
destinationApp: destination.labels["app"] | ""
meshHostname: destination.service.host | ""
hostname: api.service | ""
pod: destination.name | ""
namespace: destination.namespace | ""
apiClaims: request.auth.raw_claims | ""
protocol: request.scheme | context.protocol | "http"
method: request.method | ""
url: request.path | ""
responseCode: response.code | 0
responseFlags: context.proxy_error_code | ""
responseSize: response.size | 0
requestSize: request.size | 0
requestId: request.headers["x-request-id"] | ""
latency: response.duration | "0ms"
connection_security_policy: conditional((context.reporter.kind | "inbound") == "outbound", "unknown", conditional(connection.mtls | false, "mutual_tls", "none"))
requestedServerName: connection.requested_server_name | ""
userAgent: request.headers["x-user-agent"] | request.useragent | ""
responseTimestamp: response.time
receivedBytes: request.total_size | 0
sentBytes: response.total_size | 0
xForwardedFor: request.headers["x-forwarded-for"] | "unknown"
monitored_resource_type: '"global"'
---
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
name: access-log-cloudwatch
labels:
app: mixer
spec:
match: context.protocol == "http" && context.reporter.kind == "inbound"
actions:
- handler: cloudwatch
instances:
- accesslog@studees Would you mind also supplying your handler? I am still running into failed to evaluate template for log instance errors. Thank you for your reply!
Sure - here you go, fair bit of escaping in there as I wanted the logs as JSON:
apiVersion: "config.istio.io/v1alpha2"
kind: handler
metadata:
name: cloudwatch
labels:
app: mixer
chart: mixer
release: istio
spec:
compiledAdapter: cloudwatch
params:
namespace: istio-metrics
metricInfo:
totalrequests.instance.istio-system:
unit: Count
requests.instance.istio-system:
unit: Count
clientrequests.instance.istio-system:
unit: Count
duration.instance.istio-system:
unit: Milliseconds
logGroupName: /istio/istio-system/istio-access.json
logStreamName: mixer
logs:
accesslog.instance.istio-system:
payloadTemplate: '{ "sourceApp": "{{"{{"}} .sourceApp {{"}}"}}", "sourcePrincipal": "{{"{{"}} .sourcePrincipal {{"}}"}}", "sourcePod": "{{"{{"}} .sourcePod {{"}}"}}", "destinationApp": "{{"{{"}} .destinationApp {{"}}"}}", "meshHostname": "{{"{{"}} .meshHostname {{"}}"}}", "hostname": "{{"{{"}} .hostname {{"}}"}}", "pod": "{{"{{"}} .pod {{"}}"}}", "namespace": "{{"{{"}} .namespace {{"}}"}}", "apiClaims": "{{"{{"}} .apiClaims {{"}}"}}", "protocol": "{{"{{"}} .protocol {{"}}"}}", "method": "{{"{{"}} .method {{"}}"}}", "url": "{{"{{"}} .url {{"}}"}}", "responseCode": "{{"{{"}} .responseCode {{"}}"}}", "responseFlags": "{{"{{"}} .responseFlags {{"}}"}}", "responseSize": "{{"{{"}} .responseSize {{"}}"}}", "requestSize": "{{"{{"}} .requestSize {{"}}"}}", "requestId": "{{"{{"}} .requestId {{"}}"}}", "latency": "{{"{{"}} .latency {{"}}"}}", "connection_security_policy": "{{"{{"}} .connection_security_policy {{"}}"}}", "requestedServerName": "{{"{{"}} .requestedServerName {{"}}"}}", "userAgent": "{{"{{"}} .userAgent {{"}}"}}", "responseTimestamp": "{{"{{"}} .responseTimestamp {{"}}"}}", "receivedBytes": "{{"{{"}} .receivedBytes {{"}}"}}", "sentBytes": "{{"{{"}} .sentBytes {{"}}"}}", "xForwardedFor": "{{"{{"}} .xForwardedFor {{"}}"}}" }'Thank you very much, I believe my issue was that I wasn’t addressing fields in the payload template with their name before their value. So for payload template I just had .sourceApp instead of sourceApp: .sourceApp
Hello there - Do you know if there is any better documentation around this topic? Im struggling to understand what adapter these resources refer to in cloud watch is it the container insights adapter, if not which agent do I need to install to scrape istio metics envoy logs into cloud watch?
Is this the bets way to get logs and metrics from istio into cloudwatch or would using Fluent D and prometheus cloudwatch adapter be a better path?
Thanks in advance
@fostpau at this point, using a fluentd + prom->cloudwatch solution would be a better path. the mixer adapters are deprecated (and completely removed in 1.8).
thanks mate - yep came to the same conclusion after time.