Disable gc logs from mixer istio-telemetry pods

Policies and Telemetry
1

Hello.

I have a plethora of garbage collection logs emitted to stderr from istio-telemetry pods – for example:

gc 1748368 @4319896.596s 0%: 0.017+8.3+0.011 ms clock, 0.10+2.2/9.1/24+0.066 ms cpu, 27->28->14 MB, 28 MB goal, 6 P

Helm settings look like this:

# Common settings used among istio subcharts.
global:
...
  # Comma-separated minimum per-scope logging level of messages to output, in the form of <scope>:<level>,<scope>:<level>
  # The control plane has different scopes depending on component, but can configure default log level across all components
  # If empty, default scope and level will be used as configured in code
  logging:
    level: "default:error"

Is there a way to disable this behavior? My logging budget will thank you.

1 Like
2

Hi, we got the same problem. The logs are filled with the GC-messages and I need a way to turn it off.

3

I believe that is controlled by an env var. IIRC, the helm manifests used to add a GODEBUG: gctrace=2 env to Mixer deployments. Removing that, or setting it differently should help.

4

Saw Douglas’ response and decided to investigate a bit more. Looks like John Howard fixed in 1.4.X. I confirmed the symptoms are gone in our environments.

commit c269a435093f74baf4af9af34223156d4a157de6
Author: John Howard <howardjohn@google.com>
Date:   Thu Sep 5 16:48:02 2019 -0700

    Remove GODEBUG from pilot and mixer (#16852)

    A bug/feature in golang 1.13 causes GODEBUG to output thousands of
    messages. Since we will release 1.4 off of 1.13, I think the best course
    of action is to turn this off by default.

    See #16635

diff --git a/install/kubernetes/helm/istio/charts/mixer/values.yaml b/install/kubernetes/helm/istio/charts/mixer/values.yaml
index 4d9452356..d335c36c3 100644
--- a/install/kubernetes/helm/istio/charts/mixer/values.yaml
+++ b/install/kubernetes/helm/istio/charts/mixer/values.yaml
@@ -4,7 +4,6 @@
 image: mixer

 env:
-  GODEBUG: gctrace=1
   # max procs should be ceil(cpu limit + 1)
   GOMAXPROCS: "6"