Remove default resources limits for istio-proxy sidecar

1

Hi,

Is it possible to remove resources limits at all? By default they are set to cpu: 2000m and memory: 1024Mi. I understand that, I can override those values, but my question is: is it possible to completely remove these settings?

Cheers,
Pawel

2

Anyone? cc @ostromart?

3

There should be a way to delete items with k8s.overlays and in general with settings in overlay files for all paths, but I just verified it doesn’t work.
The rather clunky way of making this work in 1.5 would be to use external charts and edit the default.yaml profile to remove the values there. I’ve created https://github.com/istio/istio/issues/23429 to track it, I think it should make it into 1.6.

4

fyi it’s fixed in https://github.com/istio/istio/pull/23596.

1 Like
5

@ostromart, I’ve just checked with Istio 1.6.2 and still it doesn’t work.

Here is my manifest (integration-testing.yaml):

apiVersion: operator.istio.io/v1alpha1
kind: IstioOperator
spec:
  tag: 1.6.2
  components:
    pilot:
      k8s:
        hpaSpec:
          minReplicas: 1

  values:
    global:
      proxy:
        autoInject: enabled
        resources:
          limits:
            memory: 256Mi
    
    grafana:
      enabled: false

    kiali:
      enabled: false

    prometheus:
      enabled: false

And I install Istio this way:

$ istioctl manifest apply -f integration-testing.yaml

Then I can see, that CPU limits are set, but it shouldn’t be:

$ kubectl -n istio-system get istiooperator installed-state -o json | jq .spec.values.global.proxy.resources.limits
{
  "cpu": "2000m",
  "memory": "256Mi"
}

And istio-proxy containers in my pods have those CPU limits:

$ kubectl -n default describe pod application-bf595fc58-69m7k
...
  istio-proxy:
    Container ID:  containerd://a84348116f71c35928c3ba383204339b03ef837ef81d4797fe824d8248ffd0ec
    Image:         docker.io/istio/proxyv2:1.6.2
    Image ID:      docker.io/istio/proxyv2@sha256:ac25e39a130f5678bef302adea9e22163cd9d4737b0777e4e5e8353d916e223f
    Port:          15090/TCP
    Host Port:     0/TCP
    Args:
      proxy
      sidecar
      --domain
      $(POD_NAMESPACE).svc.cluster.local
      --serviceCluster
      application.$(POD_NAMESPACE)
      --proxyLogLevel=warning
      --proxyComponentLogLevel=misc:error
      --trust-domain=cluster.local
      --concurrency
      2
    State:          Running
      Started:      Tue, 16 Jun 2020 10:01:18 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  256Mi
    Requests:
      cpu:      100m
      memory:   128Mi

As you can see the istio-proxy container has CPU limits set to 2.

1 Like
6

Hi Pawel,
That’s correct, there’s no way to do it through the regular overlay, but you can now do it in the output manifest using the components.X.k8s.overlays path. See examples here: https://istio.io/latest/docs/setup/install/istioctl/#patching-the-output-manifest
To make this possible in the rest of the API we’d probably need to recognize a special “null” string. It’s doable but I’m not sure if anyone on the team would have time to prioritize it given the current workload. Feel free to open a feature request in istio/istio and assign to me to triage though.
You can also do it at the general API level by using the external charts i.e. use the charts in the tarball and run istioctl --charts /manifests to point to them. Then you can edit the profiles under manfiests/profiles/default.yaml to remove the settings there.

7

Hi @ostromart,

Thank you for the response. When I set null I can see, that CPU limits are not set.

Cheers,
Pawel

8

I wasn’t expecting that to work :slight_smile: Luckily for us the yaml unmarshaler does the right thing and creates an in memory structure with nil in it.

9

In 1.7.0 and also in the latest version (which is at the moment 1.7.3), there is a regression. At this moment it’s not possible to remove resource limits. Can you advise how can I do that?

EDIT

I tried to set:

  • spec.values.global.proxy.limits.cpu: null
  • spec.values.global.proxy.limits.cpu: ~

When I try to set other values, I can that limits are set correctly. The problem is, that it cannot completely remove CPU limits. However, it’s interesting, because in the IstioOperator CPU limits are not set:

$ kubectl --context kind-kind -n istio-system get istiooperators.install.istio.io installed-state -o json | jq .spec.values.global.proxy.resources
{
  "limits": {
    "memory": "300Mi"
  },
  "requests": {
    "cpu": "100m",
    "memory": "128Mi"
  }
}

EDIT2

Also, when I set spec.values.global.proxy.limits.cpu: '', the sidecar injector doesn’t work properly (as expected):

$ kubectl --context kind-kind --namespace istio-system logs -f istiod-6c5f6f55ff-cdrgw
...
2020-10-07T09:00:14.223914Z	info	AdmissionReview for Kind=/v1, Kind=Pod Namespace=test Name= (with-sidecar-6cdb4c8f89-***** (actual name not yet known)) UID=d340cca9-4a95-4760-b0da-3f2ae883807e Rfc6902PatchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller 8e876f51-ce1b-45f7-89b8-cdb8c6ba94fc [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
2020-10-07T09:00:14.228177Z	warn	Failed to unmarshal template error unmarshaling JSON: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'
...
  resources:
    limits:
      cpu: ""
      memory: 300Mi
    requests:
      cpu: 100m
      memory: 128Mi
...
2020-10-07T09:00:14.228222Z	error	Injection data: err=failed parsing generated injected YAML (check Istio sidecar injector configuration): error unmarshaling JSON: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$' spec=
...
10

Stumbled upon the same problem after upgrading from 1.6 - not possible to use null to remove resource limits and indeed it works only for iop config, but no effect on the actual injected sidecar.

11

I am able to get a solution which works for any version above 1.6 at least. Did not check for previous versions though.
Here is the manifest (overlay.yml):

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
spec:
  profile: default
  meshConfig:
    defaultConfig:
      holdApplicationUntilProxyStarts: true
    outboundTrafficPolicy:
      mode: REGISTRY_ONLY
  components:
    ingressGateways:
      - name: istio-ingressgateway
        namespace: istio-system
        enabled: true
        k8s:
          overlays:
            - kind: Deployment
              name: istio-ingressgateway
              patches:
                - path: spec.template.spec.containers.[name:istio-proxy].resources.limits.cpu
                  value:

So, an empty value for value makes it disappear in the final manifest.

12

Hello, we currently have the same issue with latest version (1.11.3)… We tried the comments above but none worked. Is there a known solution/workaround to completely remove proxy sidecar resource limits? Thanks!

1 Like
13

I am unable to find the fix for this. Is it still an open issue. We are the version 1.11.3 and cannot set the limits to “null” to change the default behavior? Do we have any fix for this?

14

Hello, I also stumbled on this issue. We would like to remove CPU limits from Istio sidecar containers, just wondering if someone manage to find any alternatives.