Nodeagent crashing

Security
1

I have Istio 1.2.0 running in GKE v1.11.8-gke.6

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-26T00:04:52Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.8-gke.6", GitCommit:"394ee507d00f15a63cef577a14026096c310698e", GitTreeState:"clean", BuildDate:"2019-03-30T19:31:43Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"}

which I have installed on my custom namespace “foo”. However, the nodeagent crashes with the below error

2019-06-24T17:11:19.156269Z     info    unalbe to fetch CA TLS root cert: failed to get CA TLS root cert: configmaps "istio-security" not found, retry in 2s
2019-06-24T17:11:19.156342Z     error   failed to create caClient: exhausted all the retries (100) to fetch the CA TLS root cert
2019-06-24T17:11:19.156360Z     error   failed to create secretFetcher for workload proxy: failed to create caClient

$ kubectl -n foo get cm istio-security -o yaml
apiVersion: v1
data:
  caTLSRootCert: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMzakNDQWNhZ0F3SUJBZ0lSQUo0OXJsY1BjbEtWY1ErMm1IckVxWmN3RFFZSktvWklodmNOQVFFTEJRQXcKR0RFV01CUUdBMVVFQ2hNTlkyeDFjM1JsY2k1c2IyTmhiREFlRncweE9UQTJNalF4TnpBNE1EQmFGdzB5T1RBMgpNakV4TnpBNE1EQmFNQmd4RmpBVUJnTlZCQW9URFdOc2RYTjBaWEl1Ykc5allXd3dnZ0VpTUEwR0NTcUdTSWIzCkRRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQzhLN0JiU0VtR3JMOTZxYUJhby9hWk42SUJ5WmcxUFFiaUIzenIKTWhhanJlSTRPQ0k0OWR6c25GNnc5bWFPRjBYb3N2T2VXMXloVTVVUUg4aUJueTJyckk3SE0yZlBRRUprcExMNQpjc0FhdjB4b1NZUEJIc2tNMk5CUWZ6ZmprSmgxbHc4ZzNyVXFCS3lvNHhlOVVtVU51MjRzQmxaQ3VxbGFLQzRpCnZUOVJzSThlS25iT2d3c0oydnFZeEFmUnBtdTVVOXJPemMrLzBGNzFWVTFWMk9zVDUwM3ZkVHhVYVM1c212cEsKWlJJT29CY2wwZ0gyZ3F6aTRHalViLy94cmlXQVpVUGc2ZTZPRWdNdk5YdmhubEdsSElacWdhSEsrVG96S2Z4TApUOGJRMitpWTdpcTBZY2JsZWhsZWFnclBCSFZrMUJhNExvY1NsSm84dmNpS25WaGRBZ01CQUFHakl6QWhNQTRHCkExVWREd0VCL3dRRUF3SUNCREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUIKQVFCM1VRVmRqdndpUW5mbG0xdERwOFNwSVZaRjdNMkhGRFVtaFY3NkMwZXJBUGkyeWMvZS9XcVVCdm9JRnVZQQpiS2FiaCthbWtvYWFuM1hDRjVKeFFST3VSY09xMjQvbWlCWG5zTzFjUnFoM2FkUjhKRmVhYTJWKy9JR1VDZkNjCjZwTTRpQnRSUmtqa2ZTeWlFSkNhTWYxcTlpTWJoZGNFYjB4L2kxditSTmdRQW5YTkNudlU0MWdmYnFHaUNkT1IKU1dqVVgwK29XOW1ueURnYzJUemZaeXhwL0VJQ2Z0VkR3MWtmK0FiMTZxNjZBNFVmbjIvT0paN1hkMnRDR0lNbQoyM2hqaWtjcU15RW1rTy93UG11VGdRZUorUTlJbzV5dDNUSWEybTBERXFhU1A4YzYwb3R2dVpmbUxuWG9yUVU3CmxqU211OUdqcVZ6WEtZMVFUN09nMERIawotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
kind: ConfigMap
metadata:
  creationTimestamp: "2019-06-24T17:08:00Z"
  name: istio-security
  namespace: foo
  resourceVersion: "28623497"
  selfLink: /api/v1/namespaces/foo/configmaps/istio-security
  uid: 9eb31555-96a2-11e9-8301-42010a800fc1

Am I doing something that is causing the issue?

Btw, I have used the below helm for the installation

helm template install/kubernetes/helm/istio --name istio --namespace foo \
    --set gateways.istio-ingressgateway.sds.enabled=true \
   	--set security.enabled=true \
   	--set galley.enabled=true \
   	--set sidecarInjectorWebhook.enabled=false \
   	--set mixer.enabled=false \
   	--set mixer.policy.enabled=false \
   	--set mixer.telemetry.enabled=false \
   	--set prometheus.enabled=false \
   	--set pilot.sidecar=false \
   	--values install/kubernetes/helm/istio/values-istio-sds-auth.yaml | kubectl -f

And my values-istio-sds-auth.yaml as below where I’ve modified the mtls enables as false

$ cat install/kubernetes/helm/istio/values-istio-sds-auth.yaml
global:
  controlPlaneSecurityEnabled: false

  mtls:
    # Default setting for service-to-service mtls. Can be set explicitly using
    # destination rules or service annotations.
    enabled: false

  sds:
    enabled: true
    udsPath: "unix:/var/run/sds/uds_path"
    useNormalJwt: true

nodeagent:
  enabled: true
  image: node-agent-k8s
  env:
    CA_PROVIDER: "Citadel"
    CA_ADDR: "istio-citadel:8060"
    VALID_TOKEN: true

Any pointer will be much appreciated. Thank you!!

2

Looks like it works fine if “istio-system” namespace is used. Can we remove this “isto-system” namespace dependency?

3

istio-system is a reserved namespace for Istio control plane components and their resources. Why do you want to use foo or something else?

4

Yes, that namespace is reserved but I don’t feel it is mandatory. If you have given the option in templating where it can be modified, it should support any namespace

5

I’m afraid the namespace is hard-coded in a lot of places, at least that’s what I saw browsing through the code base. Maybe it’s not meant to be mandatory to use istio-system, but at this point it certainly is.

6

I agree with @dgn point. FYI @Quanjie_Lin

7

thanks for raising the issue, sent a fix https://github.com/istio/istio/pull/15199

8

Components should be able to be installed in any namespace, this is a hard requirement for https://github.com/istio/installer. Thanks for bringing this up