Istio-ingressgateway fails to start

Hi all,
I have ai Istio 1.6.3 upgraded from 1.6.1 upgraded from 1.6.0

Everyday I shut down the k8s cluster and restart it the next day.

Suddenly today the istio-ingressgateway deployment and pod fail to start

From the Kubernetes dashboard I see, for the deployment:

endpoint-controller

Failed to update endpoint istio-system/istio-ingressgateway: Operation cannot be fulfilled on endpoints “istio-ingressgateway”: the object has been modified; please apply your changes to the latest version and try again

endpoint-slice-controller

Error updating Endpoint Slices for Service istio-system/istio-ingressgateway: Error updating istio-ingressgateway-wknwr EndpointSlice for Service istio-system/istio-ingressgateway: Operation cannot be fulfilled on endpointslices.discovery.k8s.io “istio-ingressgateway-wknwr”: the object has been modified; please apply your changes to the latest version and try again

Then I tried to reinstall istio 1.6.3 obtaining the error

sysop@hdev:~/software/istio-1.6.3$ istioctl manifest apply --set profile=demo
:heavy_check_mark: Istio core installed
:heavy_check_mark: Istiod installed
:heavy_check_mark: Egress gateways installed
:heavy_check_mark: Addons installed
✘ Ingress gateways encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the condition
Deployment/istio-system/istio-ingressgateway

  • Pruning removed resources Error: failed to apply manifests: errors occurred during operation
    sysop@hdev:~/software/istio-1.6.3$

The istio-ingressgateway pod log is full of:

|2020-07-02T17:46:13.786890Z|warn|Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected|
|---|---|---|
|2020-07-02T17:46:15.786562Z|warn|Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected|
|2020-07-02T17:46:17.786890Z|warn|Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected|
|2020-07-02T17:46:19.788339Z|warn|Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected|
|2020-07-02T17:46:21.787993Z|warn|Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected|
|2020-07-02T17:46:23.786105Z|warn|Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected|
|2020-07-02T17:46:25.786699Z|warn|Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected|
|2020-07-02T17:46:27.787527Z|warn|Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected|

What can I do?

What are the logs from your istiod container?

I’ll try to post the logs asap

I see that there is an error related to “https://www.h.net:30000/auth/realms/hproject” that is a Keycloak server hosted on the same cluster.
https://www.h.net” is the address of the ingressgateway

I used this requestauthentication file:

sysop@hdev:~/software/hproject$ cat h-reqaut-ingress.yaml 
apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
  name: "h-ingress-jwt"
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  jwtRules:
  - issuer: "https://www.h.net:30000/auth/realms/hproject"
    jwksUri: "https://www.h.net:30000/auth/realms/hproject/protocol/openid-connect/certs"

---

sysop@hdev:~/software/hproject$ 

The two uri seem to respond correctly

I’m doing something wrong?

Here is the part of the log. The entire log is here

2020-07-03T08:24:56.119865Z	info	FLAG: --appNamespace=""
2020-07-03T08:24:56.119959Z	info	FLAG: --caCertFile=""
2020-07-03T08:24:56.119969Z	info	FLAG: --clusterID="Kubernetes"
2020-07-03T08:24:56.119974Z	info	FLAG: --clusterRegistriesNamespace="istio-system"
2020-07-03T08:24:56.119977Z	info	FLAG: --configDir=""
2020-07-03T08:24:56.119981Z	info	FLAG: --consulserverURL=""
2020-07-03T08:24:56.119986Z	info	FLAG: --ctrlz_address="localhost"
2020-07-03T08:24:56.119992Z	info	FLAG: --ctrlz_port="9876"
2020-07-03T08:24:56.119998Z	info	FLAG: --disable-install-crds="true"
2020-07-03T08:24:56.120018Z	info	FLAG: --domain="cluster.local"
2020-07-03T08:24:56.120024Z	info	FLAG: --grpcAddr=":15010"
2020-07-03T08:24:56.120028Z	info	FLAG: --help="false"
2020-07-03T08:24:56.120031Z	info	FLAG: --httpAddr=":8080"
2020-07-03T08:24:56.120035Z	info	FLAG: --httpsAddr=":15017"
2020-07-03T08:24:56.120042Z	info	FLAG: --keepaliveInterval="30s"
2020-07-03T08:24:56.120046Z	info	FLAG: --keepaliveMaxServerConnectionAge="30m0s"
2020-07-03T08:24:56.120050Z	info	FLAG: --keepaliveTimeout="10s"
2020-07-03T08:24:56.120054Z	info	FLAG: --kubeconfig=""
2020-07-03T08:24:56.120058Z	info	FLAG: --log_as_json="false"
2020-07-03T08:24:56.120061Z	info	FLAG: --log_caller=""
2020-07-03T08:24:56.120065Z	info	FLAG: --log_output_level="default:info"
2020-07-03T08:24:56.120069Z	info	FLAG: --log_rotate=""
2020-07-03T08:24:56.120081Z	info	FLAG: --log_rotate_max_age="30"
2020-07-03T08:24:56.120103Z	info	FLAG: --log_rotate_max_backups="1000"
2020-07-03T08:24:56.120111Z	info	FLAG: --log_rotate_max_size="104857600"
2020-07-03T08:24:56.120117Z	info	FLAG: --log_stacktrace_level="default:none"
2020-07-03T08:24:56.120130Z	info	FLAG: --log_target="[stdout]"
2020-07-03T08:24:56.120146Z	info	FLAG: --mcpInitialConnWindowSize="1048576"
2020-07-03T08:24:56.120152Z	info	FLAG: --mcpInitialWindowSize="1048576"
2020-07-03T08:24:56.120160Z	info	FLAG: --mcpMaxMsgSize="4194304"
2020-07-03T08:24:56.120166Z	info	FLAG: --meshConfig="/etc/istio/config/mesh"
2020-07-03T08:24:56.120172Z	info	FLAG: --monitoringAddr=":15014"
2020-07-03T08:24:56.120183Z	info	FLAG: --namespace="istio-system"
2020-07-03T08:24:56.120187Z	info	FLAG: --networksConfig="/etc/istio/config/meshNetworks"
2020-07-03T08:24:56.120199Z	info	FLAG: --plugins="[authn,authz,health,mixer]"
2020-07-03T08:24:56.120215Z	info	FLAG: --profile="true"
2020-07-03T08:24:56.120229Z	info	FLAG: --registries="[Kubernetes]"
2020-07-03T08:24:56.120244Z	info	FLAG: --resync="1m0s"
2020-07-03T08:24:56.120251Z	info	FLAG: --tlsCertFile=""
2020-07-03T08:24:56.120257Z	info	FLAG: --tlsKeyFile=""
2020-07-03T08:24:56.120263Z	info	FLAG: --trust-domain="cluster.local"
2020-07-03T08:24:56.448933Z	info	mesh configuration: {
    "disableMixerHttpReports": true,
    "proxyListenPort": 15001,
    "connectTimeout": "10s",
    "protocolDetectionTimeout": "0.100s",
    "ingressClass": "istio",
    "ingressService": "istio-ingressgateway",
    "ingressControllerMode": "STRICT",
    "enableTracing": true,
    "accessLogFile": "/dev/stdout",
    "defaultConfig": {
        "configPath": "./etc/istio/proxy",
        "binaryPath": "/usr/local/bin/envoy",
        "serviceCluster": "istio-proxy",
        "drainDuration": "45s",
        "parentShutdownDuration": "60s",
        "discoveryAddress": "istiod.istio-system.svc:15012",
        "proxyAdminPort": 15000,
        "statNameLength": 189,
        "concurrency": 2,
        "tracing": {
            "zipkin": {
                "address": "zipkin.istio-system:9411"
            }
        },
        "envoyAccessLogService": {

        },
        "envoyMetricsService": {

        },
        "proxyMetadata": {
            "DNS_AGENT": ""
        },
        "statusPort": 15020
    },
    "outboundTrafficPolicy": {
        "mode": "ALLOW_ANY"
    },
    "sdsUdsPath": "unix:/etc/istio/proxy/SDS",
    "enableAutoMtls": true,
    "trustDomain": "cluster.local",
    "trustDomainAliases": [
    ],
    "defaultServiceExportTo": [
        "*"
    ],
    "defaultVirtualServiceExportTo": [
        "*"
    ],
    "defaultDestinationRuleExportTo": [
        "*"
    ],
    "rootNamespace": "istio-system",
    "localityLbSetting": {
        "enabled": true
    },
    "dnsRefreshRate": "5s",
    "reportBatchMaxEntries": 100,
    "reportBatchMaxTime": "1s",
    "certificates": [
    ],
    "thriftConfig": {

    },
    "serviceSettings": [
    ],
    "enablePrometheusMerge": false
}

2020-07-03T08:33:39.800484Z	warn	ads	ADS:LDS: ACK ERROR router~10.42.1.154~istio-ingressgateway-66cc54b468-62n25.istio-system~istio-system.svc.cluster.local-6 Internal:Error adding/updating listener(s) 0.0.0.0_8080: Proto constraint validation failed (JwtAuthenticationValidationError.Providers[key]: ["embedded message failed validation"] | caused by JwtProviderValidationError.LocalJwks: ["embedded message failed validation"] | caused by DataSourceValidationError.InlineString: ["value length must be at least " '\x01' " bytes"]): providers {
  key: "origins-0"
  value {
    issuer: "https://www.h.net:30000/auth/realms/hproject"
    local_jwks {
      inline_string: ""
    }
    payload_in_metadata: "https://www.h.net:30000/auth/realms/hproject"
  }
}
rules {
  match {
    prefix: "/"
  }
  requires {
    requires_any {
      requirements {
        provider_name: "origins-0"
      }
      requirements {
        allow_missing {
        }
      }
    }
  }
}

0.0.0.0_8443: Proto constraint validation failed (JwtAuthenticationValidationError.Providers[key]: ["embedded message failed validation"] | caused by JwtProviderValidationError.LocalJwks: ["embedded message failed validation"] | caused by DataSourceValidationError.InlineString: ["value length must be at least " '\x01' " bytes"]): providers {
  key: "origins-0"
  value {
    issuer: "https://www.h.net:30000/auth/realms/hproject"
    local_jwks {
      inline_string: ""
    }
    payload_in_metadata: "https://www.h.net:30000/auth/realms/hproject"
  }
}
rules {
  match {
    prefix: "/"
  }
  requires {
    requires_any {
      requirements {
        provider_name: "origins-0"
      }
      requirements {
        allow_missing {
        }
      }
    }
  }
}


2020-07-03T08:33:42.967776Z	info	ads	Push Status: {}
2020-07-03T08:33:48.497008Z	info	grpc: Server.Serve failed to complete security handshake from "10.42.1.153:39768": EOF

Maybe you have a mismatch version of proxies / control plane?
Maybe try:

wget https://raw.githubusercontent.com/istio/tools/release-1.6/bin/root-transition.sh chmod +x root-transition.sh
$ ./root-transition.sh check-version

Seems that they are all at 1.6.3 (the namespace with RequestAuthentication is hproject)

sysop@hdev:~/software$ ./root-transition.sh check-version
Checking namespace: istio-system
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Checking namespace: default
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Checking namespace: keycloak
Checking namespace: openebs
Checking namespace: kubernetes-dashboard
Checking namespace: kube-node-lease
Checking namespace: hproject
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Checking namespace: foo
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Checking namespace: bar
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Istio proxy version: 1.6.3
Checking namespace: legacy
sysop@hdev:~/software$ 

BTW the two urls used in RequestAuthenticatnion give

sysop@hdev:~/software/hproject$ curl --insecure https://www.h.net:30000/auth/realms/hproject
{"realm":"hproject","public_key":"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAmcQm7YJZFMWVuFinL6HDwHO9q2QGqZj+TFLLWsdDmyr8eowBQwA7SZT1pTFsqdLqv6QTGJvYaxoHMLHSahwc0hcbFFnu8dsiQ+Hi0xXQBqKU+/uH4CyNZq0aLI3c2BqhV8ntLE5NKJyxDbe6BGbQhr0Te3BG4SmqZtya0WLtQ1BC/Mi+1v+C7QqRo4yNIl6Csu4gfAe8CopEudM2tynuDkQVDKrcI3qQrqrvVoJsWOKv66rPy1QejLyOHrkF0fcKKSxmWQmdyo6rkGhwLPJ/bVnI+WpZjnhubCwAxd3HuJsTG7Inwq9pR/09BnOJc1t4rlch9OoHW33EvCfKHdCYuQIDAQAB","token-service":"https://www.h.net:30000/auth/realms/hproject/protocol/openid-connect","account-service":"https://www.h.net:30000/auth/realms/hproject/account","tokens-not-before":0}sysop@hdev:~/software/hproject$ curl --insecure https://www.h.net:30000/auth/realms/hproject|jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   610  100   610    0     0  12978      0 --:--:-- --:--:-- --:--:-- 12978
{
  "realm": "hproject",
  "public_key": "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAmcQm7YJZFMWVuFinL6HDwHO9q2QGqZj+TFLLWsdDmyr8eowBQwA7SZT1pTFsqdLqv6QTGJvYaxoHMLHSahwc0hcbFFnu8dsiQ+Hi0xXQBqKU+/uH4CyNZq0aLI3c2BqhV8ntLE5NKJyxDbe6BGbQhr0Te3BG4SmqZtya0WLtQ1BC/Mi+1v+C7QqRo4yNIl6Csu4gfAe8CopEudM2tynuDkQVDKrcI3qQrqrvVoJsWOKv66rPy1QejLyOHrkF0fcKKSxmWQmdyo6rkGhwLPJ/bVnI+WpZjnhubCwAxd3HuJsTG7Inwq9pR/09BnOJc1t4rlch9OoHW33EvCfKHdCYuQIDAQAB",
  "token-service": "https://www.h.net:30000/auth/realms/hproject/protocol/openid-connect",
  "account-service": "https://www.h.net:30000/auth/realms/hproject/account",
  "tokens-not-before": 0
}
sysop@hdev:~/software/hproject$ curl --insecure https://www.h.net:30000/auth/realms/hproject/protocol/openid-connect/certs|jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1466  100  1466    0     0  20647      0 --:--:-- --:--:-- --:--:-- 20647
{
  "keys": [
    {
      "kid": "ZeZlzvgsOIgdbVyz8RF3jisSpsvrYKFyGfFpqkIuLRE",
      "kty": "RSA",
      "alg": "RS256",
      "use": "sig",
      "n": "mcQm7YJZFMWVuFinL6HDwHO9q2QGqZj-TFLLWsdDmyr8eowBQwA7SZT1pTFsqdLqv6QTGJvYaxoHMLHSahwc0hcbFFnu8dsiQ-Hi0xXQBqKU-_uH4CyNZq0aLI3c2BqhV8ntLE5NKJyxDbe6BGbQhr0Te3BG4SmqZtya0WLtQ1BC_Mi-1v-C7QqRo4yNIl6Csu4gfAe8CopEudM2tynuDkQVDKrcI3qQrqrvVoJsWOKv66rPy1QejLyOHrkF0fcKKSxmWQmdyo6rkGhwLPJ_bVnI-WpZjnhubCwAxd3HuJsTG7Inwq9pR_09BnOJc1t4rlch9OoHW33EvCfKHdCYuQ",
      "e": "AQAB",
      "x5c": [
        "MIICnzCCAYcCBgFyxvWYmTANBgkqhkiG9w0BAQsFADATMREwDwYDVQQDDAhocHJvamVjdDAeFw0yMDA2MTgxMDIzMDNaFw0zMDA2MTgxMDI0NDNaMBMxETAPBgNVBAMMCGhwcm9qZWN0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAmcQm7YJZFMWVuFinL6HDwHO9q2QGqZj+TFLLWsdDmyr8eowBQwA7SZT1pTFsqdLqv6QTGJvYaxoHMLHSahwc0hcbFFnu8dsiQ+Hi0xXQBqKU+/uH4CyNZq0aLI3c2BqhV8ntLE5NKJyxDbe6BGbQhr0Te3BG4SmqZtya0WLtQ1BC/Mi+1v+C7QqRo4yNIl6Csu4gfAe8CopEudM2tynuDkQVDKrcI3qQrqrvVoJsWOKv66rPy1QejLyOHrkF0fcKKSxmWQmdyo6rkGhwLPJ/bVnI+WpZjnhubCwAxd3HuJsTG7Inwq9pR/09BnOJc1t4rlch9OoHW33EvCfKHdCYuQIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQBM0GufqiKBWyoCFlv51mGpMsSRQ2znPzpu31zcZT98kXaBaWDguBd6GbrpSHEoAFCx5sv9lihSsn2KoBy0nFR4lHVP2NYR8no3DzDqZPuOEcJpwlM+H3QzV2pT4fKyyMH+yf0VBaeLplMLYUwV+nsYYqD9ReXWCPbhoTPhemPi5ENTugHGCVjEfOtlBtfY0o+giumltFJoRTuYkGY8V4xtk3Gdjt19dY5DhjaSDZsCxHlpf99dSHKfmBkbMlsOrkI4M9cdAm0HXl2IFisDnNYQqGGLvPFmOMbQOUNFS+yAA9OiEBL6hlIaw2rDCuFvw8vpXs9y9w7i/LLSPBgsGAMk"
      ],
      "x5t": "Njkffzqf4gZ_yqXLkqegOScyocg",
      "x5t#S256": "H2jZbULZ7epMjGLLZVm6rjizdYPEc7D4X6TvfE8aRrU"
    }
  ]
}
sysop@hdev:~/software/hproject$ 

Maybe there is a problemm with self-signed certificates?

Confirmed.
This requestauthentication creates the error

apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
  name: "h-ingress-jwt"
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  jwtRules:
  - issuer: "https://192.168.202.21:30000/auth/realms/hproject"
    jwksUri: "https://192.168.202.21:30000/auth/realms/hproject/protocol/openid-connect/certs"

---

This one works correctly

apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
  name: "h-ingress-jwt"
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  jwtRules:
  - issuer: "testing@secure.istio.io"
    jwksUri: "https://raw.githubusercontent.com/istio/istio/release-1.6/security/tools/jwt/samples/jwks.json"

---

I’ll start a new thread about using keycloak for requestauthorization

Thanks for your time