Issues faced while installing istio

I work on a big data technology called HPCC which uses linux nodes to do parallel processing. The product in a whole will involve a number of nodes(back end, middle ware, front end etc). I was checking out istio for all the inter node communication required. Once i install istio using the basic steps provided(profile demo) my cluster deployed on AKS starts off on a crashloopbackoff spree. Should i make changes any where in my cluster or istio config ? i presumed that demo config will not have any rules applied so it will atleast start up. Please let me know if i am missing some settings or config.
I have installed a basic version of hpcc https://hpccsystems.com/blog/default-azure-setup on AKS and before installing i am enabling the istio side car injection. Should i do some config changes before going ahead?
Thanks,
Manish

Hi ,
I am still unable to resolve the continuous crash loop back off error i am facing. I am pasting my logs here. Still unable to see the reasons on why it is happening. Can any one help ?

Environment:

ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_MARK=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=

Variables:

PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15090,15021,15020
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_EXCLUDE=
KUBEVIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false

Writing following contents to rules file: /tmp/iptables-rules-1595484007872258736.txt178330292

  • nat
    -N ISTIO_REDIRECT
    -N ISTIO_IN_REDIRECT
    -N ISTIO_INBOUND
    -N ISTIO_OUTPUT
    -A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
    -A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
    -A PREROUTING -p tcp -j ISTIO_INBOUND
    -A ISTIO_INBOUND -p tcp --dport 22 -j RETURN
    -A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN
    -A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN
    -A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
    -A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
    -A OUTPUT -p tcp -j ISTIO_OUTPUT
    -A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
    -A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
    -A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
    -A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
    -A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
    -A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
    -A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
    -A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
    -A ISTIO_OUTPUT -j ISTIO_REDIRECT
    COMMIT

iptables-restore --noflush /tmp/iptables-rules-1595484007872258736.txt178330292
Writing following contents to rules file: /tmp/ip6tables-rules-1595484008150281616.txt002967683

ip6tables-restore --noflush /tmp/ip6tables-rules-1595484008150281616.txt002967683
iptables-save

Generated by iptables-save v1.6.1 on Thu Jul 23 06:00:08 2020

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT

Completed on Thu Jul 23 06:00:08 2020

00000000 PRG 2020-07-23 07:59:14.425 1 1 “Using ESP configuration section [Software/EspProcess[1]]”
00000001 PRG 2020-07-23 07:59:14.426 1 1 “Esp starting community_7.10.6-1Debug[community_7.10.6-1]”
00000002 PRG 2020-07-23 07:59:14.426 1 1 “componentfiles are under /opt/HPCCSystems/componentfiles/”
00000003 OPR 2020-07-23 07:59:14.445 1 1 “WARNING: Warning: Could not load /opt/HPCCSystems/filehooks/libarchivefile.so: libarchive.so.13: cannot open shared object file: No such file or directory”
00000004 PRG 2020-07-23 07:59:14.445 1 1 “File hook library /opt/HPCCSystems/filehooks/libarchivefile.so could not be loaded”
00000005 PRG 2020-07-23 07:59:14.452 1 1 “ESP process name [eclwatch]”
00000006 PRG 2020-07-23 07:59:14.452 1 1 “Could not load DALI Attach state file [/var/lib/HPCCSystems/eclwatch/eclwatch-AttachState.xml] forESP process [eclwatch]”
00000007 PRG 2020-07-23 07:59:14.452 1 1 “ESP Process [eclwatch] State to be stored: '<ESPAttachState StateSaveTimems=‘9797091’ attached=‘1’subscribed=‘1’/>’”
00000008 PRG 2020-07-23 07:59:14.452 1 1 “Could not load DALI Attach state file [/var/lib/HPCCSystems/eclwatch/eclwatch-AttachState.xml] forESP process [eclwatch]”
00000009 PRG 2020-07-23 07:59:14.452 1 1 “Initializing DALI client [servers = mydali]”
2020-07-23T06:00:25.772895Z info FLAG: --concurrency=“2”
2020-07-23T06:00:25.772995Z info FLAG: --disableInternalTelemetry=“false”
2020-07-23T06:00:25.773010Z info FLAG: --domain=“default.svc.cluster.local”
2020-07-23T06:00:25.773017Z info FLAG: --help=“false”
2020-07-23T06:00:25.773023Z info FLAG: --id=""
2020-07-23T06:00:25.773028Z info FLAG: --ip=""
2020-07-23T06:00:25.773033Z info FLAG: --log_as_json=“false”
2020-07-23T06:00:25.773097Z info FLAG: --log_caller=""
2020-07-23T06:00:25.773105Z info FLAG: --log_output_level=“default:info”
2020-07-23T06:00:25.773110Z info FLAG: --log_rotate=""
2020-07-23T06:00:25.773115Z info FLAG: --log_rotate_max_age=“30”
2020-07-23T06:00:25.773121Z info FLAG: --log_rotate_max_backups=“1000”
2020-07-23T06:00:25.773127Z info FLAG: --log_rotate_max_size=“104857600”
2020-07-23T06:00:25.773189Z info FLAG: --log_stacktrace_level=“default:none”
2020-07-23T06:00:25.773202Z info FLAG: --log_target="[stdout]"
2020-07-23T06:00:25.773208Z info FLAG: --meshConfig="./etc/istio/config/mesh"
2020-07-23T06:00:25.773213Z info FLAG: --mixerIdentity=""
2020-07-23T06:00:25.773218Z info FLAG: --outlierLogPath=""
2020-07-23T06:00:25.773222Z info FLAG: --pilotIdentity=""
2020-07-23T06:00:25.773227Z info FLAG: --proxyComponentLogLevel=“misc:error”
2020-07-23T06:00:25.773269Z info FLAG: --proxyLogLevel=“warning”
2020-07-23T06:00:25.773328Z info FLAG: --serviceCluster=“eclwatch.default”
2020-07-23T06:00:25.773339Z info FLAG: --serviceregistry=“Kubernetes”
2020-07-23T06:00:25.773344Z info FLAG: --stsPort=“0”
2020-07-23T06:00:25.773402Z info FLAG: --templateFile=""
2020-07-23T06:00:25.773413Z info FLAG: --tokenManagerPlugin=“GoogleTokenExchange”
2020-07-23T06:00:25.773418Z info FLAG: --trust-domain=“cluster.local”
2020-07-23T06:00:25.773455Z info Version 1.6.5-f508fdd78eb0d3444e2bc2b3f36966d904c5db52-dirty-Modified
2020-07-23T06:00:25.809998Z info Obtained private IP [10.244.0.21]
2020-07-23T06:00:25.810132Z info Apply proxy config from env {“proxyMetadata”:{“DNS_AGENT”:""}}

2020-07-23T06:00:25.939284Z info Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
envoyAccessLogService: {}
envoyMetricsService: {}
parentShutdownDuration: 60s
proxyAdminPort: 15000
proxyMetadata:
DNS_AGENT: “”
serviceCluster: eclwatch.default
statNameLength: 189
statusPort: 15020
tracing:
zipkin:
address: zipkin.istio-system:9411

2020-07-23T06:00:25.939469Z info Proxy role: &model.Proxy{ClusterID:"", Type:“sidecar”, IPAddresses:string{“10.244.0.21”}, ID:“eclwatch-646f55748c-ttvqp.default”, Locality:(*envoy_api_v2_core.Locality)(nil), DNSDomain:“default.svc.cluster.local”, ConfigNamespace:"", Metadata:(*model.NodeMetadata)(nil), SidecarScope:(*model.SidecarScope)(nil), PrevSidecarScope:(*model.SidecarScope)(nil), MergedGateway:(*model.MergedGateway)(nil), ServiceInstances:*model.ServiceInstance(nil), IstioVersion:(*model.IstioVersion)(nil), ipv6Support:false, ipv4Support:false, GlobalUnicastIP:"", XdsResourceGenerator:model.XdsResourceGenerator(nil), Active:map[string]*model.WatchedResource(nil)}
2020-07-23T06:00:25.939488Z info JWT policy is third-party-jwt
2020-07-23T06:00:25.939577Z info PilotSAN string{“istiod.istio-system.svc”}
2020-07-23T06:00:25.939604Z info MixerSAN string{“spiffe://cluster.local/ns/istio-system/sa/istio-mixer-service-account”}
2020-07-23T06:00:26.518823Z info serverOptions.CAEndpoint == istiod.istio-system.svc:15012
2020-07-23T06:00:26.518890Z info Using user-configured CA istiod.istio-system.svc:15012
2020-07-23T06:00:26.518899Z info istiod uses self-issued certificate
2020-07-23T06:00:26.519076Z info the CA cert of istiod is: -----BEGIN CERTIFICATE-----
MIIC3jCCAcagAwIBAgIRAPbXVSnRWhVdeKyKW3FwKKQwDQYJKoZIhvcNAQELBQAw
GDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDAeFw0yMDA3MjMwNTI1MzVaFw0zMDA3
MjEwNTI1MzVaMBgxFjAUBgNVBAoTDWNsdXN0ZXIubG9jYWwwggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQDwzhBb0SBmQ18Fi4NLw6FG/gC4MJI52h6LPeXn
MVG403s84LUapkMbtmqgeen5qREahTYvCQJ6VX96JTbIdoG6OaEq0oTmt4pnkBVt
Ga5RFK81TGN2Xs0TbfjeHAVe+bT6KOl0Od9JVTSh5xWdXheCYwHZcTSaVLmrvcCt
UOU4hu+CUEKNCgXkitgg3Mze1W2wzoJqSiolSGKSOCcIFAsI8GwlmcXJ/KT4bqG6
5IBXO5eBQLv534EvIOpIgFcH3w5zscdM5p9Z2xAqRpEyNe+UAwZ1PrbX7i9yjEbC
zS1077RZIH5+sygna1iTZREQWRjtR1eGtmMtYEfwyIp0f89pAgMBAAGjIzAhMA4G
A1UdDwEB/wQEAwICBDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IB
AQDbFRehQnjmUsindG0G1wo8IGVSn8VjqZKONw98jP8txmx82MPs2bv81Zq0jqBz
9vHV8axo/CF0wpJg6J7bAk8xL9fL0jO4rOFoTBDyxO6QDyqEBtOqInEKiNPkLeaq
kBUYKwJVk4dYtrTNs1kwKUZcFdUY0FiXH0eam3xqe3SWy/cS4G0oT4CRnwCASqlX
XnuYkVFCPxJ/06wMl/JKxk3v+kcSU+L2y6jEx/hOwMAkx9RWHZuJlVZW5JrZQ1wa
oX0mcXDqziV9X/dhYLH311ITwqqF8vyUcQSNxWQJdON3ykN47MO/AftFBCY7tnHf
8PoA7kDNynJ4Qv8UiggLWXaL
-----END CERTIFICATE-----

2020-07-23T06:00:26.519510Z info parsed scheme: “”
2020-07-23T06:00:26.519526Z info scheme “” not registered, fallback to default scheme
2020-07-23T06:00:26.519655Z info ccResolverWrapper: sending update to cc: {[{istiod.istio-system.svc:15012 0 }] }
2020-07-23T06:00:26.519715Z info ClientConn switching balancer to “pick_first”
2020-07-23T06:00:26.519724Z info Channel switches to new LB policy “pick_first”
2020-07-23T06:00:26.522360Z info Subchannel Connectivity change to CONNECTING
2020-07-23T06:00:26.522564Z info Subchannel picks a new address “istiod.istio-system.svc:15012” to connect
2020-07-23T06:00:26.523785Z info pickfirstBalancer: HandleSubConnStateChange: 0xc0008893d0, {CONNECTING }
2020-07-23T06:00:26.533839Z info Channel Connectivity change to CONNECTING
2020-07-23T06:00:26.540553Z info sds Start SDS grpc server
2020-07-23T06:00:26.540475Z info sds SDS gRPC server for workload UDS starts, listening on “./etc/istio/proxy/SDS”

2020-07-23T06:00:26.540875Z info Opening status port 15020

2020-07-23T06:00:26.545524Z info Received new config, creating new Envoy epoch 0
2020-07-23T06:00:26.545609Z info Epoch 0 starting
2020-07-23T06:00:26.549554Z info Starting proxy agent
2020-07-23T06:00:26.618544Z info Subchannel Connectivity change to READY
2020-07-23T06:00:26.618647Z info pickfirstBalancer: HandleSubConnStateChange: 0xc0008893d0, {READY }
2020-07-23T06:00:26.618664Z info Channel Connectivity change to READY
2020-07-23T06:00:26.623557Z info Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster eclwatch.default --service-node sidecar~10.244.0.21~eclwatch-646f55748c-ttvqp.default~default.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --log-format %Y-%m-%dT%T.%fZ %l envoy %n %v -l warning --component-log-level misc:error --concurrency 2]
2020-07-23T06:00:27.954875Z warning envoy config [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-07-23T06:00:27.955018Z warning envoy config [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2020-07-23T06:00:28.100444Z warning envoy main [external/envoy/source/server/server.cc:475] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
2020-07-23T06:00:28.193612Z info sds resource:default new connection
2020-07-23T06:00:28.221984Z info sds Skipping waiting for ingress gateway secret
2020-07-23T06:00:28.295049Z warning envoy config [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-07-23T06:00:28.295200Z warning envoy config [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2020-07-23T06:00:29.740990Z info cache Root cert has changed, start rotating root cert for SDS clients
2020-07-23T06:00:29.741099Z info cache GenerateSecret default
2020-07-23T06:00:29.741376Z info sds resource:default pushed key/cert pair to proxy
2020-07-23T06:00:33.126652Z info sds resource:ROOTCA new connection
2020-07-23T06:00:33.126868Z info sds Skipping waiting for ingress gateway secret
2020-07-23T06:00:33.126898Z info cache Loaded root cert from certificate ROOTCA
2020-07-23T06:00:33.126990Z info sds resource:ROOTCA pushed root cert to proxy
2020-07-23T06:00:35.128841Z warning envoy filter [src/envoy/http/authn/http_filter_factory.cc:83] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2020-07-23T06:00:35.136702Z warning envoy filter [src/envoy/http/authn/http_filter_factory.cc:83] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2020-07-23T06:00:36.299463Z info Envoy proxy is ready
[2020-07-23T06:00:45.648Z] “- - -” 0 UF,URX “-” “-” 0 0 42 - “-” “-” “-” “-” “127.0.0.1:8888” inbound|8010||eclwatch.default.svc.cluster.local - 10.244.0.21:8888 10.244.0.1:49883 - -
[2020-07-23T06:02:18.781Z] “- - -” 0 UF,URX “-” “-” 0 0 0 - “-” “-” “-” “-” “127.0.0.1:8888” inbound|8010||eclwatch.default.svc.cluster.local - 10.244.0.21:8888 10.244.0.1:50994 - -
[2020-07-23T06:03:42.903Z] “- - -” 0 UF,URX “-” “-” 0 0 0 - “-” “-” “-” “-” “127.0.0.1:8888” inbound|8010||eclwatch.default.svc.cluster.local - 10.244.0.21:8888 10.244.0.1:51897 - -
2020-07-23T06:32:34.887810Z info Subchannel Connectivity change to CONNECTING
2020-07-23T06:32:34.888507Z info Subchannel picks a new address “istiod.istio-system.svc:15012” to connect
2020-07-23T06:32:34.888291Z info transport: loopyWriter.run returning. connection error: desc = “transport is closing”
2020-07-23T06:32:34.889051Z info pickfirstBalancer: HandleSubConnStateChange: 0xc0008893d0, {CONNECTING }
2020-07-23T06:32:34.889588Z info Channel Connectivity change to CONNECTING
2020-07-23T06:32:34.899958Z info Subchannel Connectivity change to READY
2020-07-23T06:32:34.900121Z info pickfirstBalancer: HandleSubConnStateChange: 0xc0008893d0, {READY }
2020-07-23T06:32:34.900135Z info Channel Connectivity change to READY
2020-07-23T06:32:35.335938Z warning envoy config [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 13,
2020-07-23T07:01:20.945426Z warning envoy config [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 13,
2020-07-23T07:02:10.695310Z info Subchannel Connectivity change to CONNECTING
2020-07-23T07:02:10.695409Z info transport: loopyWriter.run returning. connection error: desc = “transport is closing”
2020-07-23T07:02:10.695512Z info pickfirstBalancer: HandleSubConnStateChange: 0xc0008893d0, {CONNECTING }
2020-07-23T07:02:10.695528Z info Channel Connectivity change to CONNECTING
2020-07-23T07:02:10.695528Z info Subchannel picks a new address “istiod.istio-system.svc:15012” to connect
2020-07-23T07:02:10.708220Z info Subchannel Connectivity change to READY
2020-07-23T07:02:10.708322Z info pickfirstBalancer: HandleSubConnStateChange: 0xc0008893d0, {READY }
2020-07-23T07:02:10.708337Z info Channel Connectivity change to READY
2020-07-23T07:33:34.884479Z info Subchannel Connectivity change to CONNECTING
2020-07-23T07:33:34.884580Z info transport: loopyWriter.run returning. connection error: desc = “transport is closing”
2020-07-23T07:33:34.884655Z info pickfirstBalancer: HandleSubConnStateChange: 0xc0008893d0, {CONNECTING }
2020-07-23T07:33:34.884668Z info Channel Connectivity change to CONNECTING
2020-07-23T07:33:34.884691Z info Subchannel picks a new address “istiod.istio-system.svc:15012” to connect
2020-07-23T07:33:34.893130Z info Subchannel Connectivity change to READY
2020-07-23T07:33:34.893181Z info pickfirstBalancer: HandleSubConnStateChange: 0xc0008893d0, {READY }
2020-07-23T07:33:34.893213Z info Channel Connectivity change to READY
2020-07-23T07:33:39.907092Z warning envoy config [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 13,
[2020-07-23T07:44:03.334Z] “- - -” 0 UF,URX “-” “-” 0 0 0 - “-” “-” “-” “-” “127.0.0.1:8888” inbound|8010||eclwatch.default.svc.cluster.local - 10.244.0.21:8888 10.244.0.1:54656 - -