I am running/trying to run istio using non-root for all containers.
When running v1.3 with global sds set to true the envoy process in the ingressgateway crashes continually. Setting the securityContext n the container to runAsUser: 0 stops the failure.
ingress-gateway log for istio-proxy
2019-11-23T09:27:50.717536Z info Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingressgateway --service-node router~172.16.3.68~istio-ingressgateway-6f858f5685-xp6mc.istio-system~istio-system.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --allow-unknown-fields -l warning --component-log-level misc:error]
[2019-11-23 09:27:50.730][23][warning][config] [external/envoy/source/server/options_impl.cc:193] --allow-unknown-fields is deprecated, use --allow-unknown-static-fields instead.
[2019-11-23 09:27:50.783][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, no healthy upstream
[2019-11-23 09:27:50.783][23][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:50] Unable to establish new stream
[2019-11-23 09:27:52.195][60][critical][main] [external/envoy/source/exe/terminate_handler.cc:13] std::terminate called! (possible uncaught exception, see trace)
[2019-11-23 09:27:52.195][60][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:69] Backtrace (use tools/stack_decode.py to get line numbers):
[2019-11-23 09:27:52.198][83][critical][main] [external/envoy/source/exe/terminate_handler.cc:13] std::terminate called! (possible uncaught exception, see trace)
[2019-11-23 09:27:52.198][83][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:69] Backtrace (use tools/stack_decode.py to get line numbers):
[2019-11-23 09:27:52.201][60][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #0: Envoy::TerminateHandler::logOnTerminate()::$_0::operator()() [0x1fe6ab8]
[2019-11-23 09:27:52.204][83][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #0: Envoy::TerminateHandler::logOnTerminate()::$_0::operator()() [0x1fe6ab8]
[2019-11-23 09:27:52.207][60][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #1: [0x1fe69c9]
[2019-11-23 09:27:52.209][83][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #1: [0x1fe69c9]
[2019-11-23 09:27:52.213][60][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #2: std::__terminate() [0x22077b3]
[2019-11-23 09:27:52.215][83][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #2: std::__terminate() [0x22077b3]
[2019-11-23 09:27:52.219][60][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #3: Envoy::Config::DataSource::read() [0x1b29259]
[2019-11-23 09:27:52.220][83][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #3: Envoy::Config::DataSource::read() [0x1b29259]
[2019-11-23 09:27:52.225][60][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #4: Envoy::Extensions::GrpcCredentials::FileBasedMetadata::FileBasedMetadataAuthenticator::GetMetadata() [0xcc6930]
[2019-11-23 09:27:52.226][83][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #4: Envoy::Extensions::GrpcCredentials::FileBasedMetadata::FileBasedMetadataAuthenticator::GetMetadata() [0xcc6930]
[2019-11-23 09:27:52.231][60][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #5: grpc::MetadataCredentialsPluginWrapper::InvokePlugin() [0x1b2c60d]
[2019-11-23 09:27:52.232][83][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #5: grpc::MetadataCredentialsPluginWrapper::InvokePlugin() [0x1b2c60d]
[2019-11-23 09:27:52.237][60][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #6: std::__1::__function::__func<>::operator()() [0x1b2d3ce]
[2019-11-23 09:27:52.237][83][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #6: std::__1::__function::__func<>::operator()() [0x1b2d3ce]
[2019-11-23 09:27:52.243][60][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #7: grpc::DynamicThreadPool::ThreadFunc() [0x1b338b1]
[2019-11-23 09:27:52.243][83][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #7: grpc::DynamicThreadPool::ThreadFunc() [0x1b338b1]
[2019-11-23 09:27:52.248][83][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #8: grpc::DynamicThreadPool::DynamicThread::ThreadFunc() [0x1b33605]
[2019-11-23 09:27:52.249][60][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #8: grpc::DynamicThreadPool::DynamicThread::ThreadFunc() [0x1b33605]
[2019-11-23 09:27:52.254][83][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #9: grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix()::{lambda()#1}::__invoke() [0x1bf40d3]
[2019-11-23 09:27:52.254][83][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #10: start_thread [0x7fc3071b96db]
[2019-11-23 09:27:52.254][83][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:81] Caught Aborted, suspect faulting address 0x271100000017
[2019-11-23 09:27:52.254][83][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:69] Backtrace (use tools/stack_decode.py to get line numbers):
[2019-11-23 09:27:52.254][83][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #0: __restore_rt [0x7fc3071c4890]
[2019-11-23 09:27:52.255][60][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #9: grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix()::{lambda()#1}::__invoke() [0x1bf40d3]
[2019-11-23 09:27:52.255][60][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #10: start_thread [0x7fc3071b96db]
2019-11-23T09:27:52.260552Z warn Epoch 0 terminated with an error: signal: aborted
2019-11-23T09:27:52.260574Z warn Aborted all epochs
2019-11-23T09:27:52.260729Z info Epoch 0: set retry delay to 200ms, budget to 9
If I use the upstream image for the proxy all is well (but it looks like all processes in that run as root) or setting a securityContext for the container with runAsUser: 0 also resolves.
I am using the envoy binary from the upstream deb (I had previously been building envoy form source - but same issue).
The target container was originally scratch bit to zone in on the problem I have switched to distrolesscc.
Same experience with both. The other envoys (sidecars to control plane pilot, mixer etc) do not show this issue.
Here’s my docker file:
ARG ISTIO_VERSION
FROM myartifactory.com/dev/istio-package/istiobuilder:${ISTIO_VERSION} as builder
USER root
ENV PROXY_BUILD=/home/build/proxybuild
ENV ARTIFACTS_LOG=/home/build/artifacts.txt
ENV PROXY_FOLDER=istio-proxy
RUN mkdir -p ${PROXY_BUILD}/etc/istio/proxy && \
mkdir -p ${PROXY_BUILD}/var/log && \
mkdir -p ${PROXY_BUILD}/var/run/sds && \
echo "root:x:0:0:root:/root:/sbin/nologin" > ${PROXY_BUILD}/etc/passwd && \
echo "nobody:x:65534:65534:nobody:/nonexistent:/sbin/nologin" >> ${PROXY_BUILD}/etc/passwd && \
echo "nonroot:x:65532:65532:nonroot:/home/nonroot:/sbin/nologin" >> ${PROXY_BUILD}/etc/passwd && \
echo "istio-user:10001:10001:istio-user:/var/lib/istio" >>${PROXY_BUILD}/etc/passwd && \
tar -zxf /home/build/go/src/istio.io/istio/docker/ca-certificates.tgz -C ${PROXY_BUILD} && \
chmod 644 ${PROXY_BUILD}/etc/ssl/certs/ca-certificates.crt && \
cp ${GOPATH}/src/istio.io/istio/pilot/docker/*.yaml.tmpl ${PROXY_BUILD}/etc/istio/proxy && \
tar -C ${PROXY_BUILD} -xzf /ca-certificates.tgz
COPY envoy ${PROXY_BUILD}/envoy
FROM gcr.io/distroless/cc:latest
ARG proxy_version
ARG istio_version
ENV PROXY_BUILD=/home/build/proxybuild
COPY --from=builder --chown=10001:10001 /ca-certificates.tgz /ca-certificates.tgz
COPY --from=builder ${PROXY_BUILD}/etc/passwd /etc/passwd
COPY --from=builder ${PROXY_BUILD}/etc/ssl /etc/ssl
COPY --from=builder --chown=10001:10001 ${PROXY_BUILD}/etc/istio /etc/istio
COPY --from=builder /home/build/go/out/linux_amd64/release/pilot-agent /usr/local/bin/pilot-agent
COPY --from=builder ${PROXY_BUILD}/envoy /usr/local/bin/envoy
COPY --from=builder /home/build/go/src/istio.io/istio/tools/packaging/common/istio-iptables.sh /usr/local/bin/istio-iptables.sh
COPY --from=builder --chown=10001:10001 ${PROXY_BUILD}/var/log /var/log
COPY --from=builder --chown=10001:10001 ${PROXY_BUILD}/var/run /var/run
COPY --from=builder --chown=10001:10001 /home/build/go/src/istio.io/istio/tools/packaging/common/envoy_bootstrap_drain.json /var/lib/istio/envoy/envoy_bootstrap_drain.json
COPY --from=builder --chown=10001:10001 /home/build/go/src/istio.io/istio/tools/packaging/common/envoy_bootstrap_v2.json /var/lib/istio/envoy/envoy_bootstrap_tmpl.json
COPY --from=builder --chown=10001:10001 /home/build/go/src/istio.io/istio/tools/packaging/common/sidecar.env /var/lib/istio/envoy/sidecar.env
COPY --from=builder --chown=10001:10001 /home/build/go/src/istio.io/istio/pilot/docker/envoy_pilot.yaml.tmpl /etc/istio/proxy/envoy_pilot.yaml.tmpl
COPY --from=builder --chown=10001:10001 /home/build/go/src/istio.io/istio/pilot/docker/envoy_telemetry.yaml.tmpl /etc/istio/proxy/envoy_telemetry.yaml.tmpl
COPY --from=builder --chown=10001:10001 /home/build/go/src/istio.io/istio/pilot/docker/envoy_policy.yaml.tmpl /etc/istio/proxy/envoy_policy.yaml.tmpl
USER istio-user
ENTRYPOINT ["/usr/local/bin/pilot-agent"]
I don’t want to run as user 0 especially on the ingressgateway