Istio Ingress Gateway: 503 Service Unavailable

1

Hi,

I am trying to debug an issue with our Istio setup, all our new services registered in the last 10-15 days are failing with

< HTTP/1.1 503 Service Unavailable
< Server: istio-envoy

I enabled debug on the Istio Ingress Gateway and for the services having issue i see below exceptions.

2019-12-03T00:30:15.025025415Z [2019-12-03 00:30:15.024][39][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:243] [C10747011] new stream
2019-12-03T00:30:15.025047873Z [2019-12-03 00:30:15.024][39][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:580] [C10747011][S1582204866555343596] request headers complete (end_stream=true):
2019-12-03T00:30:15.025052394Z ‘:authority’, ‘xxxxxxx’
2019-12-03T00:30:15.025055972Z ‘:path’, ‘/oalapp/services/b2b/massorder-service/api/massorder/testService’
2019-12-03T00:30:15.025060015Z ‘:method’, ‘GET’
…
…
2019-12-03T00:30:15.025133034Z [2019-12-03 00:30:15.024][39][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1040] [C10747011][S1582204866555343596] request end stream
2019-12-03T00:30:15.025139573Z [2019-12-03 00:30:15.025][39][debug][filter] [src/envoy/http/mixer/filter.cc:39] Called Mixer::Filter : Filter
2019-12-03T00:30:15.02514303Z [2019-12-03 00:30:15.025][39][debug][filter] [src/envoy/http/mixer/filter.cc:146] Called Mixer::Filter : setDecoderFilterCallbacks
2019-12-03T00:30:15.025146415Z [2019-12-03 00:30:15.025][39][debug][filter] [src/envoy/http/mixer/filter.cc:62] Called Mixer::Filter : decodeHeaders
2019-12-03T00:30:15.025233027Z [2019-12-03 00:30:15.025][39][debug][filter] [./src/envoy/utils/header_update.h:46] Mixer forward attributes set: CjcKGGRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZRIbEhlvYWwtYjJiLW1hc3NvcmRlci1zZXJ2aWNlCjQKHWRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZXNwYWNlEhMSEW9hbGIyYi1wb3JkLWd4cGV0Ck8KCnNvdXJjZS51aWQSQRI/a3ViZXJuZXRlczovL2lzdGlvLWluZ3Jlc3NnYXRld2F5LTdkYjRiNmM4ZjQtbWdjbWQuaXN0aW8tc3lzdGVtClsKGGRlc3RpbmF0aW9uLnNlcnZpY2UuaG9zdBI/Ej1vYWwtYjJiLW1hc3NvcmRlci1zZXJ2aWNlLm9hbGIyYi1wb3JkLWd4cGV0LnN2Yy5jbHVzdGVyLmxvY2FsClkKF2Rlc3RpbmF0aW9uLnNlcnZpY2UudWlkEj4SPGlzdGlvOi8vb2FsYjJiLXBvcmQtZ3hwZXQvc2VydmljZXMvb2FsLWIyYi1tYXNzb3JkZXItc2VydmljZQ==
2019-12-03T00:30:15.025260156Z [2019-12-03 00:30:15.025][39][debug][filter] [src/envoy/http/mixer/filter.cc:154] Called Mixer::Filter : check complete OK
2019-12-03T00:30:15.025264391Z [2019-12-03 00:30:15.025][39][debug][router] [external/envoy/source/common/router/router.cc:308] [C10747011][S1582204866555343596] unknown cluster 'outbound|9000||oal-b2b-massorder-service.oalb2b-pord-gxpet.svc.cluster.local’
2019-12-03T00:30:15.025268151Z [2019-12-03 00:30:15.025][39][debug][filter] [src/envoy/http/mixer/filter.cc:133] Called Mixer::Filter : encodeHeaders 2
2019-12-03T00:30:15.025287685Z [2019-12-03 00:30:15.025][39][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1305] [C10747011][S1582204866555343596] encoding headers via codec (end_stream=true):
2019-12-03T00:30:15.025299023Z ‘:status’, '503’
2019-12-03T00:30:15.025303916Z ‘date’, ‘Tue, 03 Dec 2019 00:30:14 GMT’
2019-12-03T00:30:15.025308634Z ‘server’, 'istio-envoy’
*2019-12-03T00:30:15.025313832Z *
2019-12-03T00:30:15.02531835Z [2019-12-03 00:30:15.025][39][debug][filter] [src/envoy/http/mixer/filter.cc:205] Called Mixer::Filter : onDestroy state: 2
2019-12-03T00:30:15.02537308Z [2019-12-03 00:30:15.025][39][debug][filter] [src/envoy/http/mixer/filter.cc:219] Called Mixer::Filter : log
2019-12-03T00:30:15.025379327Z [2019-12-03 00:30:15.025][39][debug][filter] [./src/envoy/http/mixer/report_data.h:132] No dynamic_metadata found for filter envoy.filters.http.rbac
2019-12-03T00:30:15.025964245Z [2019-12-03 00:30:15.025][39][debug][filter] [src/istio/control/client_context_base.cc:125] Report attributes: attributes {
2019-12-03T00:30:15.025986818Z key: “connection.mtls”
2019-12-03T00:30:15.025991427Z value {
2019-12-03T00:30:15.025994837Z bool_value: false
2019-12-03T00:30:15.025998319Z }
2019-12-03T00:30:15.026001678Z }
2019-12-03T00:30:15.026005091Z attributes {
2019-12-03T00:30:15.0260083Z key: “context.protocol”
2019-12-03T00:30:15.026011981Z value {
2019-12-03T00:30:15.026015105Z string_value: “http”
2019-12-03T00:30:15.026018404Z }
2019-12-03T00:30:15.026021454Z }
2019-12-03T00:30:15.026024526Z attributes {
2019-12-03T00:30:15.026027658Z key: "context.prox

Can someone kindly help here, what does the above message highlighted in bold indicate ?

Regards
Subhankar

2

Hi,

Tried to debug this further by doing a config_dump and between working and non-working services I can see below difference.

For non-working service, below section is missing.

cluster: {

  • name: “outbound_.9000_._.oal-b2b-commons-service.oalb2b-pord-gxpet.svc.cluster.local”,
  • type: “EDS”,
  • eds_cluster_config: {
    • eds_config: {
      • ads: { }},
    • service_name: “outbound_.9000_._.oal-b2b-commons-service.oalb2b-pord-gxpet.svc.cluster.local”},
  • connect_timeout: “10s”,
  • circuit_breakers: {
    • thresholds: [
      • {
        • max_retries: 1024}]}
          }

Any idea how this can be fixed on the Istio Ingress Gateway?

3

This should help you out. :slight_smile:

4

Hi Mike,

I don’t think the service definition has an issue. I ran the same service against a different Ingress Gateway (custom ingress on same cluster) and it works fine.

I wanted to check if there is a way the IngressGateway configuration can be modified, the registration of the service (cluster) in it isn’t correct.

apiVersion: v1
kind: Service
metadata:
name: oal-b2b-massorder-service
namespace: oalb2b-pord-gxpet
spec:
ports:

  • port: 9000
    protocol: TCP
    targetPort: 9000
    selector:
    app: oal-massorder-service
    sessionAffinity: None
    type: ClusterIP