Cannot get bookinfo and httpbin ingressgateway to work on new install 1.2.*

Installed istio on a bare-metal cluster. Followed setup guide (tried all of the install types): https://istio.io/docs/setup/kubernetes/install/helm/#option-2-install-with-helm-and-tiller-via-helm-install

We use metallb for LoadBalancer service types (i saw other users reported that things work well for them with metallb), so istio-ingressgateway is exposed via LoadBalancer service type.

Steps taken:

  • install using helm with global k8sIngress and gateway options enabled egress+ingress. all successful:
    kubectl -n istio-system get po,svc | grep -e NAME -e ingress

    NAME READY STATUS RESTARTS AGE
    pod/istio-ingressgateway-545dd78c-487kb 1/1 Running 0 16h
    pod/istio-ingressgateway-545dd78c-579wn 1/1 Running 0 16h
    pod/istio-ingressgateway-545dd78c-5n54c 1/1 Running 0 16h
    pod/istio-ingressgateway-545dd78c-7bm7z 1/1 Running 0 16h
    pod/istio-ingressgateway-545dd78c-qxgml 1/1 Running 0 16h

    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    service/istio-ingressgateway LoadBalancer 10.43.54.158 10.255.42.73 15020:31648/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:30676/TCP,15030:31322/TCP,15031:31251/TCP,15032:32599/TCP,15443:30342/TCP 17h

  • installed httpbin - all installed correctly:
    kubectl get po,ep,svc | grep productpage
    pod/productpage-v1-7868c48878-nmfqr 2/2 Running 0 16h
    endpoints/productpage 10.42.4.140:9080 5d21h
    service/productpage ClusterIP 10.43.50.42 9080/TCP 5d21h

  • can access httpbin service inside cluster successfully:
    kubectl run -ti --rm --restart=Never dummy --image=tutum/curl:alpine sh
    curl httpbin:8000/headers
    {
    “headers”: {
    “Accept”: “/”,
    “Content-Length”: “0”,
    “Host”: “httpbin:8000”,
    “User-Agent”: “curl/7.54.0”,
    “X-B3-Parentspanid”: “1c9071e907d379e1”,
    “X-B3-Sampled”: “0”,
    “X-B3-Spanid”: “619666bc9fddfda2”,
    “X-B3-Traceid”: “43e4ea231eb3e7801c9071e907d379e1”,
    “X-Forwarded-Client-Cert”: “By=spiffe://cluster.local/ns/default/sa/default;Hash=a8b8a2125289bd547d68d64d516ddb15e50ec7686e888c114c3097919c86d7ef;Subject=”";URI=spiffe://cluster.local/ns/default/sa/default"
    }
    }

  • applied manifests with VirtualService and Gateway - everything successfully created but cannot access from outside the cluster:
    trying to access via ingress:
    curl -I -HHost:httpbin.example.com http://10.255.42.73:80/headers
    HTTP/1.1 404 Not Found
    location: http…://httpbin.example.com/headers
    date: Thu, 18 Jul 2019 14:07:00 GMT
    server: istio-envoy
    transfer-encoding: chunked

apparently i’m hitting istio-envoy, but the request doesn’t travel past that point.

Any idea what I’m missing or how to troubleshoot further?

Show us the yaml for gateway and virtual service. Make sure virtual service references gateway with correct name spacing.

copy/pasted from here https://istio.io/docs/tasks/traffic-management/ingress/ingress-control/#configuring-ingress-using-an-istio-gateway

Gateway:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin-gateway
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "httpbin.example.com"

VirtualService:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
  - "httpbin.example.com"
  gateways:
  - httpbin-gateway
  http:
  - match:
    - uri:
        prefix: /status
    - uri:
        prefix: /delay
    route:
    - destination:
        port:
          number: 8000
        host: httpbin

re-installed (wiped clean, then installed) istio and the namespace istio-system several times. same result
testing httpbin in “default” namespace

after 5 days of re-install, i am now getting 503 error and a completely different behavior, played with values.yaml install options, here’s a copy of it as well:

global:

  k8sIngress:
    enabled: true
    enableHttps: false
    gatewayName: ingressgateway

  controlPlaneSecurityEnabled: false

  mtls:
    # Default setting for service-to-service mtls. Can be set explicitly using
    # destination rules or service annotations.
    enabled: false

  proxy:
    accessLogFile: "/dev/stdout"
    resources:
      requests:
        cpu: 10m
        memory: 40Mi

  disablePolicyChecks: false

sidecarInjectorWebhook:
  enabled: true
  # If true, webhook or istioctl injector will rewrite PodSpec for liveness
  # health check to redirect request to sidecar. This makes liveness check work
  # even when mTLS is enabled.
  rewriteAppHTTPProbe: true

certmanager:
  enabled: true
  email: alex@****.com
  extraArgs: ['--default-issuer-name=letsencrypt', '--default-issuer-kind=ClusterIssuer']

prometheus:
  ingress:
    enabled: true
    hosts:
      - cluster.app.mycluster.com

pilot:
  traceSampling: 20.0
  resources:
    requests:
      cpu: 10m
      memory: 100Mi
#    limits:
#      cpu: 100m
#      memory: 200Mi

mixer:
  policy:
    enabled: true
    resources:
      requests:
        cpu: 10m
        memory: 100Mi
#      limits:
#        cpu: 100m
#        memory: 100Mi

  telemetry:
    enabled: true
    resources:
      requests:
        cpu: 50m
        memory: 100Mi
#      limits:
#        cpu: 100m
#        memory: 100Mi
 
  adapters:
    stdio:
      enabled: true
 
grafana:
  enabled: true
  ingress:
    enabled: true
    hosts:
      - cluster.app.mycluster.com

tracing:
  enabled: true
#  provider: zipkin
  ingress:
    enabled: true
    hosts:
      - cluster.app.mycluster.com
  jaeger:
    tag: 1.13

kiali:
  enabled: true
  createDemoSecret: true
  ingress:
    enabled: true
    hosts:
      - cluster.app.mycluster.com

gateways:
  istio-ingressgateway:
    externalTrafficPolicy: Local #change to Local to preserve source IP or Cluster for default behaviour or leave commented out
    loadBalancerIP: "10.255.42.73"
    sds:
      enabled: false
    resources:
      requests:
        cpu: 10m
        memory: 40Mi
#      limits:
#        cpu: 100m
#        memory: 512Mi

  istio-egressgateway:
    enabled: true
    resources:
      requests:
        cpu: 10m
        memory: 40Mi
#      limits:
#        cpu: 100m
#        memory: 512Mi

Now running curl throws a different error:
curl -HHost:httpbin.example.com http://10.255.42.73:80/status/200

upstream connect error or disconnect/reset before headers. reset reason: connection failure

Also tried to change listening host to “*” and getting same result from accessing from outside.

And in ingress logs i see this:

[2019-07-18T20:59:58.240Z] "GET /status/200 HTTP/1.1" 503 UF,URX "-" "-" 0 91 30036 - "10.255.254.128" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36" "79aa724d-2326-4e85-8839-288351bf49cf" "10.255.42.73" "10.42.7.15:80" outbound|8000||httpbin.default.svc.cluster.local - 10.42.7.193:80 10.255.254.128:50673 -