Cannot get bookinfo and httpbin ingressgateway to work on new install 1.2.*

1

Installed istio on a bare-metal cluster. Followed setup guide (tried all of the install types): https://istio.io/docs/setup/kubernetes/install/helm/#option-2-install-with-helm-and-tiller-via-helm-install

We use metallb for LoadBalancer service types (i saw other users reported that things work well for them with metallb), so istio-ingressgateway is exposed via LoadBalancer service type.

Steps taken:

  • install using helm with global k8sIngress and gateway options enabled egress+ingress. all successful:
    kubectl -n istio-system get po,svc | grep -e NAME -e ingress

    NAME READY STATUS RESTARTS AGE
    pod/istio-ingressgateway-545dd78c-487kb 1/1 Running 0 16h
    pod/istio-ingressgateway-545dd78c-579wn 1/1 Running 0 16h
    pod/istio-ingressgateway-545dd78c-5n54c 1/1 Running 0 16h
    pod/istio-ingressgateway-545dd78c-7bm7z 1/1 Running 0 16h
    pod/istio-ingressgateway-545dd78c-qxgml 1/1 Running 0 16h

    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    service/istio-ingressgateway LoadBalancer 10.43.54.158 10.255.42.73 15020:31648/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:30676/TCP,15030:31322/TCP,15031:31251/TCP,15032:32599/TCP,15443:30342/TCP 17h

  • installed httpbin - all installed correctly:
    kubectl get po,ep,svc | grep productpage
    pod/productpage-v1-7868c48878-nmfqr 2/2 Running 0 16h
    endpoints/productpage 10.42.4.140:9080 5d21h
    service/productpage ClusterIP 10.43.50.42 9080/TCP 5d21h

  • can access httpbin service inside cluster successfully:
    kubectl run -ti --rm --restart=Never dummy --image=tutum/curl:alpine sh
    curl httpbin:8000/headers
    {
    “headers”: {
    “Accept”: “/”,
    “Content-Length”: “0”,
    “Host”: “httpbin:8000”,
    “User-Agent”: “curl/7.54.0”,
    “X-B3-Parentspanid”: “1c9071e907d379e1”,
    “X-B3-Sampled”: “0”,
    “X-B3-Spanid”: “619666bc9fddfda2”,
    “X-B3-Traceid”: “43e4ea231eb3e7801c9071e907d379e1”,
    “X-Forwarded-Client-Cert”: “By=spiffe://cluster.local/ns/default/sa/default;Hash=a8b8a2125289bd547d68d64d516ddb15e50ec7686e888c114c3097919c86d7ef;Subject=”";URI=spiffe://cluster.local/ns/default/sa/default"
    }
    }

  • applied manifests with VirtualService and Gateway - everything successfully created but cannot access from outside the cluster:
    trying to access via ingress:
    curl -I -HHost:httpbin.example.com http://10.255.42.73:80/headers
    HTTP/1.1 404 Not Found
    location: http…://httpbin.example.com/headers
    date: Thu, 18 Jul 2019 14:07:00 GMT
    server: istio-envoy
    transfer-encoding: chunked

apparently i’m hitting istio-envoy, but the request doesn’t travel past that point.

Any idea what I’m missing or how to troubleshoot further?

2

Show us the yaml for gateway and virtual service. Make sure virtual service references gateway with correct name spacing.

3

copy/pasted from here https://istio.io/docs/tasks/traffic-management/ingress/ingress-control/#configuring-ingress-using-an-istio-gateway

Gateway:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin-gateway
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "httpbin.example.com"

VirtualService:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
  - "httpbin.example.com"
  gateways:
  - httpbin-gateway
  http:
  - match:
    - uri:
        prefix: /status
    - uri:
        prefix: /delay
    route:
    - destination:
        port:
          number: 8000
        host: httpbin

re-installed (wiped clean, then installed) istio and the namespace istio-system several times. same result
testing httpbin in “default” namespace

after 5 days of re-install, i am now getting 503 error and a completely different behavior, played with values.yaml install options, here’s a copy of it as well:

global:

  k8sIngress:
    enabled: true
    enableHttps: false
    gatewayName: ingressgateway

  controlPlaneSecurityEnabled: false

  mtls:
    # Default setting for service-to-service mtls. Can be set explicitly using
    # destination rules or service annotations.
    enabled: false

  proxy:
    accessLogFile: "/dev/stdout"
    resources:
      requests:
        cpu: 10m
        memory: 40Mi

  disablePolicyChecks: false

sidecarInjectorWebhook:
  enabled: true
  # If true, webhook or istioctl injector will rewrite PodSpec for liveness
  # health check to redirect request to sidecar. This makes liveness check work
  # even when mTLS is enabled.
  rewriteAppHTTPProbe: true

certmanager:
  enabled: true
  email: alex@****.com
  extraArgs: ['--default-issuer-name=letsencrypt', '--default-issuer-kind=ClusterIssuer']

prometheus:
  ingress:
    enabled: true
    hosts:
      - cluster.app.mycluster.com

pilot:
  traceSampling: 20.0
  resources:
    requests:
      cpu: 10m
      memory: 100Mi
#    limits:
#      cpu: 100m
#      memory: 200Mi

mixer:
  policy:
    enabled: true
    resources:
      requests:
        cpu: 10m
        memory: 100Mi
#      limits:
#        cpu: 100m
#        memory: 100Mi

  telemetry:
    enabled: true
    resources:
      requests:
        cpu: 50m
        memory: 100Mi
#      limits:
#        cpu: 100m
#        memory: 100Mi
 
  adapters:
    stdio:
      enabled: true
 
grafana:
  enabled: true
  ingress:
    enabled: true
    hosts:
      - cluster.app.mycluster.com

tracing:
  enabled: true
#  provider: zipkin
  ingress:
    enabled: true
    hosts:
      - cluster.app.mycluster.com
  jaeger:
    tag: 1.13

kiali:
  enabled: true
  createDemoSecret: true
  ingress:
    enabled: true
    hosts:
      - cluster.app.mycluster.com

gateways:
  istio-ingressgateway:
    externalTrafficPolicy: Local #change to Local to preserve source IP or Cluster for default behaviour or leave commented out
    loadBalancerIP: "10.255.42.73"
    sds:
      enabled: false
    resources:
      requests:
        cpu: 10m
        memory: 40Mi
#      limits:
#        cpu: 100m
#        memory: 512Mi

  istio-egressgateway:
    enabled: true
    resources:
      requests:
        cpu: 10m
        memory: 40Mi
#      limits:
#        cpu: 100m
#        memory: 512Mi

Now running curl throws a different error:
curl -HHost:httpbin.example.com http://10.255.42.73:80/status/200

upstream connect error or disconnect/reset before headers. reset reason: connection failure

Also tried to change listening host to “*” and getting same result from accessing from outside.

And in ingress logs i see this:

[2019-07-18T20:59:58.240Z] "GET /status/200 HTTP/1.1" 503 UF,URX "-" "-" 0 91 30036 - "10.255.254.128" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36" "79aa724d-2326-4e85-8839-288351bf49cf" "10.255.42.73" "10.42.7.15:80" outbound|8000||httpbin.default.svc.cluster.local - 10.42.7.193:80 10.255.254.128:50673 -