Openshift ingressgateway `http` vs `https`

#1

I have installed istio on my openshift cluster. To test, I got sample bookinfo application working on http.
When I tried it for my actual application http works but not https. I am not sure if I need to change anything on router config?

The following is the yaml

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: asdfgapi-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
    hosts:
    - "*.istio.myurl.myexample.com"
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*.istio.myurl.myexample.com"

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: asdfgapi
spec:
  hosts:
  - "asdfgapi.istio.myurl.myexample.com"
  gateways:
  - asdfgapi-gateway
  http:
  - route:
    - destination:
        host: asdfgapi.myexample-dt.svc.cluster.local
        port:
          number: 8080


---
apiVersion: v1
kind: Service
metadata:
  name: asdfgapi
  labels:
    app: asdfgapi
    service: asdfgapi
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: asdfgapi

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: asdfgapi
  labels:
    app: asdfgapi
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: asdfgapi
    spec:
      containers:
        - env:
            - name: SPRING_PROFILES_ACTIVE
              value: dt
          image: 'nexus.myexample.com:9083/asdfgapi:162db1e'
          imagePullPolicy: Always
          name: asdfgapi
          ports:
            - containerPort: 8080
              protocol: TCP
          volumeMounts:
            - mountPath: /var/log/app
              name: volume1
      volumes:
        - emptyDir: {}
          name: volume1

---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: asdfgapi
spec:
  host: asdfgapi
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

---

The following shows the correct certs on ingressgateway

kubectl exec -ti $(kubectl get po -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system -- curl  127.0.0.1:15000/certs

The following is the route

 oc get route istio-wildcard-ingress -n istio-system
NAME                     HOST/PORT                         PATH      SERVICES               PORT      TERMINATION   WILDCARD
istio-wildcard-ingress   www.istio.myurl.myexample.com             istio-ingressgateway   http2                   Subdomain

When I try curl on http

curl -kv http://asdfgapi.istio.myurl.myexample.com/api/v1/applicant/searchByANumber?aNumber=A123456789

I get a 200 and a correct response back. I see the correct logs in the pods and in envoy proxy

BUT The following curl on https doesnt work

curl -kv https://asdfgapi.istio.myurl.myexample.com/api/v1/applicant/searchByANumber?aNumber=A123456789

I get 503 back along with the following

<div>
      <h1>Application is not available</h1>
      <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>

      <div class="alert alert-info">
        <p class="info">
          Possible reasons you are seeing this page:
        </p>
        .....
      </div>
    </div>

The following doesnt return any logs related to my pod

kubectl logs istio-ingressgateway-7f8dd8f46f-xnlwq -n istio-system

Obviously the request is not reaching the pod

Any suggestions on the issue why https doesnt work ?

#2

When I do a

istioctl pc -n istio-system routes istio-ingressgateway-7f8dd8f46f-k5zfh -o json

I see the following

[
    {
        "name": "http.80",
        "virtualHosts": [
            {
                "name": "asdfgapi.istio.myurl.myexample.com:80",
                "domains": [
                    "asdfgapi.istio.myurl.myexample.com",
                    "asdfgapi.istio.myurl.myexample.com:80"
                ],
                "routes": [
                    {
                        "match": {
                            "prefix": "/"
                        },
                        "route": {
                            "cluster": "outbound|8080||asdfgapi.mynamespace.svc.cluster.local",
                            "timeout": "0s",
                            "retryPolicy": {
                                "retryOn": "connect-failure,refused-stream,unavailable,cancelled,resource-exhausted",
                                "numRetries": 10,
                                "retryHostPredicate": [
                                    {
                                        "name": "envoy.retry_host_predicates.previous_hosts"
                                    }
                                ],
                                "hostSelectionRetryMaxAttempts": "3",
                                "retriableStatusCodes": [
                                    503
                                ]
                            },
                            "maxGrpcTimeout": "0s"
                        },
                        "metadata": {
                            "filterMetadata": {
                                "istio": {
                                    "config": "/apis/networking/v1alpha3/namespaces/mynamespace/virtual-service/asdfgapi"
                                }
                            }
                        },
                        "decorator": {
                            "operation": "asdfgapi.mynamespace.svc.cluster.local:8080/*"
                        },
                        "typedPerFilterConfig": {
                            "mixer": {
                                "@type": "type.googleapis.com/istio.mixer.v1.config.client.ServiceConfig",
                                "disableCheckCalls": true,
                                "mixerAttributes": {
                                    "attributes": {
                                        "destination.service.host": {
                                            "stringValue": "asdfgapi.mynamespace.svc.cluster.local"
                                        },
                                        "destination.service.name": {
                                            "stringValue": "asdfgapi"
                                        },
                                        "destination.service.namespace": {
                                            "stringValue": "mynamespace"
                                        },
                                        "destination.service.uid": {
                                            "stringValue": "istio://mynamespace/services/asdfgapi"
                                        }
                                    }
                                },
                                "forwardAttributes": {
                                    "attributes": {
                                        "destination.service.host": {
                                            "stringValue": "asdfgapi.mynamespace.svc.cluster.local"
                                        },
                                        "destination.service.name": {
                                            "stringValue": "asdfgapi"
                                        },
                                        "destination.service.namespace": {
                                            "stringValue": "mynamespace"
                                        },
                                        "destination.service.uid": {
                                            "stringValue": "istio://mynamespace/services/asdfgapi"
                                        }
                                    }
                                }
                            }
                        }
                    }
                ]
            }
        ],
        "validateClusters": false
    },
    {
        "name": "https.443.https-asdfgapi",
        "virtualHosts": [
            {
                "name": "asdfgapi.istio.myurl.myexample.com:443",
                "domains": [
                    "asdfgapi.istio.myurl.myexample.com",
                    "asdfgapi.istio.myurl.myexample.com:443"
                ],
                "routes": [
                    {
                        "match": {
                            "prefix": "/"
                        },
                        "route": {
                            "cluster": "outbound|8080||asdfgapi.mynamespace.svc.cluster.local",
                            "timeout": "0s",
                            "retryPolicy": {
                                "retryOn": "connect-failure,refused-stream,unavailable,cancelled,resource-exhausted",
                                "numRetries": 10,
                                "retryHostPredicate": [
                                    {
                                        "name": "envoy.retry_host_predicates.previous_hosts"
                                    }
                                ],
                                "hostSelectionRetryMaxAttempts": "3",
                                "retriableStatusCodes": [
                                    503
                                ]
                            },
                            "maxGrpcTimeout": "0s"
                        },
                        "metadata": {
                            "filterMetadata": {
                                "istio": {
                                    "config": "/apis/networking/v1alpha3/namespaces/mynamespace/virtual-service/asdfgapi"
                                }
                            }
                        },
                        "decorator": {
                            "operation": "asdfgapi.mynamespace.svc.cluster.local:8080/*"
                        },
                        "typedPerFilterConfig": {
                            "mixer": {
                                "@type": "type.googleapis.com/istio.mixer.v1.config.client.ServiceConfig",
                                "disableCheckCalls": true,
                                "mixerAttributes": {
                                    "attributes": {
                                        "destination.service.host": {
                                            "stringValue": "asdfgapi.mynamespace.svc.cluster.local"
                                        },
                                        "destination.service.name": {
                                            "stringValue": "asdfgapi"
                                        },
                                        "destination.service.namespace": {
                                            "stringValue": "mynamespace"
                                        },
                                        "destination.service.uid": {
                                            "stringValue": "istio://mynamespace/services/asdfgapi"
                                        }
                                    }
                                },
                                "forwardAttributes": {
                                    "attributes": {
                                        "destination.service.host": {
                                            "stringValue": "asdfgapi.mynamespace.svc.cluster.local"
                                        },
                                        "destination.service.name": {
                                            "stringValue": "asdfgapi"
                                        },
                                        "destination.service.namespace": {
                                            "stringValue": "mynamespace"
                                        },
                                        "destination.service.uid": {
                                            "stringValue": "istio://mynamespace/services/asdfgapi"
                                        }
                                    }
                                }
                            }
                        }
                    }
                ]
            }
        ],
        "validateClusters": false
    },
    {
        "virtualHosts": [
            {
                "name": "backend",
                "domains": [
                    "*"
                ],
                "routes": [
                    {
                        "match": {
                            "prefix": "/stats/prometheus"
                        },
                        "route": {
                            "cluster": "prometheus_stats"
                        }
                    }
                ]
            }
        ]
    }
]

appreciate any suggestions on debugging

#3

@ravi_gude, Could you define the ingressgateway route edge as with a redirect policy from http to https as following:
oc create route edge --service=istio-ingressgateway --insecure-policy=Redirect
When checking the route, you should have the termination set to edge/Redirect

oc get route istio-ingressgateway
NAME                   HOST/PORT                                                                    PATH      SERVICES               PORT              TERMINATION     WILDCARD
istio-ingressgateway   istio-ingressgateway-istio-system.my-domain             istio-ingressgateway   http2             edge/Redirect   None