(NOTE: This is used to report product bugs:
To report a security vulnerability, please visit https://istio.io/about/security-vulnerabilities/
To ask questions about how to use Istio, please visit https://discuss.istio.io
)
Bug description
- The same gateway, virtualservice configurations used to work but suddenly stop working.
- Routes not seen in
proxy-configdespite correct Gateway and VirtualService configs - I have deleted
istio-systemnamespace and re-installed istio from scratch, then re-injected sidecar bykubectl rollout restart deployment --namespace staging. But no luck. - Connectivity within mesh works. Just external traffic through ingress gateway doesn’t work with gateway and virtual service.
Using EKS v1.6
Versions
$ istioctl version
client version: 1.6.7
privateingressgateway version:
pilot version: 1.6.7
data plane version: 1.6.7 (6 proxies)
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.10-eks-bac369", GitCommit:"bac3690554985327ae4d13e42169e8b1c2f37226", GitTreeState:"clean", BuildDate:"2020-02-21T23:37:18Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.13-eks-2ba888", GitCommit:"2ba888155c7f8093a1bc06e3336333fbdb27b3da", GitTreeState:"clean", BuildDate:"2020-07-17T18:48:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Installed istio
istioctl install -f overrides.yaml
Where overrides.yaml is below
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
gateways:
istio-ingressgateway:
sds:
enabled: true
components: # ref: https://istio.io/latest/docs/reference/config/istio.operator.v1alpha1/#IstioOperatorSpec
# sidecarInjector:
# enabled: true
ingressGateways:
- enabled: true
name: istio-ingressgateway
k8s:
resources:
requests:
cpu: 10m
memory: 40Mi
service:
ports:
- name: status-port
port: 15020
targetPort: 15020
- name: http2
port: 80
targetPort: 8080
- name: https
port: 443
targetPort: 8443
- name: tcp
port: 31400
targetPort: 31400
- name: tls
port: 15443
targetPort: 15443
serviceAnnotations:
# enable ELB access log
# ref: https://www.giantswarm.io/blog/load-balancer-service-use-cases-on-aws
service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: "true"
# The interval for publishing the access logs (can be 5 or 60 minutes).
service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval: "5"
service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: "xxx"
service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: "public-elb"
# enable TLS termination at AWS ELB level
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-east-1:xxx:certificate/xxx"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
- enabled: true
name: istio-private-ingressgateway
label:
istio: privateingressgateway # this will be needed as gateway will look for this selector
app: istio-private-ingressgateway
k8s:
resources:
requests:
cpu: 10m
memory: 40Mi
service:
ports:
- name: status-port
port: 15020
targetPort: 15020
- name: http2
port: 80
targetPort: 8080
- name: https
port: 443
targetPort: 8443
- name: tcp
port: 31400
targetPort: 31400
- name: tls
port: 15443
targetPort: 15443
serviceAnnotations:
# ref: https://medium.com/swlh/public-and-private-istio-ingress-gateways-on-aws-f968783d62fe
service.beta.kubernetes.io/aws-load-balancer-internal: "true" # make this CLB private. Refs for service annotations for AWS ELB: https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/#aws, https://docs.aws.amazon.com/eks/latest/userguide/load-balancing.html
# enable ELB access log
# ref: https://www.giantswarm.io/blog/load-balancer-service-use-cases-on-aws
service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: "true"
# The interval for publishing the access logs (can be 5 or 60 minutes).
service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval: "60"
service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: "xxxx"
service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: "internal-elb"
# enable TLS termination at AWS ELB level
# service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-east-1:xxx:certificate/xxx"
# service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
# service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
values:
gateways: # configure gateways: https://istio.io/latest/docs/setup/install/istioctl/#configure-gateways
istio-ingressgateway: # for internal ELB
applicationPorts: ""
autoscaleEnabled: false
debug: info
domain: ""
env: {}
meshExpansionPorts:
- name: tcp-pilot-grpc-tls
port: 15011
targetPort: 15011
- name: tcp-istiod
port: 15012
targetPort: 15012
- name: tcp-citadel-grpc-tls
port: 8060
targetPort: 8060
- name: tcp-dns-tls
port: 853
targetPort: 8853
name: istio-private-ingressgateway
secretVolumes:
- mountPath: /etc/istio/ingressgateway-certs
name: ingressgateway-certs
secretName: istio-ingressgateway-certs
- mountPath: /etc/istio/ingressgateway-ca-certs
name: ingressgateway-ca-certs
secretName: istio-ingressgateway-ca-certs
type: LoadBalancer
gateway.yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: example-gateway
namespace: staging
spec:
selector:
istio: ingressgateway # use istio default controller
servers: # defines L7 host, port, and protocol
- port:
number: 80
name: http
protocol: HTTP
hosts: # all other external requests will be rejected with a 404 response.
- "example.co"
- "*.example.co"
# tls:
# httpsRedirect: true # sends 301 redirect for http requests
VirtualService
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: example-api-virtualservice
namespace: staging
spec:
hosts:
- "example.co"
- "*.example.co"
- example-api # for internal request among apps within Istio mesh
gateways: # names of gateways and sidecars that should apply these routes
- example-gateway.staging.svc.cluster.local
# - mesh # applies to all the sidecars in the mesh. The reserved word mesh is used to imply all the sidecars in the mesh. When gateway field is omitted, the default gateway (mesh) will be used, which would apply the rule to all sidecars in the mesh. If a list of gateway names is provided, the rules will apply only to the gateways. To apply the rules to both gateways and sidecars, specify mesh as one of the gateway names. Ref: https://istio.io/latest/docs/reference/config/networking/virtual-service/#VirtualService
http:
- route: # default route
- destination:
host: example-api.staging.svc.cluster.local # specify service name
port:
number: 9999
Public ingress gateway’s route config
istioctl proxy-config routes istio-ingressgateway-59d7487bd6-55x58 -n istio-system -o json
[
{
"name": "http.80",
"virtualHosts": [
{
"name": "blackhole:80",
"domains": [
"*"
],
"routes": [
{
"name": "default",
"match": {
"prefix": "/"
},
"directResponse": {
"status": 404
}
}
]
}
],
"validateClusters": false
},
{
"virtualHosts": [
{
"name": "backend",
"domains": [
"*"
],
"routes": [
{
"match": {
"prefix": "/healthz/ready"
},
"route": {
"cluster": "agent"
}
}
]
}
]
},
{
"virtualHosts": [
{
"name": "backend",
"domains": [
"*"
],
"routes": [
{
"match": {
"prefix": "/stats/prometheus"
},
"route": {
"cluster": "prometheus_stats"
}
}
]
}
]
}
]
When I curl, I get 404
curl example.com/status.html -v
* Trying xx.xx.xx.xx...
* TCP_NODELAY set
* Connected to example.com (xx.xx.xx.xx) port 80 (#0)
> GET /status.html HTTP/1.1
> Host: example.com
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< date: Fri, 04 Sep 2020 19:44:02 GMT
< server: istio-envoy
< Content-Length: 0
< Connection: keep-alive
<
* Connection #0 to host example.com left intact
Istio-proxy log shows request is reaching envoy in ingressgateway pod, but not sending it to virtual service
2020-09-04T19:44:12.323089827Z [2020-09-04T19:44:03.154Z] "GET /status.html HTTP/1.1" 404 - "-" "-" 0 0 0 - "1.20.48.191,10.1.103.200" "curl/7.54.0" "73a3b7d3-b2d1-9fe9-aaf5-c558bd2f4901" "stagingapi.peerwell.co" "-" - - 10.1.103.100:8080 10.1.103.200:6402 - default
I can connect to this backend pod through service or directly from another curl pod from within the cluster
What baffles me the most is gateway and virtual service configs haven’t changed.
DNS is okay, it’s resolving to AWS ELB’s IP
$ host example.com
xx.xx.xx.xx
I don’t know why route configs are not seen in proxy-config, this seems to be the cause, although I see gateway and virtualservice yamls on Kiali dashboard without any errors.
$ kubectl get gw,vs -n staging
NAME AGE
gateway.networking.istio.io/example-gateway 57m
NAME GATEWAYS HOSTS AGE
virtualservice.networking.istio.io/example-api-virtualservice [example-gateway.staging.svc.cluster.local] [stagingapi.example.co example.co example-api] 57m
istioctl proxy-config routes istio-ingressgateway-59d7487bd6-55x58 -n istio-system -o json
[
{
"name": "http.80",
"virtualHosts": [
{
"name": "blackhole:80",
"domains": [
"*"
],
"routes": [
{
"name": "default",
"match": {
"prefix": "/"
},
"directResponse": {
"status": 404
}
}
]
}
],
"validateClusters": false
},
Affected product area (please put an X in all that apply)
Docs
Installation
[x ] Networking
Performance and Scalability
Extensions and Telemetry
Security
Test and Release
[x ] User Experience
Developer Infrastructure
Affected features (please put an X in all that apply)
Multi Cluster
Virtual Machine
Multi Control Plane
Expected behavior
Steps to reproduce the bug
Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
How was Istio installed?
Environment where bug was observed (cloud vendor, OS, etc)
Additionally, please consider attaching a cluster state archive by attaching
the dump file to this issue.