Istio version: 1.6.9
I have an HTTP service which is exposed to the public internet using AWS ELB and Istio Ingress Gateway resource with ELB doing SSL termination. Here are the definitions:
Ingress Gateway:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: monitoring-gateway
namespace: istio-system
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- api.staging.example.com
port:
name: http
number: 80
protocol: HTTP
tls:
httpsRedirect: true
- hosts:
- api.staging.example.com
port:
name: https
number: 443
protocol: HTTP
a VirtualService:
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: api
namespace: istio-system
spec:
gateways:
- monitoring-gateway
hosts:
- api.staging.example.com
http:
- route:
- destination:
host: api-service.default.svc.cluster.local
and Service:
---
apiVersion: v1
kind: Service
metadata:
name: api-service
namespace: "default"
spec:
ports:
- name: http
port: 80
targetPort: http
protocol: TCP
type: NodePort
selector:
app: api
and trying to access my service API from a browser yields:
Access to XMLHttpRequest at 'https://api.example.com/api/user/login' from origin 'https://example.com' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
and using cURL:
> GET /api/user/login HTTP/1.1
> Host: api.example.com
> User-Agent: curl/7.64.1
> Accept: */*
> origin: example.com
>
< HTTP/1.1 504 GATEWAY_TIMEOUT
< Content-Length: 0
< Connection: keep-alive
<
* Connection #0 to host api.example.com left intact
* Closing connection 0
I was investigating 504 but most of the resources explain cases where a timeout happens after some amount of time: for me it’s immediate response. Investigating ELB access logs I can see:
2020-09-17T12:30:03.407841Z a822518c9a16344e1b7442258d5bdce5 <REMOVED>:63608 - -1 -1 -1 504 0 0 0 "OPTIONS https://api.example.com:443/api/user/login HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2
Here is how my ELB is configured using serviceAnnotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <REMOVED>
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
service.beta.kubernetes.io/aws-load-balancer-type: "elb"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "false"
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
and the entire istio operator definition:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: istio-operator
namespace: istio-system
spec:
addonComponents:
istiocoredns:
enabled: false
kiali:
enabled: true
k8s:
replicaCount: 1
prometheus:
enabled: true
k8s:
replicaCount: 1
tracing:
enabled: true
components:
base:
enabled: true
cni:
enabled: false
egressGateways:
- enabled: false
k8s:
env:
- name: ISTIO_META_ROUTER_MODE
value: sni-dnat
hpaSpec:
maxReplicas: 5
metrics:
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: istio-egressgateway
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 100m
memory: 128Mi
service:
ports:
- name: http2
port: 80
targetPort: 8080
- name: https
port: 443
targetPort: 8443
- name: tls
port: 15443
targetPort: 15443
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
name: istio-egressgateway
ingressGateways:
- enabled: true
k8s:
env:
- name: ISTIO_META_ROUTER_MODE
value: sni-dnat
hpaSpec:
maxReplicas: 5
metrics:
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: istio-ingressgateway
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 100m
memory: 128Mi
serviceAnnotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-west-1:829064998962:certificate/ca56bc5b-d2cc-4264-838c-6465f7b1f919
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
service.beta.kubernetes.io/aws-load-balancer-type: "elb"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "false"
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
service:
externalTrafficPolicy: Cluster
ports:
- name: status-port
port: 15021
targetPort: 15021
- name: http2
port: 80
targetPort: 8080
- name: https
port: 443
targetPort: 8443
- name: tls
port: 15443
targetPort: 15443
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
name: istio-ingressgateway
istiodRemote:
enabled: false
pilot:
enabled: true
k8s:
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 1
periodSeconds: 3
timeoutSeconds: 5
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
policy:
enabled: false
k8s:
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
hpaSpec:
maxReplicas: 5
metrics:
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: istio-policy
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
telemetry:
enabled: true
k8s:
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: GOMAXPROCS
value: "6"
hpaSpec:
maxReplicas: 5
metrics:
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: istio-telemetry
replicaCount: 1
resources:
limits:
cpu: 4800m
memory: 4G
requests:
cpu: 1000m
memory: 1G
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
hub: docker.io/istio
meshConfig:
defaultConfig:
proxyMetadata: {}
enablePrometheusMerge: true
profile: default
tag: 1.6.9
values:
base:
enableCRDTemplates: false
validationURL: ""
clusterResources: true
gateways:
istio-egressgateway:
autoscaleEnabled: true
env: {}
name: istio-egressgateway
secretVolumes:
- mountPath: /etc/istio/egressgateway-certs
name: egressgateway-certs
secretName: istio-egressgateway-certs
- mountPath: /etc/istio/egressgateway-ca-certs
name: egressgateway-ca-certs
secretName: istio-egressgateway-ca-certs
type: ClusterIP
zvpn: {}
istio-ingressgateway:
applicationPorts: ""
autoscaleEnabled: true
debug: info
domain: ""
env: {}
meshExpansionPorts:
- name: tcp-istiod
port: 15012
targetPort: 15012
- name: tcp-dns-tls
port: 853
targetPort: 8853
name: istio-ingressgateway
secretVolumes:
- mountPath: /etc/istio/ingressgateway-certs
name: ingressgateway-certs
secretName: istio-ingressgateway-certs
- mountPath: /etc/istio/ingressgateway-ca-certs
name: ingressgateway-ca-certs
secretName: istio-ingressgateway-ca-certs
type: LoadBalancer
zvpn: {}
global:
arch:
amd64: 2
ppc64le: 2
s390x: 2
configValidation: true
controlPlaneSecurityEnabled: true
defaultNodeSelector: {}
defaultPodDisruptionBudget:
enabled: true
defaultResources:
requests:
cpu: 10m
enableHelmTest: false
imagePullPolicy: ""
imagePullSecrets: []
istioNamespace: istio-system
istiod:
enableAnalysis: false
jwtPolicy: third-party-jwt
logAsJson: false
logging:
level: default:info
meshExpansion:
enabled: false
useILB: false
meshNetworks: {}
mountMtlsCerts: false
multiCluster:
clusterName: ""
enabled: false
network: ""
omitSidecarInjectorConfigMap: false
oneNamespace: false
operatorManageWebhooks: false
pilotCertProvider: istiod
priorityClassName: ""
proxy:
autoInject: enabled
clusterDomain: cluster.local
componentLogLevel: misc:error
enableCoreDump: false
excludeIPRanges: ""
excludeInboundPorts: ""
excludeOutboundPorts: ""
image: proxyv2
includeIPRanges: '*'
logLevel: warning
privileged: false
readinessFailureThreshold: 30
readinessInitialDelaySeconds: 1
readinessPeriodSeconds: 2
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 100m
memory: 128Mi
statusPort: 15020
tracer: zipkin
proxy_init:
image: proxyv2
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 10m
memory: 10Mi
sds:
token:
aud: istio-ca
sts:
servicePort: 0
tracer:
datadog:
address: $(HOST_IP):8126
lightstep:
accessToken: ""
address: ""
stackdriver:
debug: false
maxNumberOfAnnotations: 200
maxNumberOfAttributes: 200
maxNumberOfMessageEvents: 200
zipkin:
address: ""
trustDomain: cluster.local
useMCP: false
grafana:
accessMode: ReadWriteMany
contextPath: /grafana
dashboardProviders:
dashboardproviders.yaml:
apiVersion: 1
providers:
- disableDeletion: false
folder: istio
name: istio
options:
path: /var/lib/grafana/dashboards/istio
orgId: 1
type: file
datasources:
datasources.yaml:
apiVersion: 1
env: {}
envSecrets: {}
image:
repository: grafana/grafana
tag: 7.0.5
nodeSelector: {}
persist: false
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
security:
enabled: false
passphraseKey: passphrase
secretName: grafana
usernameKey: username
service:
annotations: {}
externalPort: 3000
name: http
type: ClusterIP
storageClassName: ""
tolerations: []
istiocoredns:
coreDNSImage: coredns/coredns
coreDNSPluginImage: istio/coredns-plugin:0.2-istio-1.1
coreDNSTag: 1.6.2
istiodRemote:
injectionURL: ""
kiali:
contextPath: /kiali
createDemoSecret: false
dashboard:
auth:
strategy: login
grafanaInClusterURL: http://grafana.monitoring
jaegerInClusterURL: http://tracing/jaeger
passphraseKey: passphrase
secretName: kiali
usernameKey: username
viewOnlyMode: false
hub: quay.io/kiali
nodeSelector: {}
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
security:
cert_file: /kiali-cert/cert-chain.pem
enabled: false
private_key_file: /kiali-cert/key.pem
service:
annotations: {}
tag: v1.22
mixer:
adapters:
kubernetesenv:
enabled: true
prometheus:
enabled: true
metricsExpiryDuration: 10m
stackdriver:
auth:
apiKey: ""
appCredentials: false
serviceAccountPath: ""
enabled: false
tracer:
enabled: false
sampleProbability: 1
stdio:
enabled: false
outputAsJson: false
useAdapterCRDs: false
policy:
adapters:
kubernetesenv:
enabled: true
useAdapterCRDs: false
autoscaleEnabled: true
image: mixer
sessionAffinityEnabled: false
telemetry:
autoscaleEnabled: true
env:
GOMAXPROCS: "6"
image: mixer
loadshedding:
latencyThreshold: 100ms
mode: enforce
nodeSelector: {}
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
replicaCount: 1
sessionAffinityEnabled: false
tolerations: []
pilot:
appNamespaces: []
autoscaleEnabled: true
autoscaleMax: 5
autoscaleMin: 1
configMap: true
configNamespace: istio-config
cpu:
targetAverageUtilization: 80
enableProtocolSniffingForInbound: true
enableProtocolSniffingForOutbound: true
env:
PILOT_HTTP10: "1"
image: pilot
keepaliveMaxServerConnectionAge: 30m
nodeSelector: {}
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
policy:
enabled: false
replicaCount: 1
tolerations: []
traceSampling: 1
prometheus:
contextPath: /prometheus
hub: docker.io/prom
nodeSelector: {}
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
provisionPrometheusCert: true
retention: 6h
scrapeInterval: 15s
security:
enabled: true
tag: v2.19.2
tolerations: []
sidecarInjectorWebhook:
enableNamespacesByDefault: false
injectLabel: istio-injection
objectSelector:
autoInject: true
enabled: false
rewriteAppHTTPProbe: true
telemetry:
enabled: true
v1:
enabled: false
v2:
enabled: true
metadataExchange:
wasmEnabled: false
prometheus:
enabled: true
wasmEnabled: false
stackdriver:
configOverride: {}
enabled: false
logging: false
monitoring: false
topology: false
tracing:
jaeger:
accessMode: ReadWriteMany
hub: docker.io/jaegertracing
memory:
max_traces: 50000
persist: false
spanStorageType: badger
storageClassName: ""
tag: "1.18"
nodeSelector: {}
opencensus:
exporters:
stackdriver:
enable_tracing: true
hub: docker.io/omnition
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: 200m
memory: 400Mi
tag: 0.1.9
podAntiAffinityLabelSelector: []
podAntiAffinityTermLabelSelector: []
provider: jaeger
service:
annotations: {}
externalPort: 9411
name: http-query
type: ClusterIP
zipkin:
hub: docker.io/openzipkin
javaOptsHeap: 700
maxSpans: 500000
node:
cpus: 2
probeStartupDelay: 10
queryPort: 9411
resources:
limits:
cpu: 1000m
memory: 2048Mi
requests:
cpu: 150m
memory: 900Mi
tag: 2.20.0
version: ""
PS: It does work when I switch to:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
but then ELB does not set X-Forwarded-* headers and client IP is lost. I would also like to make use of session stickiness so TCP is not an option for me.