Istio with AWS ELB - switching to backend-protocol HTTP triggers CORS errors for browser and 504 for cURL

Istio version: 1.6.9

I have an HTTP service which is exposed to the public internet using AWS ELB and Istio Ingress Gateway resource with ELB doing SSL termination. Here are the definitions:

Ingress Gateway:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: monitoring-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - api.staging.example.com
    port:
      name: http
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true
  - hosts:
    - api.staging.example.com
    port:
      name: https
      number: 443
      protocol: HTTP

a VirtualService:

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api
  namespace: istio-system
spec:
  gateways:
  - monitoring-gateway
  hosts:
  - api.staging.example.com
  http:
  - route:
    - destination:
        host: api-service.default.svc.cluster.local

and Service:

---
apiVersion: v1
kind: Service
metadata:
  name: api-service
  namespace: "default"
spec:
  ports:
  - name: http
    port: 80
    targetPort: http
    protocol: TCP
  type: NodePort
  selector:
    app: api

and trying to access my service API from a browser yields:

Access to XMLHttpRequest at 'https://api.example.com/api/user/login' from origin 'https://example.com' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

and using cURL:

> GET /api/user/login HTTP/1.1
> Host: api.example.com
> User-Agent: curl/7.64.1
> Accept: */*
> origin: example.com
>
< HTTP/1.1 504 GATEWAY_TIMEOUT
< Content-Length: 0
< Connection: keep-alive
<
* Connection #0 to host api.example.com left intact
* Closing connection 0

I was investigating 504 but most of the resources explain cases where a timeout happens after some amount of time: for me it’s immediate response. Investigating ELB access logs I can see:

2020-09-17T12:30:03.407841Z a822518c9a16344e1b7442258d5bdce5 <REMOVED>:63608 - -1 -1 -1 504 0 0 0 "OPTIONS https://api.example.com:443/api/user/login HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2

Here is how my ELB is configured using serviceAnnotations:

          service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <REMOVED>
          service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
          service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
          service.beta.kubernetes.io/aws-load-balancer-type: "elb"
          service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "false"
          service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"

and the entire istio operator definition:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-operator
  namespace: istio-system
spec:
  addonComponents:
    istiocoredns:
      enabled: false
    kiali:
      enabled: true
      k8s:
        replicaCount: 1
    prometheus:
      enabled: true
      k8s:
        replicaCount: 1
    tracing:
      enabled: true
  components:
    base:
      enabled: true
    cni:
      enabled: false
    egressGateways:
    - enabled: false
      k8s:
        env:
        - name: ISTIO_META_ROUTER_MODE
          value: sni-dnat
        hpaSpec:
          maxReplicas: 5
          metrics:
          - resource:
              name: cpu
              targetAverageUtilization: 80
            type: Resource
          minReplicas: 1
          scaleTargetRef:
            apiVersion: apps/v1
            kind: Deployment
            name: istio-egressgateway
        resources:
          limits:
            cpu: 2000m
            memory: 1024Mi
          requests:
            cpu: 100m
            memory: 128Mi
        service:
          ports:
          - name: http2
            port: 80
            targetPort: 8080
          - name: https
            port: 443
            targetPort: 8443
          - name: tls
            port: 15443
            targetPort: 15443
        strategy:
          rollingUpdate:
            maxSurge: 100%
            maxUnavailable: 25%
      name: istio-egressgateway
    ingressGateways:
    - enabled: true
      k8s:
        env:
        - name: ISTIO_META_ROUTER_MODE
          value: sni-dnat
        hpaSpec:
          maxReplicas: 5
          metrics:
          - resource:
              name: cpu
              targetAverageUtilization: 80
            type: Resource
          minReplicas: 1
          scaleTargetRef:
            apiVersion: apps/v1
            kind: Deployment
            name: istio-ingressgateway
        resources:
          limits:
            cpu: 2000m
            memory: 1024Mi
          requests:
            cpu: 100m
            memory: 128Mi
        serviceAnnotations:
          service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-west-1:829064998962:certificate/ca56bc5b-d2cc-4264-838c-6465f7b1f919
          service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
          service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
          service.beta.kubernetes.io/aws-load-balancer-type: "elb"
          service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "false"
          service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
        service:
          externalTrafficPolicy: Cluster
          ports:
          - name: status-port
            port: 15021
            targetPort: 15021
          - name: http2
            port: 80
            targetPort: 8080
          - name: https
            port: 443
            targetPort: 8443
          - name: tls
            port: 15443
            targetPort: 15443
        strategy:
          rollingUpdate:
            maxSurge: 100%
            maxUnavailable: 25%
      name: istio-ingressgateway
    istiodRemote:
      enabled: false
    pilot:
      enabled: true
      k8s:
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 1
          periodSeconds: 3
          timeoutSeconds: 5
        strategy:
          rollingUpdate:
            maxSurge: 100%
            maxUnavailable: 25%
    policy:
      enabled: false
      k8s:
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        hpaSpec:
          maxReplicas: 5
          metrics:
          - resource:
              name: cpu
              targetAverageUtilization: 80
            type: Resource
          minReplicas: 1
          scaleTargetRef:
            apiVersion: apps/v1
            kind: Deployment
            name: istio-policy
        strategy:
          rollingUpdate:
            maxSurge: 100%
            maxUnavailable: 25%
    telemetry:
      enabled: true
      k8s:
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: GOMAXPROCS
          value: "6"
        hpaSpec:
          maxReplicas: 5
          metrics:
          - resource:
              name: cpu
              targetAverageUtilization: 80
            type: Resource
          minReplicas: 1
          scaleTargetRef:
            apiVersion: apps/v1
            kind: Deployment
            name: istio-telemetry
        replicaCount: 1
        resources:
          limits:
            cpu: 4800m
            memory: 4G
          requests:
            cpu: 1000m
            memory: 1G
        strategy:
          rollingUpdate:
            maxSurge: 100%
            maxUnavailable: 25%
  hub: docker.io/istio
  meshConfig:
    defaultConfig:
      proxyMetadata: {}
    enablePrometheusMerge: true
  profile: default
  tag: 1.6.9
  values:
    base:
      enableCRDTemplates: false
      validationURL: ""
    clusterResources: true
    gateways:
      istio-egressgateway:
        autoscaleEnabled: true
        env: {}
        name: istio-egressgateway
        secretVolumes:
        - mountPath: /etc/istio/egressgateway-certs
          name: egressgateway-certs
          secretName: istio-egressgateway-certs
        - mountPath: /etc/istio/egressgateway-ca-certs
          name: egressgateway-ca-certs
          secretName: istio-egressgateway-ca-certs
        type: ClusterIP
        zvpn: {}
      istio-ingressgateway:
        applicationPorts: ""
        autoscaleEnabled: true
        debug: info
        domain: ""
        env: {}
        meshExpansionPorts:
        - name: tcp-istiod
          port: 15012
          targetPort: 15012
        - name: tcp-dns-tls
          port: 853
          targetPort: 8853
        name: istio-ingressgateway
        secretVolumes:
        - mountPath: /etc/istio/ingressgateway-certs
          name: ingressgateway-certs
          secretName: istio-ingressgateway-certs
        - mountPath: /etc/istio/ingressgateway-ca-certs
          name: ingressgateway-ca-certs
          secretName: istio-ingressgateway-ca-certs
        type: LoadBalancer
        zvpn: {}
    global:
      arch:
        amd64: 2
        ppc64le: 2
        s390x: 2
      configValidation: true
      controlPlaneSecurityEnabled: true
      defaultNodeSelector: {}
      defaultPodDisruptionBudget:
        enabled: true
      defaultResources:
        requests:
          cpu: 10m
      enableHelmTest: false
      imagePullPolicy: ""
      imagePullSecrets: []
      istioNamespace: istio-system
      istiod:
        enableAnalysis: false
      jwtPolicy: third-party-jwt
      logAsJson: false
      logging:
        level: default:info
      meshExpansion:
        enabled: false
        useILB: false
      meshNetworks: {}
      mountMtlsCerts: false
      multiCluster:
        clusterName: ""
        enabled: false
      network: ""
      omitSidecarInjectorConfigMap: false
      oneNamespace: false
      operatorManageWebhooks: false
      pilotCertProvider: istiod
      priorityClassName: ""
      proxy:
        autoInject: enabled
        clusterDomain: cluster.local
        componentLogLevel: misc:error
        enableCoreDump: false
        excludeIPRanges: ""
        excludeInboundPorts: ""
        excludeOutboundPorts: ""
        image: proxyv2
        includeIPRanges: '*'
        logLevel: warning
        privileged: false
        readinessFailureThreshold: 30
        readinessInitialDelaySeconds: 1
        readinessPeriodSeconds: 2
        resources:
          limits:
            cpu: 2000m
            memory: 1024Mi
          requests:
            cpu: 100m
            memory: 128Mi
        statusPort: 15020
        tracer: zipkin
      proxy_init:
        image: proxyv2
        resources:
          limits:
            cpu: 2000m
            memory: 1024Mi
          requests:
            cpu: 10m
            memory: 10Mi
      sds:
        token:
          aud: istio-ca
      sts:
        servicePort: 0
      tracer:
        datadog:
          address: $(HOST_IP):8126
        lightstep:
          accessToken: ""
          address: ""
        stackdriver:
          debug: false
          maxNumberOfAnnotations: 200
          maxNumberOfAttributes: 200
          maxNumberOfMessageEvents: 200
        zipkin:
          address: ""
      trustDomain: cluster.local
      useMCP: false
    grafana:
      accessMode: ReadWriteMany
      contextPath: /grafana
      dashboardProviders:
        dashboardproviders.yaml:
          apiVersion: 1
          providers:
          - disableDeletion: false
            folder: istio
            name: istio
            options:
              path: /var/lib/grafana/dashboards/istio
            orgId: 1
            type: file
      datasources:
        datasources.yaml:
          apiVersion: 1
      env: {}
      envSecrets: {}
      image:
        repository: grafana/grafana
        tag: 7.0.5
      nodeSelector: {}
      persist: false
      podAntiAffinityLabelSelector: []
      podAntiAffinityTermLabelSelector: []
      security:
        enabled: false
        passphraseKey: passphrase
        secretName: grafana
        usernameKey: username
      service:
        annotations: {}
        externalPort: 3000
        name: http
        type: ClusterIP
      storageClassName: ""
      tolerations: []
    istiocoredns:
      coreDNSImage: coredns/coredns
      coreDNSPluginImage: istio/coredns-plugin:0.2-istio-1.1
      coreDNSTag: 1.6.2
    istiodRemote:
      injectionURL: ""
    kiali:
      contextPath: /kiali
      createDemoSecret: false
      dashboard:
        auth:
          strategy: login
        grafanaInClusterURL: http://grafana.monitoring
        jaegerInClusterURL: http://tracing/jaeger
        passphraseKey: passphrase
        secretName: kiali
        usernameKey: username
        viewOnlyMode: false
      hub: quay.io/kiali
      nodeSelector: {}
      podAntiAffinityLabelSelector: []
      podAntiAffinityTermLabelSelector: []
      security:
        cert_file: /kiali-cert/cert-chain.pem
        enabled: false
        private_key_file: /kiali-cert/key.pem
      service:
        annotations: {}
      tag: v1.22
    mixer:
      adapters:
        kubernetesenv:
          enabled: true
        prometheus:
          enabled: true
          metricsExpiryDuration: 10m
        stackdriver:
          auth:
            apiKey: ""
            appCredentials: false
            serviceAccountPath: ""
          enabled: false
          tracer:
            enabled: false
            sampleProbability: 1
        stdio:
          enabled: false
          outputAsJson: false
        useAdapterCRDs: false
      policy:
        adapters:
          kubernetesenv:
            enabled: true
          useAdapterCRDs: false
        autoscaleEnabled: true
        image: mixer
        sessionAffinityEnabled: false
      telemetry:
        autoscaleEnabled: true
        env:
          GOMAXPROCS: "6"
        image: mixer
        loadshedding:
          latencyThreshold: 100ms
          mode: enforce
        nodeSelector: {}
        podAntiAffinityLabelSelector: []
        podAntiAffinityTermLabelSelector: []
        replicaCount: 1
        sessionAffinityEnabled: false
        tolerations: []
    pilot:
      appNamespaces: []
      autoscaleEnabled: true
      autoscaleMax: 5
      autoscaleMin: 1
      configMap: true
      configNamespace: istio-config
      cpu:
        targetAverageUtilization: 80
      enableProtocolSniffingForInbound: true
      enableProtocolSniffingForOutbound: true
      env: 
        PILOT_HTTP10: "1"
      image: pilot
      keepaliveMaxServerConnectionAge: 30m
      nodeSelector: {}
      podAntiAffinityLabelSelector: []
      podAntiAffinityTermLabelSelector: []
      policy:
        enabled: false
      replicaCount: 1
      tolerations: []
      traceSampling: 1
    prometheus:
      contextPath: /prometheus
      hub: docker.io/prom
      nodeSelector: {}
      podAntiAffinityLabelSelector: []
      podAntiAffinityTermLabelSelector: []
      provisionPrometheusCert: true
      retention: 6h
      scrapeInterval: 15s
      security:
        enabled: true
      tag: v2.19.2
      tolerations: []
    sidecarInjectorWebhook:
      enableNamespacesByDefault: false
      injectLabel: istio-injection
      objectSelector:
        autoInject: true
        enabled: false
      rewriteAppHTTPProbe: true
    telemetry:
      enabled: true
      v1:
        enabled: false
      v2:
        enabled: true
        metadataExchange:
          wasmEnabled: false
        prometheus:
          enabled: true
          wasmEnabled: false
        stackdriver:
          configOverride: {}
          enabled: false
          logging: false
          monitoring: false
          topology: false
    tracing:
      jaeger:
        accessMode: ReadWriteMany
        hub: docker.io/jaegertracing
        memory:
          max_traces: 50000
        persist: false
        spanStorageType: badger
        storageClassName: ""
        tag: "1.18"
      nodeSelector: {}
      opencensus:
        exporters:
          stackdriver:
            enable_tracing: true
        hub: docker.io/omnition
        resources:
          limits:
            cpu: "1"
            memory: 2Gi
          requests:
            cpu: 200m
            memory: 400Mi
        tag: 0.1.9
      podAntiAffinityLabelSelector: []
      podAntiAffinityTermLabelSelector: []
      provider: jaeger
      service:
        annotations: {}
        externalPort: 9411
        name: http-query
        type: ClusterIP
      zipkin:
        hub: docker.io/openzipkin
        javaOptsHeap: 700
        maxSpans: 500000
        node:
          cpus: 2
        probeStartupDelay: 10
        queryPort: 9411
        resources:
          limits:
            cpu: 1000m
            memory: 2048Mi
          requests:
            cpu: 150m
            memory: 900Mi
        tag: 2.20.0
    version: ""

PS: It does work when I switch to:

          service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"

but then ELB does not set X-Forwarded-* headers and client IP is lost. I would also like to make use of session stickiness so TCP is not an option for me.