Apply EnvoyFilter to one out of multiple Ingress Gateways

Is there a way to apply an EnvoyFilter to a specific gateway when there are more than one gateway running?

The workloadSelector appears to go by label but anything other than istio: ingressgateway or app: istio-ingressgateway seems to result in the filter not being applied.

There doesn’t seem to be a way to assign additional labels when creating the ingress gateways and reversely, if you manually add a label to an existing deployment it seems that Istio either doesn’t see it or doesn’t care about it.

Does anyone know a way to achieve this? I’m ultimately looking to have one gateway that requires authentication and one that doesn’t.