How to route azure application gateway to a service in different namespace?

My ingress for azure application gateway, so that it will use istio gateway internally.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: server-ingress
  namespace: productnamespace
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/backend-protocol: "http"
    appgw.ingress.kubernetes.io/cookie-based-affinity: "true"
    cert-manager.io/cluster-issuer: letsencrypt-prod
    appgw.ingress.kubernetes.io/health-probe-status-codes: "200-399, 401, 403"
spec:
  tls:
  - hosts:
    - dev1.myproductnamespace.com
    secretName: productnamespace-cert-secret
  rules:
  - host: dev1.myproductnamespace.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          namespace: istio-system
          service:
            name: istio-gateway
            port:
              number: 80

istio-gateway is in istio-system namespace, And above ingress is in productnamespace. When I did port-forwarding for istio-gateway, it is able to access the product pages. So, istio gateway is good.

But route from azure app gateway ingress to istio-gateway ingress is failing.

In the events, it shows as below.

It is trying to search for istio-gateway in productnamespace instead of istio-system and failing.

different namespace event

That error in the screenshot suggests that the azure application gateway ingress controller cannot find the endpoints (AKA IP address) of the istio-ingressgateway pods. Are you able to see the endpoints with this command?

$ kubectl get endpoints -n istio-system istio-ingressgateway

I installed with helm charts.

helm install istio/base --namespace istio-system --create-namespace --generate-name
helm install istio/istiod --namespace istio-system --create-namespace --generate-name

The name shows as istio-gateway now.

Something is wrong with istio-gateway. There should be endpoints. Make sure the istio-gateway pods are healthy and that the label selector on the istio-gateway service matches the labels on the istio-gateway pods.