Nginx Ingress controller fails to communicate with apiserver

Networking
1

I have enabled MTLS for my cluster using namespace level policy and destinationrules.
When i inject envoy in the ingress controller i start seeing below error in logs and ingress contoller pod goes in crashloopback state.
Istio version 1.4.7
kubernetes version 1.15.7

I0513 10:36:56.383730 6 request.go:848] Got a Retry-After 1s response for attempt 1 to https://10.20.0.1:443/version?timeout=32s
I0513 10:36:57.385092 6 request.go:848] Got a Retry-After 1s response for attempt 2 to https://10.20.0.1:443/version?timeout=32s
I0513 10:36:58.387623 6 request.go:848] Got a Retry-After 1s response for attempt 3 to https://10.20.0.1:443/version?timeout=32s

Have annotated nginx ingress controller with below

  traffic.sidecar.istio.io/includeInboundPorts: ""
  traffic.sidecar.istio.io/excludeInboundPorts: "80,443"
  # Exclude outbound traffic to kubernetes master from redirection.
  traffic.sidecar.istio.io/excludeOutboundIPRanges: kube_api_server_ip