Enabling Envoy Statistics lead to very slow start time (2m+!)

Hello!

We are debugging very long istio-proxy start times. Most deployments are up and read in 5-10seconds. But we found that if we include envoy statistics the startup times explodes to consitently 1m45s - 2m30s consistently.

For example we have a the following label on the deployment:

 proxy.istio.io/config: |-
          proxyStatsMatcher:
            inclusionPrefixes:
            - "cluster"
            - "http"
            - "tcp"
            - "http_mixer_filter"
            - "tcp_mixer_filter"

And we see:

Readiness succeeded in 2m11.25443902s

We are digging through documentation and logs to see why the delay is so long but wanted to ask the community if this is known behavior? Just to be clear, if we remove the proxyStatsMatcher field it starts in ~20secs (Still too slow in my opinion but ok)

Istio Version: 1.12.5

More info:
After comparing startup log lines between the one configured with the prefixes and one without the only lines I can see different are

{"level":"debug","time":"2022-10-05T21:55:30.958401Z","scope":"xdsproxy","msg":"accepted XDS connection from Envoy, forwarding to upstream XDS server"}                                                                                                                       {"level":"info","time":"2022-10-05T21:55:30.963723Z","scope":"xdsproxy","msg":"connected to upstream XDS server: istiod-1-12-5.istio-system.svc:15012"}
{"level":"debug","time":"2022-10-05T21:55:30.963755Z","scope":"xdsproxy","msg":"request for type url type.googleapis.com/envoy.config.cluster.v3.Cluster"}
{"level":"debug","time":"2022-10-05T21:55:30.992498Z","scope":"xdsproxy","msg":"response for type url type.googleapis.com/envoy.config.cluster.v3.Cluster"}
{"level":"debug","time":"2022-10-05T21:56:10.959326Z","scope":"xdsproxy","msg":"downstream [1] terminated with status rpc error: code = Canceled desc = context canceled"}
{"level":"debug","time":"2022-10-05T21:56:10.959362Z","scope":"xdsproxy","msg":"disconnected from XDS server: istiod-1-12-5.istio-system.svc:15012"}
{"level":"warning","time":"2022-10-05T21:56:39.427588Z","scope":"envoy config","msg":"StreamAggregatedResources gRPC config stream closed: 13, "}

Notice the time jump, in timestamps