Trying to get envoyExtAuthzHttp working with my OAuth2 Proxy instance

I have a simple application based on the httpbin application in the example. It is setup to use Istio through a simple gateway…

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin-gateway
  namespace: foo
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
    - port:
        number: 80
        name: http2
        protocol: HTTP2
      hosts:
        - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
  namespace: foo
spec:
  hosts:
    - "*"
  gateways:
    - httpbin-gateway
  http:
    - match:
        - uri:
            prefix: /
      route:
        - destination:
            host: httpbin
            port:
              number: 8000

I can run minikube tunnel and access via port 80. I would now like to add authentication using OAuth2 Proxy and OIDC. I configure OAuth2 Proxy and confirm it is working by running minikube service -n istio-system oauth-proxy. Now I would like to connect the 2 so I try adding the following to the config may (per the instructions)…

extensionProviders:
- name: "oauth2-proxy"
  envoyExtAuthzHttp:
    service: "oauth-proxy.istio-system.svc.cluster.local"
    port: "4180" # The default port used by oauth2-proxy.
    includeRequestHeadersInCheck: ["authorization", "cookie"]
    headersToUpstreamOnAllow: ["authorization", "path", "x-auth-request-user", "x-auth-request-email", "x-auth-request-access-token"]
    headersToDownstreamOnDeny: ["content-type", "set-cookie"]

Everything works until I get to the OAuth. Then it redirects to http://127.0.0.1/oauth2/start?rd=%2Fip which throws a 404 (since it is not being forwarded by Istio). How would I go about handling this? Do I need another gateway/virtual service for the OAuth2 proxy deployment running in istio-system?