ServiceRoleBinding for unauthenticated users only

Is it possible to authorize an unauthenticated, but disable access for an authenticated one?

I can trigger both authenticated and unauthenticated with the following in ServiceRoleBinding

  subjects:
  - user: "*"

But cannot get this restricted only for unauthenticated users