After a 8h debugging the issue, I finally figure out the issue.
- Istio was good
- Axios behaviour was sending cookies and getting denied by the BE
My Istio config was actually fine, but I had a misconfiguration on the BE/FE shaking hands for cors.
I had configured most of the cors in the BE, but set-cookie (which I’m still battling to know why is not working) was not set.
There is a long thread talking about the issue in Axios (How to request anonymously via Axios? · Issue #2455 · axios/axios · GitHub). Basically it always sets the cookies as part of the request if both services are in the same domain.
The ways to avoid this is:
- Configure cookies
- Move to fetch instead
I decide to defer the decision for later and removed cookies from the session.
How I found this:
Istio didn’t have much logs to help me (or I still need to figure out how to enhance it to have), so in the ingress I was able to see income and outbound calls.
Tcpdump was literally inefficient to see anything.
I decided then to remote debug the pod in kubernetes by proxying the call to my ide. That’s when I started seeing that localhost kube proxy was working while dns based it was failing.
With google chrome, I extract a curl from the XHR (see github link above for relationship with it). When I called the service with cookies, I would get the same issue. While removing passed the flow.
Ideally, it would be nice to have more logs from ingress on cors shake hand issues. One day I figure it out.