Envoy to micro service

1

How will the communication happen between envoy side car and actual micro services because as soon as I enable mutual tls the envoy was not able to talk to the actual micro service. I am hitting the https application though https ingress load balancer and able to hit the service but once I enabled MTLS, I am getting 503 error, error log in the envoy proxy of the container says, upstream unavailable

2

MTLS enabled communication between sidecars, not between sidecars and the application running the same pod, Once enabled in the service mesh all communication between the sidecars will happen on tls where in certs managed through citadel. When accessing a microservice from outside the mesh you are bound to get an error as your request as you might have not enabled destination rules. So incase the client wants to talk to a an mtls enabaled mesh, you ll need to add destination rules for that.

 kubectl apply -f - <<EOF
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  host: "*.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
EOF
3

Hi Adheip Singh, Thanks for the reply. I have a microservice which accepts only https traffic, I was able to reach the microservice from my ingress gateway using https if mTLS is not enabled. If mTLS is enabled and when I try to access my service, I am getting the below error.

upstream connect error or disconnect/reset before headers. reset reason: connection failure

Also I am able to see the below log in the istio-proxy, does this mean after enabling mTLS, the communication between the envoy and the actual microservice happens in a different way.

[2019-07-24 13:34:39.708][27][warning][filter] [src/istio/mixerclient/report_batch.cc:106] Mixer Report failed with: UNAVAILABLE:upstream connect error or disconnect/reset before headers. reset reason: connection failure

4

Have you enabled destination rules ? Can you enable debug level for mixer logs.

5

Hi Adheip, Yes I have enabled destination rule only then I was able to connect to my https service else I can’t connect to the service even if mTLS is not enabled.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: test-http-dr
spec:
host: isamconfig-service
trafficPolicy:
tls:
mode: SIMPLE

6

So then in this case , i assume you are able to reach to your istio-proxy and its throughing an upstream error, i have faced upstreams errors a lot, so see if the following helps :-

  1. Proxy is able to connect to your application service. ( inside your proxy curl service.namespace.svc.cluster.local:yourport )
  2. Inside your proxy curl localhost:15000/config_status -s, you can see how envoy is connecting to the proxy.
  3. Inside your service deployment maifest, enable istio-proxy logs to debug level and see the error logs.
  4. Make sure your service in container is running and all the ports, target ports are rightly mentioned.
7

Hi Adheip Singh, Thanks for the reply, Below is the case now .

Only if I enable the destination rule for tls mode to be simple, I am able to access my HTTPS application atleast without mTLS.

Dest Rule:
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: test-http-dr
spec:
host: isamconfig-service
trafficPolicy:
tls:
mode: simple.

But when I enable mTLS the above solution itself causing probelem the reason being it is making the configuration to be conflict

HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
isamconfig-service.apps.svc.cluster.local:9443 CONFLICT mTLS TLS default/apps

The status is conflict since I have enabled the above destination rule. So in this case, if I change the tls mode to ISTIO_MUTUAL.