Istio Canary Deploy - Don't send traffic to version with unhealthly pods?


#1

Hello Guys,

Today I works to implement istio in production but i have a problem,
because i dont find the answer in istio docs.

When i configure istio canary, this works fine and the traffic is distribute correctly but if I scale v2 to 0 containers, the istio canary continue send traffic to the service without containers.
I trying implement circuit breaking but don’t work for me.

EXAMPLE USE 2 differents services:

---
apiVersion: v1
kind: Service
metadata:
  name: hello-dev-stable
  namespace: hello-dev
  labels:
    app: 'hello'
    env: 'dev'
    version: 'stable'
  annotations: { }
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 80
  selector:
    app: 'hello'
    env: 'dev'
    version: 'stable'
---
apiVersion: v1
kind: Service
metadata:
  name: hello-dev
  namespace: hello-dev
  labels:
    app: 'hello'
    env: 'dev'
    version: 'canary'
  annotations: { }
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 80
  selector:
    app: 'hello'
    env: 'dev'
    version: 'canary'

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: hello-dev
  namespace: hello-dev
spec:
  gateways:
  - hello-dev
  hosts:
  - 'hello-dev.prd-k8s.pepito.com'
  http:
  - match:
    - headers:
       user-x:
        regex: ".*Eze.*"
    route:
    - destination:
        host: hello-dev-stable
        subset: v1
      weight: 70
    - destination:
        host: hello-dev
        subset: v2
      weight: 30
  - route:
    - destination:
        host: hello-dev-stable
        subset: v1


---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: hello-dev-stable
  namespace: hello-dev
spec:
  host: hello-dev-stable
  trafficPolicy:
    outlierDetection:
      consecutiveErrors: 1
      interval: 1s
      baseEjectionTime: 1h
      maxEjectionPercent: 100
  subsets:
  - name: v1
    labels:
      version: stable

---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: hello-dev
  namespace: hello-dev
spec:
  host: hello-dev
  trafficPolicy:
    outlierDetection:
      consecutiveErrors: 1
      interval: 1s
      baseEjectionTime: 1h
      maxEjectionPercent: 100
  subsets:
  - name: v2
    labels:
      version: canary

➜  ~ k scale --replicas=0 deployment/hello-dev -n hello-dev

➜  ~ k get po -n hello-dev

NAME                                READY     STATUS    RESTARTS   AGE

hello-dev-stable-6667775dc8-6ndtt   2/2       Running   0          11h

➜  ~ k get deploy -n hello-dev

NAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE

hello-dev          0         0         0            0           12h

hello-dev-stable   1         1         1            1           15h

➜  ~ for i in {1..1000}; do curl hello-dev.prd-k8s.pepito.com/api/helloprim-ms/ -H"user-x: Eze" ; done

stable
no healthy upstreamstable
stable
stable
no healthy upstreamstable
stable
stable
no healthy upstreamstable

I trying the replicate the problem using one host(service) and 2 host differents.

EXAMPLE USE 1 service same host:

---
apiVersion: v1
kind: Service
metadata:
  name: hello-dev
  namespace: hello-dev
  labels:
    app: 'hello'
    env: 'dev'
  annotations: { }
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 80
  selector:
    app: 'hello'
    env: 'dev'

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: hello-dev
  namespace: hello-dev
  labels:
    app: hello
    env: dev
spec:
  gateways:
  - hello-dev
  hosts:
  - 'services.prd-k8s.pepito.com'
  http:
  - match:
    - uri:
        prefix: /api/helloprim-ms/
    route:
    - destination:
        host: hello-dev
        subset: stable
      weight: 40
    - destination:
        host: hello-dev
        subset: canary
      weight: 60
    timeout: 30s
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: hello-dev
  namespace: hello-dev
spec:
  host: hello-dev
  trafficPolicy:
    outlierDetection:
      consecutiveErrors: 1
      interval: 1s
      baseEjectionTime: 1h
      maxEjectionPercent: 100
  subsets:
  - name: stable
    labels:
      version: stable
  - name: canary
    labels:
      version: canary

➜  ~ for i in {1..1000}; do curl services.prd-k8s.pepito.com/api/helloprim-ms/ ; done
no healthy upstreamno healthy upstreamstable
stable
no healthy upstream
stable
no healthy upstream
no healthy upstream
no healthy upstream
stable
stable
stable
stable
no healthy upstream
no healthy upstream
stable
no healthy upstream
stable
stable

It’s possible don’t send traffic to version down using canary??