Multiple Issues with Istio 1.1.1

1

This morning I reinstalled our cluster with istio 1.1.1 (1.0.6 previously). The cluster is running with mtls enabled and sds enabled. Initially I was able to successfully test intra cluster communication using the sleep/httpbin example pods. Upon adding more servicesa/apps to the cluster this is no longer the case. I now get:

kubectl -n default exec $(kubectl -n default get pod -l app=sleep -o jsonpath={.items..metadata.name}) -- curl -s http://httpbin.default.svc.cluster.local:8000/ip
Defaulting container name to sleep.
Use 'kubectl describe pod/sleep-88ddbcfdd-fgn45 -n default' to see all of the containers in this pod.
upstream connect error or disconnect/reset before headers%

Secondly, I have the cluster configured to REGISTRY_ONLY to be the same as what it was in 1.0.6. (note installed istio using helm template). Yet with this setting from a pod I can still access external sites without needing a service entry, and in fact having those entries in place was preventing the traffic from working.

Third, in our testing of the installation for istio1.1.1 using helm template, I was never able to create a cluster where SDS would work from scratch. What I mean by this is that using a values file with the desired settings, if no istio mesh was in place, I could not successfully bring up the mesh. If I created a no auth mesh first and then updated it with the sds config, then sds would work.

Here is my values file for sds.

---
certmanager:
  enabled: true
  email: platformengineering@alienvault.com
  podAnnotations:
    iam.amazonaws.com/role: cert-manager
  replicaCount: 2
gateways:
  enabled: true
  istio-ingressgateway:
    enabled: true
    autoscaleMin: 2
    serviceAnnotations:
      service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: 'true'
      service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: '60'
      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '4000'
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
    sds.enabled: true
    ports:
    - name: http2
      port: 80
      targetPort: 80
    - name: https
      port: 443
      targetPort: 443
    - name: tcp
      port: 31400
      targetPort: 31400
    - name: tls
      port: 15443
      targetPort: 15443
    - name: status-port
      port: 15020
      targetPort: 15020
  istio-ilbgateway:
    enabled: true
    autoscaleMin: 2
    sds.enabled: true
    serviceAnnotations:
      service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
      service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: 'true'
      service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: '60'
      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '4000'
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
    ports:
    - name: http2
      port: 80
      targetPort: 80
    - name: https
      port: 443
      targetPort: 443
    - name: tcp
      port: 31400
      targetPort: 31400
    - name: tls
      port: 15443
      targetPort: 15443
    - name: status-port
      port: 15020
      targetPort: 15020
    - name: https-kiali
      port: 15029
      targetPort: 15029
    - name: https-prometheus
      port: 15030
      targetPort: 15030
    - name: https-grafana
      port: 15031
      targetPort: 15031
    - name: https-tracing
      port: 15032
      targetPort: 15032
sidecarInjectorWebhook:
  enabled: true
  rewriteAppHTTPProbe: true
galley:
  enabled: true
  replicaCount: 2
mixer:
  enabled: true
  policy:
    enabled: true
    autoscaleMin: 2
  telemetry:
    enabled: true
    autoscaleMin: 2
pilot:
  enabled: true
  autoscaleMin: 2
security:
  enabled: true
nodeagent:
  enabled: true
  image: node-agent-k8s
  env:
    CA_PROVIDER: Citadel
    CA_ADDR: istio-citadel:8060
    VALID_TOKEN: true
grafana:
  enabled: true
prometheus:
  enabled: true
servicegraph:
  enabled: false
tracing:
  enabled: true
kiali:
  enabled: true
istio_cni:
  enabled: false
istiocoredns:
  enabled: false
global:
  hub: docker.io/istio
  tag: 1.1.1
  monitoringPort: 15014
  k8sIngress:
    enabled: false
    gatewayName: ingressgateway
    enableHttps: false
  proxy:
    image: proxyv2
    clusterDomain: cluster.local
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 2000m
        memory: 128Mi
    concurrency: 2
    accessLogFile: ''
    accessLogFormat: ''
    accessLogEncoding: TEXT
    dnsRefreshRate: 5s
    privileged: false
    enableCoreDump: false
    statusPort: 15020
    readinessInitialDelaySeconds: 1
    readinessPeriodSeconds: 2
    readinessFailureThreshold: 30
    includeIPRanges: "*"
    excludeIPRanges: ''
    kubevirtInterfaces: ''
    includeInboundPorts: "*"
    excludeInboundPorts: ''
    autoInject: enabled
    envoyStatsd:
      enabled: false
      host: 
      port: 
    envoyMetricsService:
      enabled: false
      host: 
      port: 
    tracer: zipkin
  proxy_init:
    image: proxy_init
  imagePullPolicy: IfNotPresent
  controlPlaneSecurityEnabled: false
  disablePolicyChecks: true
  policyCheckFailOpen: false
  enableTracing: true
  tracer:
    lightstep:
      address: ''
      accessToken: ''
      secure: true
      cacertPath: ''
    zipkin:
      address: ''
  mtls:
    enabled: true
  imagePullSecrets: 
  arch:
    amd64: 2
    s390x: 2
    ppc64le: 2
  oneNamespace: false
  defaultNodeSelector: {}
  configValidation: true
  meshExpansion:
    enabled: false
    useILB: false
  multiCluster:
    enabled: false
  defaultResources:
    requests:
      cpu: 10m
  defaultPodDisruptionBudget:
    enabled: true
  priorityClassName: ''
  useMCP: true
  trustDomain: ''
  outboundTrafficPolicy:
    mode: REGISTRY_ONLY
  sds:
    enabled: true
    udsPath: "/var/run/sds/uds_path"
    useTrustworthyJwt: false
    useNormalJwt: true
  meshNetworks: {}
  enableHelmTest: false

And finally, istioctl authn tls-check no longer seems to work:

 istioctl authn tls-check sleep-88ddbcfdd-fgn45.default  httpbin.default.svc.cluster.local
Error: nothing to output
---
with debug on
---
2019-04-02T18:50:16.694061Z	debug	[Config loaded from file /home/rdyer/.kube/config]
2019-04-02T18:50:16.695446Z	debug	[Config loaded from file /home/rdyer/.kube/config]
2019-04-02T18:50:16.696302Z	info	curl -k -v -XGET  -H "Accept: application/json, */*" -H "Authorization: Basic FOOBAR" 'https://api.kube.domain/api/v1/namespaces/istio-system/pods?fieldSelector=status.phase%3DRunning&labelSelector=istio%3Dpilot'
2019-04-02T18:50:17.049084Z	info	GET https://api.kube.domain/api/v1/namespaces/istio-system/pods?fieldSelector=status.phase%3DRunning&labelSelector=istio%3Dpilot 200 OK in 352 milliseconds
2019-04-02T18:50:17.049167Z	info	Response Headers:
2019-04-02T18:50:17.049237Z	info	    Content-Type: application/json
2019-04-02T18:50:17.049314Z	info	    Date: Tue, 02 Apr 2019 18:50:21 GMT
2019-04-02T18:50:17.124839Z	info	Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/istio-system/pods","resourceVersion":"8718457"},"items":[{"metadata":{"name":"istio-pilot-55d5f6464f-dcb8v","generateName":"istio-pilot-55d5f6464f-","namespace":"istio-system","selfLink":"/api/v1/namespaces/istio-system/pods/istio-pilot-55d5f6464f-dcb8v","uid":"af900585-5567-11e9-b7fc-1238749b0fea","resourceVersion":"8698794","creationTimestamp":"2019-04-02T16:52:22Z","labels":{"app":"pilot","chart":"pilot","heritage":"Tiller","istio":"pilot","pod-template-hash":"1181920209","release":"istio"},"annotations":{"scheduler.alpha.kubernetes.io/critical-pod":"","sidecar.istio.io/inject":"false"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"istio-pilot-55d5f6464f","uid":"5601957b-5541-11e9-b7fc-1238749b0fea","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"config-volume","configMap":{"name":"istio","defaultMode":420}},{"name":"istio-certs","secret":{"secretName":"istio.istio-pilot-service-account","defaultMode":420,"optional":true}},{"name":"istio-pilot-service-account-token-qm7tz","secret":{"secretName":"istio-pilot-service-account-token-qm7tz","defaultMode":420}}],"containers":[{"name":"discovery","image":"docker.io/istio/pilot:1.1.1","args":["discovery","--monitoringAddr=:15014","--domain","cluster.local","--secureGrpcAddr","","--keepaliveMaxServerConnectionAge","30m"],"ports":[{"containerPort":8080,"protocol":"TCP"},{"containerPort":15010,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"GODEBUG","value":"gctrace=1"},{"name":"PILOT_PUSH_THROTTLE","value":"100"},{"name":"PILOT_TRACE_SAMPLING","value":"1"},{"name":"PILOT_DISABLE_XDS_MARSHALING_TO_ANY","value":"1"}],"resources":{"requests":{"cpu":"500m","memory":"2Gi"}},"volumeMounts":[{"name":"config-volume","mountPath":"/etc/istio/config"},{"name":"istio-certs","readOnly":true,"mountPath":"/etc/certs"},{"name":"istio-pilot-service-account-token-qm7tz","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"readinessProbe":{"httpGet":{"path":"/ready","port":8080,"scheme":"HTTP"},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":30,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"istio-proxy","image":"docker.io/istio/proxyv2:1.1.1","args":["proxy","--domain","$(POD_NAMESPACE).svc.cluster.local","--serviceCluster","istio-pilot","--templateFile","/etc/istio/proxy/envoy_pilot.yaml.tmpl","--controlPlaneAuthPolicy","NONE"],"ports":[{"containerPort":15003,"protocol":"TCP"},{"containerPort":15005,"protocol":"TCP"},{"containerPort":15007,"protocol":"TCP"},{"containerPort":15011,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"INSTANCE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}}],"resources":{"limits":{"cpu":"2","memory":"128Mi"},"requests":{"cpu":"100m","memory":"128Mi"}},"volumeMounts":[{"name":"istio-certs","readOnly":true,"mountPath":"/etc/certs"},{"name":"istio-pilot-service-account-token-qm7tz","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"istio-pilot-service-account","serviceAccount":"istio-pilot-service-account","nodeName":"ip-10-1-9-84.ec2.internal","securityContext":{},"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"beta.kubernetes.io/arch","operator":"In","values":["amd64","ppc64le","s390x"]}]}]},"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":2,"preference":{"matchExpressions":[{"key":"beta.kubernetes.io/arch","operator":"In","values":["amd64"]}]}},{"weight":2,"preference":{"matchExpressions":[{"key":"beta.kubernetes.io/arch","operator":"In","values":["ppc64le"]}]}},{"weight":2,"preference":{"matchExpressions":[{"key":"beta.kubernetes.io/arch","operator":"In","values":["s390x"]}]}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-04-02T16:52:22Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-04-02T16:52:42Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":null},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-04-02T16:52:22Z"}],"hostIP":"10.1.9.84","podIP":"100.114.202.85","startTime":"2019-04-02T16:52:22Z","containerStatuses":[{"name":"discovery","state":{"running":{"startedAt":"2019-04-02T16:52:23Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"istio/pilot:1.1.1","imageID":"docker-pullable://istio/pilot@sha256:573506316664cad19ed9d2640c52955294811163c4e3c849a305555fca22759c","containerID":"docker://231ace88f34919f9ff92a345ac286a22a5957e748eaf9c986528e5d055c0676d"},{"name":"istio-proxy","state":{"running":{"startedAt":"2019-04-02T16:52:23Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"istio/proxyv2:1.1.1","imageID":"docker-pullable://istio/proxyv2@sha256:ca41c2c715db0bf2369f892451008a631de6af519ad0bebf3e657bcbb5afb9df","containerID":"docker://9657e40f02e5d6f8f27e7e39af6b78b81fe21b30e77179dec200070844746919"}],"qosClass":"Burstable"}},{"metadata":{"name":"istio-pilot-55d5f6464f-s87r7","generateName":"istio-pilot-55d5f6464f-","namespace":"istio-system","selfLink":"/api/v1/namespaces/istio-system/pods/istio-pilot-55d5f6464f-s87r7","uid":"af9d3807-5567-11e9-b7fc-1238749b0fea","resourceVersion":"8698849","creationTimestamp":"2019-04-02T16:52:22Z","labels":{"app":"pilot","chart":"pilot","heritage":"Tiller","istio":"pilot","pod-template-hash":"1181920209","release":"istio"},"annotations":{"scheduler.alpha.kubernetes.io/critical-pod":"","sidecar.istio.io/inject":"false"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"istio-pilot-55d5f6464f","uid":"5601957b-5541-11e9-b7fc-1238749b0fea","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"config-volume","configMap":{"name":"istio","defaultMode":420}},{"name":"istio-certs","secret":{"secretName":"istio.istio-pilot-service-account","defaultMode":420,"optional":true}},{"name":"istio-pilot-service-account-token-qm7tz","secret":{"secretName":"istio-pilot-service-account-token-qm7tz","defaultMode":420}}],"containers":[{"name":"discovery","image":"docker.io/istio/pilot:1.1.1","args":["discovery","--monitoringAddr=:15014","--domain","cluster.local","--secureGrpcAddr","","--keepaliveMaxServerConnectionAge","30m"],"ports":[{"containerPort":8080,"protocol":"TCP"},{"containerPort":15010,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"GODEBUG","value":"gctrace=1"},{"name":"PILOT_PUSH_THROTTLE","value":"100"},{"name":"PILOT_TRACE_SAMPLING","value":"1"},{"name":"PILOT_DISABLE_XDS_MARSHALING_TO_ANY","value":"1"}],"resources":{"requests":{"cpu":"500m","memory":"2Gi"}},"volumeMounts":[{"name":"config-volume","mountPath":"/etc/istio/config"},{"name":"istio-certs","readOnly":true,"mountPath":"/etc/certs"},{"name":"istio-pilot-service-account-token-qm7tz","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"readinessProbe":{"httpGet":{"path":"/ready","port":8080,"scheme":"HTTP"},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":30,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"istio-proxy","image":"docker.io/istio/proxyv2:1.1.1","args":["proxy","--domain","$(POD_NAMESPACE).svc.cluster.local","--serviceCluster","istio-pilot","--templateFile","/etc/istio/proxy/envoy_pilot.yaml.tmpl","--controlPlaneAuthPolicy","NONE"],"ports":[{"containerPort":15003,"protocol":"TCP"},{"containerPort":15005,"protocol":"TCP"},{"containerPort":15007,"protocol":"TCP"},{"containerPort":15011,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"INSTANCE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}}],"resources":{"limits":{"cpu":"2","memory":"128Mi"},"requests":{"cpu":"100m","memory":"128Mi"}},"volumeMounts":[{"name":"istio-certs","readOnly":true,"mountPath":"/etc/certs"},{"name":"istio-pilot-service-account-token-qm7tz","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"istio-pilot-service-account","serviceAccount":"istio-pilot-service-account","nodeName":"ip-10-1-62-112.ec2.internal","securityContext":{},"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"beta.kubernetes.io/arch","operator":"In","values":["amd64","ppc64le","s390x"]}]}]},"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":2,"preference":{"matchExpressions":[{"key":"beta.kubernetes.io/arch","operator":"In","values":["amd64"]}]}},{"weight":2,"preference":{"matchExpressions":[{"key":"beta.kubernetes.io/arch","operator":"In","values":["ppc64le"]}]}},{"weight":2,"preference":{"matchExpressions":[{"key":"beta.kubernetes.io/arch","operator":"In","values":["s390x"]}]}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-04-02T16:52:22Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-04-02T16:52:55Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":null},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-04-02T16:52:22Z"}],"hostIP":"10.1.62.112","podIP":"100.96.16.26","startTime":"2019-04-02T16:52:22Z","containerStatuses":[{"name":"discovery","state":{"running":{"startedAt":"2019-04-02T16:52:42Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"istio/pilot:1.1.1","imageID":"docker-pullable://istio/pilot@sha256:573506316664cad19ed9d2640c52955294811163c4e3c849a305555fca22759c","containerID":"docker://87cf1a12d81d77b6dac54bc5df7662299bb8304bf540adf5b5e1dd12293acae2"},{"name":"istio-proxy","state":{"running":{"startedAt":"2019-04-02T16:52:42Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"istio/proxyv2:1.1.1","imageID":"docker-pullable://istio/proxyv2@sha256:ca41c2c715db0bf2369f892451008a631de6af519ad0bebf3e657bcbb5afb9df","containerID":"docker://6121ea39e6eb7aa94ef22692c4de1287e299a545f1c99f5dbf669dcaa318230f"}],"qosClass":"Burstable"}}]}
2019-04-02T18:50:17.149412Z	info	curl -k -v -XPOST  -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "Authorization: Basic FOOBAR" 'https://api.kube.domain/api/v1/namespaces/istio-system/pods/istio-pilot-55d5f6464f-dcb8v/exec?command=sh&command=-c&command=GODEBUG%3D+%2Fusr%2Flocal%2Fbin%2Fpilot-discovery+request+GET+%2Fdebug%2Fauthenticationz%3FproxyID%3Dsleep-88ddbcfdd-fgn45.default+&container=discovery&container=discovery&stderr=true&stdout=true'
2019-04-02T18:50:17.537412Z	info	POST https://api.kube.domain/api/v1/namespaces/istio-system/pods/istio-pilot-55d5f6464f-dcb8v/exec?command=sh&command=-c&command=GODEBUG%3D+%2Fusr%2Flocal%2Fbin%2Fpilot-discovery+request+GET+%2Fdebug%2Fauthenticationz%3FproxyID%3Dsleep-88ddbcfdd-fgn45.default+&container=discovery&container=discovery&stderr=true&stdout=true 101 Switching Protocols in 387 milliseconds
2019-04-02T18:50:17.537498Z	info	Response Headers:
2019-04-02T18:50:17.537541Z	info	    Connection: Upgrade
2019-04-02T18:50:17.537592Z	info	    Upgrade: SPDY/3.1
2019-04-02T18:50:17.537629Z	info	    X-Stream-Protocol-Version: v4.channel.k8s.io
2019-04-02T18:50:17.537665Z	info	    Date: Tue, 02 Apr 2019 18:50:21 GMT
ERROR: nothing to output

Any assistance would be appreciated.

2

Minor update in regards to issue 1. When I originally tested the service I did not have the policy and dr which are supposed to be in place for mtls. With the below in place, I cannot communicate with the service. This would lead me to believe that mtls with sds is not working as I originally thought. And with the istioctl authn tls-check I dont know how to verify the traffic.

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "default"
  namespace: "default"
spec:
  peers:
  - mtls: {}

---

apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "default"
  namespace: "default"
spec:
  host: "*.default.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
3

Hi Rdyer,
Thanks for your report. Did you get a chance to take a look at this example: https://istio.io/docs/tasks/security/auth-sds/? It shows how to config SDS, with authnPolicy and DestinationRule set properly. It should solve issue 1.

For issue 3, not sure what you mean by "if no istio **mesh** was in place, i could not successfully bring up the **mesh**". Can you explain more on this please?

And for issue 2, I will talk to the team and get back to you soon.

Thanks
Tao
4

RE: 1 . Yes I read the doc which is where I came up with the policy and dr that I posted. And with those in place, I cannot perform the basic sleep/httpbin test. I just get the following:

 kubectl -n default exec $(kubectl -n default get pod -l app=sleep -o jsonpath={.items..metadata.name}) -- curl -s http://httpbin.default:8000/ip
upstream connect error or disconnect/reset before headers%

RE #3: by mesh I mean any istio being installed or not. This may be entirely related to issue #1, so until that is resolved, working this issue is probably irrelevant.

5

More info for #1. Note there are no DRs or policys defined in any namespace. Just the default meshpolicy which is set to permissive.

> cat mtls.yaml
---
apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "httpbin"
  namespace: default
spec:
  targets:
  - name: httpbin
  peers:
  - mtls: {}


---
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "httpbin"
  namespace: default
spec:
  host: "httpbin.default.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
> kubectl -n default exec $(kubectl -n default get pod -l app=sleep -o jsonpath={.items..metadata.name}) -- curl -s http://httpbin.default:8000/ip
Defaulting container name to sleep.
Use 'kubectl describe pod/sleep-88ddbcfdd-x2hml -n default' to see all of the containers in this pod.
{
  "origin": "127.0.0.1"
}
> istioctl authn tls-check sleep-88ddbcfdd-x2hml.default httpbin.default.svc.cluster.local
HOST:PORT                                  STATUS     SERVER        CLIENT     AUTHN POLICY     DESTINATION RULE
httpbin.default.svc.cluster.local:8000     OK         HTTP/mTLS     HTTP       default/         -
> kubectl apply -f mtls.yaml
policy.authentication.istio.io/httpbin created
destinationrule.networking.istio.io/httpbin created
> istioctl authn tls-check sleep-88ddbcfdd-x2hml.default httpbin.default.svc.cluster.local
HOST:PORT                                  STATUS     SERVER     CLIENT     AUTHN POLICY        DESTINATION RULE
httpbin.default.svc.cluster.local:8000     OK         mTLS       mTLS       httpbin/default     httpbin/default
> kubectl -n default exec $(kubectl -n default get pod -l app=sleep -o jsonpath={.items..metadata.name}) -- curl -s http://httpbin.default:8000/ip
Defaulting container name to sleep.
Use 'kubectl describe pod/sleep-88ddbcfdd-x2hml -n default' to see all of the containers in this pod.
upstream connect error or disconnect/reset before headers%
6

Hi Rdyer:
To clarify: let’s put aside mtls.yaml and everything else, does the sds example https://istio.io/docs/tasks/security/auth-sds/#before-you-begin work for you?

 If not, I would try on a new cluster for debugging purpose. And can you provide the output of below commands:

 $ kubectl get meshpolicy -o yaml
 $ kubectl get destinationrule default -n istio-system -o yaml
7

@Tao_Li Perhaps I’m misunderstanding that SDS is a replacement for using key/certs with mtls? Otherwise, why would we ignore my having an mtls policy in place given thats a requirement for mtls?

As I show in my previous comment, if I do not have the mtls policy and DR in place, the connection does work. I believe this would be in the spirit of following the SDS example. But no where in this example is there proof that mtls is working. Nor is there anything in this example which enforces mtls. The default meshpolicy for this example is permissive. The example only shows that the services can talk to one another. In this scenario istioctl shows that the connection is not actually mtls, but just http. Is using istioctl not a valid way to verify mtls comms between services when sds is enabled? Just curling from one service to another within the mesh doesnt prove to me that the comms are mtls. Other than trying to wireshark on my nodes or trusting istioctl, I’m not aware of any way to verify that actual network traffic is TLS.

8

To make sure we are on the same page, can you summarize the command you used to create new mesh as well as to upgrade.
Thanks.

9

helm template istio-init --name istio-init --namespace istio-system -f values.yaml | kubectl apply -f -
helm template istio --name istio --namespace istio-system -f values.yaml | kubectl apply -f -

The values.yaml is in my original comment. It contains the sds configuration. I have another values file(noauth) which contains everything in the above except with mtls disabled and the sds disabled(including nodeagent)

I started with the noauth config, did validation, then applied the sds config both using the command above (just istio template; not init)

10

Do you restart the app after applying sds config? If sidecar was injected with the old configuration (i.e no sds), it will not work with the new one, and vice versa.

11

The app was installed post istio installation.

12

Hi Ryder,
Please provide at least the output of

$ kubectl get meshpolicy -o yaml

If it looks something like

spec:
peers:
- mtls: {}

It means is mTLS enforced, not permissive.