Hi:
I try mtls feature on the local cluster. As following setup:
- install istio with default profile
istioctl install --set profile=default --set values.global.jwtPolicy=first-party-jwt
- create httpbin and sleep on foo namespace
kubectl apply -f <(istioctl kube-inject -f samples/httpbin/httpbin.yaml) -n foo
kubectl apply -f <(istioctl kube-inject -f samples/sleep/sleep.yaml) -n foo
- after that I try accessing httpbin through sleep. everything was ok.
- enable mTLS
kubectl apply -n foo -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: "foo"
namespace: foo
spec:
mtls:
mode: STRICT
EOF
- after enabled mtls I access httpbin through sleep retrurn 503. when I capture package on the httpbin. I found the http-request was plaintext. not encrypted.
.......,GET / HTTP/1.1
host: httpbin:8000
user-agent: curl/8.0.1-DEV
accept: */*
x-forwarded-proto: http
x-request-id: bb7b6009-d8dd-4874-bea3-651b0b9d4a13
x-envoy-peer-metadata: ChkKDkFQUF9DT05UQUlORVJTEgcaBXNsZWVwChoKCkNMVVNURVJfSUQSDBoKS3ViZXJuZXRlcwoZCg1JU1RJT19WRVJTSU9OEggaBjEuMTIuOQrEAQoGTEFCRUxTErkBKrYBCg4KA2FwcBIHGgVzbGVlcAohChFwb2QtdGVtcGxhdGUtaGFzaBIMGgo2NmJiN2Y3NDQ2CiQKGXNlY3VyaXR5LmlzdGlvLmlvL3Rsc01vZGUSBxoFaXN0aW8KKgofc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtbmFtZRIHGgVzbGVlcAovCiNzZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1yZXZpc2lvbhIIGgZsYXRlc3QKGgoHTUVTSF9JRBIPGg1jbHVzdGVyLmxvY2FsCiAKBE5BTUUSGBoWc2xlZXAtNjZiYjdmNzQ0Ni03ZGg5bgoSCglOQU1FU1BBQ0USBRoDZm9vChcKEVBMQVRGT1JNX01FVEFEQVRBEgIqAA==
x-envoy-peer-metadata-id: sidecar~10.244.0.27~sleep-66bb7f7446-7dh9n.foo~foo.svc.cluster.local
x-envoy-attempt-count: 1
x-b3-traceid: 775cee7167c4038301baa745f8b91835
x-b3-spanid: 01baa745f8b91835
x-b3-sampled: 0
I think when I enabled mtls. istio-proxy on sleep would auto convert http to https. But in fact it’s not. any body can help me?
BTW: I also didn’t find dir /etc/certs on istio-proxy.
here is the log of istio-proxy on sleep:
2023-03-22T11:52:27.491217Z info FLAG: --concurrency="2"
2023-03-22T11:52:27.491240Z info FLAG: --domain="foo.svc.cluster.local"
2023-03-22T11:52:27.491246Z info FLAG: --help="false"
2023-03-22T11:52:27.491251Z info FLAG: --log_as_json="false"
2023-03-22T11:52:27.491256Z info FLAG: --log_caller=""
2023-03-22T11:52:27.491262Z info FLAG: --log_output_level="default:info"
2023-03-22T11:52:27.491267Z info FLAG: --log_rotate=""
2023-03-22T11:52:27.491273Z info FLAG: --log_rotate_max_age="30"
2023-03-22T11:52:27.491278Z info FLAG: --log_rotate_max_backups="1000"
2023-03-22T11:52:27.491282Z info FLAG: --log_rotate_max_size="104857600"
2023-03-22T11:52:27.491286Z info FLAG: --log_stacktrace_level="default:none"
2023-03-22T11:52:27.491294Z info FLAG: --log_target="[stdout]"
2023-03-22T11:52:27.491302Z info FLAG: --meshConfig="./etc/istio/config/mesh"
2023-03-22T11:52:27.491306Z info FLAG: --outlierLogPath=""
2023-03-22T11:52:27.491309Z info FLAG: --proxyComponentLogLevel="misc:error"
2023-03-22T11:52:27.491313Z info FLAG: --proxyLogLevel="warning"
2023-03-22T11:52:27.491316Z info FLAG: --serviceCluster="istio-proxy"
2023-03-22T11:52:27.491319Z info FLAG: --stsPort="0"
2023-03-22T11:52:27.491322Z info FLAG: --templateFile=""
2023-03-22T11:52:27.491326Z info FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2023-03-22T11:52:27.491331Z info FLAG: --vklog="0"
2023-03-22T11:52:27.491335Z info Version 1.12.9-d85042dbcde912bd21e26c19f283414fc6346f9f-Clean
2023-03-22T11:52:27.491500Z info Proxy role ips=[10.244.0.27 fe80::90fd:a4ff:fe1d:533] type=sidecar id=sleep-66bb7f7446-7dh9n.foo domain=foo.svc.cluster.local
2023-03-22T11:52:27.491572Z info Apply proxy config from env {}
2023-03-22T11:52:27.493273Z info Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: istio-proxy
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
zipkin:
address: zipkin.istio-system:9411
2023-03-22T11:52:27.493294Z info JWT policy is first-party-jwt
2023-03-22T11:52:27.496066Z info CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2023-03-22T11:52:27.496101Z info Opening status port 15020
2023-03-22T11:52:27.496142Z info Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2023-03-22T11:52:27.496240Z info citadelclient Citadel client using custom root cert: istiod.istio-system.svc:15012
2023-03-22T11:52:27.516821Z info ads All caches have been synced up in 28.822519ms, marking server ready
2023-03-22T11:52:27.517083Z info sds SDS server for workload certificates started, listening on "etc/istio/proxy/SDS"
2023-03-22T11:52:27.517120Z info xdsproxy Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2023-03-22T11:52:27.517136Z info sds Starting SDS grpc server
2023-03-22T11:52:27.517562Z info Pilot SAN: [istiod.istio-system.svc]
2023-03-22T11:52:27.517565Z info starting Http service at 127.0.0.1:15004
2023-03-22T11:52:27.519554Z info Pilot SAN: [istiod.istio-system.svc]
2023-03-22T11:52:27.522088Z info Starting proxy agent
2023-03-22T11:52:27.522126Z info Epoch 0 starting
2023-03-22T11:52:27.522168Z info Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --drain-strategy immediate --parent-shutdown-time-s 60 --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --log-format %Y-%m-%dT%T.%fZ %l envoy %n %v -l warning --component-log-level misc:error --concurrency 2]
2023-03-22T11:52:27.659962Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2023-03-22T11:52:27.692214Z info ads ADS: new connection for node:sleep-66bb7f7446-7dh9n.foo-1
2023-03-22T11:52:27.702423Z info ads ADS: new connection for node:sleep-66bb7f7446-7dh9n.foo-2
2023-03-22T11:52:27.746898Z info cache generated new workload certificate latency=229.651606ms ttl=23h59m59.253120723s
2023-03-22T11:52:27.746929Z info cache Root cert has changed, start rotating root cert
2023-03-22T11:52:27.746943Z info ads XDS: Incremental Pushing:0 ConnectedEndpoints:2 Version:
2023-03-22T11:52:27.747031Z info cache returned workload trust anchor from cache ttl=23h59m59.25297511s
2023-03-22T11:52:27.747054Z info cache returned workload certificate from cache ttl=23h59m59.25294982s
2023-03-22T11:52:27.747167Z info cache returned workload trust anchor from cache ttl=23h59m59.25284273s
2023-03-22T11:52:27.747483Z info ads SDS: PUSH request for node:sleep-66bb7f7446-7dh9n.foo resources:1 size:8.9kB resource:default
2023-03-22T11:52:27.747596Z info ads SDS: PUSH request for node:sleep-66bb7f7446-7dh9n.foo resources:1 size:1.8kB resource:ROOTCA
2023-03-22T11:52:27.747699Z info cache returned workload trust anchor from cache ttl=23h59m59.252313819s
2023-03-22T11:52:27.747750Z info ads SDS: PUSH for node:sleep-66bb7f7446-7dh9n.foo resources:1 size:1.8kB resource:ROOTCA
2023-03-22T11:52:29.374176Z info Readiness succeeded in 1.88534115s
2023-03-22T11:52:29.375586Z info Envoy proxy is ready
2023-03-22T12:23:55.550892Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2023-03-22T12:55:45.055014Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
istio-proxy log of httpbin:
2023-03-22T11:52:05.194145Z info FLAG: --concurrency="2"
2023-03-22T11:52:05.194171Z info FLAG: --domain="foo.svc.cluster.local"
2023-03-22T11:52:05.194178Z info FLAG: --help="false"
2023-03-22T11:52:05.194182Z info FLAG: --log_as_json="false"
2023-03-22T11:52:05.194186Z info FLAG: --log_caller=""
2023-03-22T11:52:05.194189Z info FLAG: --log_output_level="default:info"
2023-03-22T11:52:05.194194Z info FLAG: --log_rotate=""
2023-03-22T11:52:05.194200Z info FLAG: --log_rotate_max_age="30"
2023-03-22T11:52:05.194205Z info FLAG: --log_rotate_max_backups="1000"
2023-03-22T11:52:05.194211Z info FLAG: --log_rotate_max_size="104857600"
2023-03-22T11:52:05.194217Z info FLAG: --log_stacktrace_level="default:none"
2023-03-22T11:52:05.194227Z info FLAG: --log_target="[stdout]"
2023-03-22T11:52:05.194231Z info FLAG: --meshConfig="./etc/istio/config/mesh"
2023-03-22T11:52:05.194234Z info FLAG: --outlierLogPath=""
2023-03-22T11:52:05.194238Z info FLAG: --proxyComponentLogLevel="misc:error"
2023-03-22T11:52:05.194241Z info FLAG: --proxyLogLevel="warning"
2023-03-22T11:52:05.194245Z info FLAG: --serviceCluster="istio-proxy"
2023-03-22T11:52:05.194248Z info FLAG: --stsPort="0"
2023-03-22T11:52:05.194252Z info FLAG: --templateFile=""
2023-03-22T11:52:05.194255Z info FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2023-03-22T11:52:05.194262Z info FLAG: --vklog="0"
2023-03-22T11:52:05.194275Z info Version 1.12.9-d85042dbcde912bd21e26c19f283414fc6346f9f-Clean
2023-03-22T11:52:05.194451Z info Proxy role ips=[10.244.0.213 fe80::6caa:71ff:febf:db30] type=sidecar id=httpbin-7887ccc466-d48lh.foo domain=foo.svc.cluster.local
2023-03-22T11:52:05.194525Z info Apply proxy config from env {}
2023-03-22T11:52:05.195845Z info Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: istio-proxy
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
zipkin:
address: zipkin.istio-system:9411
2023-03-22T11:52:05.195862Z info JWT policy is first-party-jwt
2023-03-22T11:52:05.232301Z info CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2023-03-22T11:52:05.232410Z info Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2023-03-22T11:52:05.232385Z info Opening status port 15020
2023-03-22T11:52:05.232591Z info citadelclient Citadel client using custom root cert: istiod.istio-system.svc:15012
2023-03-22T11:52:05.266217Z info ads All caches have been synced up in 75.954681ms, marking server ready
2023-03-22T11:52:05.267061Z info sds SDS server for workload certificates started, listening on "etc/istio/proxy/SDS"
2023-03-22T11:52:05.267096Z info xdsproxy Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2023-03-22T11:52:05.267107Z info sds Starting SDS grpc server
2023-03-22T11:52:05.267498Z info Pilot SAN: [istiod.istio-system.svc]
2023-03-22T11:52:05.267556Z info starting Http service at 127.0.0.1:15004
2023-03-22T11:52:05.269024Z info Pilot SAN: [istiod.istio-system.svc]
2023-03-22T11:52:05.271146Z info Starting proxy agent
2023-03-22T11:52:05.271180Z info Epoch 0 starting
2023-03-22T11:52:05.271213Z info Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --drain-strategy immediate --parent-shutdown-time-s 60 --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --log-format %Y-%m-%dT%T.%fZ %l envoy %n %v -l warning --component-log-level misc:error --concurrency 2]
2023-03-22T11:52:05.430315Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2023-03-22T11:52:05.459190Z info ads ADS: new connection for node:httpbin-7887ccc466-d48lh.foo-1
2023-03-22T11:52:05.469586Z info ads ADS: new connection for node:httpbin-7887ccc466-d48lh.foo-2
2023-03-22T11:52:05.543775Z info cache generated new workload certificate latency=276.69181ms ttl=23h59m59.456250185s
2023-03-22T11:52:05.543918Z info cache Root cert has changed, start rotating root cert
2023-03-22T11:52:05.543950Z info ads XDS: Incremental Pushing:0 ConnectedEndpoints:2 Version:
2023-03-22T11:52:05.544058Z info cache returned workload trust anchor from cache ttl=23h59m59.455949012s
2023-03-22T11:52:05.544082Z info cache returned workload certificate from cache ttl=23h59m59.455922072s
2023-03-22T11:52:05.544265Z info cache returned workload trust anchor from cache ttl=23h59m59.455749876s
2023-03-22T11:52:05.544607Z info ads SDS: PUSH request for node:httpbin-7887ccc466-d48lh.foo resources:1 size:8.9kB resource:default
2023-03-22T11:52:05.544704Z info ads SDS: PUSH request for node:httpbin-7887ccc466-d48lh.foo resources:1 size:1.8kB resource:ROOTCA
2023-03-22T11:52:05.544836Z info cache returned workload trust anchor from cache ttl=23h59m59.455174897s
2023-03-22T11:52:05.544901Z info ads SDS: PUSH for node:httpbin-7887ccc466-d48lh.foo resources:1 size:1.8kB resource:ROOTCA
2023-03-22T11:52:07.018786Z info Readiness succeeded in 1.827522001s
2023-03-22T11:52:07.019174Z info Envoy proxy is ready
2023-03-22T12:22:38.220740Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2023-03-22T12:51:46.252793Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2023-03-22T13:20:39.209658Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
env:
istio: 1.12.9
k8s: v1.19.4