Hi,
our goal is to use istio as an ingress for our http(s) traffic with certificates from letsencrypt via cert-manager.
We want to use K8S ingress resources to setup virtual gateway and rules automatically.
We are using helm to setup our resources and most helm charts only support ingress resources out of the box.
Our Setup details you can find at the bottom.
We tried to find documentation about istio and K8s ingress resources, but most of it are github issues. We set global.k8sIngress.enabled in helm values.
To test istio we use the bookinfo sample in default namesapce, where we enable sidecar injection
$ kubectl label namespace default istio-injection=enabled
$ kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/bookinfo/platform/kube/bookinfo.yaml
$ kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/bookinfo/platform/kube/bookinfo-ingress.yaml
$ kc get ingresses.extensions gateway
NAME HOSTS ADDRESS PORTS AGE
gateway * 80 1h
But nothing happened. Ingress resource did NOT get any address. Checking istio-ingress if it’s ok:
$ kubectl -n istio-system get pod istio-ingressgateway-d57f5f484-pcvbk
NAME READY STATUS RESTARTS AGE
istio-ingressgateway-d57f5f484-pcvbk 1/1 Running 0 17h
$ kubectl -n istio-system get service istio-ingressgateway
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-ingressgateway LoadBalancer 100.64.0.151 internal-HASH-NUMBERS.eu-central-1.elb.amazonaws.com 80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:31821/TCP,15030:30521/TCP,15031:32503/TCP,15032:30272/TCP,15443:30582/TCP,15020:31460/TCP 17h
All good so, we debugged the istio-pilot discovery pod and we’ve found no trace that it’s watching for k8s ingress resources. We digged into the code and we realized that the mesh config from the helm template miss some settings ?BUG?.
I manually added the configmap to add 2 parameters to the mesh config and restarted pilot discovery process / container.
$ kubectl -n istio-system edit cm istio
ingressControllerMode: DEFAULT
ingressClass: istio
And the magic convertIngress happened.
$ kc get ingresses.extensions gateway
NAME HOSTS ADDRESS PORTS AGE
gateway * internal-HASH-NUMBERS.eu-central-1.elb.amazonaws.com 80 32m
Questions:
- Is the missing parameters in mesh config we’ve found in helm chart template a bug or desired?
- How to get HTTPS working with ingress resources?
- If we enable
global.k8sIngress.enableHttpsin helm chart, pilot fails since the secret to mount key files doesn’t exist, which is referenced in Gateway. This is desired, since we want to use cert-manager and k8s secrets, not mounted files. https://github.com/istio/istio/blob/master/install/kubernetes/helm/istio/charts/gateways/templates/preconfigured.yaml#L22- port: number: 443 protocol: HTTPS name: https-default tls: mode: SIMPLE serverCertificate: /etc/istio/ingress-certs/tls.crt privateKey: /etc/istio/ingress-certs/tls.key hosts: - "*" - How to setup istio-autogenerated-k8s-ingress to use cert-manager secrets from ingress resources?
- Does istio require to have the cert-manager certificates be created in istio system namespace?
- If we enable
We can succesfully create certificates via cert-manager defined by annotations in ingress resources.
This is how a ingress resource with TLS looks like. (Ports list 80 + 443)
$ kubectl -n monitoring get ingresses.extensions kube-prometheus-prometheus-alertmanager
NAME HOSTS ADDRESS PORTS AGE
kube-prometheus-prometheus-alertmanager alertmanager.sub.domain.com internal-HASH-NUMBERS.eu-central-1.elb.amazonaws.com 80, 443 23h
$ kubectl -n monitoring get ingresses.extensions kube-prometheus-prometheus-alertmanager -oyaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
certmanager.k8s.io/acme-challenge-type: dns01
certmanager.k8s.io/acme-dns01-provider: route53
certmanager.k8s.io/cluster-issuer: letsencrypt-staging
kubernetes.io/ingress.class: istio
creationTimestamp: "2019-03-28T10:39:32Z"
generation: 2
labels:
app: prometheus-operator-alertmanager
chart: prometheus-operator-5.0.3
heritage: Tiller
release: kube-prometheus
name: kube-prometheus-prometheus-alertmanager
namespace: monitoring
resourceVersion: "2042745"
selfLink: /apis/extensions/v1beta1/namespaces/monitoring/ingresses/kube-prometheus-prometheus-alertmanager
uid: c5dda5e2-5145-11e9-b1da-066ff4a1fb28
spec:
rules:
- host: alertmanager.sub.domain.example.com
http:
paths:
- backend:
serviceName: kube-prometheus-prometheus-alertmanager
servicePort: 9093
tls:
- hosts:
- alertmanager.sub.domain.example.com
secretName: alertmanager-tls
status:
loadBalancer:
ingress:
- hostname: internal-HASH-NUMBERS.eu-central-1.elb.amazonaws.com
And the certificate
$ kubectl -n monitoring get certificates alertmanager-tls -o yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
creationTimestamp: "2019-03-28T13:01:36Z"
generation: 1
name: alertmanager-tls
namespace: monitoring
ownerReferences:
- apiVersion: extensions/v1beta1
blockOwnerDeletion: true
controller: true
kind: Ingress
name: kube-prometheus-prometheus-alertmanager
uid: c5dda5e2-5145-11e9-b1da-066ff4a1fb28
resourceVersion: "2003681"
selfLink: /apis/certmanager.k8s.io/v1alpha1/namespaces/monitoring/certificates/alertmanager-tls
uid: 9ec0df5b-5159-11e9-b1da-066ff4a1fb28
spec:
acme:
config:
- dns01:
provider: route53
domains:
- alertmanager.sub.domain.example.com
dnsNames:
- alertmanager.sub.domain.example.com
issuerRef:
kind: ClusterIssuer
name: letsencrypt-staging
secretName: alertmanager-tls
status:
conditions:
- lastTransitionTime: "2019-03-28T13:03:18Z"
message: Certificate is up to date and has not expired
reason: Ready
status: "True"
type: Ready
notAfter: "2019-06-26T12:03:16Z"
$ kubectl -n monitoring get secrets alertmanager-tls
NAME TYPE DATA AGE
alertmanager-tls kubernetes.io/tls 3 20h
Http requests work, but HTTPS won’t until the Gateway is also listening for https:
$ kubectl describe gateways.networking.istio.io istio-autogenerated-k8s-ingress
Name: istio-autogenerated-k8s-ingress
Namespace: istio-system
Labels: app=gateways
chart=gateways
heritage=Tiller
release=istio
Annotations: <none>
API Version: networking.istio.io/v1alpha3
Kind: Gateway
Metadata:
Creation Timestamp: 2019-03-28T14:20:16Z
Generation: 1
Resource Version: 2019287
Self Link: /apis/networking.istio.io/v1alpha3/namespaces/istio-system/gateways/istio-autogenerated-k8s-ingress
UID: 9bbb8910-5164-11e9-b1da-066ff4a1fb28
Spec:
Selector:
Istio: ingressgateway
Servers:
Hosts:
*
Port:
Name: http
Number: 80
Protocol: HTTP2
Events: <none>
Setup
Kubernetes Cluster in AWS setup via KOPS 1.11.1 on EC2
K8S Version = Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.8",
GitCommit:"4e209c9383fa00631d124c8adcc011d617339b3c", GitTreeState:"clean",
BuildDate:"2019-02-28T18:40:05Z", GoVersion:"go1.10.8", Compiler:"gc",
Platform:"linux/amd64"}
Helm Version = v2.13.1
| NAME | REVISION | UPDATED | STATUS | CHART | APP VERSION | NAMESPACE |
|---|---|---|---|---|---|---|
| cert-manager | 9 | Thu Mar 28 14:00:36 2019 | DEPLOYED | cert-manager-v0.7.0 | v0.7.0 | cert-manager |
| external-dns | 11 | Tue Mar 26 16:28:24 2019 | DEPLOYED | external-dns-1.7.0 | 0.5.9 | system-addons |
| istio | 3 | Thu Mar 28 15:46:58 2019 | DEPLOYED | istio-1.1.0 | 1.1.0 | istio-system |
| istio-cni | 22 | Thu Mar 28 15:46:33 2019 | DEPLOYED | istio-cni-0.1.0 | 0.1.0 | istio-system |
| istio-init | 31 | Thu Mar 28 15:46:34 2019 | DEPLOYED | istio-init-1.1.1 | 1.1.1 | istio-system |
| kube-prometheus | 33 | Thu Mar 28 13:50:37 2019 | DEPLOYED | prometheus-operator-5.0.3 | 0.29.0 | monitoring |
| kube2iam | 10 | Tue Mar 26 11:06:41 2019 | DEPLOYED | kube2iam-0.10.0 | 0.10.4 | system-addons |
| metrics-server | 6 | Tue Mar 26 16:30:11 2019 | DEPLOYED | metrics-server-2.5.0 | 0.3.1 | kube-system |
Helm repos (Jenkinsfile snippet)
sh "helm init --client-only"
sh "helm repo add istio.io https://storage.googleapis.com/istio-release/releases/1.1.1/charts"
sh "helm repo update"
sh "helm upgrade --install istio-cni istio.io/istio-cni --namespace ${NAMESPACE} -f provision/k8s/charts/istio/istio-cni-values.yaml"
sh "helm upgrade --install istio-init istio.io/istio-init --namespace ${NAMESPACE} -f provision/k8s/charts/istio/values.yaml"
sh "sleep 20"
sh "helm upgrade --install istio istio.io/istio --namespace ${NAMESPACE} -f provision/k8s/charts/istio/values.yaml"
Helm values for istio + istio-init
---
#
# addon grafana configuration
#
grafana:
enabled: false
#
# addon prometheus configuration
#
prometheus:
enabled: false
# addon Istio CoreDNS configuration
#
istiocoredns:
enabled: false
#
# addon jaeger tracing configuration
#
tracing:
enabled: false
#
# install citadel
#
security:
enabled: true
#
# Istio CNI plugin enabled
# This must be enabled to use the CNI plugin in Istio. The CNI plugin is installed separately.
# If true, the privileged initContainer istio-init is not needed to perform the traffic redirect
# settings for the istio-proxy.
#
istio_cni:
enabled: true
#
# Gateways Configuration, refer to the charts/gateways/values.yaml
# for detailed configuration
#
gateways:
enabled: true
istio-ingressgateway:
enabled: true
serviceAnnotations:
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
istio-ilbgateway:
enabled: false
serviceAnnotations:
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
# Common settings used among istio subcharts.
global:
hub: gcr.io/istio-release
tag: release-1.1-latest-daily
# Use the Mesh Control Protocol (MCP) for configuring Mixer and
# Pilot. Requires galley (`--set galley.enabled=true`).
useMCP: true
k8sIngress:
enabled: true
# Gateway used for k8s Ingress resources. By default it is
# using 'istio:ingressgateway' that will be installed by setting
# 'gateways.enabled' and 'gateways.istio-ingressgateway.enabled'
# flags to true.
gatewayName: ingressgateway
# enableHttps will add port 443 on the ingress.
# It REQUIRES that the certificates are installed in the
# expected secrets - enabling this option without certificates
# will result in LDS rejection and the ingress will not work.
enableHttps: false
sds:
# SDS enabled. IF set to true, mTLS certificates for the sidecars will be
# distributed through the SecretDiscoveryService instead of using K8S secrets to mount the certificates.
enabled: true
udsPath: "unix:/var/run/sds/uds_path"
useTrustworthyJwt: false
useNormalJwt: false
#
# nodeagent configuration
#
nodeagent:
enabled: true
image: node-agent-k8s
env:
CA_PROVIDER: "Citadel"
CA_ADDR: "istio-citadel:8060"
VALID_TOKEN: true