Since the upgrade from Istio 1.7 to 1.8 we’re seeing issues with specially the communication from our Prometheus to our Alertmanager (pushing alarms via HTTP).
level=error ts=2021-05-31T07:16:45.210Z caller=notifier.go:527 component=notifier alertmanager=http://100.96.9.151:9093/api/v2/alerts count=1 msg="Error sending alert" err="bad response status 404 Not Found"
Both, Prometheus & Alertmanager are installed via the
prometheus-operator and therefore two headless Kubernetes Service exist called
alertmanager-operated , those services are hard-coded into the
prometheus-operator and can not be changed.
Next to it each “installation” of those two components has an own service, which is not headless and under full-control by us. (
One thing we tried and also worked is to down scale the
prometheus-operator to 0 and add the
appProtocol tags on the
-operated Kubernetes services with the value
http which worked, but is overwritten, as long as the operator is not scaled to zero.
If we’re setting the
appProtocol on the by us controlled service, it is not working out.
Any clue how to to fix it?
Version (include the output of
istioctl version --remote and
kubectl version --short and
helm version --short if you used Helm)
➜ istioctl version --remote client version: 1.10.0 control plane version: 1.8.5 data plane version: 1.8.5 (78 proxies)
➜ kubectl version --short Client Version: v1.19.7 Server Version: v1.19.10
How was Istio installed?
istio-operator also in version
Environment where the bug was observed (cloud vendor, OS, etc)
Running on a
kOps cluster on AWS.