MicroK8s operability add-on

- 2 minutes read - 398 words

Spent time today yak-shaving which resulted in an unplanned migration from MicroK8s ‘prometheus’ add-on to the new and not fully-documented ‘observability’ add-on:

sudo microk8s.enable prometheus

Infer repository core for addon prometheus
DEPRECATION WARNING: 'prometheus' is deprecated and will soon be removed. Please use 'observability' instead.
...

The reason for the name change is unclear.

It’s unclear whether there’s a difference in the primary components that are installed too (I’d thought Grafana wasn’t included in ‘prometheus’), (Grafana) Loki and (Grafana) Tempo definitely weren’t included and I don’t want them either.

Fundamentally, the stack is represented by kube-prometheus with which I’m familiar so, that’s good.

There’s a namespace change though from monitoring to observability.

This post documents my first experiences with ‘observability’ and tweaking it for my needs.

NOTE After the fact (!) I discovered the installation script (enable) for the ‘observability’ add-on.

Googling for “microk8s observability uninstall grafana”, I found Upgrade required from grafana in observability which led me to the realization that the stack is installed by Helm.

microk8s.helm3 list \
--namespace=observability

NAME           	NAMESPACE    	REVISION	STATUS  	CHART                       	APP VERSION
kube-prom-stack	observability	1       	deployed	kube-prometheus-stack-45.5.0	v0.63.0    
loki           	observability	1       	deployed	loki-stack-2.9.9            	v2.6.1     
tempo          	observability	1       	deployed	tempo-1.0.0                 	2.0.0

NOTE I’ve deleted the UPDATED (date) column from the above for presentation purposes.

This made it simple to remove loki and tempo:

microk8s.helm3 uninstall tempo --namespace=observability
release "tempo" uninstalled

microk8s.helm3 uninstall loki --namespace=observability
release "loki" uninstalled

kube-prometheus-stack values.yaml includes a grafana section:

grafana:
  enabled: true
  ...

The documentation for Multiple releases explains|confirms that it should be able to disable grafana by setting a property during deployment --set=grafana.enabled=false.

But, how to intervene in this deployment when I’m using microk8s.enable observability?

I suspect there’s a better way to do this but, I’m simply re-upping the deployment (thanks to the ) enable script for helping determine the correct way to do this:

VERSION="45.5.0"

NODE_ENDPOINTS=$(\
  kubectl get nodes \
  --output=jsonpath={.items[*].status.addresses[?\(@.type==\"InternalIP\"\)].address} \
  | sed 's/\s\+/,/g') && echo ${NODE_ENDPOINTS}

microk8s.helm3 upgrade kube-prom-stack kube-prometheus-stack \
--namespace=observability \
--set=grafana.enabled=false \
--set=kubeControllerManager.endpoints={${NODE_ENDPOINTS}} \
--set=kubeScheduler.endpoints={${NODE_ENDPOINTS}} \
--repo=https://prometheus-community.github.io/helm-charts \
--version=${VERSION}

NOTE NODE_ENDPOINTS is a comma-separated list of the node IPs and is enclosed within braces ({...}) for the helm set value to represent a list.

Additionally, I decided to truly minimize the stack and (temporarily) delete AlertManager and (permanently) delete Node Exporter:

microk8s.helm3 upgrade kube-prom-stack kube-prometheus-stack \
--namespace=observability \
--set=alertmanager.enabled=false \
--set=grafana.enabled=false \
--set=nodeExporter.enabled=false \
--set=kubeControllerManager.endpoints={${NODE_ENDPOINTS}} \
--set=kubeScheduler.endpoints={${NODE_ENDPOINTS}} \
--repo=https://prometheus-community.github.io/helm-charts \
--version=${VERSION}