Quiet is not the same as healthy.
kubeaid-cli installs Prometheus, Alertmanager, Grafana and their alert rules on every cluster it builds unless you opt out.
Where a check is opt-in, this page says so.
An alert nobody receives is still silence.
On a cluster kubeaid-cli builds with an Obmondo subscription, Alertmanager sends warning and critical alerts to Obmondo over mutual TLS. Without a subscription, you choose where alerts go.
Watchdog always fires, as a check that the alerting pipeline works. If it stops arriving from a subscribed cluster, Obmondo raises an alert of its own.
Who answers an alert, and how fast, depends on your Obmondo plan.
route: group_by: ['namespace'] group_wait: 10s group_interval: 10s repeat_interval: 1h receiver: 'obmondo' routes: - receiver: 'obmondo' matchers: - 'alertname = Watchdog' repeat_interval: 5mEach layer, and the alert rules that watch it.
Metrics
node-exporter and kube-state-metrics feed Prometheus, and Grafana ships dashboards for cluster, namespace, pod and node resources.
- Watchdog
- TargetDown
Nodes and disks
Alerts on a nearly full filesystem or a degraded software RAID array, with critical RAID disk alerts after 15 minutes.
- NodeFilesystemAlmostOutOfSpace
- NodeRAIDDegraded
- HostRaidDiskFailure
Control plane and workloads
Alerts when the API server is down, a node is not Ready or a pod keeps crash-looping. etcd's rules ship too, but nothing scrapes etcd until you expose etcd's metrics and set etcd_metrics for the cluster.
- KubeAPIDown
- KubeNodeNotReady
- KubePodCrashLooping
Git drift
Warns once an app KubeAid manages has been out of sync with Git, or unhealthy, for two hours.
- ArgoCdAppOutOfSync
- ArgoCdAppUnhealthy
Certificates and volumes
Warns before a cert-manager or kubelet certificate expires, and when a volume claim in a namespace KubeAid manages sits unused for an hour.
- CertManagerCertExpirySoon
- KubeletClientCertificateExpiration
- KubeDetectOrphanPvc
A backup stops running. This is what notices.
Opt-in on a subscribed clusterThe backup exporter covers five backends and ships 18 alert rules. Each backend needs S3 credentials. MongoDB and MariaDB need their collector switched on, and the MongoDB, MariaDB and Sealed Secrets rules need their own switch.
All 18 rules on GitHubkubeaid-backup-exporter: enabled: true exporter: postgres: {s3: {secretName: s3-credentials}} velero: {s3: {secretName: s3-credentials}} mongodb: {enabled: true, s3: {secretName: s3-credentials}} mariadb: {enabled: true, s3: {secretName: s3-credentials}} sealedSecrets: {s3: {bucket: <bucket>, endpoint: <url>, secretName: s3-credentials}} prometheusRule: mongodb: {enabled: true} mariadb: {enabled: true} sealedSecrets: {enabled: true}kubeaid-backup-exporter: enabled: true exporter: postgres: s3: {secretName: s3-credentials} velero: s3: {secretName: s3-credentials} mongodb: enabled: true s3: {secretName: s3-credentials} mariadb: enabled: true s3: {secretName: s3-credentials} sealedSecrets: s3: bucket: <bucket> endpoint: <url> secretName: s3-credentials prometheusRule: mongodb: {enabled: true} mariadb: {enabled: true} sealedSecrets: {enabled: true}A scheduled backup does not run
KubeAid's Velero chart backs up at 03:00, Monday to Saturday. The backup exporter checks the newest backup's age every two hours by default.
The newest backup passes 26 hours
max_rpo is the maximum acceptable recovery point objective (RPO), 26 hours by default, so VeleroBackupExceededRPO fires as critical after five minutes. It can fire up to one exporter check late.
Someone checks the PostgreSQL and Velero backups at once
kubeaid-cli backup status reads the exporter and prints the backup state of each resource. It changes nothing on the cluster.
collected 2h 57m ago: cnpg | velero Operator errors: cnpg: s3_list_failed NAMESPACE RESOURCE TYPE STREAM METHOD LATEST AGE STATUSdemo demo-pgsql cnpg_cluster logical CronJob none collector_error (cronjob_not_found)demo demo-pgsql cnpg_cluster wal Barman 4h 9m healthydemo demo-uploads pvc volume PodVolumeBackup 14h 57m healthymonitoring obs-postgres cnpg_cluster logical cronjob 3d 2h exceeds_rpomonitoring prom-data pvc volume CSISnapshot 14h 57m healthy4 resources: 2 healthy, 1 exceeds_rpo, 1 collector_errorRESOURCE STATUSdemo-pgsql collector_errordemo-uploads healthyobs-postgres exceeds_rpoprom-data healthyRestore, by platform
| Platform | Restore |
|---|---|
| Self-managed Azure | One command, kubeaid-cli cluster recover. It needs a disasterRecovery block. |
| AKS | By hand. kubeaid-cli installs no Velero on AKS, so add KubeAid's Velero chart. |
| Hetzner | By hand: add KubeAid's Velero chart, re-bootstrap with kubeaid-cli, then restore a Velero backup. A recover command is in progress. |
| Self-managed AWS, EKS and generic bare metal | By hand: add KubeAid's Velero chart, re-bootstrap with kubeaid-cli, then restore a Velero backup. |
| PostgreSQL, any platform | Set recover.enabled on the CloudNativePG cluster in kubeaid-addons to restore it from its object-store backup, which is off by default. KubeAid's Velero chart skips the volume data of RabbitMQ, Redis, MongoDB, OpenSearch and CloudNativePG. |
What we do not claim.
No recovery time
Nothing in KubeAid's code or docs times a restore. max_rpo is an alert threshold, not a promise about how much data a restore brings back.
Hubble metrics are exposed, not collected
Cloud and bare-metal clusters kubeaid-cli builds run Cilium with Hubble and the Hubble UI for watching network flows. KubeAid's Prometheus does not collect Hubble's metrics, so no dashboard shows them and no alert fires on a dropped packet.
Monitoring runs one replica
Prometheus and Alertmanager each run one replica by default, so neither is highly available. To run more, set prometheus.replicas or alertmanager_replicas in the cluster's vars file.
Bring us a cluster you are not sure you could restore.
An Obmondo engineer who runs this stack in production will look at what alerts today and what a restore would take.