Monitoring in KubeAid
KubeAid monitoring has two layers:
- Metrics -
kube-prometheus(Prometheus, Alertmanager, Grafana) - Logs - one of OpenObserve, Graylog, or OpenSearch + Kibana
Per-stack setup and operations are documented in each application's Helm chart README (linked below).
Metrics: kube-prometheus
kube-prometheus is the default monitoring stack. It provides:
- Prometheus - scrapes metrics from ServiceMonitors, PodMonitors, and exporters across the cluster
- Alertmanager - routes metric-based alerts to notification channels
- Grafana - dashboards for metrics visualization
Configuration is managed per cluster via Jsonnet (<cluster-name>-vars.jsonnet) and built into Kubernetes manifests.
See Prometheus Configuration for details.
Log monitoring
Log monitoring runs alongside kube-prometheus. Each option handles log ingestion, search, and log-based alerting on
its own - none of them replace Prometheus for metrics.
| Option | Scope | Log collection | Prometheus integration | Application docs |
|---|---|---|---|---|
| OpenObserve | Logs, metrics, and alerts | OpenTelemetry (OTLP) | Pulls metrics and alerts from Prometheus; alerts on logs | openobserve |
| Graylog | Logs only | Fluent Bit, Fluentd, Beats, Syslog, GELF, etc. | None (metrics stay in kube-prometheus) | graylog |
| OpenSearch + Kibana | Logs only (ELK-style) | Fluent Bit, Fluentd, OpenTelemetry Collector, etc. | None (metrics stay in kube-prometheus) | opensearch, opensearch-dashboards |
OpenObserve
OpenObserve uses the OpenTelemetry standard for ingestion. It can ingest logs via OpenTelemetry Collector, pull metrics and alerts from Prometheus, and provide log-based search and alerting.
Graylog
Graylog focuses on log collection and management. Logs are shipped using agents such as Fluent Bit or other supported inputs. Graylog does not integrate with Prometheus for metrics or alerting.
OpenSearch and Kibana
OpenSearch with Kibana (or OpenSearch Dashboards) is an alternative to Graylog: centralized log storage, search, and visualization without a Graylog management layer. KubeAid includes Helm charts for OpenSearch and OpenSearch Dashboards; Kibana itself is not packaged.
Alerting strategy
- Metric alerts - Prometheus rules evaluated by Prometheus, routed by Alertmanager
- Log alerts - evaluated by the log stack (OpenObserve, Graylog, or OpenSearch)
Metrics tell you that something is unhealthy; logs help explain why.
Additional monitoring capabilities
Orphan PVC detection
The orphan-pvc kube-prometheus mixin (in build/kube-prometheus/mixins/orphan-pvc/) fires the
KubeDetectOrphanPvc alert when one or more PersistentVolumeClaims are Bound but not mounted by
any running Pod for over 1 hour. This helps catch orphaned volumes consuming storage unnecessarily.
Enable it in your cluster's Jsonnet vars:
addMixins: {
'orphan-pvc': true,
},
ZFS snapshot replication alerts
The prometheus-linuxaid chart includes alerts for ZFS snapshot replication health on bare-metal
nodes:
| Alert | Fires When |
|---|---|
monitor::system::zfs::snapshot_missing_on_replica | A snapshot exists on the source but is missing on the replica |
monitor::system::zfs::replica_snapshot_stale | The newest replicated snapshot for a dataset is older than 2 hours |
Both alerts fire at critical severity after 30 minutes.
Backup exporter
The obmondo-backup-exporter chart monitors Velero and PostgreSQL backup health via dedicated Prometheus metrics and alerts. See the Backup Exporter guide for details.
Security scan alert duration
The SecurityScanFailing alert duration was changed from 1 hour to 24 hours (commit 6e29177d2)
to reduce noise from transient scan failures. Adjust in your values if you need a different threshold.
Further reading
- Prometheus Configuration - kube-prometheus setup and Jsonnet build
- Prometheus Namespaces - namespace scrape scope
- Pod Autoscaling - HPA with custom metrics from Prometheus
- Backup Exporter - Velero and PostgreSQL backup monitoring