Skip to main content

Prometheus Configuration in KubeAid

KubeAid uses kube-prometheus to provide a comprehensive monitoring stack including Prometheus, Grafana, and Alertmanager. This guide explains how to configure monitoring for your cluster.

Overview​

The monitoring stack is configured using Jsonnet files that generate Kubernetes manifests. This approach provides:

  • Customizable configuration per cluster
  • CI/CD integration for automated updates
  • Reusable mixins for common monitoring patterns

Configuration Files​

Cluster Variables File​

Each cluster has a Jsonnet variables file at:

kubeaid-config/k8s/<cluster-name>/<cluster-name>-vars.jsonnet

Example configuration:

{
  // Cluster identification
  cluster_name: 'production',
  cluster_domain: 'example.com',
  
  // kube-prometheus version to use
  kube_prometheus_version: 'v0.18.0',
  
  // Prometheus configuration
  prometheus_replicas: 2,
  prometheus_retention: '30d',
  prometheus_storage_size: '100Gi',
  
  // Alertmanager configuration
  alertmanager_replicas: 3,
  
  // Grafana configuration
  grafana_domain: 'grafana.example.com',
  
  // Namespaces to scrape metrics from
  prometheus_scrape_namespaces: [
    'default',
    'kube-system',
    'monitoring',
    'rook-ceph',
    'logging',
  ],
  
  // Enable custom metrics API (for HPA)
  enable_custom_metrics_apiservice: true,
  
  // Enable mixins
  addMixins: {
    ceph: true,
    sealedsecrets: true,
    etcd: true,
    velero: true,
    'cert-manager': true,
    'orphan-pvc': true,
  },
}

Building Prometheus Manifests​

Prerequisites​

Install the required tools:

# On macOS
brew install bash jsonnet

# On Linux (using Go)
go install github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest
go install github.com/brancz/gojsontoyaml@latest
go install github.com/google/go-jsonnet/cmd/jsonnet@latest

Running the Build​

From the KubeAid repository root:

./build/kube-prometheus/build.sh ../kubeaid-config/k8s/<cluster-name>

This generates manifests in:

kubeaid-config/k8s/<cluster-name>/kube-prometheus/

Applying to Cluster​

Commit the generated manifests and sync via ArgoCD, or apply manually:

kubectl apply -f ../kubeaid-config/k8s/<cluster-name>/kube-prometheus/

Common Configuration Options​

Scraping Additional Namespaces​

Add namespaces to the scrape list:

prometheus_scrape_namespaces: [
  'default',
  'kube-system',
  'monitoring',
  'my-application',  // Add your namespace
],

Custom Alerting Rules​

Add Prometheus adapter rules for custom metrics:

prometheus_adapter_additional_rules: [
  {
    seriesQuery: 'my_custom_metric',
    name: { as: 'custom_metric_rate' },
    resources: {
      overrides: {
        pod: { resource: 'pod' },
        namespace: { resource: 'namespace' },
      },
    },
    metricsQuery: 'rate(my_custom_metric{<<.LabelMatchers>>}[5m])',
  },
],

Custom Grafana Dashboards​

Add your own dashboards:

grafana_dashboards: {
  'Custom Folder': {
    'my-dashboard.json': (import '../dashboards/my-dashboard.json'),
  },
},

See the build/kube-prometheus README for detailed instructions.

Alertmanager Configuration​

Configure alerting channels by creating a sealed secret:

kubectl create secret generic alertmanager-main \
  --dry-run=client \
  --namespace monitoring \
  --from-literal=slack-url='https://your-slack-webhook-url' \
  -o yaml | \
  kubeseal --controller-namespace system \
           --controller-name sealed-secrets \
           --namespace monitoring \
           -o yaml > alertmanager-main.yaml

See alertmanager configuration examples.

CI/CD Integration​

KubeAid supports automatic PR creation when monitoring configurations change.

GitHub Actions​

Set these repository secrets:

VariableDescription
API_TOKEN_GITHUBGitHub PAT with repo permission
OBMONDO_DEPLOY_REPO_TARGETTarget repo (e.g., org/kubeaid-config)
OBMONDO_DEPLOY_REPO_TARGET_BRANCHBranch name (e.g., main)

GitLab CI​

Set these CI/CD variables:

VariableDescription
KUBERNETES_CONFIG_REPO_TOKENGitLab token with api and read_repository
KUBERNETES_CONFIG_REPO_URLFull repo URL

Upgrading kube-prometheus​

To upgrade to a new version:

  1. Update kube_prometheus_version in your vars file:
kube_prometheus_version: 'v0.18.0',  // Update this
  1. Run the build script:
./build/kube-prometheus/build.sh ../kubeaid-config/k8s/<cluster-name>

  1. Review generated changes and commit

  2. Sync via ArgoCD

Troubleshooting​

Build Errors​

If you encounter dependency errors, clean and rebuild:

rm -rf ./build/kube-prometheus/libraries/<version>/
./build/kube-prometheus/build.sh ../kubeaid-config/k8s/<cluster-name>

Missing Metrics​

Verify the namespace is in prometheus_scrape_namespaces and that ServiceMonitors exist:

kubectl get servicemonitors -A

Grafana Password Reset​

GrafanaPod=$(kubectl get pods -n monitoring | grep grafana | awk '{print $1}')
kubectl exec -it $GrafanaPod -n monitoring -- grafana-cli admin reset-admin-password <new-password>

See Also​