KubeAid CLI - Architecture
A GitOps-native, multi-cloud Kubernetes provisioning tool. Go • ClusterAPI • ArgoCD • AWS / Azure / Hetzner / KubeOne
License: Apache-2.0 • Runtime: single kubeaid-cli binary (+ Docker for K3D) • Source: Obmondo/kubeaid-cli
Who should read this
This document is the single entry point for understanding how KubeAid CLI is put together. It is written for three audiences:
- Operators who want to know what the tool does before trusting it with a production cluster.
- Contributors who need to find the right file to edit for a given feature.
- Integrators who are adding a new cloud provider, template, or addon.
If you only want to use the CLI, start at the README. This doc assumes you already understand Kubernetes, ClusterAPI, and GitOps at a conceptual level.
Table of contents
- Introduction
- Architecture at a glance
- Two-binary model
- Bootstrap flow
- Cloud providers
- Configuration system
- Template engine
- GitOps with ArgoCD
- Secrets & identity flow
- Cluster lifecycle
- Codebase map
- Development guide
1. Introduction
KubeAid CLI provisions, upgrades, tests, and recovers production Kubernetes clusters across multiple clouds, from a single YAML config. Everything it does eventually lands as commits in a Git repository called KubeAid Config, and ArgoCD reconciles that repository onto the target cluster.
What makes it different:
- GitOps-native. Day-2 operations are Git commits, not
kubectl apply. - One config, many clouds. The same
general.yamlshape provisions AWS, Azure, Hetzner, or bare-metal clusters. - Opinionated addon stack. Cilium, Cert-Manager, Sealed Secrets, KubePrometheus, Rook-Ceph, Velero - pre-wired.
- Self-managing result. After
clusterctl move, the target cluster manages itself; the bootstrap environment can be thrown away.
Supported providers:
| Provider | Provisioning | Management cluster |
|---|---|---|
| AWS | ClusterAPI (CAPA) | K3D (local) |
| Azure | ClusterAPI (CAPZ) + CrossPlane | K3D (local) |
| Hetzner HCloud | ClusterAPI (CAPH) | K3D (local) |
| Hetzner bare metal | ClusterAPI (CAPH) + Robot API | K3D (local) |
| Hetzner hybrid | ClusterAPI (CAPH) | K3D (local) |
| Generic bare metal | KubeOne | None (direct) |
2. Architecture at a glance
Five diagrams. Each zooms in on one concern; together they cover the whole system.
2.1 System overview
2.2 Bootstrap sequence
2.3 Config → templates → GitOps data flow
3. Binary model
kubeaid-cli is a single self-contained binary. cmd/kubeaid-cli embeds the cmd/kubeaid-core command tree and runs it in-process, so Cluster API, ArgoCD, templating, and the cloud SDKs are all compiled in — there is no separate engine process or container to pull. A second small binary, kubeaid-storagectl, runs on bare-metal hosts to apply storage plans.
| Binary | Role | Where it runs |
|---|---|---|
kubeaid-cli | The whole tool — parses config, renders manifests, drives Cluster API / KubeOne and ArgoCD. The kubeaid-core engine is compiled in. | Operator workstation |
kubeaid-storagectl | Applies storage plans (disk partition, ZFS pool, Ceph prep). | On each bare-metal host |
Operators install only kubeaid-cli. Go, the CAPI providers, kubectl, clusterctl, and helm are vendored into the binary; Docker is required only to run a local K3D cluster (Kubernetes-in-Docker) — and only for the Cluster API clouds, since the bare-metal (KubeOne) path installs directly onto the hosts.
4. Bootstrap flow
kubeaid-core cluster bootstrap is the most important code path. It runs four phases sequentially. Entry point: pkg/core/bootstrap_cluster.go.
Why only Hetzner has a "prerequisite infra" phase
AWS and Azure are provisioned declaratively by ClusterAPI + CrossPlane. Hetzner bare-metal has no such provider - networks, VSwitches, and OS installs must be done imperatively via the Robot API before CAPH can reconcile machines.
The pivot
After Phase 3, the target cluster hosts its own CAPI controllers and ArgoCD. clusterctl move transfers all CAPI custom resources from the K3D management cluster to the target, and the K3D cluster can be disposed.
5. Cloud providers
Every provider implements the CloudProvider interface. The interface is deliberately small - it only covers operations that differ meaningfully between clouds; provisioning is delegated to ClusterAPI/KubeOne, and addons to ArgoCD.
| Provider | Package | Distinctive work |
|---|---|---|
| AWS | pkg/cloud/aws | CloudFormation IAM stack; CAPA provider |
| Azure | pkg/cloud/azure | CrossPlane provisions resource group, VNet, OIDC blob; Workload Identity |
| Hetzner | pkg/cloud/hetzner | Network, VSwitch, NAT GW, failover IP; Robot API for bare-metal OS install and storage plans |
| BareMetal | embedded in core (KubeOne) | No CAPI; kubeone apply runs directly against target hosts, for both bootstrap and Kubernetes version upgrades (see upgrade-bare-metal.md) |
| Local | not applicable | The K3D management cluster is the main cluster |
6. Configuration system
Two YAML files drive everything: general.yaml (shape of the cluster) and secrets.yaml (API tokens, SSH keys). They are parsed, defaults are filled in, the result is validated, and the parsed structs become globals for the rest of the run.
Key files:
- pkg/config/general.go -
GeneralConfigstruct tree. - pkg/config/secrets.go -
SecretsConfigstruct tree. - pkg/config/parser.go - orchestrates parse → defaults → validate → hydrate.
Top-level GeneralConfig fields:
| Field | Purpose |
|---|---|
Git | KubeAid Config repo URL, branch, credentials reference |
Cluster | Name, Kubernetes version, pod/service CIDRs, feature flags |
Cloud | Discriminated union: exactly one of aws/azure/hetzner/bareMetal/local |
Forks | Optional forks of KubeAid/KubeAid-Config repos |
MonitoringSetup | KubePrometheus, Grafana, alerting config |
DisasterRecovery | Velero backup target (S3/Azure Blob) |
Why a two-file split?
general.yaml is expected to land in Git (the KubeAid Config repo itself). secrets.yaml stays on the operator's machine and is referenced via Sealed Secrets inside the cluster.
7. Template engine
Go templates (pkg/templates) embed every YAML manifest the CLI will ever emit. At runtime, templates are rendered with ParsedGeneralConfig as their data context and written into outputs/configs/*.yaml, which is the working copy of the KubeAid Config repo.
Template categories:
| Category | Example output path | Purpose |
|---|---|---|
| CAPI cluster | argocd-apps/capi-cluster/values.yaml | Cluster CR, control-plane, node groups |
| ArgoCD apps | argocd-apps/<addon>/Chart.yaml, values.yaml | One directory per addon |
| Root app | argocd-apps/root/templates/*.yaml | App-of-apps manifest |
| Sealed Secrets | sealed-secrets/*.yaml | Encrypted secret material |
| KubeOne (bare metal) | kubeone.yaml | Direct install manifest |
Rendering is deterministic: re-running bootstrap on the same config regenerates the same files byte-for-byte, which makes the PR workflow reviewable.
8. GitOps with ArgoCD
KubeAid uses the app-of-apps pattern. A single "root" ArgoCD Application manages many child Applications, each representing one addon.
Why app-of-apps? Operators only hand ArgoCD one manifest (root); every other addon is reconciled transitively. Enabling or disabling an addon is a one-line flip in general.yaml.
Related code:
- pkg/kubernetes/argocd - ArgoCD install, client, sync helpers.
- pkg/constants/constants.go -
ArgoCDApp*names. - pkg/core/setup_cluster.go - the sync orchestration.
9. Secrets & identity flow
One of the most important architectural properties of KubeAid CLI is that no long-lived secret is ever committed to Git. There are two parallel flows: user-provided secrets (API tokens, SSH keys, registry creds) go through Sealed Secrets; cloud access for in-cluster controllers goes through provider-native identity (Workload Identity on Azure, kube2iam-brokered IAM roles on AWS, Basic Auth on Hetzner Robot). Both flows converge at the target cluster without leaving plaintext material in the KubeAid Config repo.
Two-plane model:
| Plane | Source | Transit | At rest in cluster |
|---|---|---|---|
| User secrets | secrets.yaml on operator host | Encrypted by kubeseal → Git → ArgoCD | Secret decrypted by controller |
| Cloud identity (AWS) | CLI flags / env | Used to bootstrap the CloudFormation IAM stack; then CAPA instance profiles + kube2iam take over | IAM role assumed via kube2iam pod annotation |
| Cloud identity (Azure) | CLI flags / env | Used to create CrossPlane + UAMI; then Workload Identity | Federated token, no secret stored |
| Cloud identity (Hetzner) | ROBOT_USER/ROBOT_PASSWORD + HCLOUD_TOKEN | Mounted into kubeaid-core; wired to hetzner-robot addon via SealedSecret | Controller reads SealedSecret |
Why this matters:
- Git repo is safe to share. The KubeAid Config repo contains only manifests and ciphertext; operators can open PRs, grant team-wide read access, or mirror it publicly without leaking credentials.
- Rotation is surgical. Rotating a cloud credential means editing
secrets.yamland re-runningbootstrap(or a dedicated rotation command). The sealed-secrets controller's keypair is the only long-lived secret, and it is backed up by thebackup-sealed-secretsCronJob. - Disaster recovery preserves identity. Velero backs up the sealed-secrets controller's private key; a recovered cluster can decrypt the same SealedSecret manifests from Git without operator intervention.
10. Cluster lifecycle
Every lifecycle command has its own entry point under pkg/core. The commands share configuration parsing, template rendering, and ArgoCD sync; they differ in which subset of phases they run.
| Command | Entry point | What it does |
|---|---|---|
cluster bootstrap | bootstrap_cluster.go | Four-phase provision (see §4) |
cluster upgrade | upgrade_cluster.go | Bump K8s version: update values file, recreate MachineTemplates, rolling replace |
cluster test | test_cluster.go | Smoke-test a provisioned cluster (Cilium, DNS, storage) |
cluster delete | delete_cluster.go | Delete Cluster CR, wait for CAPI cleanup, tear down infra |
cluster recover | recover_cluster.go | Restore from Velero backup onto a fresh cluster |
The shared primitives - create dev env, setup cluster, setup KubeAid Config - live alongside them (create_dev_env.go, setup_cluster.go, setup_kubeaid_config.go).
11. Codebase map
kubeaid-cli/
├── cmd/
│ ├── kubeaid-cli/ # Binary entry point (embeds kubeaid-core root)
│ ├── kubeaid-core/ # Engine: Cobra command tree, compiled into kubeaid-cli
│ └── kubeaid-storagectl/ # Bare-metal storage plan executor
├── pkg/
│ ├── core/ # Lifecycle orchestration (bootstrap, upgrade, delete…)
│ ├── cloud/
│ │ ├── aws/ # IAM, CAPA wiring
│ │ ├── azure/ # CrossPlane, OIDC, Workload Identity
│ │ ├── hetzner/ # Network, VSwitch, Robot API, storage plans
│ │ └── cloud_provider.go
│ ├── kubernetes/
│ │ ├── argocd/ # ArgoCD client, install, sync
│ │ ├── capi/ # ClusterAPI helpers, clusterctl move
│ │ └── ...
│ ├── config/ # GeneralConfig, SecretsConfig, parser, validator
│ ├── templates/ # embed.FS of Go templates
│ ├── constants/ # Shared names, env vars, flag names, timeouts
│ ├── globals/ # Process-wide state (parsed configs, CP instance)
│ └── utils/
│ ├── assert/ # Fail-fast helpers (os.Exit on error)
│ ├── git/ # Clone, commit, PR
│ ├── commandexecutor/ # Run external CLIs (kubeone, clusterctl, helm)
│ ├── kubernetes/ # Client factories, apply, wait
│ ├── logger/ # slog setup, context-attached attrs
│ └── templates/ # Template rendering primitives
├── docs/ # This file + per-feature guides
├── tools/generators/ # Code generation for config schema
└── Makefile # Build, lint, image, run targets
Global state lives in pkg/globals/globals.go - intentionally small: the cloud provider instance, parsed configs, the ArgoCD client, and a handful of cloud-specific handles. New state should have a strong reason before going here.
Error handling is fail-fast. pkg/utils/assert wraps assert.AssertErrNil / assert.Assert and exits the process with a structured log line on any unexpected failure. This keeps call sites free of repetitive error-plumbing while still producing readable incident logs.
12. Development guide
12.1 Build
All build targets live in the Makefile. Version, commit, and build date are injected into the binary via -ldflags -X.
| Target | Output |
|---|---|
make build | ./build/kubeaid-cli (CGO-off, ready to ship) |
make build-storagectl | ./build/kubeaid-storagectl |
make lint | golangci-lint run ./... |
make format | golangci-lint fmt - imports, golines, etc. |
make test | Unit tests; writes coverage.out |
make check-coverage | Enforce testcoverage.yaml thresholds |
make addlicense | Adds Apache-2.0 headers to any Go file that lacks one |
make run-generators | Regenerates config artifacts from the struct definitions |
make fetch-k8s-eol | Refreshes embedded Kubernetes end-of-life data |
make management-cluster-delete | Deletes the local K3D management cluster |
12.2 Local dev loop
# 1. Build the CLI.
make build
# 2. Generate a config, then bootstrap against it. The CLI pulls the matching
# kubeaid-core image and runs the engine in a container.
./build/kubeaid-cli config generate --configs-directory ./outputs/configs/<cluster>/
./build/kubeaid-cli cluster bootstrap --configs-directory ./outputs/configs/<cluster>/
# 3. Tear down the local management cluster when done.
make management-cluster-delete
12.3 Coding standards
- Follow Google's Go style decisions and best practices.
- Run
make lintandmake formatbefore pushing; CI is strict. - Fail fast via pkg/utils/assert instead of bubbling errors up through every caller.
- Use
log/slogeverywhere; attach per-entity attrs withlogger.AppendSlogAttributesToCtxso parallel flows remain filterable.
12.4 Contributing
- Open an issue describing the problem or feature.
- Fork the repo and create a topic branch.
- Run
make lintand any relevant tests (go test ./...). - Open a PR that references the issue; describe the why in the body, not just the what.
- For multi-step features, include a short architecture note in docs/