Keycloak bootstrap on a VPN cluster

Companion to netbird-vpn-architecture.md. That doc covers what NetBird + Keycloak give end users; this one covers how kubeaid-cli bootstrap provisions (or wires up) Keycloak's realm

• clients on a VPN cluster, and how workload clusters join that Keycloak.

Update (v0.29.6): the kube-apiserver OIDC integration described in parts of this doc — the --authentication-config / AuthenticationConfiguration delivery, apiServer.oidc, kubelogin for cluster access, and the Obmondo multi-issuer / session-timeout designs — was removed. No cluster's kube-apiserver trusts an OIDC issuer anymore; cluster access is now via the NetBird clusterProxy (see post-bootstrap.md and netbird-vpn-architecture.md). Keycloak remains only as NetBird's SSO IdP. The Keycloak realm/client provisioning below (realm, netbird-client, netbird-backend, the groups/api scopes) is still current; the kube-apiserver-auth sections are retained as historical design context.

VPN clusters can run in either of two Keycloak modes, set via cluster.keycloak.mode in general.yaml:

mode	what kubeaid-cli does	who runs Keycloak
managed	installs Keycloak via the keycloakx chart, generates the admin password, runs the gocloak realm reconciler post-sync	this cluster (kubeaid-cli)
external	configures kube-apiserver / NetBird Mgmt to trust an existing Keycloak; the realm + clients must be set up by hand before bootstrap	operator (somewhere else)

Most of this doc covers managed mode — that's the path most operators take. The External Keycloak mode section calls out exactly which steps differ when the operator brings their own Keycloak.

What this design has to do

When a customer runs kubeaid-cli bootstrap against cluster.type=vpn with managed Keycloak:

1. Stand Keycloak up on the cluster (the existing keycloakx Helm chart) backed by CloudNativePG.
2. Create a realm named after the customer's domain, populated with NetBird's OIDC clients and a kubernetes-<vpn-cluster> client for kubelogin.
3. Hand NetBird Mgmt the credentials it needs to talk to Keycloak.

When workload clusters join later, they get their own kubernetes-<cluster> client added to the same realm — without re-running anything against the Keycloak install.

The decision: kubeaid-cli + gocloak

Several alternatives were considered:

• Crossplane provider-keycloak with Compositions/XRDs — what a peer initially proposed. Heavy: full Crossplane operator + provider package + XRDs + Compositions + a permanent in-cluster admin Secret, just to call Keycloak admin API.
• Official Keycloak Operator — its CRDs (Keycloak, KeycloakRealmImport) only manage Keycloak instances the operator itself deployed. Doesn't work against a Helm-installed Keycloak.
• Archived Red Hat realm-operator — would have fit ("external-Keycloak-friendly per-resource CRDs") but unmaintained since 2022.
• A community Keycloak resource operator — none currently maintained that match the requirements.
• kubeaid-agent calling Keycloak admin API — would put admin credentials in an in-cluster pod 24/7 and expand the agent's blast radius.

The chosen path: kubeaid-cli calls Keycloak admin API directly using Nerzal/gocloak. kubeaid-cli runs on the operator's machine, not in-cluster — admin credentials live in process memory only during the bootstrap operation. No operator, no CRDs, no Crossplane. Fits the existing pattern where kubeaid-cli orchestrates bootstrap by calling external APIs (Hetzner, AWS, Kubernetes).

What gets created in the realm

At VPN-cluster bootstrap (Phase 0)

Resource	Purpose
Realm <derived>	Named after the customer's domain; e.g. keycloak.vpn.acme.com → acme. Derived in kubeaid-cli via golang.org/x/net/publicsuffix.EffectiveTLDPlusOne so multi-part TLDs (.co.uk) resolve correctly. User can override cluster.keycloak.realm.
Client netbird-client (PUBLIC, PKCE)	Browser-based SSO for NetBird's UI / mesh login.
Client netbird-backend (CONFIDENTIAL, serviceAccountsEnabled: true)	NetBird Mgmt uses this client's secret AND its auto-generated service-account user for admin-style calls (looking up users, checking group membership). The client secret is pre-generated by kubeaid-cli and set on the client at create time (gocloak's Client.Secret field) — not auto-generated by Keycloak. Lets kubeaid-cli write the SealedSecret to kubeaid-config in the same git push. The service-account user replaces a dedicated netbird user that the peer's composition created.
Client kubernetes-<vpn-cluster> (PUBLIC, PKCE)	Still created, but vestigial (v0.29.6) — nothing consumes its OIDC tokens. VPN-cluster kube-api access is via the NetBird clusterProxy, not kubelogin.
ClientScope api	Standardised scope assigned to each client.
ClientDefaultScopes (×3)	Each client gets profile, email, roles, web-origins, api.
ProtocolMapper ("Audience for NetBird Management API")	On the api scope; adds the netbird-backend audience claim to tokens.
ClientServiceAccountRole	netbird-backend's service-account → view-users on the built-in realm-management client.
Groups (e.g. k8s-<vpn-cluster>-admin, …)	Seeded so admin RBAC bindings reference real groups from day one. List configurable via cluster.keycloak.groups in general.yaml.

Not created: a dedicated netbird User. The confidential client's service-account user covers it; saves a Secret (netbird-user-init-password) and avoids the chicken-and-egg of where that initial password comes from.

At each workload-cluster bootstrap (Phase 1)

One additional resource added to the parent VPN cluster's realm:

Resource	Purpose
Client kubernetes-<workload-cluster> (PUBLIC, PKCE)	Removed (v0.29.6). Workload clusters don't use Keycloak at all — the config parser rejects a keycloak block on type=workload. Cluster access is via the NetBird clusterProxy (mesh identity → RBAC impersonation), not kubelogin/OIDC.

Locked design decisions

Keycloak install: existing keycloakx Helm chart, no chart changes

The kubeaid/argocd-helm-charts/keycloakx chart already does everything we need: Keycloak Deployment, dependent CNPG Cluster CR (rendered via kubeaid-addons/templates/postgresql-cluster.yaml), ingress hooks, monitoring probes. No new chart, no Keycloak-Operator switch, no Helm hook to add.

Run-once secrets: pre-generated by kubeaid-cli, single git push

All run-once values needed by Phase 0 are generated by kubeaid-cli upfront, in process memory on the operator's laptop, before anything is pushed to kubeaid-config:

• Keycloak admin password (32 bytes random)
• netbird-backend client secret (32 bytes random — set on the Keycloak client during reconcile, not auto-generated by Keycloak)
• DATASTORE_ENCRYPTION_KEY, RELAY_PASSWORD, TURN_SERVER_PASSWORD, etc. — every key the netbird chart references via envFromSecret

Two delivery paths, by chart convention:

1. keycloak-admin Secret — keycloakx chart references this directly via secretKeyRef{name: keycloak-admin, key: KEYCLOAK_PASSWORD} in the Pod env. kubeaid-cli kubectl applys it to the main cluster before keycloakx syncs. Fast first-deploy without a sealed-secrets round-trip.
2. netbird Secret (with all the IDP/TURN/RELAY/etc. keys) — netbird chart references via envFromSecret: netbird/<key> (resolved by chart templates to secretKeyRef{name: netbird, key: <key>}). kubeaid-cli writes a SealedSecret named netbird into kubeaid-config; sealed-secrets-controller materialises the in-cluster Secret on sync.

Pre-generating up front — particularly the netbird-backend client secret, which gocloak passes to Keycloak via the Client.Secret field on CreateClient — means the SealedSecret in kubeaid-config and the Keycloak client end up with the same value in one git push. No second push to capture-and-write a Keycloak-auto-generated value.

kubeaid-cli keeps all generated values in process memory for the gocloak admin login + reconcile that follow in the same run, then drops them on exit. On re-runs the existing in-cluster Secrets are read back; new values are generated only if missing (idempotent).

None of these are in the customer's secrets.yaml (which is local-only and holds customer-owned credentials). All are backed up via the existing sealed-secrets backup CronJob → DR bucket (pkg/core/bootstrap_cluster.go already triggers this; constants.CRONJobNameBackupSealedSecrets).

Admin API access: port-forward, not public URL

Matches pkg/utils/kubernetes/argo.go's NewArgoCDClient pattern at line 210 (which port-forwards to argocd-server). kubeaid-cli opens a port-forward to the keycloakx Service on port 8080 and points gocloak at http://localhost:<forwarded-port>. Avoids depending on Traefik / cert-manager / DNS being up during the bootstrap window — port-forward needs only the Keycloak Pod and Service. Keeps admin traffic off the public LB.

The public https://<keycloak-dns>/ URL stays in use for kubelogin and end-user browsers — it's just not in the kubeaid-cli admin path.

Reconciliation: idempotent, on-demand, not continuous

Every Reconcile<Resource> function follows the pattern read; if missing, create; if present, no-op or update. Re-running kubeaid-cli bootstrap is safe and converges. No continuous in-cluster reconciler — drift via direct Keycloak UI edits is not auto-corrected. Acceptable because operators are the only authorised mutators of Keycloak state at Obmondo's scale; if drift becomes a problem, a CronJob running kubeaid-cli reconcile-keycloak on a schedule can be added later.

kube-apiserver --authentication-config: delivered via CAPI files

The OIDC config kube-apiserver consumes is rendered into the existing apiServer.files / apiServer.extraArgs / apiServer.extraVolumes schema in pkg/config/general.go — same pattern as the existing audit-policy delivery (pkg/config/parser/audit_logging.go). CAPI's KubeadmControlPlane.spec.kubeadmConfigSpec carries them via cloud-init. Day-2 changes go through git → ArgoCD → CAPI rolling update. No DaemonSet, no ConfigMap, no inotify polling.

The trade-off: rolling-update latency (~10-15 min on a 3-node HA control plane; brief downtime on single-node CP) instead of k8s 1.30's hot-reload. Acceptable for low-frequency AuthenticationConfiguration changes. Emergency revocation: SSH to control-plane nodes and edit /etc/kubernetes/auth-config.yaml directly — kube-apiserver hot-reload (1.30+) picks it up immediately; commit the same change to kubeaid-config so the next CAPI rolling update doesn't revert it.

Phase 0 — VPN-cluster bootstrap

1. Provision Hetzner infra
2. Create + setup k3d management cluster
3. CAPI provisions main (Hetzner) cluster

4. kubeaid-cli generates ALL run-once secrets in process memory:
     - keycloak-admin password
     - netbird-backend client secret  (pre-generated; SET on Keycloak in step 11)
     - DATASTORE_ENCRYPTION_KEY, RELAY_PASSWORD, TURN_SERVER_PASSWORD, ...

5. kubeaid-cli kubectl-applies Secret keycloak-admin on main cluster
   (chart references it directly via secretKeyRef)

6. kubeaid-cli writes ONCE to kubeaid-config (single git push):
     - SealedSecret "netbird"   (with all the IDP/TURN/RELAY/etc. keys
                                  including the pre-generated client secret)
     - argocd-apps/values-cloudnative-pg.yaml
     - argocd-apps/values-keycloakx.yaml
     - argocd-apps/values-netbird.yaml      (single file for the entire
                                              NetBird stack — Mgmt, Signal,
                                              Relay, Dashboard, coturn,
                                              Postgres via kubeaid-addons)
   git commit + push (SINGLE PUSH)

7. kubeaid-cli syncs apps in dependency order (not relying on ArgoCD waves):
     a. cnpg                 → CNPG operator running
     b. sealed-secrets       → SealedSecret "netbird" decrypted, in-cluster
                                Secret "netbird" exists with all keys
     c. keycloakx            → Cluster keycloak-pgsql up, Keycloak up
     wait for Healthy after each.

8. Port-forward to keycloakx Service (port 8080) via main cluster's kubeconfig

9. Probe http://localhost:<fwd>/.well-known/openid-configuration
   (master realm — sanity check, Keycloak is alive)

10. gocloak.LoginAdmin("admin", <generated-password>, "master") → admin token

11. Reconcile in realm <derived> via gocloak:
      Realm <derived>
      Client netbird-client       (PUBLIC, PKCE)
      Client netbird-backend      (CONFIDENTIAL, SA enabled,
                                    Secret: <pre-generated value>)
                                   ↑ we pass it; Keycloak doesn't auto-generate.
                                     Same value already in the SealedSecret
                                     synced in step 7b.
      Client kubernetes-<vpn>     (PUBLIC, PKCE)
      ClientScope api
      ClientDefaultScopes         (each of the three clients)
      ProtocolMapper              ("Audience for NetBird")
      ClientServiceAccountRole    (netbird-backend SA → view-users on realm-management)
      Groups                      (k8s-<vpn>-admin, ... seeded list)

12. Probe http://localhost:<fwd>/realms/<derived>/.well-known/openid-configuration
    (the new realm is reachable)

13. kubeaid-cli syncs the netbird ArgoCD app (the single-chart NetBird stack
    that includes Mgmt, Signal, Relay, Dashboard, coturn, Postgres):
      Cluster netbird-pgsql up (via kubeaid-addons postgresql-cluster template)
      NetBird pods start; envFromSecret resolves IDP_CLIENT_MGMT_SECRET et al.
      from in-cluster Secret "netbird" (already there from step 7b)
      management.json template substitutes the values; client_credentials
      grant against Keycloak succeeds (the pre-generated secret matches what
      gocloak set on netbird-backend in step 11) ✓

14. Trigger sealed-secrets backup CronJob → DR bucket

Phase 1 — Workload-cluster bootstrap

1. Pre-flight: probe https://<parent-vpn-dns>/realms/<parent-realm>/.well-known/openid-configuration
2. Provision Hetzner infra
3. CAPI / kubeadm-init  (kube-apiserver starts with --authentication-config wired
                          via apiServer.files; trusts parent's realm)
4. Read parent VPN cluster's kubeconfig (operator's laptop already has it)
5. kubectl get secret keycloak-admin -n keycloakx --context=<parent>
6. Port-forward to parent's keycloakx Service via parent's kubeconfig
7. gocloak.LoginAdmin against http://localhost:<fwd>/
8. ReconcileClient(realm=parent.realm, spec={
     clientID:    "kubernetes-<this-cluster>",
     publicClient: true,
     redirectURIs: ["http://localhost:8000"],
   })
9. Render kubeaid-config (workload apps; no Keycloak install)
10. ArgoCD syncs apps; NetBird host agents on every node join parent's mesh

External Keycloak mode

When the operator already runs Keycloak elsewhere (corporate IdP, existing SSO instance), set cluster.keycloak.mode: external in general.yaml. kubeaid-cli still installs the surrounding stack — cnpg (for NetBird's Postgres), traefik, the LE ClusterIssuer, the netbird + netbird-turn-credentials SealedSecrets, the NetBird Mgmt + Signal + Relay + Dashboard + Coturn ArgoCD app — but skips everything that talks to Keycloak as an installer or admin:

	managed	external
keycloakx chart rendered	✓	—
keycloak-admin SealedSecret	✓	—
Post-sync gocloak realm reconcile	✓	—
netbird Secret rendered	✓	✓
postgres DSN patch	✓	✓
NetBird Mgmt + Signal + Relay + Dashboard + Coturn	✓	✓
traefik + cert-manager LE issuer	✓	✓

Operator prerequisites

The realm + OIDC clients have to exist in the external Keycloak before bootstrap runs — otherwise NetBird Mgmt can't complete its client_credentials flow and kube-apiserver rejects every JWT for an unknown issuer. The exact list of resources is documented next to the chart that consumes them:

kubeaid/argocd-helm-charts/netbird/README.md — "Keycloak realm prerequisites"

That section enumerates the three OIDC clients (netbird-client public PKCE, netbird-backend confidential + service-account-enabled, kubernetes-<cluster-name> public PKCE), the api client scope, the audience mapper, and the view-users role grant on netbird-backend's service account — i.e. exactly what the gocloak reconciler creates in managed mode.

Configuring kubeaid-cli for external mode

In general.yaml:

cluster:
  type: vpn
  acmeEmail: ops@acme.com
  keycloak:
    mode: external
    dns: auth.acme.com         # operator's existing Keycloak public DNS
    realm: acme                # the realm they created the clients in
  netbird:
    dns: netbird.vpn.acme.com  # NetBird Mgmt's public DNS (this cluster)

In secrets.yaml — kubeaid-cli has no way to mint or look up the client secret in an external Keycloak, so the operator hands it over:

keycloak:
  netBirdBackendClientSecret: <copy from external Keycloak admin UI>

The validator rejects bootstrap with a clear error when this is missing; no chance of a silent "everything synced but NetBird Mgmt can't talk to Keycloak" failure mode.

Phase 0 differences

Steps 4-12 of the managed-mode sequence collapse: kubeaid-cli skips the keycloakx app render, the admin Secret, the port-forward to the in-cluster Keycloak, and the gocloak Reconcile* calls. Step 6's git push still goes out (with the netbird SealedSecret using the operator-supplied client secret), step 7 syncs cnpg + traefik + the NetBird ArgoCD app, and step 13's NetBird Mgmt comes up directly against the external Keycloak.

Phase 1 (workload clusters joining the mesh)

Identical to managed mode in shape — the workload cluster's kube-apiserver trusts the same external Keycloak issuer URL set in apiServer.oidc. kubeaid-cli does NOT port-forward into the parent's Keycloak; the operator instead creates the workload's kubernetes-<cluster> OIDC client manually in their external Keycloak (per the realm-prerequisites doc) before running the workload bootstrap.

Phase 2 — User onboarding (operator-driven, no kubeaid-cli code)

1. Operator → https://<keycloak-dns>/admin
   logs in as `admin` with the password kubeaid-cli generated in Phase 0
2. Create user (e.g. alice@acme.com), or — if upstream IdP federation is on —
   the user signs in via Google/Azure once and is auto-created
3. Assign Keycloak group(s):
     k8s-acme-prod-dev      → access to acme-prod
     k8s-acme-vpn-admin     → admin on the VPN cluster
   ...
Done.

What enforces it:

• Mesh access: NetBird Mgmt is configured with groupsClaim: groups. On netbird up, the JWT carries groups: ["k8s-acme-prod-dev", ...]. NetBird derives membership from the JWT — it doesn't keep a parallel NetBird-side group assignment. NBPolicy references the group name directly.
• kube-api access: kube-apiserver's AuthenticationConfiguration reads the same groups claim. ClusterRoleBindings reference the group name directly.

Same group name appears at two enforcement points (NetBird NBPolicy, k8s ClusterRoleBinding), but membership is assigned in only one place (Keycloak). Single source of truth.

For the operator's own first onboarding: log in as admin once, create your own user (e.g. bob@acme.com), assign yourself to k8s-<vpn>-admin, then use that user for all subsequent kube-api access. Reserve admin for Keycloak admin operations only.

Implementation surface

The plan is implemented entirely in this repo (kubeaid-cli); the kubeaid chart repo needs no changes.

New code

Path	Purpose
pkg/keycloak/ (new package)	gocloak-driven idempotent reconciler. One file per resource type — realm.go, client_resource.go, scope.go, mapper.go, role.go — plus top-level entry points netbird.go and kubernetes.go.
pkg/config/parser/keycloak.go	hydrateKeycloakDefaults() (publicsuffix-based realm derivation) + validateKeycloakConfig() (mode=managed requires cluster.type=vpn).
pkg/core/templates/argocd-apps/templates/keycloakx.yaml.tmpl	Conditional ArgoCD Application gated on cluster.type=vpn AND cluster.keycloak.mode=managed.
pkg/core/templates/argocd-apps/values-keycloakx.yaml.tmpl	Per-cluster values (DNS, ingress, admin Secret name).

Modified code

Path	Change
pkg/config/general.go	Add KeycloakConfig struct + Cluster.Keycloak *KeycloakConfig field.
pkg/config/parser/parse.go	Call hydrateKeycloakDefaults(); auto-derive apiServer.oidc.issuerUrl from keycloak.dns + realm when unset.
pkg/config/parser/keycloak.go	Render the AuthenticationConfiguration YAML body into apiServer.files instead of emitting --oidc-* flags; set --authentication-config in apiServer.extraArgs; mount via apiServer.extraVolumes.
pkg/config/parser/keycloak.go	Skip the pre-bootstrap probe when keycloak.mode=managed (the issuer doesn't exist yet).
pkg/core/bootstrap_cluster.go	After SyncAllArgoCDApps, when managed Keycloak is enabled: wait for keycloakx Healthy, port-forward, gocloak login, reconcile NetBird's resources, capture netbird-backend secret, write the consumer Secret, probe the new realm.
go.mod	github.com/Nerzal/gocloak/v13, golang.org/x/net/publicsuffix.

Branch sequence

The work ships as a stack of small PRs:

#	Branch	Scope
1	feat/keycloak-config-schema	Schema + parser defaults for cluster.keycloak.{mode,dns,realm,groups}. Realm derivation + validation. No bootstrap behaviour change.
2	feat/keycloak-argocd-apps	Conditional ArgoCD app rendering for keycloakx + cloudnative-pg. Bootstrap renders them; ArgoCD syncs them; Keycloak comes up. Still no realm/client management.
3	feat/keycloak-admin-secret-bootstrap	kubeaid-cli pre-creates the keycloak-admin Secret so the chart finds it on first sync. Idempotent.
4	feat/keycloak-reconciler	The pkg/keycloak/ package itself: gocloak-driven Reconcile* functions, unit-tested via httptest.NewServer. No bootstrap-pipeline integration yet.
5	feat/oidc-render-auth-config	Switch pkg/config/parser/oidc.go to render --authentication-config YAML instead of --oidc-* flags. Reuses existing OIDCConfig schema.
6	feat/keycloak-bootstrap-integration	Wire the reconciler into BootstrapCluster: port-forward, login, reconcile NetBird's resources, capture the client secret.
7	feat/keycloak-workload-client	Workload-cluster bootstrap port-forwards to the parent's Keycloak and applies a kubernetes-<this-cluster> client.

Out of scope — captured for future work

• Multi-issuer kube-apiserver auth (Obmondo SRE access). When obmondo.monitoring: true, the AuthenticationConfiguration gains a second jwt: entry trusting Obmondo's Keycloak realm in addition to the customer's. Trust lives only on kube-apiserver — no IdP federation, no second login button on customer's login screen. Setting obmondo.monitoring: false removes it via the same render-and-redeploy path. Customer or Obmondo can flip the switch; same code path either way.
• Customer session-timeout enforcement on Obmondo SRE tokens (raised by the security team for ISO 27001 A.9). AuthenticationConfiguration's claimValidationRules (CEL) can reject Obmondo-issued tokens older than a customer-configured cap regardless of Obmondo's Keycloak settings. Customer enforcement is at acceptance, not issuance.
• Upstream IdP federation in Keycloak (Google / Azure / Okta as upstream IdPs in the customer realm). gocloak's IdentityProvider methods cover this; expose a cluster.keycloak.upstreamIdP field and reconcile.
• Operator's first realm user via CLI. Currently operator clicks through Keycloak admin UI; could be a kubeaid-cli subcommand (or prompt step) that creates the user via gocloak.
• Drift correction. If continuous reconciliation is needed (e.g. customers manually editing Keycloak via UI), wrap kubeaid-cli reconcile-keycloak in a CronJob. No new operators.

Reference

• netbird-vpn-architecture.md — what NetBird + Keycloak give end users; this doc fills in how Keycloak is provisioned.
• config-reference.md — schema reference (auto-generated; will include cluster.keycloak after Branch 1 lands).
• pkg/utils/kubernetes/argo.go:210 — port-forward pattern (NewArgoCDClient) we mirror for Keycloak admin access.
• pkg/config/parser/audit_logging.go — pattern for apiServer.files / extraArgs / extraVolumes delivery via CAPI; AuthenticationConfiguration follows the same path.
• Nerzal/gocloak — Go client for Keycloak admin API.
• kubeaid/argocd-helm-charts/netbird/README.md — Keycloak realm prerequisites for mode=external; lists the exact OIDC clients, scopes, mappers, and role grants the operator must create by hand.
• Plan source: /home/ashish/.claude/plans/or-user-can-also-lively-sloth.md (local; this doc is the in-repo committed version).