Hetzner k8s with NetBird-gated kube-api — architecture

Hetzner HCloud cluster (single-node or 3-node HA control plane) where kube-apiserver:6443 is never exposed publicly.
NetBird gates network access; Keycloak is NetBird's SSO IdP; kube-apiserver is reached over the mesh via the netbird-operator clusterProxy — no OIDC on the apiserver.
Outcome for end users: Teleport-style "session with role", built from off-the-shelf parts.

See also: architecture.md for the broader KubeAid CLI architecture.

Context

The setup, at a glance:

• Cluster — Hetzner HCloud Kubernetes. Either a single control-plane node, or a 3-node HA control plane.
• Public surface — kube-apiserver:6443 is sealed off from the public internet. It lives only on the NetBird WireGuard mesh.
• Network gate — NetBird. The laptop must be on the mesh before kube-api is even reachable.
• Identity gate — NetBird + the clusterProxy. Mesh membership authenticates you; the netbird-operator clusterProxy impersonates your NetBird groups into Kubernetes RBAC. kube-apiserver does no JWT/OIDC validation.
• kubectl flow — once the laptop is on the mesh, kubectl talks to the clusterProxy peer; there is no OIDC token — your mesh identity is the credential and the proxy impersonates you into RBAC.

Topology notes:

• Single-node and 3-node HA share the exact same architecture.
• The only thing that changes between them is how many control-plane VMs run the NetBird host agent.
• Workers always run the agent, in either topology.

Already in this repo

• HCloud control-plane LB is created with PublicInterface parameterized (PublicInterface: ptr.To(enablePublicInterface), pkg/cloud/hetzner/loadbalancer.go:46) — VPN clusters pass false to keep the LB private.
• A vpn cluster type is defined in config (pkg/config/general.go:130) — this design wires it up so kubeaid-cli bootstrap with cluster.type=vpn provisions the NetBird/VPN server itself (Phase 0).

Not yet present (this design adds)

• NetBird
• Keycloak (VPN clusters — NetBird's dashboard IdP)
• The netbird-operator clusterProxy for kubectl access

The stacked code branch implements all of them.

Three planes

The whole design splits into three independent planes, each with one responsibility:

1. NetBird / VPN Server — the only thing publicly reachable.

• Separate VM, own failure domain (so a workload-cluster outage can't take down auth).
• Provisioned by kubeaid-cli with cluster.type=vpn.
• Sits behind one Hetzner LB + Traefik (TLS termination + path routing).
• Hosts Keycloak (OIDC IdP) + NetBird Mgmt / Signal / Relay.

2. Workload k8s cluster — the actual Kubernetes cluster, with no public surface.

• 1 or 3 control-plane nodes + N workers, all in a Hetzner private network.
• NetBird host agent on every node (control-plane and workers).
• NetBird Operator in-cluster — exposes kube-apiserver as a NetBird mesh resource.
• kube-apiserver reachable only through the mesh; the netbird-operator clusterProxy proxies kubectl to it under an impersonated identity. Cert SANs include k8s-X.netbird (the mesh DNS name).

3. User laptop — the developer's workstation.

• NetBird client — joins the WireGuard mesh on netbird up.
• kubectl — talks to the netbird-operator clusterProxy peer over the mesh; the proxy impersonates the caller's NetBird identity and maps NetBird groups to RBAC. No OIDC, no login helper on the kubectl path.

What NetBird gives us natively (and what it doesn't)

NetBird has an official Kubernetes Operator. It covers the network side cleanly but stops short of identity.

What we use from it:

• Host agent on every node — joins each node to the mesh via a setup key (installed on master + workers via cloud-init).
• netbird.io/expose: "true" Service annotation — Operator publishes kubernetes.default as a mesh DNS resource (k8s-X.netbird).
• NBPolicy CRD — declarative ACLs: group k8s-cluster-X-* reaches resource k8s-X.netbird.
• NBSetupKey CRD + sidecar injection — annotate a pod with netbird.io/setup-key to inject a mesh sidecar. Optional, for pods that need outbound mesh egress.

What's missing — the clusterProxy fills it:

• No kubectl access path — the netbird-operator clusterProxy proxies kubectl over the mesh, impersonating the caller's NetBird identity.
• No k8s RBAC integration — NetBird group membership maps to standard ClusterRoleBindings via the proxy's impersonation.

NetBird = network gate + kubectl identity. Keycloak remains the NetBird dashboard's SSO IdP on VPN clusters.

Network diagrams

Two diagrams. The first shows the precondition (get on the NetBird mesh — done once per session). The second shows the main flow (kubectl over the mesh via the netbird-operator clusterProxy).

Diagram 1 — NetBird mesh login (precondition)

netbird up puts the laptop on the WireGuard mesh by SSO-ing the user against Keycloak. After this step the laptop has a NetBird IP and can resolve / reach k8s-X.netbird:6443. Nothing else in this design works until this finishes.

Result: the laptop holds a NetBird-issued IP and a peer relationship with the master.

Diagram 2 — kubectl over the mesh (the main flow)

Assumes the NetBird mesh is already up (Diagram 1). kubectl reaches the cluster through the netbird-operator's clusterProxy peer, which impersonates the caller's NetBird identity and maps NetBird groups to Kubernetes RBAC. No kubeaid-cli login, no klist, no OIDC on the kubectl path — the mesh identity is the credential.

kubectl talks only to the clusterProxy peer's mesh IP; the proxy forwards to the in-cluster apiserver under an impersonated identity derived from the caller's NetBird groups. The host firewall (CCNP) keeps 6443 closed to the public internet, so the mesh is the only path in.

The full flow

Four phases. Phase 0 happens once. Phase 1 happens per cluster. Phase 2 happens per user. Phase 3 happens every workday.

Each phase below is the user-visible flow. Anything kubeaid-cli does internally (Hetzner API calls, kubeadm init, manifest rendering, ArgoCD wiring, etc.) is collapsed into a single bootstrap step — the operator/developer only types a small number of commands.

Phase 0 — NetBird/VPN Server provisioning (once)

Phase 1 — Cluster bootstrap (per cluster)

Phase 2 — Onboard a user (per user)

Phase 3 — Day-2 access (every workday)

Two enforcement points, one identity

The same NetBird group name (k8s-cluster-X-dev) gates both layers — belt and suspenders:

• Network — NetBird NBPolicy on resource k8s-X.netbird. Remove user from the NetBird group → reachability drops immediately.
• Auth/RBAC — NetBird group → clusterProxy impersonation → ClusterRoleBinding. Remove user from the NetBird group → the proxy stops impersonating those groups, so kube-apiserver 403s.

Namespace-scoped access

Keycloak handles identity; k8s RBAC handles authorization. To give a user rights in only some namespaces, encode the namespace in the group name and bind it via a RoleBinding (namespace-scoped) rather than a ClusterRoleBinding.

• Group naming convention (Keycloak + NetBird): k8s-<cluster>-<namespace>-<role> — e.g. k8s-prod-payments-developer, k8s-prod-billing-readonly.
• Bind in the target namespace: a RoleBinding in payments referencing the impersonated NetBird group k8s-prod-payments-developer → ClusterRole developer grants only that namespace's rights to anyone in that group.
• A user can be in multiple namespace groups at once. k8s RBAC ORs the matches, so the user gets the union of rights.
• Revoking a single namespace = removing the user from that one group. The other namespace memberships are untouched, no cluster-level change needed.

For cluster-wide roles (cluster-admin, view across everything), keep using ClusterRoleBinding with broader group names (e.g. k8s-prod-admin). The two coexist — pick whichever scope each role needs.

Session lifetime — "X hours per login"

Keycloak governs only the NetBird mesh login (Keycloak is NetBird's SSO IdP). The token/session TTLs on Keycloak's NetBird SSO client decide how often a user must re-authenticate to NetBird — i.e. how often netbird up triggers a fresh browser SSO:

Setting	Effect
Access token lifespan	NetBird refreshes its mesh session token at this cadence
Client session max	Hard cap — re-auth via browser SSO required after this
SSO session idle	Refresh fails after this idle window → forces re-auth earlier

Tune these to whatever mesh-session policy you want; the exact values are a NetBird/Keycloak SSO concern and don't touch kube-API access.

kube-API access has no separate session. It lasts exactly as long as you are on the mesh and your NetBird group is still bound via clusterProxy.rbac. There is no OIDC token, no kubelogin refresh, and no per-cluster kube-API TTL on the kubectl path — drop off the mesh or lose the group binding and the clusterProxy stops impersonating you.

For revoking access mid-session or for the next session, see Two enforcement points, one identity above.

Why this shape (the missing elements made explicit)

• Identity provider exists at all. Keycloak is the NetBird dashboard's SSO IdP on VPN clusters — it authenticates users onto the mesh. kubectl access to workload clusters does not touch Keycloak; the mesh identity is the credential.
• kube-api access via the clusterProxy. No OIDC broker, no CSRs on the kubectl path. The netbird-operator clusterProxy proxies kubectl over the mesh and impersonates the caller's NetBird identity — RBAC binds to the impersonated groups.
• NetBird/VPN Server is separate from the workload cluster. Avoids the loop where you need cluster access to fix cluster access. If the workload cluster dies, you can still authenticate, debug, and re-bootstrap.
• Single public ingress. Only the NetBird/VPN Server is reachable on the internet. Workload cluster has zero public attack surface beyond NetBird's WireGuard handshake on the relay.
• Bootstrap chicken-and-egg solved by cloud-init. NetBird agent is installed and enrolled on each node during boot via a setup key. The laptop running kubeaid-cli is itself a NetBird peer, so the local k3d bootstrap cluster reaches the new master through the same mesh path kubectl will later use. No temporary public 6443 needed.
• Network and identity are decoupled but consistent. Same group name in both systems. NetBird drops packets if you're not in the group; kube-apiserver returns 403 if the clusterProxy is impersonating a NetBird group with no matching RBAC (no token or claim is involved). Defense in depth.

Trade-offs accepted

• NetBird Mgmt needs Keycloak for mesh SSO (OIDC discovery / JWKS). kube-apiserver needs nothing from Keycloak — it does no OIDC, so there's no JWKS fetch on the kube-api path at all. Acceptable.
• Revocation is effectively instant. Removing the user from the NetBird group both drops their mesh access and stops the clusterProxy impersonating their groups — there's no OIDC token TTL to wait out on the kube path.
• Single-control-plane topology trades availability for cost. For the 1-node variant, a master reboot or NetBird-agent failure on that node makes kube-api unreachable until the agent and operator come back via systemd. Acceptable for non-prod / cost-sensitive use; pick the 3-node HA topology when that downtime is unacceptable. The HA path inherits the same network/auth shape — only the count of NetBird-enrolled control-plane VMs changes.