Skip to main content

KubeAid Cluster Design Guide

Overview​

This document describes the design principles, architectural decisions, and best practices for Kubernetes clusters managed with KubeAid. It covers:

  1. Control plane architecture and security isolation
  2. Storage strategies (Cloud CSI vs Local PV vs Ceph)
  3. High availability patterns for stateful workloads
  4. Cloud provider considerations and limitations

Table of Contents​

Cluster Architecture Patterns​

Standard Production Cluster​

Key Design Principles:

ComponentArchitecturePurpose
Control Plane3× ARM64 (Ampere)Dedicated cluster management, cost-effective
Worker NodesN× AMD64Application workloads, scalable
TaintNoSchedulePrevents workloads on control plane

Control Plane Design​

Dedicated Controllers with ARM64 (Ampere)​

Why use dedicated control plane nodes?

BenefitDescription
Security IsolationWorkloads cannot access control plane components directly
Resource ProtectionControl plane has guaranteed resources, no OOM from workloads
StabilityMisbehaving applications cannot crash the cluster
Attack Surface ReductionCompromised workload cannot easily pivot to etcd/apiserver

Why ARM64 (Ampere) for control planes?

BenefitDescription
Cost EffectiveARM servers are typically 20-40% cheaper than x86 equivalents
Lower PowerBetter power efficiency for always-on control plane
Security BoundaryDifferent architecture = harder to exploit from x86 workloads
Right-SizedControl plane doesn't need heavy compute, ARM is sufficient

Control Plane Taints (CRITICAL)​

Control plane nodes MUST have taints to prevent workload scheduling:

# Required taint on all control plane nodes
taints:
  - key: node-role.kubernetes.io/control-plane
    effect: NoSchedule

Verify taints are in place:

kubectl get nodes -l node-role.kubernetes.io/control-plane -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

Add missing taints:

kubectl taint nodes <control-plane-node> node-role.kubernetes.io/control-plane:NoSchedule --overwrite

⚠️ Risks of NOT Having Dedicated Controllers​

RiskImpactSeverity
OOM KillsWorkload pods consume memory, causing control plane OOM🔴 Critical
CPU StarvationScheduler/controller-manager become slow or unresponsive🔴 Critical
etcd LatencyDisk I/O from workloads impacts etcd performance🔴 Critical
Security BreachCompromised pod can access etcd data/apiserver🔴 Critical
Cluster InstabilityRandom failures, kubectl timeouts, deployment failures🟡 High

Real-world example: When worker nodes are full (e.g., volume limit reached), pods may get scheduled on control plane nodes if taints are removed, causing OOM kills and cluster instability.

Storage Design​

Storage Types Comparison​

Storage TypeFailoverPer-Node LimitHA StrategyBest For
Hetzner CSI✅ Automatic❌ 16 volumesSingle replica DBSmall/Dev clusters
AWS EBS✅ Automatic❌ 28 volumesSingle replica DBAWS workloads
Azure Disk✅ Automatic❌ VariesSingle replica DBAzure workloads
Local PV❌ None✅ UnlimitedMulti-replica DBHA databases
Ceph (Rook)✅ Automatic✅ UnlimitedSingle replica DBProduction clusters

Storage Decision Tree​

Design Philosophy: Why We Keep It Simple (KISS)​

We intentionally limit supported storage solutions to Cloud CSI and Ceph.

Why not support other options?

"Too much choice just means too many combinations and options for problems in production."

  1. Maturity Wins: Ceph is the industry standard for self-hosted Kubernetes storage. It is battle-tested at massive scale.
  2. Reduced Complexity: By standardizing on fewer, better tools, we reduce the "surface area" for bugs.
  3. Easier Debugging: When things fail (and they will), having a simple, well-understood stack makes recovery faster.
  4. Production Safety: We prioritize stability and data integrity over "easy setup" solutions.

Our Rule: If it's not rock-solid for production, it's not in KubeAid.

Hetzner CSI Limitations​

⚠️ Critical Limitation: Each Hetzner Cloud server can only attach 16 volumes maximum.

This is a hardware limitation of Hetzner Cloud (HCloud) servers, not the CSI driver:

Impact:

ScenarioVolumes UsedIssues
5 Odoo instances (Odoo PV + 2 PG replicas each)5 × 3 = 15 volumes⚠️ Almost at limit
6 Odoo instances18 volumes❌ Exceeds limit
Add monitoring/logging PVs+2-4 volumes❌ No capacity

Solutions:

  1. Short-term: Reduce database replicas to 1 (see below)
  2. Medium-term: Add more worker nodes
  3. Long-term: Migrate to Ceph storage (no per-node limit)

Database High Availability Strategies​

The Key Principle​

Use database-level replication (2+ replicas) ONLY when using local storage. Use single replica when storage provides failover (cloud CSI, Ceph).

Why This Matters​

Storage TypeFailover MechanismDB Replicas NeededReason
Local PV❌ None2+ replicasData loss if node fails
Hetzner CSI✅ Volume reattaches1 replicaVolume moves to new node
Ceph✅ Data replicated 3x1 replicaStorage handles HA
AWS EBS✅ AZ-level redundancy1 replicaStorage handles HA

When Do You Actually Need Read Replicas?​

💡 Key Insight: Read replicas are for performance scaling, not for HA when using failover-capable storage.

ScenarioNeed Read Replicas?Reason
Read-heavy workload (10,000+ queries/sec)✅ YesDistribute read load
HA with cloud storage❌ NoStorage handles failover
HA with local storage✅ YesNo storage-level failover
Small/Medium apps❌ NoSingle instance sufficient
Backups available❌ NoCan restore from backup

Real-world example: None of our current workloads (Obmondo, Image2Work) have the scale requiring read replicas. Backups + failover storage provides sufficient data protection.

Backups vs Replicas​

Recommendation: Use scheduled backups (to SeaweedFS/S3) instead of read replicas for data protection.

Visual Comparison​

❌ Wrong: 2 PG Replicas with Cloud Storage (Wastes Resources)​

✅ Correct: 1 PG Instance with Cloud Storage​

✅ Correct: 2 PG Replicas with Local Storage​

PostgreSQL Configuration Examples​

Cloud Storage (Hetzner CSI, AWS EBS, Ceph)​

# values.yaml for PostgreSQL operator
postgres:
  instance: 1  # Single instance - storage handles HA
  storage:
    storageClass: hcloud-volumes  # or ceph-block, gp3, etc.
    size: 10Gi

Local Storage (Bare Metal, Local PV)​

# values.yaml for PostgreSQL operator
postgres:
  instance: 2  # Two instances for HA - storage has no failover
  storage:
    storageClass: local-path
    size: 100Gi

Resource Management Best Practices​

Memory Limits vs Requests​

⚠️ CRITICAL: Never reduce memory limits to save resources. Only adjust requests.

SettingPurposeSafe to Reduce?
Memory RequestScheduling guarantee✅ Yes (carefully)
Memory LimitOOM kill threshold❌ NO
CPU RequestScheduling guarantee✅ Yes
CPU LimitThrottling threshold⚠️ Careful

Why this matters:

Resource Configuration Guidelines​

# ✅ CORRECT: High limit, lower request
resources:
  requests:
    memory: "256Mi"   # Can be reduced for scheduling
    cpu: "100m"
  limits:
    memory: "512Mi"   # Keep this high - prevents OOMKilled
    cpu: "500m"

# ❌ WRONG: Low limit causes OOMKilled
resources:
  requests:
    memory: "256Mi"
    cpu: "100m"
  limits:
    memory: "256Mi"   # Too low - will get OOMKilled!
    cpu: "500m"

Monitoring Resource Usage​

# Check actual node memory usage
kubectl top nodes

# Check pod memory usage
kubectl top pods --all-namespaces --sort-by=memory

# Find OOMKilled pods
kubectl get pods --all-namespaces -o json | jq -r '
  .items[] | 
  select(.status.containerStatuses[]?.lastState.terminated.reason=="OOMKilled") | 
  "\(.metadata.namespace)/\(.metadata.name)"'

Cloud Provider Considerations​

Hetzner Cloud (HCloud)​

AspectDetailsImpact
Volume Limit16 volumes per serverPlan capacity carefully
Volume Failover✅ Automatic reattachmentUse single DB replica
StabilityOccasional API timeoutsMay cause PV attach delays
CostLowGood for dev/small prod

Recommended for: Development clusters, small production workloads

When to move away: When approaching volume limits, large production workloads

Hetzner Bare Metal (Dedicated) - Recommended for Production​

AspectDetailsImpact
Volume Limit✅ No cloud volume limitUse local PV or Ceph
Local StorageFast NVMe/SSDGreat for databases
Ceph✅ RecommendedNo per-node limits
Network10G NIC availableFast Ceph replication
CostHigherBetter for production
ComponentSpecificationPurpose
CPUAMD Ryzen 9 5950X (16c/32t)High-performance workloads
RAM128 GB DDR4 ECCLarge memory for apps + Ceph
Storage2× 3.84 TB NVMeCeph OSD + Local PV
Network10 Gbps NICFast Ceph replication, low latency

Why 10G NIC?

  • Ceph replication requires high bandwidth between nodes
  • Reduces latency for distributed storage operations
  • Enables fast recovery when OSD fails
  • Critical for production databases

Recommended for: Production clusters, Image2Work prod, stateful workloads at scale

Comparison Matrix​

Key Insight: Hetzner Cloud CSI is suitable for dev/small workloads but hits limitations (16 volumes/node) for larger deployments. Migrate to Ceph on bare metal for production scale.

Quick Decision Matrix​

Storage Selection​

QuestionIf YesIf No
Over 10 stateful applications?Use CephCloud CSI may work
Need >16 volumes per node?Use Ceph/Local PVCloud CSI OK
Using Hetzner dedicated servers?Use CephN/A
Production-critical data?Use Ceph with 3x replicationCloud CSI acceptable

Database Replica Count​

Storage TypeUse 1 ReplicaUse 2+ Replicas
Hetzner CSI✅❌
AWS EBS/EFS✅❌
Azure Disk✅❌
Ceph (Rook)✅❌
Local PV❌✅
hostPath❌✅

Control Plane Configuration​

EnvironmentDedicated CP?ARM64 CP?Taints?
DevelopmentOptionalOptional✅ Required
Staging✅ RecommendedOptional✅ Required
Production✅ Required✅ Recommended✅ Required

Configuration Examples​

# Cluster configuration
controlPlane:
  count: 3
  machineType: cax31          # ARM64 Ampere
  taints:
    - key: node-role.kubernetes.io/control-plane
      effect: NoSchedule

workers:
  count: 3+                    # Scale based on workload
  machineType: cpx41           # AMD64 for compatibility

storage:
  # For small clusters
  primary: hcloud-volumes
  # For larger clusters - add Ceph
  # primary: rook-ceph-block

databases:
  postgresql:
    instances: 1               # Single instance - CSI handles failover
    storageClass: hcloud-volumes
# Cluster configuration
controlPlane:
  count: 3
  hardware: AX41-NVMe          # Dedicated servers
  taints:
    - key: node-role.kubernetes.io/control-plane
      effect: NoSchedule

workers:
  count: 3+
  hardware: AX41               # With Ceph OSD disks

storage:
  primary: rook-ceph-block     # Ceph for all persistent storage

databases:
  postgresql:
    instances: 1               # Single instance - Ceph handles failover
    storageClass: rook-ceph-block

Troubleshooting Common Issues​

"Pods scheduled on control plane"​

Cause: Missing taints on control plane nodes

Fix:

kubectl taint nodes <cp-node> node-role.kubernetes.io/control-plane:NoSchedule

"Cannot schedule pod - volume limit reached"​

Cause: Hetzner CSI 16-volume limit

Fix:

  1. Reduce DB replicas: postgres.instance: 1
  2. Add more worker nodes
  3. Migrate to Ceph storage

See: Hetzner CSI Volume Limit Debugging

"Database not recovering after node failure"​

Cause: Using local PV with single replica

Fix: With local PV, use 2+ replicas:

postgres:
  instance: 2

Hetzner Storage Guides:

Alert Handling:

Application Guides:

Summary​

Design DecisionRecommendation
Control PlaneDedicated ARM64 nodes with NoSchedule taint
Storage (HCloud)Hetzner CSI for small, Ceph for large
Storage (Bare Metal)Always Ceph
DB Replicas (Cloud Storage)1 replica (storage handles failover)
DB Replicas (Local PV)2+ replicas (application-level HA)
Volume PlanningMax 16/node with Hetzner CSI