Skip to main content

Traefik - a loadbalancer

Traefik dashboard​

you can access it by doing kubectl -n traefik port-forward <podname> 9000:9000 and opening http://localhost:9000/dashboard/ in your browser

Setup for Internal use​

  • We setup an internal service which will have an internal loadbalance IP

AWS​

service:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"

AKS​

service:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"

Public​

  • We setup an internal service which will have an internet-facing loadbalance IP. you will have to add these service annotation in your yaml file

AWS Public​

service:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"

AKS Public​

service:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "false"
    service.beta.kubernetes.io/azure-load-balancer-resource-group: <your-resource-group-name>

TLS Auth​

  • Doc
  • NOTE: Make sure the key is either ca.crt or tls.ca
kubectl create secret generic internalca-cert --namespace traefik --dry-run=client --from-file=/path/to/ca.crt -o yaml | kubeseal --controller-namespace system --controller-name sealed-secrets -o yaml
  • Add this in the values file
tlsOptions:
  tls-client-auth:
    clientAuth:
      clientAuthType: VerifyClientCertIfGiven
      secretNames:
        - internalca-cert

Troubleshooting​

  • Traefik has a default limit on Request Body that can affect file uploads. To re-configure it using a middleware, see this example.

  • If you want to run multiple traefik instances, ensure that each traefik deployment has the additional argument which binds it to a specific ingress class --providers.kubernetesingress.ingressclass=<ingress-class>. This ensures that multiple instances of Traefik don't try to update ingress resources at the same time leading to the hostname of the ingress switching from one ingress class to another.

Upgrading Traefik​

While upgrading to Traefik Helm chart v25.0.0 and Traefik v2.10.5, the Traefik deployment needs to be deleted, or else ArgoCD throws a Sync error like this:

Deployment.apps "traefik" is invalid: spec.selector: Invalid value: 
v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"traefik-traefik",
"app.kubernetes.io/name":"traefik"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: 
field is immutable

This happens due to new label selectors in the new version of the Traefik Helm chart and hence a rolling upgrade is not possible. Link to Upstream k8s issue

The procedure for the upgrade is:

  • [Pre-Requisite] Use kubectl port forward to argocd-server pod and open the UI on localhost. This is recommended as Traefik will go down and ArgoCD's Ingress will not be able to handle requests.
  • [Pre-Requisite] If you don't have access to port forward the argocd service, then use helm template command to keep a copy of the Deployment and Service YAML of Traefik.
  • Refresh the Traefik ArgoCD app on the cluster
  • Verify the diff once the ArgoCD app is in OutOfSync state
  • Click on Sync in the ArgoCD app, and select all the CRD (Custom Resource Definition) resources which are OutOfSync.
  • Click on Sync in the ArgoCD app, and select all the resources which are OutOfSync except the Deployment, Service, and Service Account.
  • Once the sync is completed successfully, there will be only the Traefik Deployment, Service and Service Account will be shown as OutOfSync.
  • Sync the Service and Service Account.
  • Delete the Traefik deployment from ArgoCD or using kubectl cli.
  • Once the pods are deleted successfully, sync the Traefik Deployment from ArgoCD.
  • If you did not create a port forward to argocd, the ArgoCD UI will stop responding as the Traefik Ingress will not be able to respond to the requests due to the Traefik pod being deleted. Use kubectl apply -f traefik-deployment.yaml to create the deployment from the [Step 2] above.
  • Traefik pods should come up with the latest version.

Known ArgoCD Drift​

Service clusterIP, Deployment annotations, ClusterRole, IngressRoute, PodDisruptionBudget​

Several Traefik resources drift from the chart-rendered state at runtime:

  • Service: clusterIP and clusterIPs are assigned by Kubernetes after creation and are not in the Helm chart
  • Deployment: checksum/config annotations update when Traefik config changes
  • ClusterRole: additional rules may be added by the cluster
  • IngressRoute (dashboard): spec may drift from chart defaults
  • PodDisruptionBudget: spec modified by cluster autoscaler

Add ignoreDifferences to your ArgoCD Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: traefik
  namespace: argocd
spec:
  ignoreDifferences:
    - group: ""
      kind: Service
      jsonPointers:
        - /spec/clusterIP
        - /spec/clusterIPs
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/template/metadata/annotations
    - group: rbac.authorization.k8s.io
      kind: ClusterRole
      jsonPointers:
        - /rules
    - group: rbac.authorization.k8s.io
      kind: ClusterRoleBinding
      jsonPointers:
        - /subjects
    - group: traefik.io
      kind: IngressRoute
      jsonPointers:
        - /spec
    - group: policy
      kind: PodDisruptionBudget
      jsonPointers:
        - /spec
  sources:
    - repoURL: https://gitea.obmondo.com/EnableIT/KubeAid
      path: argocd-helm-charts/traefik
      targetRevision: HEAD
      helm:
        valueFiles:
          - $values/k8s/<cluster>/argocd-apps/values-traefik.yaml
    - repoURL: <your-config-repo>
      targetRevision: HEAD
      ref: values