#Upgrading Kubernetes on Bare Metal

This guide explains how to complete Phase 2 of the upgrade workflow for clusters on bare metal. Before you upgrade Kubernetes, complete the Distribution Version upgrade described in Upgrading Clusters.

#Key Considerations

Bare-metal upgrades replace every node — they do not run kubeadm upgrade on the existing OS. The mechanism is:

1. CAPI deletes one Machine according to the rollout strategy.
2. The provider writes a clean plan to the inventory's plan secret.
3. The host stops kubelet + CRI workload + containerd, then returns to Available in the same pool.
4. CAPI creates a replacement Machine.
5. The provider picks an Available inventory from the same pool, resolves the new Kubernetes version against elemental-image-catalog, and writes a reprovision plan.
6. The host runs cloud-init clean → elemental upgrade --system <new-image> → reboot. initramfs clears Kubernetes persistent state; cloud-init re-executes and runs kubeadm init / kubeadm join against the new control plane.

Two structural consequences operators must internalize before starting:

• Bare-metal does not preserve Kubernetes-managed disk state. /var/lib/kubelet, /var/lib/containerd, /var/lib/etcd, /etc/kubernetes are cleared by the initramfs cleanup step of every reprovision. There is no equivalent of the DCS provider's pool-managed persistent disks today.
• The same MachineInventory may not be re-picked. The provider does not guarantee that the inventory released by a clean plan is the same one re-allocated to the replacement Machine. Replacements are performed as delete-then-add when maxSurge=0 (no overlap between old and new node), so pool capacity sizing need only accommodate the desired replica count — old and new nodes do not coexist during the rollout.

#Upgrade Sequence

Upgrade bare-metal clusters in the following order:

1. (Prerequisite) Upgrade the ACP platform on the global cluster. This brings the bare-metal provider, elemental-operator, and the related CAPI components to versions that understand the new schema. Trigger workload-cluster upgrades only after the management-side controllers have rolled out and become Ready.
2. Upgrade the Distribution Version (Aligned Extensions) on the workload cluster. See Upgrading Distribution Version.
3. Ensure the target Kubernetes version is present in elemental-image-catalog (and the matching -iso repository is available for any future host re-registration).
4. Upgrade Kube-OVN to the chart version required by the target ACP release and wait for the AppRelease to reach Success.
5. Upgrade the control plane Kubernetes version (replaces all control-plane nodes one at a time).
6. Upgrade worker nodes to the target Kubernetes version (replaces all worker nodes within the maxUnavailable budget).

Cluster API orchestrates the rolling replacement.

#Prerequisites

Before you start, ensure all of the following prerequisites are met:

• The Distribution Version upgrade is complete.
• The control plane is reachable through the existing <control-plane-vip>:<control-plane-port>.
• All current nodes are healthy and Ready.
• The target Kubernetes version is a key in elemental-image-catalog. If it is not, add it before starting (see Update the Image Catalog).
• The target kube-ovn (chart) version has been read from the matching ACP row in the OS Support Matrix.
• BaremetalCluster.spec.networkType is kube-ovn; the provider skips Kube-OVN reconciliation for any other value.
• The platform registry is reachable from every host in the cluster.
• Both KubeadmControlPlane.spec.rolloutStrategy.rollingUpdate.maxSurge = 0 and MachineDeployment.spec.strategy.rollingUpdate.maxSurge = 0 are set — bare-metal does not over-provision physical hosts.
• The relevant pools have enough capacity to replace one node at a time without falling below the desired replica count.

#Update the Image Catalog

elemental-image-catalog is the resource that introduces a Kubernetes version to the bare-metal provider.

Option A — Add a chart override. Append the new version under provider.imageCatalog.images (uses global.registry.address as the registry):

provider:
  imageCatalog:
    images:
      v1.33.7-2:
        repository: tkestack/baremetal-base-image
        tag: v0.0.0-beta-1.33.7-2
      <new-kubernetes-version>:                # for example, v1.34.5
        repository: tkestack/baremetal-base-image
        tag: <new-tag>                         # for example, v0.0.0-dev.0-1.34.5

Reapply the bare-metal provider plugin. The chart re-renders the ConfigMap; the provider's in-process watch hot-reloads the cache without restarting.

Option B — Patch the ConfigMap directly. Useful for digest-pinned images or out-of-band test versions:

kubectl -n cpaas-system patch configmap elemental-image-catalog \
  --type='json' \
  -p='[{"op":"add","path":"/data/<new-kubernetes-version>","value":"<registry-address>/tkestack/baremetal-base-image:<new-tag>"}]'

In either case, verify before continuing:

kubectl -n cpaas-system get configmap elemental-image-catalog -o yaml

The key must include the leading v (for example v1.34.5). If the key is missing when CAPI creates the replacement Machine, the resulting BaremetalMachine enters Failed with ImageResolved=False / Reason=ImageCatalogMiss and no reprovision plan is written — adding the key later restarts the reconciliation automatically.

When the new version corresponds to a new Alauda OS / base-image release, the ISO variant for that release should also be available before the upgrade. The bare-metal provider does not rebuild SeedImages automatically, but you may want to refresh MachineRegistration / SeedImage for future host onboarding so the new hosts come up on the new release. The ISO repository is the same as the base-image repository with -iso appended.

#Upgrade Kube-OVN Before the Control Plane

Upgrade Kube-OVN and wait for it to become healthy before you change KubeadmControlPlane.spec.version. This ordering ensures that the target network components are already running before bare-metal control-plane nodes begin delete-reprovision replacement.

The bare-metal provider's reconciliation behavior makes two operations necessary:

• It reconciles Kube-OVN only when BaremetalCluster.spec.networkType is exactly kube-ovn.
• It reads Cluster.metadata.annotations["cpaas.io/kube-ovn-version"] before falling back to the kube-ovn ModulePlugin version. However, after cni-kube-ovn already exists, the provider updates only AppRelease.spec.values and preserves spec.source, including the existing chart targetRevision. Therefore, changing the annotation alone records the intended version but does not upgrade the existing chart.

1. Verify that provider-managed Kube-OVN reconciliation is enabled

   Run this command against the management (global) cluster:

   kubectl -n cpaas-system get baremetalcluster <cluster-name> \
     -o jsonpath='{.spec.networkType}{"\n"}'
   # Expected output: kube-ovn

   If the output is empty or different, correct BaremetalCluster.spec.networkType before continuing. Otherwise, the provider skips Kube-OVN reconciliation, including the control-plane node list updates needed during replacement.

2. Record the target chart version on the CAPI Cluster

   Use the kube-ovn (chart) value from the target ACP row in the OS Support Matrix:

   kubectl annotate cluster <cluster-name> -n cpaas-system \
     cpaas.io/kube-ovn-version=<kube-ovn-version-from-matrix> --overwrite

   Run this command against the management (global) cluster where the CAPI Cluster exists. The annotation takes precedence over the provider's ModulePlugin fallback and keeps the cluster's recorded target deterministic.

3. Patch the existing AppRelease chart revision

   Run this command against the workload cluster's API server, not the management cluster:

   kubectl patch apprelease cni-kube-ovn -n cpaas-system --type='json' \
     -p='[{"op":"replace","path":"/spec/source/charts/0/targetRevision","value":"<kube-ovn-version-from-matrix>"}]'

   Use the same chart version that you set in the annotation. Do not change the provider-managed chart name (acp/chart-cpaas-kube-ovn) or release name (cpaas-kube-ovn).

4. Wait for reconciliation to complete before upgrading the control plane

   # Overall AppRelease state — Sync and Health must reach a Success-equivalent reason
   kubectl get apprelease cni-kube-ovn -n cpaas-system
   
   # Installed revision and chart phase
   kubectl get apprelease cni-kube-ovn -n cpaas-system \
     -o jsonpath='Installed: {.status.charts.*.installedRevision}{"\n"}Phase: {.status.charts.*.phase}{"\n"}'

   The normal sequence is Upgrading → HealthChecking → Success. Do not start the KCP rollout based on installedRevision alone: it changes during HealthChecking, before Kube-OVN pods have been verified Ready. Continue only when phase is Success and installedRevision matches the target chart version.

   If the AppRelease reaches DownloadFailed, DeployFailed, or NotReady, stop before changing KubeadmControlPlane and inspect the condition messages:

   kubectl describe apprelease cni-kube-ovn -n cpaas-system

#Upgrade the Control Plane

Before continuing, complete Upgrade Kube-OVN Before the Control Plane and verify that cni-kube-ovn is at the target installed revision with phase=Success.

Patch KubeadmControlPlane.spec.version to the new Kubernetes version. Where component image tags are pinned (DNS, etcd), update them in the same edit:

kubectl -n cpaas-system edit kubeadmcontrolplane <cluster-name>-control-plane

• spec.version ← target Kubernetes version (must match an elemental-image-catalog key).
• spec.kubeadmConfigSpec.clusterConfiguration.dns.imageTag ← matching CoreDNS image tag for the new release.
• spec.kubeadmConfigSpec.clusterConfiguration.etcd.local.imageTag ← matching etcd image tag for the new release.
• When the target is Kubernetes 1.35 or later, update /etc/kubernetes/patches/kubeletconfiguration0+strategic.json in spec.kubeadmConfigSpec.files in this same edit, as described in Required kubelet patch for Kubernetes 1.35.

The bare-metal provider does not require a new BaremetalMachineTemplate for a Kubernetes-only upgrade: the template only carries the pool reference, and the upgrade image is resolved from Machine.spec.version through the catalog. A new template is only required when you want to move the control plane to a different pool.

Cluster API rolls control-plane nodes one at a time (because maxSurge=0):

kubectl -n cpaas-system get kubeadmcontrolplane <cluster-name>-control-plane -w
kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io
kubectl -n cpaas-system get machineinventorypools.infrastructure.cluster.x-k8s.io <cluster-name>-control-plane-pool
kubectl get nodes -o wide                                                    # workload cluster

The expected sequence on each node:

1. CAPI marks one old Machine for deletion; corresponding BaremetalMachine moves to Preparing.
2. The provider writes a clean plan; the inventory ends up Available with cleared owner annotations.
3. CAPI creates a new Machine (with the target version); a new BaremetalMachine is created.
4. The provider allocates an Available inventory from the control-plane pool, resolves the new image, and writes a reprovision plan.
5. The host runs elemental upgrade --system <new-image>, reboots, and kubeadm joins the surviving control plane. BaremetalMachine.status.phase becomes Running.

Repeat steps 1–5 until every control-plane node has been replaced.

Throughout the rollout, alive continues to manage the VIP. As control-plane membership changes, the bare-metal provider re-renders the alive chart values (the peer list and ipvs.ips) and rolls out the static-pod manifests.

#Upgrade Workers

Patch each MachineDeployment.spec.template.spec.version to the new Kubernetes version. CAPI replaces workers within the maxUnavailable budget — the bare-metal provider re-uses the same worker pool and the same image-catalog resolution logic as the control plane.

For Kubernetes 1.35 or later, first create a new worker KubeadmConfigTemplate containing the required kubelet patch, then update spec.template.spec.bootstrap.configRef.name in the same MachineDeployment edit as the version bump. The command below includes that reference. For an earlier target with no bootstrap change, omit the bootstrap object. Follow Updating Bootstrap Templates to create the immutable template.

kubectl -n cpaas-system patch machinedeployment <cluster-name>-workers \
  --type='merge' \
  -p='{"spec":{"template":{"spec":{"version":"<new-kubernetes-version>","bootstrap":{"configRef":{"name":"<new-bootstrap-template>"}}}}}}'

Watch:

kubectl -n cpaas-system get machinedeployments.cluster.x-k8s.io -w
kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io
kubectl get nodes -o wide

Worker upgrades carry fewer knobs than the control plane: there is no clusterConfiguration block on MachineDeployment, so no DNS / etcd tags to update. Kubernetes component versions on worker nodes follow the new base image.

For earlier versions, a bootstrap-template change is needed only when the release changes settings such as cloud-init or kubeletExtraArgs.

#Cross-Version Upgrades

Kubernetes control planes must move through supported minor-version hops. Use Preparing for Cross-Version Upgrades to identify and pre-stage every intermediate ACP row.

For each intermediate row, and finally for the target row:

1. Ensure that row's Kubernetes version exists in elemental-image-catalog.
2. Upgrade Kube-OVN to that row's kube-ovn (chart) version and wait for phase=Success.
3. Upgrade the control plane to that row's Kubernetes version and wait for every control-plane node to become Ready.
4. Upgrade every worker MachineDeployment to the same Kubernetes version and wait for the rollout to complete before starting the next hop.

The version values always come from the OS Support Matrix; they are not fixed examples in this guide. The bare-metal rollout strategy already serializes node replacement, so cross-version upgrades do not require additional pool capacity beyond what is needed for a same-minor upgrade.

#Verification

After the rollout completes:

kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io
kubectl -n cpaas-system get machines.cluster.x-k8s.io
kubectl -n cpaas-system get apprelease cni-kube-ovn
kubectl get nodes -o wide

The cni-kube-ovn AppRelease should report the target installed revision with phase=Success. All BaremetalMachine objects should be Running. Every CAPI Machine should report the new spec.version. Every workload Node should be Ready with the new kubelet version, and every MachineInventory plan secret should carry baremetal.alauda.io/plan.type=reprovision (the most recent plan applied).

MachineInventoryPool.status should satisfy available + allocated + preparing + reprovisioning + unavailable = total with preparing = reprovisioning = 0.

#Troubleshooting

For the full operator-side state machine reference (every condition reason and recovery action), see Provider Overview.

#Additional Resources

• Cluster API Upgrade Documentation
• Kubernetes Version Skew Policy