Every image in the repository is pinned by tag and digest, and every bundle the bootstrap downloads is checked against a sha256 checksum. An upgrade changes both the version and its pin, so the cluster only ever runs exactly what was reviewed.
#Before you begin
- Read the component’s release notes for the versions between yours and the target. Some components, Longhorn among them, only support upgrading one minor version at a time.
- Make sure the platform is healthy. Follow verify a tenant first.
#Upgrade an image
Look up the digest of the new tag. The digest of the multi-arch index is the one to pin:
docker buildx imagetools inspect grafana/loki:<new tag>Copy the
Digest:line at the top.Change the image in the component’s file to the new
tag@digest:image: grafana/loki:<new tag>@sha256:<digest>Check that the registry is allowed. Images in
application,node-agentsandreloadermust come from a publisher in the registry allowlist. A new version from the same publisher is already allowed.Apply the file and wait for the rollout:
kubectl apply -f tenant-workload/observability/loki/loki.yaml kubectl -n application rollout status statefulset/loki
The same steps work for the PostgreSQL image, imageName in the database cluster files. CloudNativePG restarts the replicas first and then switches over the primary, because primaryUpdateStrategy is unsupervised. A minor PostgreSQL version can be changed this way; a major version needs a new cluster.
#Upgrade a bundled operator
Kyverno, Longhorn, CloudNativePG, the Prometheus operator and Strimzi are installed from their release manifests by tenant-workload/bootstrap/bedrock-bootstrap.yml. The bootstrap does more than apply them: it checks each download against a checksum, removes Longhorn’s wildcard RBAC and its cluster-wide Secret read, moves the Prometheus operator out of default, and gives Kyverno’s admission controller one replica per node, the top priority class and anti-affinity. Never apply a release URL directly, because that undoes all of it.
Download the new manifest and compute its checksum:
curl -sL <new release URL> | sha256sumIn the bootstrap task, change the version in the task name and the URL, and replace the
checksumwithsha256:<new sum>.Run only that operator’s tasks. They are tagged
kyverno,longhorn,cnpg-operator,prometheus-operatorandstrimzi:ansible-playbook -i tenant-workload/bootstrap/hosts.ini tenant-workload/bootstrap/bedrock-bootstrap.yml --tags longhornStrimzi only installs with Kafka turned on, so add
-e kafka_enabled=truefor it. A tagged run skips the database tasks, so it also works after a failover, when a full bootstrap refuses to run.
The bootstrap task is the record of which version a tenant runs, so a fresh install uses the new version from then on.
#Upgrade RKE2
The RKE2 version is the rke2_version variable in the bootstrap. Upgrade the nodes one at a time, servers first, and wait for each to report Ready before moving on, so etcd keeps its majority throughout. RKE2’s own documentation describes the per-node steps.