bedrock

Bootstrap

#Commands

CommandRuns
make deployansible-playbook -i tenant-workload/bootstrap/hosts.ini tenant-workload/bootstrap/bedrock-bootstrap.yml
make teardownansible-playbook -i tenant-workload/bootstrap/hosts.ini tenant-workload/bootstrap/bedrock-teardown.yml
make redeployTeardown, then deploy
make checkSyntax check of the bootstrap playbook against the example inventory, and make check-allowlist
make check-allowlistFails if the three registry allowlist patterns are not identical
make rotate-exchange-certSigns a new exchange certificate for the observers and restarts them, see rotate the exchange certificate
make rotate-proof-keyRotates the failover controller’s proof key with an overlap, no downtime
make rotate-grafana-passwordSets a new Grafana admin password in Grafana, OpenBao and on the controller, no restart, see rotate a secret
make add-replica REPLICA=<name>Adds a live replica behind the current primary, see add a live replica

Set INVENTORY=path/to/hosts.ini to use another inventory file. The default hosts.ini is ignored by git; copy it from hosts.example.ini.

The bootstrap refuses to run once a failover has happened, because it would turn the new primary back into a replica. Tagged runs, --tags kyverno, longhorn, cnpg-operator, prometheus-operator or strimzi, skip the database and still work, as described in upgrade a component.

#Inventory groups

GroupMembers
server_initThe first node, which creates the cluster
server_joinThe other server nodes, joined one at a time
rke2_serversBoth groups together

#Variables

VariableDefaultEffect
rke2_versionv1.36.4+rke2r1RKE2 release to install
nats_enabledfalseDeploy NATS JetStream
valkey_enabledfalseDeploy Valkey with Sentinel
kafka_enabledfalseDeploy Strimzi and Kafka
kubeconfig~/.bedrock/kubeconfigWhere the admin kubeconfig is written on the controller
failover_enabledtrueDeploy the failover controller
failover_dir~/.bedrock/failoverWhere the proof keys and the exchange certificates are kept on the controller

#Stages

StageWhat runs
1Node packages (open-iscsi, nfs-common, cryptsetup, lvm2), the etcd user, CIS kernel settings, the dm_crypt module
2Audit policy, RKE2 configuration, Cilium configuration, RKE2 install from a checksummed release
3The first server, then the others one at a time
4kubectl, crictl and ctr links and the login profile on every node
5Namespaces, priority classes and the ValidatingAdmissionPolicy
6Kyverno, scaled to one admission replica per node, then its policies
7Longhorn, with its RBAC rewritten without wildcards
8CloudNativePG operator and prometheus-operator
9Longhorn encryption key and storage classes, then OpenBao: three pods on raft, initialized with 5 key shares and a threshold of 3, and unsealed. Then the Secrets Store CSI driver and the OpenBao CSI provider, the platform passwords made once on the controller, and OpenBao’s kubernetes auth, the apps key value engine and one role per component. Then Garage and the bucket job, which read their secrets from OpenBao
10cluster-1, the failover functions and the proof key loaded into it, the observer password set from OpenBao, the openbao and bedrock_app roles handed to OpenBao’s database engine, its first backup, the certificate sync, then cluster-2 and cluster-3
11Poolers, scheduled backups and disruption budgets
12Event buses, if enabled
13Loki, Alloy, Tempo, Prometheus and exporters, alerting, Grafana, OpenCost, Falco, Reloader, dashboards
14Token automount turned off on every default service account
15The failover controller: its namespace, admission policies, the database CA sync, proof key, exchange certificates, then the workloads and their alerts
16Network policies, then a check that every namespace running pods has one

#Files the bootstrap writes on the nodes

PathContents
/etc/rancher/rke2/config.yamlRKE2 configuration, rendered from rke2-config.yaml.j2 for the first server and the joining ones
/etc/rancher/rke2/bedrock-audit.yamlAPI audit policy
/var/lib/rancher/rke2/server/manifests/rke2-cilium-config.yamlCilium settings
/etc/sysctl.d/60-rke2-cis.confCIS kernel settings and inotify limits
/etc/modules-load.d/dm_crypt.confLoads dm_crypt at boot for volume encryption
/etc/profile.d/rke2.shSets KUBECONFIG and CRI_CONFIG_FILE at login
/var/lib/rancher/rke2/server/logs/audit.logThe API audit log, shipped to Loki

#Files the bootstrap writes on the controller

PathContents
~/.bedrock/kubeconfigThe admin kubeconfig
~/.bedrock/openbao-init.jsonOpenBao’s unseal key shares and root token, mode 0600
~/.bedrock/grafana-adminGrafana’s admin password, also stored in OpenBao, mode 0600
~/.bedrock/valkey-passwordValkey’s password, also stored in OpenBao, mode 0600
~/.bedrock/nats-route-password, nats-sys-password, nats-app-passwordNATS passwords, also stored in OpenBao, mode 0600
~/.bedrock/garage-rpc-secret, garage-admin-tokenGarage’s RPC secret and admin token, also stored in OpenBao, mode 0600
~/.bedrock/failover/The failover controller’s proof keys and exchange certificates