#Commands
| Command | Runs |
|---|---|
make deploy | ansible-playbook -i tenant-workload/bootstrap/hosts.ini tenant-workload/bootstrap/bedrock-bootstrap.yml |
make teardown | ansible-playbook -i tenant-workload/bootstrap/hosts.ini tenant-workload/bootstrap/bedrock-teardown.yml |
make redeploy | Teardown, then deploy |
make check | Syntax check of the bootstrap playbook against the example inventory, and make check-allowlist |
make check-allowlist | Fails if the three registry allowlist patterns are not identical |
make rotate-exchange-cert | Signs a new exchange certificate for the observers and restarts them, see rotate the exchange certificate |
make rotate-proof-key | Rotates the failover controller’s proof key with an overlap, no downtime |
make rotate-grafana-password | Sets a new Grafana admin password in Grafana, OpenBao and on the controller, no restart, see rotate a secret |
make add-replica REPLICA=<name> | Adds a live replica behind the current primary, see add a live replica |
Set INVENTORY=path/to/hosts.ini to use another inventory file. The default hosts.ini is ignored by git; copy it from hosts.example.ini.
The bootstrap refuses to run once a failover has happened, because it would turn the new primary back into a replica. Tagged runs, --tags kyverno, longhorn, cnpg-operator, prometheus-operator or strimzi, skip the database and still work, as described in upgrade a component.
#Inventory groups
| Group | Members |
|---|---|
server_init | The first node, which creates the cluster |
server_join | The other server nodes, joined one at a time |
rke2_servers | Both groups together |
#Variables
| Variable | Default | Effect |
|---|---|---|
rke2_version | v1.36.4+rke2r1 | RKE2 release to install |
nats_enabled | false | Deploy NATS JetStream |
valkey_enabled | false | Deploy Valkey with Sentinel |
kafka_enabled | false | Deploy Strimzi and Kafka |
kubeconfig | ~/.bedrock/kubeconfig | Where the admin kubeconfig is written on the controller |
failover_enabled | true | Deploy the failover controller |
failover_dir | ~/.bedrock/failover | Where the proof keys and the exchange certificates are kept on the controller |
#Stages
| Stage | What runs |
|---|---|
| 1 | Node packages (open-iscsi, nfs-common, cryptsetup, lvm2), the etcd user, CIS kernel settings, the dm_crypt module |
| 2 | Audit policy, RKE2 configuration, Cilium configuration, RKE2 install from a checksummed release |
| 3 | The first server, then the others one at a time |
| 4 | kubectl, crictl and ctr links and the login profile on every node |
| 5 | Namespaces, priority classes and the ValidatingAdmissionPolicy |
| 6 | Kyverno, scaled to one admission replica per node, then its policies |
| 7 | Longhorn, with its RBAC rewritten without wildcards |
| 8 | CloudNativePG operator and prometheus-operator |
| 9 | Longhorn encryption key and storage classes, then OpenBao: three pods on raft, initialized with 5 key shares and a threshold of 3, and unsealed. Then the Secrets Store CSI driver and the OpenBao CSI provider, the platform passwords made once on the controller, and OpenBao’s kubernetes auth, the apps key value engine and one role per component. Then Garage and the bucket job, which read their secrets from OpenBao |
| 10 | cluster-1, the failover functions and the proof key loaded into it, the observer password set from OpenBao, the openbao and bedrock_app roles handed to OpenBao’s database engine, its first backup, the certificate sync, then cluster-2 and cluster-3 |
| 11 | Poolers, scheduled backups and disruption budgets |
| 12 | Event buses, if enabled |
| 13 | Loki, Alloy, Tempo, Prometheus and exporters, alerting, Grafana, OpenCost, Falco, Reloader, dashboards |
| 14 | Token automount turned off on every default service account |
| 15 | The failover controller: its namespace, admission policies, the database CA sync, proof key, exchange certificates, then the workloads and their alerts |
| 16 | Network policies, then a check that every namespace running pods has one |
#Files the bootstrap writes on the nodes
| Path | Contents |
|---|---|
/etc/rancher/rke2/config.yaml | RKE2 configuration, rendered from rke2-config.yaml.j2 for the first server and the joining ones |
/etc/rancher/rke2/bedrock-audit.yaml | API audit policy |
/var/lib/rancher/rke2/server/manifests/rke2-cilium-config.yaml | Cilium settings |
/etc/sysctl.d/60-rke2-cis.conf | CIS kernel settings and inotify limits |
/etc/modules-load.d/dm_crypt.conf | Loads dm_crypt at boot for volume encryption |
/etc/profile.d/rke2.sh | Sets KUBECONFIG and CRI_CONFIG_FILE at login |
/var/lib/rancher/rke2/server/logs/audit.log | The API audit log, shipped to Loki |
#Files the bootstrap writes on the controller
| Path | Contents |
|---|---|
~/.bedrock/kubeconfig | The admin kubeconfig |
~/.bedrock/openbao-init.json | OpenBao’s unseal key shares and root token, mode 0600 |
~/.bedrock/grafana-admin | Grafana’s admin password, also stored in OpenBao, mode 0600 |
~/.bedrock/valkey-password | Valkey’s password, also stored in OpenBao, mode 0600 |
~/.bedrock/nats-route-password, nats-sys-password, nats-app-password | NATS passwords, also stored in OpenBao, mode 0600 |
~/.bedrock/garage-rpc-secret, garage-admin-token | Garage’s RPC secret and admin token, also stored in OpenBao, mode 0600 |
~/.bedrock/failover/ | The failover controller’s proof keys and exchange certificates |