bedrock

Rotate the proof key

Postgres signs every proof of life with the active key in bedrock.bedrock_proof_key. The verifier checks each signature against the keys in the bedrock-proof-keys Secret. The keys are kept on the controller machine in ~/.bedrock/failover/proof-keys/, one file per key, named by the time it was made, such as k20261101180000.

A rotation adds the new key everywhere before postgres uses it, so no signature is ever made with a key the verifier does not know.

#Run it

From the repository root, on the controller machine:

shell
make rotate-proof-key

The playbook is tenant-workload/bootstrap/rotate-proof-key.yml. It runs these steps:

StepEffect
1Generates a new 32-byte key in ~/.bedrock/failover/proof-keys/
2Inserts it into bedrock.bedrock_proof_key on the primary, inactive. The primary is found from the bedrockctl.com/db-role labels, so this works after a failover too
3Rebuilds bedrock-proof-keys with the old and the new key and restarts the verifier
4Makes the new key the only active key, so postgres signs with it from the next round
5Waits 60 seconds, so every round signed with the old key has closed
6Deletes the old keys from postgres and from the controller machine
7Rebuilds bedrock-proof-keys with the new key only and restarts the verifier

The key never appears on a command line. It goes to postgres through standard input and to the Secret from the file.

#Check it

shell
kubectl -n bedrock-failover logs deploy/bedrock-verifier | grep 'verifier started'
kubectl -n bedrock-failover get configmap bedrock-round -o jsonpath='{.data.round}'

The verifier logs how many keys it loaded when it starts. After the rotation it is 1. The round’s validUps still lists observers for every primary, which means the new signatures verify.

#If it stops half way

Every step can be run again. If the playbook fails after step 3, the verifier trusts both keys, so nothing breaks. Run make rotate-proof-key again, or finish by hand: make sure only one key file is left in ~/.bedrock/failover/proof-keys/, that the same key is the only row in bedrock_proof_key, and rebuild the Secret from that directory.