bedrock

Buckets and keys

All buckets are created by the garage-init Job in tenant-workload/storage/garage/buckets.yaml. Each bucket has its own access key, scoped to that bucket only, and no key has owner rights, so no workload can change bucket settings or delete a bucket. The Job writes each key into a Secret in the namespace that uses it, with the fields AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

#Buckets

BucketUsed forKey permissionsSecret
<cluster>-backupsWAL archive and base backups of each database clusterRead and write<cluster>-backups in the cluster’s namespace
The primary’s <cluster>-backupsRestoring the replica clusters from the primary’s archiveRead only<primary>-backups-ro in each replica’s namespace
loki-dataLoki chunks and indexRead and writeloki-data in application
tempo-dataTempo trace blocksRead and writetempo-data in application

With the clusters in the repository, that means one -backups bucket per database cluster, plus cluster-1-backups-ro handed to each replica cluster.

#Endpoint

SettingValue
Endpointhttp://garage.garage.svc:3900
Regiongarage

Clients built on recent AWS SDKs must set AWS_REQUEST_CHECKSUM_CALCULATION and AWS_RESPONSE_CHECKSUM_VALIDATION to when_required, because Garage does not accept the newer default checksums. The database clusters set both in their env.

#How the Job behaves

  • It waits until every Garage node is up, then applies the cluster layout if none exists.
  • It skips any bucket whose Secret already exists, so running it again never replaces a working key.
  • If a bucket exists but its Secret is gone, it creates a new key for the existing bucket. The old key keeps working until it is removed with the Garage admin API.
  • Its service account can create Secrets, but can only read the ones listed above, and only in the namespaces it is bound to.

#Who can reach Garage

Network policy limits S3 traffic to Loki, Tempo, the database namespaces, the blackbox exporter and Prometheus. Holding a key is not enough on its own: a pod in any other namespace cannot open a connection to port 3900.