bedrock

Design decisions

Most parts of bedrock replaced something else that was tried or considered first. This page lists the main decisions and the reasons for them.

#Platform

ChosenInstead ofReason
RKE2 with the CIS profilek3sAudit logging, Pod Security, secrets encryption and CIS compliance out of the box
One cluster per tenantNamespaces on a shared clusterIsolation as a property of the design, not of configuration
Hand-written, pinned manifestsHelmWhat is in the repository is exactly what runs
Bootstrap once, then reviewed kubectl diff and applyArgo CD or another GitOps controllerWrite access to the repository would equal cluster admin, and a git host becomes attack surface

#Storage

ChosenInstead ofReason
GarageMinIOMinIO’s enterprise edition is paid and licence gated, and the community edition is no longer maintained
GarageCeph RGWCeph is heavy in memory and operations for a job that only needs S3
Three Garage pods, each on a local volumeOne Garage pod on a replicated volumeNo single point of failure, and no data replicated twice
LonghornCeph RBD and CephFSLonghorn already provides block and shared volumes and runs on any node

#Network

ChosenInstead ofReason
CiliumNetworkPolicyKubernetes NetworkPolicyCluster-wide deny, naming the API server as an entity, FQDN rules and layer 7 rules
Cilium kube-proxy replacementkube-proxy alongside CiliumWith both handling services, traffic from the node was rewritten and lost its identity, so host rules never matched
ClusterIP and port-forwardDashboards on NodePortsA datacenter has no trusted LAN
Cluster DNS with no egressLetting CoreDNS reach an upstream resolverA path out for DNS is a path out for data
Per-component policy in applicationAllow-all inside the namespaceOne compromised pod could otherwise silence alerts and delete logs

#Policy and security

ChosenInstead ofReason
Kyverno on application namespaces onlyKyverno on every namespaceStorage and database recovery must never depend on a policy engine
Priority classes set in each manifestA Kyverno mutationPriority is scheduling, not security, so it should not sit behind a webhook
A small privileged node-agents namespaceHost agents inside applicationTwo agents needing the host made every monitoring pod eligible to run as root
OpenBao, one per tenantSealed secrets, SOPS or a shared vaultNo shared component holding every tenant’s keys
Falco to stdout, collected by AlloyfalcosidekickOne component fewer

#Operations

ChosenInstead ofReason
A single Alertmanager per tenantA highly available AlertmanagerPrometheus keeps evaluating and showing firing alerts while Alertmanager restarts
Local etcd snapshotsSnapshots to external S3The bootstrap must work with no outside services, and etcd can be rebuilt from the repository
Single replicas for Tempo, Grafana and OpenCostHigh availability everywhereThey lose nothing on restart