bedrock

Inside a tenant cluster

A tenant cluster is a set of nodes running a fixed stack. Each layer builds on the one below it, and the order matters both when the cluster is installed and when something fails.

Tenant applicationsEach app in its own restricted namespace, reaching only what its policy namesObservabilityPrometheus, Alertmanager, Loki, Tempo, GrafanaAlloy, OpenCost, exportersRuntimeFalco on every nodeReloaderEvent busesNATS, Valkey, Kafkaoff until neededDataCloudNativePG: cluster-1 primary, cluster-2 live replica, cluster-3 delayed replica, PgBouncer poolersGarage S3: backups, logs, traces, replicated, encrypted at restBlock storageLonghorn, replicated volumes across the nodesAdmissionPod Security restricted, Kyverno policies, ValidatingAdmissionPolicy, registry allowlistNetworkCilium, kube-proxy replacement, default deny, WireGuard between nodes, HubbleRKE2 CIS on Debian nodes: etcd, secrets encryption, audit log
A tenant cluster from the machine upwards. Applications run in their own namespaces on top of the platform services.

#Nodes and Kubernetes

The nodes are plain Linux machines (Debian 12 in the reference setup). bedrock does not assume a hypervisor or a cloud. It needs SSH access with keys and passwordless sudo, nothing more.

Kubernetes is RKE2 with its CIS profile turned on, secrets encrypted at rest and an audit policy that records who did what at the API. Every node is a server, so etcd keeps a majority as long as most of the nodes are up.

#Network

Cilium is the network plugin. It replaces kube-proxy, so one component handles all service traffic, and it encrypts pod traffic between nodes with WireGuard. Every pod in every namespace starts with all traffic denied, and each flow the platform needs is allowed by name. Hubble records flows so that a blocked connection is visible. See networking.

#Admission

Before a pod runs, three checks look at it:

  • Pod Security Admission enforces the restricted level on almost every namespace, so root, privileged and host-path pods are refused
  • Kyverno checks images, tags and resource limits
  • a native ValidatingAdmissionPolicy keeps the high priority classes for the components that need them

See the security model for how these fit together.

#Storage

Longhorn provides block volumes for everything that needs a disk. Garage runs on top of Longhorn and provides S3 for backups, logs and traces. Garage keeps a copy of every object on several nodes, and its disks are encrypted. See data and storage.

#Databases

CloudNativePG runs a primary PostgreSQL cluster and replica clusters that follow it, one live and one delayed. Applications connect through a connection pooler and a stable service name, so they do not need to know which cluster is active. If the primary cluster is lost, bedrock’s failover controller promotes the live replica.

#Secrets

OpenBao runs as three pods on raft in its own namespace. The bootstrap initializes it and unseals it, and keeps the key shares on the controller machine only.

#Observability and runtime security

Prometheus collects metrics, Loki stores logs and audit events, Tempo stores traces and Grafana shows all three. Prometheus evaluates the alert rules and Alertmanager routes the ones that fire. Falco watches system calls on every node and reports suspicious behaviour. See observability.

#Namespaces

Components are grouped by what they need, not by product. The rule is that a namespace is only as privileged as its most privileged pod, so the few components that need the host live apart from everything else.

NamespaceHoldsPod Security
applicationPrometheus, Alertmanager, Grafana, Loki, Tempo, OpenCost, exportersrestricted
node-agentsAlloy and node-exporter, which read from the hostprivileged
cluster-1, cluster-2, cluster-3The PostgreSQL clusters and their poolersrestricted
bedrock-failoverThe failover controllerrestricted
openbaoOpenBaorestricted
garageS3 object storagerestricted
falcoRuntime detectionprivileged
longhorn-systemBlock storageprivileged
reloaderRestarts workloads when their configuration changesrestricted

The full list is in the namespaces reference.