bedrock

Architecture

A bedrock tenant is built in layers. Each one sits on the one below and assumes the one above can be broken into. Open a layer to see what runs in it and why it was picked.

  1. RuntimeFalcoReloader

    Falco

    Watches every node for runtime events such as a shell spawned in a container or a sensitive file read.

    Why Events go to stdout and Alloy ships them to Loki. Dropping falcosidekick left one less component to run.

    Reloader

    Restarts a component when the ConfigMap it reads changes.

    Why Scoped to ConfigMaps only. Secrets come from OpenBao and are read again from memory without restarts.

  2. ObservabilityPrometheus and AlertmanagerLoki and Tempo and AlloyGrafana

    Prometheus and Alertmanager

    Scrape every component and fire the alert rules that ship with the platform.

    Why Two Prometheus replicas spread across nodes. Alertmanager has no login of its own so only the network policy decides who reaches it.

    Loki and Tempo and Alloy

    Logs and traces. Alloy runs on every node and ships them in.

    Why Loki deletion is disabled so nobody inside the cluster can erase the logs.

    Grafana

    Dashboards for every layer read from Prometheus and Loki and Tempo.

    Why One replica by choice. Dashboards and datasources are files in ConfigMaps so a restart loses nothing.

  3. SecretsOpenBaoSecrets Store CSI driver

    OpenBao

    Three pods on raft hold every component's secrets. Each pod logs in as its own service account and reads only its own path.

    Why An audit trail and rotating credentials without a paywall. External Secrets was rejected because it copies secrets into etcd.

    When it fails Running pods keep their secrets in memory and keep working. A new pod waits until OpenBao is back.

    Secrets Store CSI driver

    Mounts each pod's secrets as files in memory and refreshes them every two minutes.

    Why Built from reviewed upstream parts in days instead of a custom security product in months.

  4. DataCloudNativePGFailover controllerBackups

    CloudNativePG

    Three PostgreSQL clusters. A primary with synchronous replication plus a live replica and a replica five minutes behind.

    Why The delayed replica keeps a copy from before a bad DROP for five minutes. Point in time restore covers the rest.

    When it fails One instance lost and CloudNativePG promotes inside the cluster. The whole cluster lost and the failover controller promotes the live replica.

    Failover controller

    bedrock's own controller. Observers vote with signed proof and a verifier checks four witnesses before anything moves.

    Why Timeouts promote on a guess. A network blip can look like a dead primary and two primaries means data that never merges back.

    When it fails Measured at 66 seconds with 0 bytes lost when cluster-1 was killed.

    Backups

    Write ahead logs stream to object storage and base backups run every six hours. Kept for 30 days.

    Why Restore any second of the last 30 days into a new cluster beside the running one.

  5. StorageLonghornGarage

    Longhorn

    Replicated block storage with every volume encrypted with LUKS.

    Why Ceph was rejected. Longhorn already does block and shared volumes for far less to operate.

    Garage

    S3 object storage for the backups and the logs. Three pods with three copies of every object.

    Why MinIO's community edition is unmaintained and its successor is paid. Ceph's gateway is heavy for an S3 only job.

    When it fails No single pod holds the only copy so losing one loses nothing.

  6. AdmissionKyvernoPod SecurityValidatingAdmissionPolicy

    Kyverno

    Admission policies written in CEL. Resource limits and the registry allowlist and the copies of each database's CA.

    Why Policies are code in the repo and every image must be pinned by digest from an allowed registry.

    Pod Security

    Platform services run at the restricted level. Only the node agents that need the host run privileged in their own namespace.

    Why Built into Kubernetes with nothing extra to run.

    ValidatingAdmissionPolicy

    Gates every action of the failover controller inside the API server itself.

    Why It lives in the API server so the controller's pods can always be recreated even while Kyverno is down.

  7. NetworkCiliumHubble

    Cilium

    Default deny in every namespace. Each flow is allowed by name and every plain HTTP call is checked at layer 7. Traffic between nodes is encrypted with WireGuard.

    Why No pod can silence an alert or delete logs or reach an admin API. Nothing reaches the internet.

    Hubble

    Records every flow and every flow that a policy dropped.

    Why Probing gets seen. A dropped flow is evidence.

  8. KubernetesRKE2The bootstrap

    RKE2

    Three control plane nodes with the CIS profile and audit logging and secrets encrypted in etcd.

    Why k3s was rejected. Audit and Pod Security and the CIS profile come built in.

    The bootstrap

    One Ansible run from empty machines to a verified tenant.

    Why No GitOps controller in the cluster. Write access to a repo it watches would equal cluster admin.