bedrock

Alert rules

Most rules live in tenant-workload/observability/alerting/alert-rules.yaml as one PrometheusRule. Falco’s rules live next to Falco in tenant-workload/observability/falco/falco.yaml. Three more, PostgresDropStatement, FalcoCriticalEventLog and AuditLogSilent, are Loki rules in tenant-workload/observability/loki/loki.yaml. They read log lines instead of metrics.

#Nodes

AlertSeverityForFires when
NodeHighCPUwarning10mCPU use is above 85%
NodeHighMemorywarning10mMemory use is above 90%
NodeMemoryCriticalcritical5mMemory use is above 95%
NodeDiskSpaceLowwarning10mA filesystem is above 85%
NodeDiskSpaceCriticalcritical5mA filesystem is above 92%
NodeDowncritical3mnode-exporter on a node cannot be scraped

#Workloads

AlertSeverityForFires when
PodCrashLoopingwarning5mA container restarted more than 3 times in 15 minutes
PodNotReadywarning15mA pod is Pending or Unknown
ContainerNearMemoryLimitwarning5mA container uses more than 90% of its memory limit
PVCAlmostFullwarning10mA volume is above 85%
PVCCriticalcritical5mA volume is above 92%
PodFailedwarning5mA pod is in phase Failed
JobFailedwarning5mA Job has failed pods
DaemonSetIncompletewarning10mA DaemonSet has nodes without a ready pod
StatefulSetNotReadywarning15mA StatefulSet has fewer ready replicas than desired

#Databases

AlertSeverityForFires when
CNPGReplicationLagHighwarning5mA replica is more than 30 seconds behind. Instances with recovery_min_apply_delay set, the delayed replicas, are left out
CNPGReplicationLagCriticalcritical5mA replica is more than 120 seconds behind, delayed replicas left out
CNPGInstanceDowncritical3mAn instance’s metrics cannot be scraped
CNPGDelayedReplicaStuckwarning10mA delayed replica, found by its recovery_min_apply_delay, is more than 600 seconds behind instead of about 300
WALArchivingFailingcriticalimmediatelyWAL archiving to Garage failed in the last 10 minutes
CNPGRowsDeletedSpikecritical1mRows are deleted at more than 10 times the usual rate on whichever cluster is primary. Standbys never count deletes they replay
CNPGRowsUpdatedSpikecritical1mRows are updated at more than 10 times the usual rate
CNPGStackedInstanceswarning15mMore than one instance of any database cluster runs on the same node
CNPGBackupTooOldcritical10mThe last base backup is more than 8 hours old

#Platform

AlertSeverityForFires when
ScrapeTargetDownwarning5mPrometheus cannot scrape a target
EndpointDowncritical2mA blackbox probe fails
KyvernoAdmissionDegradedcritical2mFewer than a majority of Kyverno admission replicas are ready
KyvernoAdmissionDowncritical1mNo Kyverno admission replica is ready
RuleEvaluationFailingcritical5mPrometheus or Loki’s ruler fails to evaluate a rule group. A rule that cannot evaluate can never fire
AlertSendingFailingcritical10mPrometheus or Loki’s ruler cannot deliver alerts to Alertmanager
AlertmanagerNotificationsFailingcritical10mAlertmanager fails to deliver through one of its integrations
PrometheusConfigReloadFailedwarning10mPrometheus rejected its last configuration and keeps the previous one
AlertmanagerConfigReloadFailedwarning10mAlertmanager rejected its last configuration and keeps the previous one
WatchdognonealwaysAlways. It proves the alerting pipeline works.

#Storage, certificates and etcd

AlertSeverityForFires when
LonghornVolumeDegradedwarning10mA Longhorn volume has lost replicas
LonghornVolumeFaultedcritical1mA Longhorn volume has no healthy replica
GarageDiskLowwarning15mLess than 15% of a Garage data disk is free
GarageDiskCriticalcritical5mLess than 5% of a Garage data disk is free
CertificateExpiringSoonwarning1hA probed certificate expires within 30 days
CertificateExpiryCriticalcritical10mA probed certificate expires within 7 days
EtcdNoLeadercritical1mAn etcd member has no leader
EtcdFrequentLeaderChangeswarning5mMore than 3 leader changes in an hour
EtcdDatabaseNearQuotawarning10mThe etcd database uses more than 80% of its quota
EtcdSlowDiskwarning10mThe 99th percentile WAL fsync takes more than 500 ms

#Runtime security

AlertSeverityForFires when
FalcoCriticalEventcriticalimmediatelyA Falco rule of priority Emergency, Alert or Critical matched in the last 5 minutes
FalcoWarningEventwarningimmediatelyA Falco rule of priority Error or Warning matched in the last 15 minutes
FalcoDowncritical5mFalco on a node cannot be scraped
FalcoCriticalEventLogcriticalimmediatelyA Falco event of priority Emergency, Alert or Critical, or a reverse shell, appears in Loki in the last 5 minutes. It reads the event itself, so it fires on the very first one

#Failover

These live in tenant-workload/failover/monitoring.yaml.

AlertSeverityForFires when
BedrockFailoverPagecriticalimmediatelyThe decider or the verifier needs a person, with the reason
BedrockPrimaryProvenDeadcriticalimmediatelyA primary is proven dead
BedrockPrimaryContradictioncritical1mA primary signed that it is alive while three witnesses say dead; nothing will act
BedrockFallbackDatabaseDowncriticalimmediatelyThe normal path is broken and the checker sees the database down
BedrockFailoverLostWriteswarningimmediatelyA failover promoted a replica that was behind the last signed write. The signed position includes the observers’ own heartbeat writes, so a few bytes after a hard crash are usually heartbeats
BedrockFailoverLostWritesLargecriticalimmediatelyThe gap is over 64 KiB, more than heartbeats alone can explain, or it is unknown (-1)
BedrockExchangeCertExpiringwarning1hThe observers’ exchange certificate expires within 30 days
BedrockExchangeCertExpiryCriticalcritical10mIt expires within 7 days
BedrockPrimaryNotWritingcriticalimmediatelyEvery observer has seen the primary answer without committing a write for over two minutes
BedrockFailoverClaimStuckcritical10mA failover has been open for ten minutes
BedrockDeciderNoLeadercritical1mThe deciders do not have exactly one leader
BedrockVerifierStalledcritical1mThe verifier has not closed a round for a minute
BedrockCheckerSilentwarning1mThe checker has not finished a round for a minute
BedrockObserverBlindwarning2mAn observer cannot discover the clusters or publish its vote
BedrockObserverAlonewarning5mAn observer reaches no peers
BedrockFailoverBlindcritical2mThe verifier has neither a signed up nor a proof of death for a primary, so a real failover cannot happen
BedrockNoLiveReplicacritical10mA primary has no ready live replica to promote. Add one with make add-replica
BedrockCheckerHoldsFailovercritical10mThe checker has held every failover for ten minutes

#Logs

AlertSeverityForFires when
PostgresDropStatementcriticalimmediatelyAny postgres container logs a DROP statement, whatever its cluster is called
AuditLogSilentcritical5mNo API audit log line has reached Loki for 15 minutes. PostgresDropStatement cannot fire while this fires
AlloyDroppingLogscriticalimmediatelyAlloy gave up on log lines after every retry, in the last 10 minutes
AlloyPushFailingwarning10mAlloy’s pushes to Loki get a non-2xx answer. A 403 means a layer 7 rule refuses the push