Run these checks after every deploy. They take a few minutes and confirm the cluster is both healthy and locked down.
#Before you begin
You need kubectl pointed at the tenant, either on the controller with export KUBECONFIG=~/.bedrock/kubeconfig or on any node, where the login profile sets it for you.
#Check the pods
kubectl get pods -AEvery pod should be Running. A handful of Completed jobs in kube-system are the RKE2 installers and are normal. To list only the pods that are not healthy:
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=SucceededThe output should be empty.
#Check the network policies
kubectl get cnp,ccnp -AConfirm three things:
cluster-default-denyexists as a cluster-wide policy- every namespace that runs pods has its own policy
- every policy shows
VALIDasTrue
The bootstrap already fails if a namespace has no policy, so a gap here usually means a namespace was added by hand afterwards.
#Check the databases
kubectl get clusters.postgresql.cnpg.io -AEvery cluster should report Cluster in healthy state. To confirm synchronous replication on cluster-1, ask its primary:
kubectl -n cluster-1 exec cluster-1-1 -- psql -U postgres -c "show synchronous_standby_names;"The result should list the standby instances, for example ANY 1 ("cluster-1-2"). If cluster-1-1 is not the current primary, run the command on the instance that is, shown by kubectl -n cluster-1 get cluster cluster-1.
#Check storage encryption
kubectl -n longhorn-system get volumes.longhorn.io \
-o custom-columns=NAME:.metadata.name,ENCRYPTED:.spec.encryptedEvery volume shows true.
#Run the CIS benchmark
On one of the nodes:
curl -sSLO https://github.com/aquasecurity/kube-bench/releases/download/v0.16.0/kube-bench_0.16.0_linux_amd64.tar.gz
tar xzf kube-bench_0.16.0_linux_amd64.tar.gz
./kube-bench run --config-dir ./cfg --benchmark rke2-cis-1.9 --skip 4.3.1 | tail -20Expect 85 checks passed and 1 failed (5.1.6). Any other failure is a regression.
#Check the alerts
Forward Prometheus and open its alerts page:
kubectl -n application port-forward svc/prometheus-operated 9090:9090At http://localhost:9090/alerts, only Watchdog should be firing.