Ceph and PostgreSQL

Ceph

Rook runs Ceph across the three nodes that have disks. Every object is written three times, so a disk or a node failing costs replication traffic instead of data.

storage class	backed by	used for
ceph-block	RBD	databases, single-writer volumes
ceph-filesystem	CephFS	build caches, shared scratch (Tekton workspaces)

The honest trade: Ceph wants three failure domains and adds latency a local SSD does not have. Replication plus a recovery story beats raw throughput for a homelab platform.

PostgreSQL

Databases are Cluster resources, not StatefulSets with a hand-written init script. CloudNativePG creates the primary, the replicas, the credentials and the backup schedule; a Pooler puts PgBouncer in front where a service needs connection fan-in.

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: grafana-db
spec:
  instances: 2
  storage:
    storageClass: ceph-block
    size: 10Gi
  backup:
    barmanObjectStore:
      destinationPath: s3://cnpg/grafana

Continuous WAL shipping to object storage means a bad migration is a point-in-time restore rather than a lost evening. Application credentials are not in the manifests: the operator mints users, External Secrets pulls the connection string, and the workload mounts a Secret.

What this buys

A node can die while I sleep. Ceph re-replicates, Postgres promotes a replica, Flux schedules the workloads onto what capacity remains, and the next plan shows no drift.