Nine months to a boring platform

An earlier version of this post claimed the foundation took days. Git disagrees. The first commit is ac900836 on 2026-01-19, the repository holds over three thousand commits, and one afternoon in July I deleted every application namespace at once.

Getting to a stack this ordinary took nine months.

January: one host, applied by hand

The cluster began as virtual machines on a single host that I configured by clicking. Talos stuck from the first week, because it hands you an API instead of a shell. Rook arrived on 2026-01-21 (539137e5) with its charts copied into the repository, since a single-node Ceph install needed patches that upstream would not take. Gitea joined two weeks later (3cda697e, 2026-02-03) next to Argo CD.

Storage came first, the forge second, automation last. Every later experiment needed both, and the order saved me a rewrite.

For the first months the repository was a diary with manifests in it. That habit is why I can still name the day each thing broke.

February: a network and a reconciler

Cilium replaced the default CNI on 2026-02-12 (509e31e8), installed from Terraform. Argo CD arrived on 2026-02-03 (3cda697e) and deployed its first workload six days later (f2f43a8d). Tailscale showed up on 2026-02-21 (d68c688d) and turned a LAN cluster into something I could reach from another country.

March: I ruined a cluster

The commit from 2026-03-08 (13235e0e) reads: "Sadly, the cluster got ruined during kubespan deployment, cluster was recreated." Cilium and KubeSpan fought over routing, and I could not debug the failure yet, so I rebuilt the cluster.

That same week the Oracle node started as a manual OCI Terraform deployment (2d93b31e). By 2026-03-20 a multi-site Traefik deployment spanned Oracle, worker-infra and worker-svt (5df84cec). Tailscale carried the cross-region link without a VPN server.

On 2026-03-21 native routing with KubeSpan landed (e28c795c). This configuration is the least obvious part of the stack, and both halves are required:

# Talos machine config
machine:
  kubespan:
    enabled: true
    advertiseKubernetesNetworks: true

# Cilium
routingMode: native
bpf:
  masquerade: true
  hostLegacyRouting:
    true # required: eBPF host routing bypasses the kernel
    # routing table and conflicts with KubeSpan

MTU returns to 1500 once the VXLAN header is gone, and remote-node masquerade stays enabled for nodes behind another NAT. Sidero documents the Cilium constraint in the KubeSpan guide. I found it by losing a cluster first.

Rebuilding Ceph that week rebuilt its storage classes too. I renamed rook-ceph-block to ceph-block (7e374cad, e7a42375), and the claims pointing at the old name came back bound to nothing. Gitea and JupyterHub moved to the new class the same day (7e374cad, ba52ef71) and started with empty volumes.

May: Argo CD out, Flux in

With Argo CD I spent my time inventing an app-of-apps convention and debugging sync waves. Flux ships one opinionated layout and a dependency graph. On 2026-05-20 the migration landed (2b236920): apps/, infrastructure/ and clusters/, reconciled as sources then secrets then controllers then configs then apps.

Velocity improved. Consistency improved more, because the same pattern repeated across forty applications and nothing outside the graph could exist. The history records the wobble: on 2026-05-12, eight days before the migration, a commit message says "convert Flux to ArgoCD" (33ab5fb9).

One day before that switch, external-dns went in (aa17aa59, 2026-05-19). I had never heard of the project. It watches Gateway and HTTPRoute objects and writes the matching Cloudflare records itself, so a hostname exists because an object exists. The Gateway carries the public IP as an annotation (2f72e37c, 2026-05-29) and the records follow it when the address changes. Hand-written DNS entries had been the last thing I edited outside git, and that chore disappeared in an afternoon.

July: one line deleted every namespace

On 2026-07-29 I removed ../../apps/namespaces/ from infrastructure/configs (5302803c). That listing had let the configs Kustomization own every application namespace. When I force-deleted the Kustomization, Flux pruned the namespaces, and Kubernetes removed everything inside them. The whole application layer went with it.

Three fixes followed in two days. I made the namespaces Kustomization depend on controllers (98451cee), so a prune race could not delete namespaces that other Kustomizations had yet to create. I dropped references to two repositories that no longer existed, e-nt-aras and trading-code (cbb4081f). Earlier that month the same shape appeared: I deleted git-credentials.yaml and left it listed in clusters/default/kustomization.yaml, which stalled the clusters Kustomization (da595da8).

These rules came out of that week and live in the repository now:

• a Namespace belongs to exactly one Kustomization, and moving it means turning prune off on the source first
• a resources: list cannot name a file that no longer exists
• namespaces are created before the things that need them, by dependency rather than luck

October: the same lesson at smaller scale

On 2026-10-28 I moved Namespace/monitoring between Kustomizations. Flux garbage-collected it, and the monitoring stack, Grafana and its CloudNativePG database went with it (1ceca0d6). The prune: disabled annotation on the live object did not stop the garbage collector.

October: Terraform leaves Tekton

For most of the year a hand-written PipelineRun applied Terraform. Each environment had its own .tekton/*-tf-apply.yaml, fired by Pipelines-as-Code on a path change, carrying credentials through Vault and running terraform apply inside a pod. It worked, and it meant twelve copies of the same pipeline with twelve chances to drift.

On 2026-10-24 I installed the Flux OpenTofu controller (ad9d1af1) and declared a Terraform custom resource per environment (24407ea0), starting plan-only. Three days later I deleted what that replaced (16e50f2d): fourteen inert PipelineRuns and the eventlistener, triggerbinding and triggertemplate chain behind them.

Drift correction now happens without me. An environment whose plan is clean gets approvePlan: auto (802513c4, 8a90751b, 3ec26789) and Flux applies it every time git changes. Environments with real decisions wait for a person. I keep state in S3, providers lock in git (3bd39975), and the controller that owns the plan also runs the apply.

What the repository buys me

Nine months, over three thousand commits, two data wipes, one pivot and a pivot back. The layer diagram on the front page compresses that work into eight boxes, and the wrong turns stay in the commit log with their dates attached.

Concretely, the repository gives me three things: a date and a reason for every decision, the ability to replace a node without asking anyone, and a stack I can delete and recreate in an afternoon, which October proved by accident.

Documentation on this site follows the same order as the build, from the node upward, because I need the layer below to hold before the next one matters. The milestones from this post sit on the timeline, each with its date and commit.