---
title: Nine months to a boring platform
description: The real timeline of this repository, from hand-built VMs on one host to Flux, KubeSpan, and the outage that deleted every namespace.
date: 2026-10-29
---

An earlier version of this post claimed the foundation took days. Git disagrees. The first
commit is `ac900836` on 2026-01-19, the repository holds over three thousand commits, and
one afternoon in July I deleted every application namespace at once.

Getting to a stack this ordinary took nine months.

## January: one host, applied by hand

The cluster began as virtual machines on a single host that I configured by clicking. Talos
stuck from the first week, because it hands you an API instead of a shell. Rook arrived on
2026-01-21 (`539137e5`) with its charts copied into the repository, since a single-node
Ceph install needed patches that upstream would not take. Gitea joined two weeks later
(`3cda697e`, 2026-02-03) next to Argo CD.

Storage came first, the forge second, automation last. Every later experiment needed both,
and the order saved me a rewrite.

For the first months the repository was a diary with manifests in it. That habit is why I
can still name the day each thing broke.

## February: a network and a reconciler

Cilium replaced the default CNI on 2026-02-12 (`509e31e8`), installed from Terraform. Argo
CD arrived on 2026-02-03 (`3cda697e`) and deployed its first workload six days later
(`f2f43a8d`). Tailscale showed up on 2026-02-21 (`d68c688d`) and turned a LAN cluster into
something I could reach from another country.

## March: I ruined a cluster

The commit from 2026-03-08 (`13235e0e`) reads: "Sadly, the cluster got ruined during
kubespan deployment, cluster was recreated." Cilium and KubeSpan fought over routing, and I
could not debug the failure yet, so I rebuilt the cluster.

That same week the Oracle node started as a manual OCI Terraform deployment (`2d93b31e`).
By 2026-03-20 a multi-site Traefik deployment spanned Oracle, worker-infra and worker-svt
(`5df84cec`). Tailscale carried the cross-region link without a VPN server.

On 2026-03-21 native routing with KubeSpan landed (`e28c795c`). This configuration is the
least obvious part of the stack, and both halves are required:

```yaml
# Talos machine config
machine:
  kubespan:
    enabled: true
    advertiseKubernetesNetworks: true
```

```yaml
# Cilium
routingMode: native
bpf:
  masquerade: true
  hostLegacyRouting:
    true # required: eBPF host routing bypasses the kernel
    # routing table and conflicts with KubeSpan
```

MTU returns to 1500 once the VXLAN header is gone, and remote-node masquerade stays enabled
for nodes behind another NAT. Sidero documents the Cilium constraint in the
[KubeSpan guide](https://docs.siderolabs.com/talos/v1.14/networking/kubespan). I found it
by losing a cluster first.

Rebuilding Ceph that week rebuilt its storage classes too. I renamed `rook-ceph-block` to
`ceph-block` (`7e374cad`, `e7a42375`), and the claims pointing at the old name came back
bound to nothing. Gitea and JupyterHub moved to the new class the same day (`7e374cad`,
`ba52ef71`) and started with empty volumes.

## May: Argo CD out, Flux in

With Argo CD I spent my time inventing an app-of-apps convention and debugging sync waves.
Flux ships one opinionated layout and a dependency graph. On 2026-05-20 the migration
landed (`2b236920`): `apps/`, `infrastructure/` and `clusters/`, reconciled as sources then
secrets then controllers then configs then apps.

Velocity improved. Consistency improved more, because the same pattern repeated across
forty applications and nothing outside the graph could exist. The history records the
wobble: on 2026-05-12, eight days before the migration, a commit message says "convert Flux
to ArgoCD" (`33ab5fb9`).

One day before that switch, external-dns went in (`aa17aa59`, 2026-05-19). I had never heard
of the project. It watches Gateway and HTTPRoute objects and writes the matching Cloudflare
records itself, so a hostname exists because an object exists. The Gateway carries the
public IP as an annotation (`2f72e37c`, 2026-05-29) and the records follow it when the
address changes. Hand-written DNS entries had been the last thing I edited outside git, and
that chore disappeared in an afternoon.

## July: one line deleted every namespace

On 2026-07-29 I removed `../../apps/namespaces/` from `infrastructure/configs` (`5302803c`).
That listing had let the configs Kustomization own every application namespace. When I
force-deleted the Kustomization, Flux pruned the namespaces, and Kubernetes removed
everything inside them. The whole application layer went with it.

Three fixes followed in two days. I made the namespaces Kustomization depend on controllers
(`98451cee`), so a prune race could not delete namespaces that other Kustomizations had yet
to create. I dropped references to two repositories that no longer existed, `e-nt-aras` and
`trading-code` (`cbb4081f`). Earlier that month the same shape appeared: I deleted
`git-credentials.yaml` and left it listed in `clusters/default/kustomization.yaml`, which
stalled the clusters Kustomization (`da595da8`).

These rules came out of that week and live in the repository now:

- a Namespace belongs to exactly one Kustomization, and moving it means turning `prune` off
  on the source first
- a `resources:` list cannot name a file that no longer exists
- namespaces are created before the things that need them, by dependency rather than luck

## October: the same lesson at smaller scale

On 2026-10-28 I moved `Namespace/monitoring` between Kustomizations. Flux garbage-collected
it, and the monitoring stack, Grafana and its CloudNativePG database went with it
(`1ceca0d6`). The `prune: disabled` annotation on the live object did not stop the garbage
collector.

## October: Terraform leaves Tekton

For most of the year a hand-written PipelineRun applied Terraform. Each environment had
its own `.tekton/*-tf-apply.yaml`, fired by Pipelines-as-Code on a path change, carrying
credentials through Vault and running `terraform apply` inside a pod. It worked, and it
meant twelve copies of the same pipeline with twelve chances to drift.

On 2026-10-24 I installed the Flux OpenTofu controller (`ad9d1af1`) and declared a
Terraform custom resource per environment (`24407ea0`), starting plan-only. Three days later
I deleted what that replaced (`16e50f2d`): fourteen inert PipelineRuns and the eventlistener,
triggerbinding and triggertemplate chain behind them.

Drift correction now happens without me. An environment whose plan is clean gets
`approvePlan: auto` (`802513c4`, `8a90751b`, `3ec26789`) and Flux applies it every time git
changes. Environments with real decisions wait for a person. I keep state in S3, providers
lock in git (`3bd39975`), and the controller that owns the plan also runs the apply.

## What the repository buys me

Nine months, over three thousand commits, two data wipes, one pivot and a pivot back. The
layer diagram on the front page compresses that work into eight boxes, and the wrong turns
stay in the commit log with their dates attached.

Concretely, the repository gives me three things: a date and a reason for every decision,
the ability to replace a node without asking anyone, and a stack I can delete and recreate
in an afternoon, which October proved by accident.

Documentation on this site follows the same order as the build, from the node upward,
because I need the layer below to hold before the next one matters. The milestones from this
post sit on the [timeline](/timeline), each with its date and commit.