How an application is deployed

Every application reaches this cluster the same way: a HelmRelease in git, reconciled by Flux, pointing at a chart and an image tag that automation keeps current. I do not run helm upgrade, and nobody edits an image tag by hand.

The repository layout

path	holds
clusters/default/	the Kustomizations Flux applies, and what depends on what
infrastructure/controllers/	controllers: Cilium, Flux, cert-manager, ESO, Tekton
apps/base/	applications, one directory each
apps/namespaces/	the namespaces themselves
terraform/environments/	declarative state the cluster cannot own: DNS, OIDC apps, node config
.tekton/	Pipelines-as-Code build pipelines

A Flux Kustomization per tree, applied by a root Kustomization, so the whole cluster is four documents deep from the entry point.

The release

This site's own release is the smallest complete example:

# kubexa.tech — the top-level site: documentation, blog, about.
# Container built by Pipelines-as-Code (.tekton/kubexa-site-build.yaml), tag owned
# by image automation, chart from the community generic chart.
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kubexa-site
  namespace: kubexa-site
spec:
  interval: 1h
  install:
    remediation:
      retries: 3
  upgrade:
    remediation:
      retries: 3
  chart:
    spec:
      chart: generic
      sourceRef:
        kind: HelmRepository
        name: helmforge
        namespace: kubexa-site
      interval: 1h
      version: "3.1.8"
  values:
    nameOverride: kubexa-site
    replicaCount: 2
    image:
      repository: harbor.kubexa.tech/home-ops/kubexa-site
      tag: latest # {"$imagepolicy": "flux-system:kubexa-site:tag"}
      digest: "sha256:efc613d149355eaa9ca87b5d8cbeba95e8a16f3af3ffdcc3224c579f86e2212d" # {"$imagepolicy": "flux-system:kubexa-site:digest"}
      pullPolicy: Always
    imagePullSecrets:
      - name: harbor-home-ops
    nodeSelector:
      kubernetes.io/arch: amd64
    service:
      type: ClusterIP
      port: 80
      targetPort: 8080
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 3
      periodSeconds: 20
    readinessProbe:
      httpGet:
        path: /readyz
        port: 8080
      initialDelaySeconds: 2
      periodSeconds: 10
    resources:
      requests:
        cpu: 10m
        memory: 32Mi
      limits:
        memory: 192Mi
    podSecurityContext:
      runAsNonRoot: true
      runAsUser: 65532
      runAsGroup: 65532
      seccompProfile:
        type: RuntimeDefault
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: ["ALL"]
    env:
      - name: KUBEXA_SITE_ADDR
        value: 0.0.0.0:8080
      - name: RUST_LOG
        value: info

Three things in there matter. chart.spec names a chart and a version, so a chart upgrade is a diff I can read. image.tag carries a $imagepolicy marker, which is how the image automation knows it may rewrite that field. imagePullSecrets names a Secret that External Secrets fills from OpenBao, so no credential is ever in git.

The build

The image is built by Pipelines-as-Code on a push that touches apps/kubexa-site/**, using kaniko inside the cluster with no Docker daemon and no privileged pod:

---
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: kubexa-site-build
  annotations:
    pipelinesascode.tekton.dev/on-event: "[push]"
    pipelinesascode.tekton.dev/on-target-branch: "[main]"
    # Content changes only. Flux image automation rewrites the digest in
    # apps/kubexa-site/release.yaml after every build, and on-path-change fired
    # on that rewrite too, so each build scheduled the next one.
    pipelinesascode.tekton.dev/on-cel-expression: "event == 'push' && target_branch == 'main' && (files.all.exists(f, f.startsWith('apps/kubexa-site/') && f != 'apps/kubexa-site/release.yaml') || files.all.exists(f, f == '.tekton/kubexa-site-build.yaml'))"
spec:
  taskRunTemplate:
    podTemplate:
      nodeSelector:
        site: svt
  serviceAccountName: default
  params:
    - name: repo_url
      value: "{{repo_url}}"
    - name: revision
      value: "{{revision}}"
  workspaces:
    - name: repo
      volumeClaimTemplate:
        spec:
          accessModes:
            - ReadWriteMany
          resources:
            requests:
              storage: 2Gi
          storageClassName: ceph-cephfs
    - name: basic-auth
      secret:
        secretName: "{{ git_auth_secret }}"
    - name: docker-credentials
      secret:
        secretName: docker-config-kaniko-home-ops
        items:
          - key: config.json
            path: config.json
  timeouts:
    pipeline: 1h
  pipelineSpec:
    params:
      - name: repo_url
      - name: revision
    workspaces:
      - name: repo
      - name: basic-auth
      - name: docker-credentials
    tasks:
      - name: fetch-source
        taskSpec:
          workspaces:
            - name: repo
            - name: basic-auth
          steps:
            - name: clone
              image: docker.io/alpine/git:2.54.0
              script: |
                set -euo pipefail
                # PaC mounts git credentials bound to the external FQDN; the pod
                # cannot reach that address, so remap to the internal service.
                CREDS="$(workspaces.basic-auth.path)/.git-credentials"
                sed -e 's#^https://#http://#' -e 's#@git.kubexa.tech#@gitea-http.gitea:3000#' "$CREDS" > "$HOME/.git-credentials"
                chmod 600 "$HOME/.git-credentials"
                export GIT_TERMINAL_PROMPT=0
                find "$(workspaces.repo.path)" -mindepth 1 -maxdepth 1 -exec rm -rf {} + 2>/dev/null || true
                git -c "credential.helper=store --file=$HOME/.git-credentials" clone --depth 1 --no-checkout \
                  http://gitea-http.gitea:3000/home-ops/talos-stack.git "$(workspaces.repo.path)"
                git -C "$(workspaces.repo.path)" -c "credential.helper=store --file=$HOME/.git-credentials" \
                  fetch --depth 1 origin "$(params.revision)"
                git -C "$(workspaces.repo.path)" checkout FETCH_HEAD
        workspaces:
          - name: repo
            workspace: repo
          - name: basic-auth
            workspace: basic-auth
      - name: test
        runAfter:
          - fetch-source
        taskSpec:
          steps:
            - name: cargo-test
              image: docker.io/library/rust:1.99-slim
              workingDir: /workspace/source/apps/kubexa-site
              script: |
                set -euo pipefail
                # The embed macro reads content/ and assets/ at compile time, so the
                # tests run against exactly the files this revision ships.
                cargo test --release --locked 2>&1 | tail -40
          workspaces:
            - name: source
              mountPath: /workspace/source
        workspaces:
          - name: source
            workspace: repo
      - name: build-and-push
        runAfter:
          - test
        taskSpec:
          # The cluster's egress path is verified against this bundle, so kaniko
          # must trust it explicitly: without both the mount and SSL_CERT_FILE it
          # fails every registry pull with "certificate signed by unknown
          # authority". Same wiring as .tekton/kb-build.yaml.
          volumes:
            - name: ca-bundle
              configMap:
                name: kaniko-ca-bundle
          steps:
            - name: kaniko
              image: ghcr.io/osscontainertools/kaniko:v1.29.0
              workingDir: /workspace/source
              env:
                - name: DOCKER_CONFIG
                  value: /kaniko/.docker
                - name: SSL_CERT_FILE
                  value: /kaniko/ca/ca-certificates.crt
              volumeMounts:
                - name: ca-bundle
                  mountPath: /kaniko/ca/ca-certificates.crt
                  subPath: ca-certificates.crt
                  readOnly: true
              args:
                - --dockerfile=$(workspaces.source.path)/apps/kubexa-site/Dockerfile
                - --context=$(workspaces.source.path)
                - --destination=harbor.kubexa.tech/home-ops/kubexa-site:latest
                - --destination=harbor.kubexa.tech/home-ops/kubexa-site:$(params.revision)
                - --cache=true
                - --cache-repo=harbor.kubexa.tech/home-ops/kaniko-cache
                - --skip-tls-verify-registry=harbor.kubexa.tech
                - --snapshot-mode=redo
          params:
            - name: revision
          workspaces:
            - name: source
              mountPath: /workspace/source
            - name: docker-credentials
              mountPath: /kaniko/.docker
        params:
          - name: revision
            value: $(params.revision)
        workspaces:
          - name: source
            workspace: repo
          - name: docker-credentials
            workspace: docker-credentials

The pipeline pushes two tags: latest and the commit SHA. The ImagePolicy selects latest, ImageUpdateAutomation writes the resolved digest back into the release, and Flux applies the result. A rebuild therefore reaches production without a human touching a manifest.

What happens when I change something

1. git push to main.
2. Flux's source controller fetches within its interval, the apps-site Kustomization applies the manifests it owns.
3. If the image changed, the pipeline built it first; if the pipeline failed, the old image stays and the cluster does not move.
4. Anything I delete from git is pruned from the cluster on the same reconcile.

The failure mode to watch is a Kustomization stuck on DependencyNotReady, which means an upstream tree never became ready and nothing downstream applied. That is what kubectl get kustomization -n flux-system is for.