docs: describe what is, now that the move is done
Validate Helm Charts / Detect changed charts (pull_request) Successful in 3s
Validate Helm Charts / Build ${{ fromJSON(needs.detect-changed-charts.outputs.charts) }} (pull_request) Skipped
Validate Helm Charts / Trivy config ${{ fromJSON(needs.detect-changed-charts.outputs.scan-charts) }} (pull_request) Skipped
Validate Helm Charts / Install-test ${{ fromJSON(needs.detect-changed-charts.outputs.install-charts) }} (pull_request) Skipped
Validate Helm Charts / Resource quantities (pull_request) Successful in 7s

The documentation still described a migration in progress, and the worst
line claimed the install-test "cannot run on Gitea at all" - true two
days ago, the opposite of true now, and exactly the sort of sentence
someone builds a wrong assumption on.

README and CLAUDE.md now state what holds: everything runs on Gitea,
github.com is a mirror with Actions and Renovate off, branch protection
with a signature requirement is active, and the tool is tea. The move
itself is not retold - that is Gitea #14.

"Where this repo lives" keeps its place but turns around: it is no
longer a list of what blocks us, it is what a Gitea runner does
differently from a GitHub one, with the measurements. Newly written
down because it holds permanently and stood nowhere: a matrix runs
concurrently in ONE job under ONE status context, so a required check
per chart is impossible and the context name carries the unexpanded
matrix expression; expected runtimes, so a later deviation is visible
as one, with the note that the difference to GitHub is structural - one
runner for both legs against a VM per leg - and not performance.

The six assumptions a GitHub runner had been meeting quietly are split
rather than listed. Five are environment (MTU, fs.inotify, missing
kubectl and envsubst, kubeconfig on 0.0.0.0): they fail loudly, say
nothing about our code, and are fixed on the node side. The sixth gets
its own block because it is the one that does say something about our
code - the hard-coded k3d cluster name was wrong on GitHub too and only
never showed there, because a VM per matrix leg meant two legs never saw
each other. A generous environment is no friend when you want to know
whether something is correct.

Two rules generalised out of this round, as rules rather than as a
story: a check that a broken system passes is worse than no check (with
the two vacuous forms that caused it), and green does not mean checked,
pointing at Gitea #24 as an open case of the same kind - explicitly not
claimed as solved.

Also fixes a hazard the move created: most #n in these files name closed
GitHub issues that were never migrated, while Gitea resolves a bare #n
against its own counter - #22 there is an unrelated throwaway pull
request, not the install-test issue the text means. Stated once at the
top instead of rewriting sixteen references into numbers that do not
exist.
This commit was merged in pull request #26.
This commit is contained in:
jklein committed 2026-09-17 17:13:39 +02:00
1 parent 82b90475ce
commit 60670182ff
2 files changed
+60 -26

No files matched your search

+51 -18
View File
@@ -2,7 +2,9 @@
Collection of some self-made helm-charts. Each top-level directory is a standalone Helm chart that gets packaged and uploaded to a private, classic (index-based) Helm registry by CI.
> **The repository is moving from github.com to the Gitea instance JLab-Git (#225).** The code is already on both and identical; what still runs the show is GitHub. The workflow files under `.github/workflows/` are read by both forges unchanged, and "Where this repo lives" below names the handful of places where the two behave differently — each of them measured on a probe pull request, not assumed.
> **This repo lives on the Gitea instance JLab-Git.** Issues, pull requests, CI and Renovate all run there, and `main` there is the leading copy. github.com is a push mirror with Actions and Renovate **switched off**. The workflow files under `.github/workflows/` are read by Gitea unchanged; "Where this repo lives" below names the places where a Gitea runner behaves differently from a GitHub one, each of them measured rather than assumed.
>
> **A reading note on `#n`.** Issue numbers in this file point at the **GitHub** tracker. Most of them name *closed* issues, which were not migrated and exist only there. Gitea resolves a bare `#n` against its own counter instead — where, for example, `#22` is an unrelated throwaway pull request rather than the install-test issue this file means. The seventeen issues that were still open at the move are Gitea #5–#21 and #24; the mapping is a comment on Gitea #14.
## Charts
@@ -53,21 +55,52 @@ Collection of some self-made helm-charts. Each top-level directory is a standalo
- `ci/helm-dependencies.sh <chart-dir>` — not a workflow but the step they all share: fetches a chart's declared dependencies into `charts/`, and returns silently for the charts that declare none (all but one). It runs in **five** places, because every helm command this repo builds on exits 1 while a declared dependency is missing: before lint and package in `validate-charts.yml`, before package in `publish-helm-chart.yml`, and inside `ci/run-install-test.sh`, `ci/run-quantity-render-test.sh` and `ci/run-trivy-config.sh`. See "Chart dependencies" in the style guidelines for what that costs.
- `validate-charts.yml` — PR pipeline: detects which charts a PR touches and, for exactly those, runs `helm lint --strict`, `helm package`, and a chart-version bump check (`template-chart` is exempt from the version check). PRs without chart changes skip the build matrix. After the build stage, an **install-test** job (the same changed-charts matrix, minus the charts listed in `ci/excluded-charts.txt`) installs each changed chart into an ephemeral [k3d](https://k3d.io) cluster on the runner and requires every workload to become Ready — see below. Both matrices follow the changed chart directories, so a PR that only changes the harness under `ci/` exercises nothing by itself; the Trivy scan has a fixed-sample fallback for that case, the install-test has none — bump the affected chart's patch version to pull it into the matrix. The **resource-quantity render test** is the exception to that rule: it runs over every chart on every PR — see below.
### Where this repo lives, and what Gitea Actions does differently
### Where this repo lives, and what a Gitea runner does differently
The move to the Gitea instance JLab-Git (#225) is in progress. Order matters and is not negotiable: **CI provably green on Gitea → GitHub Actions off → mirror on.** Any other order leaves a window in which either no CI exists or two of them publish into the same registry — and that registry accepts a re-upload of an existing version silently, replacing the artifact (#160).
Everything below was measured on this instance, not taken from documentation, and it is the reason for every deviation in `validate-charts.yml`. How the move got here is not repeated — that is Gitea #14.
Everything below was measured on a throwaway pull request against the Gitea instance, not taken from documentation. It is the reason for the two deviations now in `validate-charts.yml`.
**Reusable workflows with `secrets: inherit` work**, so the 23 thin `publish-<chart>.yml` hulls run unchanged. One quirk worth knowing before debugging the wrong thing: a *missing* secret does not fail the job, it fails job **creation**, so the run shows no steps at all.
- **Reusable workflows with `secrets: inherit` work.** The 23 thin `publish-<chart>.yml` hulls survive the move unchanged. (A missing secret does not fail the job — Gitea fails at *job creation*, so the run shows no steps at all, which is worth knowing before debugging the wrong thing.)
- **`ubuntu-latest` on this instance has no Docker socket; `ubuntu-docker` has.** Same label, different meaning than on GitHub's hosted runners, and only one job in this repo cares: the install-test creates a k3d cluster, and k3d is a Docker client. Measured, per job: `detect-changed-charts`, `build-chart`, the quantity render and the Trivy config scan all pass on `ubuntu-latest` — none of them needs Docker. The install-test on `ubuntu-latest` dies at `Create k3d cluster` with `Cannot connect to the Docker daemon at unix:///var/run/docker.sock`, an error that names k3d and never the runner label.
- **`ubuntu-docker` is necessary but not yet sufficient.** Its daemon (8 vCPU, 64 GB, Docker 29.5.2) is **rootless** and reports `Cgroup Driver: none` plus `WARNING: Running in rootless-mode without cgroups`. k3d gets as far as creating the network, pulling the images and starting the node; the k3s server container then restarts immediately with an empty log. k3s needs cgroup delegation. Until the instance's Docker-in-Docker gets it (rootful, or rootless with systemd delegation), **the install-test cannot run on Gitea at all** — see #225.
- **`aquasecurity/setup-trivy` cannot be used.** It runs `actions/checkout` against `aquasecurity/trivy` on github.com with `github.token`, which on Gitea is a *Gitea* token; github.com answers 401 and checkout retries twice before ending on `fatal: could not read Username for 'https://github.com'`. Its `actions/cache` step is inert on the instance as well (`Cache action is only supported on GHES version >= 3.5`), and fetching the nested actions cost 4m30s before the job started. The workflow now downloads the release tarball instead — one request, identical behaviour on both forges, and the `# renovate:` comment above `TRIVY_VERSION` still keeps it current (proved by dry run: pinned to `v0.72.0`, the lookup offers `v0.74.0` as a minor).
- **Matrix jobs lose their name in the status list.** Gitea does not expand `${{ matrix.chart }}` in a job `name:`; it substitutes the matrix *expression*, so the commit status reads `Validate Helm Charts / Install-test ${{ fromJSON(needs.detect-changed-charts.outputs.install-charts) }} (pull_request)` and **all legs of a matrix share that one context**. Consequence for branch protection: a per-chart required check is not possible on Gitea, and the required-check names differ from GitHub's — they have to be read off a real run rather than copied.
- **Actions are fetched from github.com anonymously and that works.** `actions/checkout`, `azure/setup-helm` and the 1Password action need no replacement; only an action that *itself* authenticates against github.com breaks (see Trivy above).
- **No merge queue.** Gitea has none. Its place is taken by auto-merge on green checks, which for a single maintainer is equivalent in effect and different in one way worth remembering: auto-merge merges as soon as the checks it knows about are green, so it is only as strong as the *required* checks configured on `main`. With none configured it merges immediately.
**Runner labels mean different machines than on GitHub.** `ubuntu-latest` here has **no Docker socket**; `ubuntu-docker` has one. Exactly one job cares — the install-test, because k3d is a Docker client — and its `runs-on` is therefore an expression rather than a fixed label, so the same file still works on a GitHub runner where `ubuntu-docker` does not exist. Every other job (lint/package, the quantity render, the Trivy scan) needs no socket and stays on `ubuntu-latest`.
Still on the maintainer's side and not delegable: branch protection plus the commit-signature requirement on the Gitea repo, the runner's cgroup problem above, and switching GitHub Actions off before enabling the mirror. The bot token gets `403` on `actions/runners`, `actions/secrets` and `branch_protections`, so none of that can be verified from here.
**A matrix runs concurrently inside ONE job, under ONE status context.** This is the single most consequential difference and it has two halves. Operationally: all legs share one Docker daemon, which is what makes a per-leg resource name mandatory (see the cluster name below). For branch protection: **a required check per chart is not possible**, because there is only one context for the whole matrix — and its name carries the *unexpanded* matrix expression, e.g. `Validate Helm Charts / Install-test ${{ fromJSON(needs.detect-changed-charts.outputs.install-charts) }} (pull_request)`. Required-check names have to be read off a real run; they cannot be copied from GitHub.
**Expected runtimes**, so that a later deviation is visible as one:
| | measured here | GitHub, same commit |
|---|---|---|
| `Create k3d cluster` | 19–25 s | 22–23 s |
| `install-test` job, two charts in the matrix (cyberchef + freesailarr) | 210–216 s over three runs, spread 6 s | ~145 s wall, two parallel VMs |
| `install-test` job, freesailarr alone, ten containers and a real tunnel | 225 s (measured on the MediathekArr change, Gitea #20) | not comparable — Actions there were already off |
The runner is **not slower** — cluster creation is a touch faster. The difference in total time is **structural, not performance**: two matrix legs share one runner here, while GitHub gives each leg its own VM. Reading it as "Gitea is slower" leads to the wrong conclusion.
**`aquasecurity/setup-trivy` cannot be used.** It runs `actions/checkout` against `aquasecurity/trivy` on github.com and hands it `github.token` — which here is a *Gitea* token. github.com answers 401 and checkout ends on `fatal: could not read Username for 'https://github.com'`, naming neither the action nor the token. Its `actions/cache` step is inert here as well. The workflow downloads the release tarball instead: one request, same behaviour on either forge.
**Tools are installed, not inherited — and the two cases differ on purpose.** `kubectl` is **pinned** to match `K3S_IMAGE` and guarded against version skew, because a client's distance to the API server changes what the test *means*, and a version that arrives with the runner image drifts with every image rebuild. `envsubst` is only **checked for and fetched if absent**, because it substitutes text and the release doing so does not change the result; pinning it would be upkeep without value. That GitHub's image happens to ship both does not make either unnecessary — it only makes the drift invisible there.
**Containerisation is detected, not inferred from the platform.** k3d publishes the API server port on the Docker *host*; when the job is itself a container — as every Gitea Actions job is — that host is the dind container, so the `https://0.0.0.0:<port>` k3d writes into the kubeconfig points at the job container, where nothing listens. The workflow asks the daemon about its own container instead of asking which forge it is on: a self-hosted **GitHub** runner in a container has the identical problem, and a condition that says "GitHub" while meaning "not containerised" is the kind of lie this repo exists to remove.
Still unmeasured, because they never stood in the way: `/dev/fuse` and user namespaces. Said as unmeasured rather than as fine.
#### What a GitHub runner had been providing quietly
Getting the install-test to run here surfaced six assumptions that had never been written down anywhere. Five of them say something about the **runner**, one about **our code**, and the split is the point.
The five environment ones are the harmless kind: they fail loudly, immediately, and are fixed on the node side (`cluster-config`), not here — Docker bridges built at MTU 1500 against a pod network of 1450 (small responses fit in one packet, a 17 MB tarball does not, which is why it looked like a blocked host); `fs.inotify` limits too low for k3s, so containerd never started and the node never registered; no `kubectl` and no `envsubst` in the image; and the kubeconfig pointing at `0.0.0.0`.
The sixth is different, and it is the one to remember.
> **The k3d cluster name was hard-coded as `install-test` — and that was wrong on GitHub too.** It simply could never show there: each matrix leg gets its own VM with its own Docker daemon, so two legs never saw each other's cluster. Here, where all legs share one daemon, it collides immediately — one leg logs `Re-using existing network`, the other dies on `The container name /k3d-install-test-server-0 is already in use`, and the first leg's rollback then deletes the shared network out from under the second. Two red legs, and neither message names the cause.
>
> For two years the roomier environment carried that bug instead of showing it. **A generous environment is no friend when you want to know whether something is correct.**
#### Two rules for every check added here
Both were learned by getting them wrong in this repo, and both apply to any new verification step, not just to clusters.
1. **A check that a broken system passes is worse than no check.** `kubectl get nodes` exits **0 on an empty list**, so a readiness check written that way went green against a cluster whose node had never registered — and the failure then surfaced five minutes later, somewhere unrelated. `kubectl wait --for=condition=Ready nodes --all` is not the shorter way to write it: with zero nodes the condition is vacuously satisfied. Assert the thing you mean (*at least one node, and Ready*), not that the command answered.
2. **Green does not mean checked.** A passing job earns exactly as much trust as the assertions inside it. The open case of this kind is Gitea #24: the images in `ci/<chart>/fixtures.yaml` are not followed by Renovate, so the install-test can keep passing against fixture infrastructure that is years stale. **Not solved** — named here so it is not mistaken for solved.
### Resource-quantity render test (all charts, every PR)
@@ -108,7 +141,7 @@ Fixture images avoid Docker Hub where possible (`public.ecr.aws`/`ghcr.io`) to d
## Dependency updates (Renovate)
Renovate runs as the self-hosted `jlab-renovate` GitHub App while the repo is on github.com, and on Gitea as the shared `jlab-renovate-bot` instance that already serves the other repositories there — see "Renovate on Gitea" below. **There is no Renovate workflow in this repo, and none should be added.** Which of the two runs it changes nothing about *what* gets updated: its config is **`renovate.json5`** in the repo root (JSON5 for the inline reasoning; requires Renovate >= 41 for `managerFilePatterns`/`customType`). There must be exactly one config file — do not add a second `renovate.json` or `.github/renovate.json`.
Renovate runs as the shared `jlab-renovate-bot` instance on the Gitea side, which also serves the other repositories there — see "Renovate on Gitea" below. **There is no Renovate workflow in this repo, and none should be added.** (The `jlab-renovate` GitHub App that used to do this is switched off along with the rest of GitHub Actions.) What updates is unaffected either way: the config is **`renovate.json5`** in the repo root (JSON5 for the inline reasoning; requires Renovate >= 41 for `managerFilePatterns`/`customType`). There must be exactly one config file — do not add a second `renovate.json` or `.github/renovate.json`.
What Renovate keeps current:
@@ -150,18 +183,18 @@ Everything is derived from the base commit, so the script is idempotent across t
>
> Without that entry Renovate skips the task without an error and every chart PR stays red on the version check.
If the dependency dashboard reports `Failed to look up github-releases package …: no-result` for third-party repos, that is the instance's GitHub credentials, not this config: a GitHub App installation token is not a good fit for looking up repos the app is not installed on. Give the instance a separate read-only PAT as `GITHUB_COM_TOKEN` for github.com datasource lookups.
If the dependency dashboard reports `Failed to look up github-releases package …: no-result` for third-party repos, that is the instance's github.com credentials, not this config. The instance needs a read-only PAT as `GITHUB_COM_TOKEN` for github.com datasource lookups; its own platform token is a Gitea token and is worth nothing there.
### Renovate on Gitea
**Do not add a Renovate workflow to this repository.** It is the obvious thing to reach for — Gitea has no app model, so a scheduled job in the repo looks like the only way to run the bot — and it is wrong here: a **shared `jlab-renovate-bot` instance already runs on the Gitea side** and already has this repository enrolled. A second runner against the same repo produces duplicate PRs and two schedulers rewriting each other's branches, which is not a safety net but a fault. This paragraph exists so the next reader who finds no workflow knows why instead of building one.
**Do not add a Renovate workflow to this repository.** It is the obvious thing to reach for — Gitea has no app model, so a scheduled job in the repo looks like the only way to run the bot — and it is wrong here: a **shared `jlab-renovate-bot` instance** runs outside this repository and has it enrolled. A second runner against the same repo produces duplicate PRs and two schedulers rewriting each other's branches, which is not a safety net but a fault. This paragraph exists so the next reader who finds no workflow knows why instead of building one.
The instance is outside this repository, which has two consequences worth writing down:
- **It reads our `renovate.json5` and the per-chart rules do fire.** Measured, not assumed: its first run on the migrated repo opened PRs for `searxng` and `gotenberg` plus the dependency dashboard. Those two are `customManagers` targets of ours — no built-in manager finds an `appVersion:` in a `Chart.yaml` — so the config is demonstrably in effect.
- **`allowedCommands` is already granted there, and the version-bump hook runs.** This was the expensive unknown: `postUpgradeTasks` only executes when the command is allow-listed in the *instance* config, and a missing entry is skipped **silently**, leaving every chart PR red on the version check. The proof is in that searxng PR's diff, which carries `version: 1.0.4+up2026.9.11.ffe96f8a6` → `1.0.5+up2026.9.13.96479fff6` alongside the `appVersion` change. Nothing but `ci/renovate-bump-chart.sh` writes `version:`. Nothing has to be configured for the move — but the allow-list is not visible from this repo either, so if it is ever dropped, the symptom here will be red chart PRs with no explanation.
- **It reads our `renovate.json5` and the per-chart rules do fire.** Measured, not assumed: its first run here opened PRs for `searxng` and `gotenberg` plus the dependency dashboard. Those two are `customManagers` targets of ours — no built-in manager finds an `appVersion:` in a `Chart.yaml` — so the config is demonstrably in effect.
- **`allowedCommands` is already granted there, and the version-bump hook runs.** This was the expensive unknown: `postUpgradeTasks` only executes when the command is allow-listed in the *instance* config, and a missing entry is skipped **silently**, leaving every chart PR red on the version check. The proof is in that searxng PR's diff, which carries `version: 1.0.4+up2026.9.11.ffe96f8a6` → `1.0.5+up2026.9.13.96479fff6` alongside the `appVersion` change. Nothing but `ci/renovate-bump-chart.sh` writes `version:`. The allow-list is not visible from this repo, though, so if it is ever dropped the symptom here will be red chart PRs with no explanation.
One item does not come for free on either side and is named in "The version-bump hook" above: a `GITHUB_COM_TOKEN` for github.com datasource lookups. Under the GitHub App those rode on the installation token; a Gitea token is worth nothing at github.com, and unauthenticated lookups hit the rate limit inside a single pass. The symptom is a `no-result` line on the dependency dashboard, not an error.
One item is easy to miss and is named in "The version-bump hook" above: the instance needs a `GITHUB_COM_TOKEN` for github.com datasource lookups. Every `github-releases`/`github-tags` lookup and the whole built-in `github-actions` manager resolve against github.com, the platform token is a Gitea one and worth nothing there, and unauthenticated lookups hit the rate limit inside a single pass. The symptom is a `no-result` line on the dependency dashboard, not an error.
### PR policy