CLAUDE.md

@@ -21,14 +21,15 @@ Additional rules for all commits, PRs, issues, and comments:
 
 ## Which forge, and with which tool
 
-The repository is moving from github.com to the Gitea instance JLab-Git (#225). The move is **not finished**, so both are live and the steps below differ by where the work happens. README, "Where this repo lives", holds the measured differences between the two CIs; this section holds only what changes about the *workflow*.
+Everything runs on the Gitea instance JLab-Git: issues, pull requests, CI and Renovate. github.com is a push mirror with Actions and Renovate **switched off**, and it lags until the mirror runs. README, "Where this repo lives", holds the measured differences between a Gitea runner and a GitHub one.
 
-- **Remotes are named `github` and `JLab-Git`** — there is no `origin`. Say which one you mean; `git push` without a remote is ambiguous here.
-- **Today the merge flow still runs on GitHub.** Issues, pull requests and merges happen there until the maintainer switches Actions off and turns on the mirror. Until then, **do not push to `JLab-Git/main`**: with publish workflows armed on both sides, two CIs publish into the same registry, and that registry accepts a re-upload of an existing version silently, replacing the artifact (#160). Pushing a *branch* to Gitea is fine — only `main` triggers a publish.
-- **`gh` only as `op plugin run -- gh`**, `tea` only as `op plugin run -- tea`. A bare `gh` falls back to the wrong identity. Every call costs a 1Password confirmation, so bundle them and never poll.
-- **After the move: no merge queue.** Gitea has none. Its place is taken by **auto-merge on green checks** — the PR is armed once and merges itself when the required checks pass. The manager still decides *whether* to arm it; rule 6 above is unchanged: the user merges, and an agent never merges its own PR.
-- **Issue numbers will not survive the move.** Gitea renumbers, and issues and PRs share one counter, so a `#n` written today will not land on the same number over there. The migration itself is the maintainer's one-off job over the API; how the old references are dealt with afterwards is open and decided in #225. Until that is decided, a `#n` anywhere in this repo means the GitHub issue.
-- **Renovate is already running on the Gitea side, and not from this repo.** A shared `jlab-renovate-bot` instance serves the other repositories there and has this one enrolled; it reads our `renovate.json5` and its `postUpgradeTasks` hook is allow-listed (both verified on its first run). **Do not add a Renovate workflow here** — a second runner against the same repo means duplicate PRs and two schedulers rewriting each other's branches. README, "Renovate on Gitea", has the evidence.
+- **`tea`, not `gh`** — and only as `op plugin run -- tea`. A bare invocation falls back to the wrong identity. Every call costs a 1Password confirmation, so bundle them and never poll.
+- **Remotes are named `github` and `JLab-Git`** — there is no `origin`. `JLab-Git/main` is the leading copy; say which remote you mean, because `git push` without one is ambiguous here.
+- **Branch protection with a signature requirement is active**, and signatures verify on Gitea. A commit that is not signed does not reach `main` — check before opening a PR, do not discover it at merge time.
+- **No merge queue.** Gitea has none, and its place is taken by **auto-merge on green checks**: the PR is armed once and merges itself when the required checks pass. Rule 6 above is unchanged — the user merges, and an agent never merges its own PR or arms auto-merge on it.
+- **A required check per chart is not possible.** Gitea runs a whole matrix as one job under one status context, so the install-test reports once for all charts. Relevant whenever branch protection is touched.
+- **Issue numbers changed with the move, and `#n` in this repo's files means the GitHub tracker.** The issues open at the time are Gitea #5–#21 and #24; the mapping is a comment on Gitea #14. Closed ones were not migrated and exist only on GitHub. Beware that Gitea silently resolves a bare `#n` against its own counter, so an old reference can render as a link to something unrelated. When you write a *new* reference, use the Gitea number.
+- **Renovate runs outside this repo**, as the shared `jlab-renovate-bot` instance; it reads our `renovate.json5` and its `postUpgradeTasks` hook is allow-listed there. **Do not add a Renovate workflow here** — a second runner means duplicate PRs and two schedulers rewriting each other's branches. README, "Renovate on Gitea", has the evidence.
 
 ## Keep the docs current
 
@@ -38,7 +39,7 @@ The repository is moving from github.com to the Gitea instance JLab-Git (#225).
 
 ## What this repo is
 
-A collection of self-made Helm charts, one directory per chart, published to a private, classic (index-based) Helm registry by CI (GitHub Actions today, Gitea Actions after #225; the workflow files are read by both unchanged) — an HTTP `POST` of the packaged `.tgz`, not an OCI `helm push`; the target URL and its credentials live in 1Password and never in this repository. Validation happens on three levels: locally with Helm, in the PR pipeline (`validate-charts.yml`: strict lint, package, version-bump check), and in the PR install-test stage (k3d, fixtures under `ci/` — see README):
+A collection of self-made Helm charts, one directory per chart, published to a private, classic (index-based) Helm registry by Gitea Actions — an HTTP `POST` of the packaged `.tgz`, not an OCI `helm push`; the target URL and its credentials live in 1Password and never in this repository. Validation happens on three levels: locally with Helm, in the PR pipeline (`validate-charts.yml`: strict lint, package, version-bump check), and in the PR install-test stage (k3d, fixtures under `ci/` — see README):
 
 ```bash
 helm lint --strict <chart-name>

README.md

@@ -2,7 +2,9 @@
 
 Collection of some self-made helm-charts. Each top-level directory is a standalone Helm chart that gets packaged and uploaded to a private, classic (index-based) Helm registry by CI.
 
-> **The repository is moving from github.com to the Gitea instance JLab-Git (#225).** The code is already on both and identical; what still runs the show is GitHub. The workflow files under `.github/workflows/` are read by both forges unchanged, and "Where this repo lives" below names the handful of places where the two behave differently — each of them measured on a probe pull request, not assumed.
+> **This repo lives on the Gitea instance JLab-Git.** Issues, pull requests, CI and Renovate all run there, and `main` there is the leading copy. github.com is a push mirror with Actions and Renovate **switched off**. The workflow files under `.github/workflows/` are read by Gitea unchanged; "Where this repo lives" below names the places where a Gitea runner behaves differently from a GitHub one, each of them measured rather than assumed.
+>
+> **A reading note on `#n`.** Issue numbers in this file point at the **GitHub** tracker. Most of them name *closed* issues, which were not migrated and exist only there. Gitea resolves a bare `#n` against its own counter instead — where, for example, `#22` is an unrelated throwaway pull request rather than the install-test issue this file means. The seventeen issues that were still open at the move are Gitea #5–#21 and #24; the mapping is a comment on Gitea #14.
 
 ## Charts
 
@@ -53,21 +55,52 @@ Collection of some self-made helm-charts. Each top-level directory is a standalo
 - `ci/helm-dependencies.sh <chart-dir>` — not a workflow but the step they all share: fetches a chart's declared dependencies into `charts/`, and returns silently for the charts that declare none (all but one). It runs in **five** places, because every helm command this repo builds on exits 1 while a declared dependency is missing: before lint and package in `validate-charts.yml`, before package in `publish-helm-chart.yml`, and inside `ci/run-install-test.sh`, `ci/run-quantity-render-test.sh` and `ci/run-trivy-config.sh`. See "Chart dependencies" in the style guidelines for what that costs.
 - `validate-charts.yml` — PR pipeline: detects which charts a PR touches and, for exactly those, runs `helm lint --strict`, `helm package`, and a chart-version bump check (`template-chart` is exempt from the version check). PRs without chart changes skip the build matrix. After the build stage, an **install-test** job (the same changed-charts matrix, minus the charts listed in `ci/excluded-charts.txt`) installs each changed chart into an ephemeral [k3d](https://k3d.io) cluster on the runner and requires every workload to become Ready — see below. Both matrices follow the changed chart directories, so a PR that only changes the harness under `ci/` exercises nothing by itself; the Trivy scan has a fixed-sample fallback for that case, the install-test has none — bump the affected chart's patch version to pull it into the matrix. The **resource-quantity render test** is the exception to that rule: it runs over every chart on every PR — see below.
 
-### Where this repo lives, and what Gitea Actions does differently
+### Where this repo lives, and what a Gitea runner does differently
 
-The move to the Gitea instance JLab-Git (#225) is in progress. Order matters and is not negotiable: **CI provably green on Gitea → GitHub Actions off → mirror on.** Any other order leaves a window in which either no CI exists or two of them publish into the same registry — and that registry accepts a re-upload of an existing version silently, replacing the artifact (#160).
+Everything below was measured on this instance, not taken from documentation, and it is the reason for every deviation in `validate-charts.yml`. How the move got here is not repeated — that is Gitea #14.
 
-Everything below was measured on a throwaway pull request against the Gitea instance, not taken from documentation. It is the reason for the two deviations now in `validate-charts.yml`.
+**Reusable workflows with `secrets: inherit` work**, so the 23 thin `publish-<chart>.yml` hulls run unchanged. One quirk worth knowing before debugging the wrong thing: a *missing* secret does not fail the job, it fails job **creation**, so the run shows no steps at all.
 
-- **Reusable workflows with `secrets: inherit` work.** The 23 thin `publish-<chart>.yml` hulls survive the move unchanged. (A missing secret does not fail the job — Gitea fails at *job creation*, so the run shows no steps at all, which is worth knowing before debugging the wrong thing.)
-- **`ubuntu-latest` on this instance has no Docker socket; `ubuntu-docker` has.** Same label, different meaning than on GitHub's hosted runners, and only one job in this repo cares: the install-test creates a k3d cluster, and k3d is a Docker client. Measured, per job: `detect-changed-charts`, `build-chart`, the quantity render and the Trivy config scan all pass on `ubuntu-latest` — none of them needs Docker. The install-test on `ubuntu-latest` dies at `Create k3d cluster` with `Cannot connect to the Docker daemon at unix:///var/run/docker.sock`, an error that names k3d and never the runner label.
-- **`ubuntu-docker` is necessary but not yet sufficient.** Its daemon (8 vCPU, 64 GB, Docker 29.5.2) is **rootless** and reports `Cgroup Driver: none` plus `WARNING: Running in rootless-mode without cgroups`. k3d gets as far as creating the network, pulling the images and starting the node; the k3s server container then restarts immediately with an empty log. k3s needs cgroup delegation. Until the instance's Docker-in-Docker gets it (rootful, or rootless with systemd delegation), **the install-test cannot run on Gitea at all** — see #225.
-- **`aquasecurity/setup-trivy` cannot be used.** It runs `actions/checkout` against `aquasecurity/trivy` on github.com with `github.token`, which on Gitea is a *Gitea* token; github.com answers 401 and checkout retries twice before ending on `fatal: could not read Username for 'https://github.com'`. Its `actions/cache` step is inert on the instance as well (`Cache action is only supported on GHES version >= 3.5`), and fetching the nested actions cost 4m30s before the job started. The workflow now downloads the release tarball instead — one request, identical behaviour on both forges, and the `# renovate:` comment above `TRIVY_VERSION` still keeps it current (proved by dry run: pinned to `v0.72.0`, the lookup offers `v0.74.0` as a minor).
-- **Matrix jobs lose their name in the status list.** Gitea does not expand `${{ matrix.chart }}` in a job `name:`; it substitutes the matrix *expression*, so the commit status reads `Validate Helm Charts / Install-test ${{ fromJSON(needs.detect-changed-charts.outputs.install-charts) }} (pull_request)` and **all legs of a matrix share that one context**. Consequence for branch protection: a per-chart required check is not possible on Gitea, and the required-check names differ from GitHub's — they have to be read off a real run rather than copied.
-- **Actions are fetched from github.com anonymously and that works.** `actions/checkout`, `azure/setup-helm` and the 1Password action need no replacement; only an action that *itself* authenticates against github.com breaks (see Trivy above).
-- **No merge queue.** Gitea has none. Its place is taken by auto-merge on green checks, which for a single maintainer is equivalent in effect and different in one way worth remembering: auto-merge merges as soon as the checks it knows about are green, so it is only as strong as the *required* checks configured on `main`. With none configured it merges immediately.
+**Runner labels mean different machines than on GitHub.** `ubuntu-latest` here has **no Docker socket**; `ubuntu-docker` has one. Exactly one job cares — the install-test, because k3d is a Docker client — and its `runs-on` is therefore an expression rather than a fixed label, so the same file still works on a GitHub runner where `ubuntu-docker` does not exist. Every other job (lint/package, the quantity render, the Trivy scan) needs no socket and stays on `ubuntu-latest`.
 
-Still on the maintainer's side and not delegable: branch protection plus the commit-signature requirement on the Gitea repo, the runner's cgroup problem above, and switching GitHub Actions off before enabling the mirror. The bot token gets `403` on `actions/runners`, `actions/secrets` and `branch_protections`, so none of that can be verified from here.
+**A matrix runs concurrently inside ONE job, under ONE status context.** This is the single most consequential difference and it has two halves. Operationally: all legs share one Docker daemon, which is what makes a per-leg resource name mandatory (see the cluster name below). For branch protection: **a required check per chart is not possible**, because there is only one context for the whole matrix — and its name carries the *unexpanded* matrix expression, e.g. `Validate Helm Charts / Install-test ${{ fromJSON(needs.detect-changed-charts.outputs.install-charts) }} (pull_request)`. Required-check names have to be read off a real run; they cannot be copied from GitHub.
+
+**Expected runtimes**, so that a later deviation is visible as one:
+
+| | measured here | GitHub, same commit |
+|---|---|---|
+| `Create k3d cluster` | 19–25 s | 22–23 s |
+| `install-test` job, two charts in the matrix (cyberchef + freesailarr) | 210–216 s over three runs, spread 6 s | ~145 s wall, two parallel VMs |
+| `install-test` job, freesailarr alone, ten containers and a real tunnel | 225 s (measured on the MediathekArr change, Gitea #20) | not comparable — Actions there were already off |
+
+The runner is **not slower** — cluster creation is a touch faster. The difference in total time is **structural, not performance**: two matrix legs share one runner here, while GitHub gives each leg its own VM. Reading it as "Gitea is slower" leads to the wrong conclusion.
+
+**`aquasecurity/setup-trivy` cannot be used.** It runs `actions/checkout` against `aquasecurity/trivy` on github.com and hands it `github.token` — which here is a *Gitea* token. github.com answers 401 and checkout ends on `fatal: could not read Username for 'https://github.com'`, naming neither the action nor the token. Its `actions/cache` step is inert here as well. The workflow downloads the release tarball instead: one request, same behaviour on either forge.
+
+**Tools are installed, not inherited — and the two cases differ on purpose.** `kubectl` is **pinned** to match `K3S_IMAGE` and guarded against version skew, because a client's distance to the API server changes what the test *means*, and a version that arrives with the runner image drifts with every image rebuild. `envsubst` is only **checked for and fetched if absent**, because it substitutes text and the release doing so does not change the result; pinning it would be upkeep without value. That GitHub's image happens to ship both does not make either unnecessary — it only makes the drift invisible there.
+
+**Containerisation is detected, not inferred from the platform.** k3d publishes the API server port on the Docker *host*; when the job is itself a container — as every Gitea Actions job is — that host is the dind container, so the `https://0.0.0.0:<port>` k3d writes into the kubeconfig points at the job container, where nothing listens. The workflow asks the daemon about its own container instead of asking which forge it is on: a self-hosted **GitHub** runner in a container has the identical problem, and a condition that says "GitHub" while meaning "not containerised" is the kind of lie this repo exists to remove.
+
+Still unmeasured, because they never stood in the way: `/dev/fuse` and user namespaces. Said as unmeasured rather than as fine.
+
+#### What a GitHub runner had been providing quietly
+
+Getting the install-test to run here surfaced six assumptions that had never been written down anywhere. Five of them say something about the **runner**, one about **our code**, and the split is the point.
+
+The five environment ones are the harmless kind: they fail loudly, immediately, and are fixed on the node side (`cluster-config`), not here — Docker bridges built at MTU 1500 against a pod network of 1450 (small responses fit in one packet, a 17 MB tarball does not, which is why it looked like a blocked host); `fs.inotify` limits too low for k3s, so containerd never started and the node never registered; no `kubectl` and no `envsubst` in the image; and the kubeconfig pointing at `0.0.0.0`.
+
+The sixth is different, and it is the one to remember.
+
+> **The k3d cluster name was hard-coded as `install-test` — and that was wrong on GitHub too.** It simply could never show there: each matrix leg gets its own VM with its own Docker daemon, so two legs never saw each other's cluster. Here, where all legs share one daemon, it collides immediately — one leg logs `Re-using existing network`, the other dies on `The container name /k3d-install-test-server-0 is already in use`, and the first leg's rollback then deletes the shared network out from under the second. Two red legs, and neither message names the cause.
+>
+> For two years the roomier environment carried that bug instead of showing it. **A generous environment is no friend when you want to know whether something is correct.**
+
+#### Two rules for every check added here
+
+Both were learned by getting them wrong in this repo, and both apply to any new verification step, not just to clusters.
+
+1. **A check that a broken system passes is worse than no check.** `kubectl get nodes` exits **0 on an empty list**, so a readiness check written that way went green against a cluster whose node had never registered — and the failure then surfaced five minutes later, somewhere unrelated. `kubectl wait --for=condition=Ready nodes --all` is not the shorter way to write it: with zero nodes the condition is vacuously satisfied. Assert the thing you mean (*at least one node, and Ready*), not that the command answered.
+2. **Green does not mean checked.** A passing job earns exactly as much trust as the assertions inside it. The open case of this kind is Gitea #24: the images in `ci/<chart>/fixtures.yaml` are not followed by Renovate, so the install-test can keep passing against fixture infrastructure that is years stale. **Not solved** — named here so it is not mistaken for solved.
 
 ### Resource-quantity render test (all charts, every PR)
 
@@ -108,7 +141,7 @@ Fixture images avoid Docker Hub where possible (`public.ecr.aws`/`ghcr.io`) to d
 
 ## Dependency updates (Renovate)
 
-Renovate runs as the self-hosted `jlab-renovate` GitHub App while the repo is on github.com, and on Gitea as the shared `jlab-renovate-bot` instance that already serves the other repositories there — see "Renovate on Gitea" below. **There is no Renovate workflow in this repo, and none should be added.** Which of the two runs it changes nothing about *what* gets updated: its config is **`renovate.json5`** in the repo root (JSON5 for the inline reasoning; requires Renovate >= 41 for `managerFilePatterns`/`customType`). There must be exactly one config file — do not add a second `renovate.json` or `.github/renovate.json`.
+Renovate runs as the shared `jlab-renovate-bot` instance on the Gitea side, which also serves the other repositories there — see "Renovate on Gitea" below. **There is no Renovate workflow in this repo, and none should be added.** (The `jlab-renovate` GitHub App that used to do this is switched off along with the rest of GitHub Actions.) What updates is unaffected either way: the config is **`renovate.json5`** in the repo root (JSON5 for the inline reasoning; requires Renovate >= 41 for `managerFilePatterns`/`customType`). There must be exactly one config file — do not add a second `renovate.json` or `.github/renovate.json`.
 
 What Renovate keeps current:
 
@@ -150,18 +183,18 @@ Everything is derived from the base commit, so the script is idempotent across t
 >
 > Without that entry Renovate skips the task without an error and every chart PR stays red on the version check.
 
-If the dependency dashboard reports `Failed to look up github-releases package …: no-result` for third-party repos, that is the instance's GitHub credentials, not this config: a GitHub App installation token is not a good fit for looking up repos the app is not installed on. Give the instance a separate read-only PAT as `GITHUB_COM_TOKEN` for github.com datasource lookups.
+If the dependency dashboard reports `Failed to look up github-releases package …: no-result` for third-party repos, that is the instance's github.com credentials, not this config. The instance needs a read-only PAT as `GITHUB_COM_TOKEN` for github.com datasource lookups; its own platform token is a Gitea token and is worth nothing there.
 
 ### Renovate on Gitea
 
-**Do not add a Renovate workflow to this repository.** It is the obvious thing to reach for — Gitea has no app model, so a scheduled job in the repo looks like the only way to run the bot — and it is wrong here: a **shared `jlab-renovate-bot` instance already runs on the Gitea side** and already has this repository enrolled. A second runner against the same repo produces duplicate PRs and two schedulers rewriting each other's branches, which is not a safety net but a fault. This paragraph exists so the next reader who finds no workflow knows why instead of building one.
+**Do not add a Renovate workflow to this repository.** It is the obvious thing to reach for — Gitea has no app model, so a scheduled job in the repo looks like the only way to run the bot — and it is wrong here: a **shared `jlab-renovate-bot` instance** runs outside this repository and has it enrolled. A second runner against the same repo produces duplicate PRs and two schedulers rewriting each other's branches, which is not a safety net but a fault. This paragraph exists so the next reader who finds no workflow knows why instead of building one.
 
 The instance is outside this repository, which has two consequences worth writing down:
 
-- **It reads our `renovate.json5` and the per-chart rules do fire.** Measured, not assumed: its first run on the migrated repo opened PRs for `searxng` and `gotenberg` plus the dependency dashboard. Those two are `customManagers` targets of ours — no built-in manager finds an `appVersion:` in a `Chart.yaml` — so the config is demonstrably in effect.
-- **`allowedCommands` is already granted there, and the version-bump hook runs.** This was the expensive unknown: `postUpgradeTasks` only executes when the command is allow-listed in the *instance* config, and a missing entry is skipped **silently**, leaving every chart PR red on the version check. The proof is in that searxng PR's diff, which carries `version: 1.0.4+up2026.9.11.ffe96f8a6` → `1.0.5+up2026.9.13.96479fff6` alongside the `appVersion` change. Nothing but `ci/renovate-bump-chart.sh` writes `version:`. Nothing has to be configured for the move — but the allow-list is not visible from this repo either, so if it is ever dropped, the symptom here will be red chart PRs with no explanation.
+- **It reads our `renovate.json5` and the per-chart rules do fire.** Measured, not assumed: its first run here opened PRs for `searxng` and `gotenberg` plus the dependency dashboard. Those two are `customManagers` targets of ours — no built-in manager finds an `appVersion:` in a `Chart.yaml` — so the config is demonstrably in effect.
+- **`allowedCommands` is already granted there, and the version-bump hook runs.** This was the expensive unknown: `postUpgradeTasks` only executes when the command is allow-listed in the *instance* config, and a missing entry is skipped **silently**, leaving every chart PR red on the version check. The proof is in that searxng PR's diff, which carries `version: 1.0.4+up2026.9.11.ffe96f8a6` → `1.0.5+up2026.9.13.96479fff6` alongside the `appVersion` change. Nothing but `ci/renovate-bump-chart.sh` writes `version:`. The allow-list is not visible from this repo, though, so if it is ever dropped the symptom here will be red chart PRs with no explanation.
 
-One item does not come for free on either side and is named in "The version-bump hook" above: a `GITHUB_COM_TOKEN` for github.com datasource lookups. Under the GitHub App those rode on the installation token; a Gitea token is worth nothing at github.com, and unauthenticated lookups hit the rate limit inside a single pass. The symptom is a `no-result` line on the dependency dashboard, not an error.
+One item is easy to miss and is named in "The version-bump hook" above: the instance needs a `GITHUB_COM_TOKEN` for github.com datasource lookups. Every `github-releases`/`github-tags` lookup and the whole built-in `github-actions` manager resolve against github.com, the platform token is a Gitea one and worth nothing there, and unauthenticated lookups hit the rate limit inside a single pass. The symptom is a `no-result` line on the dependency dashboard, not an error.
 
 ### PR policy